Answers
Is it cheaper to run my own social media scraper or pay for an API?
Short answer
At the thousands to tens of thousands of calls a month most teams need, paying per call is usually cheaper once engineer time is counted. A month of 3,000 Xiaohongshu searches and 2,000 comment pages costs $100.00 on okdata; a self-hosted crawler for the same job needs logged-in accounts, proxies, a server and someone to fix it each time the platform changes. Self-hosting wins at very large volume or when you need data an API does not return.
The scraper itself is free. Open-source projects for Douyin and Xiaohongshu are a git clone away, which is why the question keeps coming up on V2EX, from people asking how to collect Xiaohongshu and Douyin content to teams stuck on the data step of an agent project. What costs money is everything around it.
What a self-hosted crawler really costs
| Item | What drives it | Fill in your own |
|---|---|---|
| Engineer time to fix breakage | Signature changes, login expiry, risk-control pages | hours a month × hourly cost |
| Accounts | Logged-in crawling gets accounts flagged; you keep spares | accounts × price, plus SIM or phone setup |
| Proxies | Residential traffic billed per GB | GB a month × price per GB |
| Server | Browser automation needs memory | monthly server bill |
| Missed data | Days when the crawler is down | what a gap in the report costs you |
The first line is the one people underestimate. Platforms change request signatures and risk rules without notice, and each change means a developer stops other work to catch up. A thread asking how crawler engineers monitor and maintain their crawlers is a good picture of that ongoing load.
The API side of the same job
| Monthly job | Endpoint | Calls | Cost |
|---|---|---|---|
| 3,000 searches | Xiaohongshu note search | 3,000 | $60.00 |
| 2,000 comment pages | Xiaohongshu note comments | 2,000 | $40.00 |
| 3,000 Douyin searches | Douyin video search | 3,000 | $60.00 |
No account, cookie or proxy on your side, and failed calls are not charged:
curl -G "https://api.okdata.online/v1/xiaohongshu/notes/search" \
-H "Authorization: Bearer $OKDATA_API_KEY" \
--data-urlencode 'query=防晒霜 推荐' --data-urlencode 'sort=latest'
Break-even
Self-hosting pays off when (engineer hours × hourly cost + accounts + proxies + server) is less than the monthly API bill. Put your own volume in: every 10,000 Xiaohongshu searches cost {{cost:xiaohongshu.notes.search10000}} and every 10,000 Douyin comment pages {{cost:douyin.videos.comments.list10000}}. If one engineer-day of fixes a month costs more than that, the API is the cheaper side; if your volume is far larger, building may be.
When to build your own anyway
- Your volume is in the millions of pages a month and you have a team for it.
- You need fields or pages this catalog does not cover. okdata returns public data only, so anything that needs your own login is out of scope.
- Data must never pass through a third party.
- It is a learning project and your time is the point.
If you go the API route, the content research guide shows a typical Xiaohongshu workflow, and the pricing page has every price.
Updated