# Scraping it yourself vs a per-call data API: cost, upkeep and risk

> Running an open-source crawler yourself against paying per call: logins, proxies, signatures, account bans, licences and what a month really costs. Compared with dated sources.

Canonical page: https://okdata.online/compare/scraping-vs-data-api/

On okdata, 1,000 Xiaohongshu notes from search cost $1.00 and 1,000 Douyin comments $0.10, with one API key, no platform account and no charge for failed calls. Pay per call when the job is thousands of calls a month and nobody wants to look after logins, proxies and signatures; run your own crawler when the platform is outside any catalog, the volume is very large or you are learning how crawling works. Checked on 2026-11-12.

Updated 2026-11-12 · Other options checked on 2026-11-12

A crawler you run yourself looks free: the code is on GitHub and the data is public. The cost turns up elsewhere. An account has to stay logged in, IP addresses get rate-limited, a signature changes overnight, and the licence may not allow what you plan to do with the data. A per-call data API moves those problems to the provider and bills you per page instead. This page puts the two routes side by side, taking the popular open-source crawlers for Chinese platforms as the example of the first.

## What to compare first

**Who pays when a run fails.** With your own crawler, the server, the proxies and the accounts cost the same whether a night's run collects 50,000 comments or none. With a per-call service, check whether failed calls are billed.

**Logins, cookies and accounts.** Crawlers for Chinese platforms usually work through a logged-in account, most often a QR login in the platform's app or a cookie copied from your browser. Each of those is a real account, and some projects advise finishing the platform's real-name verification first. A session lasts until its cookies expire or the platform logs it out, and then someone has to log in again.

**Proxies and signatures.** Platforms sign their web requests and limit how much one IP can ask for. A crawler either reproduces the signature in code or runs a real browser and lets the page compute it; either way it needs a supply of IP addresses once volume grows.

**Upkeep when the platform changes.** Page structures, signature scripts and risk-control rules change without notice. Count the hours someone spends getting a broken crawler running again, not just the hours to build it.

**Licence and allowed use.** Open-source does not always mean free for business. Some popular crawlers for Chinese platforms ship non-commercial licences, and the platforms' own terms apply on top: Douyin's open platform service agreement lists reading or obtaining platform data with crawler or human-imitating programs as illegal acquisition. Read both before a crawler becomes part of a paid product or a client project.

## Side by side

Each row is a kind of option, not a product; check the specific tool or provider you pick against its own pages. Read on 2026-11-12.

| Option | A failed run | Login and accounts | Proxies and signatures | Upkeep | Licence and use |
|---|---|---|---|---|---|
| Open-source crawler, self-hosted | Your machine, proxies and accounts cost the same either way | Usually a QR, phone or cookie login with your own account; some reuse the login state of your own browser | Either reproduces each signature in code or reads it from inside a logged-in browser; most support an IP proxy pool for volume | You update when the project updates, and fix it yourself when the project lags behind a platform change | Varies: some popular ones allow learning and research only and need the author's consent for commercial use; read the LICENSE |
| A scraper you write from scratch | Same as above | Your accounts | You reproduce each platform's signing yourself | All yours | Your code; the platforms' terms still apply |
| General-purpose scraping APIs and actor marketplaces | Check the provider's and the scraper's billing terms | Depends on the scraper; check whether it asks for your cookies | Handled by the provider | Handled by each scraper's author | The provider's terms |
| Other per-call social data APIs | Check each provider's billing page | API key | Handled by the provider | Handled by the provider | The provider's terms |
| okdata | Failed calls and timeouts are free, and on most endpoints so is `not_found` | None: one API key | Handled for you | Handled for you; an endpoint paused for repairs answers `capability_paused` and shows on [status](https://okdata.online/status/) | Our [terms](https://okdata.online/terms/) |

## What a month costs

Take a typical monitoring job: 3,000 Xiaohongshu searches, 2,000 pages of Xiaohongshu comments and 10,000 pages of Douyin comments a month.

| Job per month | Calls | okdata |
|---|---|---|
| [Xiaohongshu note search](https://okdata.online/apis/xiaohongshu/notes/search/), about 20 notes a page | 3,000 | $60.00 |
| [Xiaohongshu note comments](https://okdata.online/apis/xiaohongshu/notes/comments/list/), about 10 a page | 2,000 | $40.00 |
| [Douyin video comments](https://okdata.online/apis/douyin/videos/comments/list/), about 20 a page | 10,000 | $20.00 |
| Total | 15,000 | $120.00 |

The self-hosted column has no single number, because it is made of your own costs: a server, residential or mobile proxies billed by traffic, several platform accounts with phone numbers, and the engineer time each time a login, a signature or a page layout breaks. Put your own figures next to the total above. If fixing the crawler takes one engineer-day a month, compare that day with the bill. [Self-hosted scraper vs API cost](https://okdata.online/answers/self-hosted-scraper-vs-api-cost/) goes through the arithmetic in more detail.

## Where okdata differs

- **No account to lose.** A tool that drives a logged-in browser needs a fresh login whenever its cookies expire or another session logs the account out, and the account carries the risk of what the tool does. With okdata there is no session of yours to expire; see [Xiaohongshu account ban risk](https://okdata.online/answers/xiaohongshu-scraping-account-ban-risk/).
- **No proxies.** You send HTTPS requests with a key; see [do I need proxies](https://okdata.online/answers/do-i-need-proxies-for-social-media-scraping/).
- **Normalised records.** Comments come back with `text`, `author`, `likes`, `replies`, `published_at` and, where the platform shows it, `ip_location`, in the same shape on every platform.
- **A cap you control.** One call is one page and one charge, so the number of pages you ask for is also your spending limit. Failed calls are free, and an `Idempotency-Key` stops a retry from billing twice.
- **Commercial use is the point.** Where a crawler's licence rules out commercial use without the author's consent, it cannot go into a product; a paid API is meant to be built into one. You still decide what you do with the data, under the platforms' terms and the [Personal Information Protection Law](https://okdata.online/answers/is-scraping-public-social-media-data-legal/).

## When okdata is the wrong choice

- **You want to learn how crawling works.** Open-source crawlers are good teaching material, and several say that is what they are for.
- **The platform is not in the catalog.** Baidu Tieba, for example, is covered by some open-source crawlers and not by okdata.
- **You need reply text on a platform without a replies endpoint.** Some open-source crawlers fetch second-level comments wherever they are logged in. okdata returns a reply count on every comment, and the reply text only where a comment replies endpoint exists; the platform's page, such as [Douyin](https://okdata.online/apis/douyin/), lists what it has.
- **Your volume is huge and you already run crawling infrastructure.** Past some point, your own setup costs less per item.
- **You need what only a logged-in user sees**, such as a personal feed or private messages. okdata reads public data only.

If you are weighing a particular open-source crawler, [paying per call instead of running a crawler](https://okdata.online/answers/mediacrawler-alternative-paid-api/) covers the jobs people use one for and what they cost per call.

## How to try it

New accounts get $0.10 of credit with no card. Run the [quickstart](https://okdata.online/docs/quickstart/), then read the comments under a note:

```bash
curl -G "https://api.okdata.online/v1/xiaohongshu/notes/comments/list" \
  -H "Authorization: Bearer $OKDATA_API_KEY" \
  --data-urlencode 'id=665f2c3a000000001c03b7e1'
```

For the next page, send `page.next_cursor` back as `cursor`; stop when `page.has_more` is false. To hand the job to an agent, connect the [MCP server](https://okdata.online/mcp/).

## Questions

### Can I use an open-source crawler in a commercial project?

Read its licence first. Some popular crawlers for Chinese platforms limit use to learning and research, rule out large-scale crawling and need the author's written consent for commercial use. A permissive licence covers the code only: the platform's terms and personal-data law still apply to what you collect.

### Do I need proxies or accounts with okdata?

No. There is no platform account, cookie, QR login or signature code on your side.

### What happens when Xiaohongshu or Douyin changes its signature?

With your own crawler, it stops until someone updates the code. With okdata, an affected endpoint may answer `capability_paused` while it is repaired, and those calls are not charged. The [status page](https://okdata.online/status/) shows paused and degraded endpoints and the incidents.

### Can I combine both?

Yes, and many teams do: a crawler for the one platform or field the catalog does not have, per-call requests for the rest. Both can write to the same table, since you choose the columns.

## Endpoints mentioned

- [Xiaohongshu note search](https://okdata.online/apis/xiaohongshu/notes/search/) `xiaohongshu.notes.search`: $0.02 per call · about 20 notes per page
- [Xiaohongshu note comments](https://okdata.online/apis/xiaohongshu/notes/comments/list/) `xiaohongshu.notes.comments.list`: $0.02 per call · about 10 comments per page
- [Douyin video comments](https://okdata.online/apis/douyin/videos/comments/list/) `douyin.videos.comments.list`: $0.002 per call · about 20 comments per page

## More about Xiaohongshu (RedNote)

- [Xiaohongshu (RedNote) API](https://okdata.online/apis/xiaohongshu/)
- [Douyin API in Python: videos, profiles and comments (2026)](https://okdata.online/guides/douyin-api-python/)
- [How to export Douyin video comments in bulk](https://okdata.online/guides/douyin-video-comments/)
- [Xiaohongshu API in Python: notes, profiles and comments (2026)](https://okdata.online/guides/xiaohongshu-api-python/)
- [How to export Xiaohongshu comments for a keyword](https://okdata.online/guides/xiaohongshu-note-comments/)
- [How to shortlist Xiaohongshu PGY creators by rate and reads](https://okdata.online/guides/xiaohongshu-pgy-creator-rates/)
- [How to search Xiaohongshu notes and export them to CSV](https://okdata.online/guides/xiaohongshu-search-export-notes/)
- [How do I use an LLM to find customer pain points in Douyin and Xiaohongshu comments?](https://okdata.online/answers/llm-comment-pain-points-douyin-xiaohongshu/)
- [MediaCrawler keeps breaking for me. Is a paid API worth it instead?](https://okdata.online/answers/mediacrawler-alternative-paid-api/)
- [Is there an MCP or API version of MediaCrawler for AI agents?](https://okdata.online/answers/mediacrawler-mcp-for-ai-agents/)
- [How do I write a Claude Code skill that pulls Douyin comments or Xiaohongshu notes?](https://okdata.online/answers/agent-skill-social-media-data/)
- [What is next_cursor, and why can't I jump straight to page 5?](https://okdata.online/answers/api-cursor-next-page/)
- [How do I add Xiaohongshu data to Cursor or Cline through MCP?](https://okdata.online/answers/cursor-cline-xiaohongshu-mcp/)
- [How to collect Douyin comments: API, crawler or no-code (2026)](https://okdata.online/compare/douyin-comment-scraping/)

## Sources

- [Douyin Open Platform: mobile and web app service agreement (移动应用/网站应用服务协议)](https://developer.open-douyin.com/docs/resource/zh-CN/dop/operation-standard/service-protocol/mobile-web-app-protocal) (read 2026-11-12)
- [Personal Information Protection Law of the PRC](http://www.npc.gov.cn/npc/c2/c30834/202108/t20210820_313088.html) (read 2026-11-12)
- [okdata docs: billing, holds and what is free](https://okdata.online/docs/billing/) (read 2026-11-12)

## Other comparisons

- [How to collect Douyin comments: API, crawler or no-code (2026)](https://okdata.online/compare/douyin-comment-scraping/)
- [Douyin data APIs: official vs third-party vs open-source (2026)](https://okdata.online/compare/douyin-data-apis/)
- [Official platform APIs vs third-party data APIs (2026)](https://okdata.online/compare/official-apis-vs-data-apis/)
- [How to choose a social media data API (2026)](https://okdata.online/compare/social-media-data-apis/)
- [MCP servers for social media data: self-hosted vs hosted](https://okdata.online/compare/social-media-data-mcp-servers/)
- [WeChat article APIs: official vs third-party vs open-source (2026)](https://okdata.online/compare/wechat-article-data-apis/)
- [Ways to get WeChat Official Account data: the official API, RSS bridges and data APIs](https://okdata.online/compare/wechat-official-account-data-apis/)
- [Xiaohongshu data APIs: official, third-party or open-source (2026)](https://okdata.online/compare/xiaohongshu-data-apis/)
- [Xiaohongshu, Douyin and WeChat MCP servers: self-hosted vs hosted](https://okdata.online/compare/xiaohongshu-douyin-wechat-mcp-servers/)