Answers
If I get social data through an API, do the platform's terms and robots.txt still matter?
Short answer
Yes, they still matter, in a different way. With an API such as okdata you do not log in, accept a platform's user agreement or crawl its pages yourself, so account bans and crawler rules stop being your daily problem (Douyin search is $0.02 a call). What does not change: privacy and copyright law apply to whatever you keep and publish, and platforms can object to uses that harm them. This is general information, not legal advice.
People often assume a data API launders the question away: the vendor collected it, so it is the vendor's problem. That is too simple. An API changes who does the collecting. It does not change what the data is, or what you are allowed to do with it. This page is general information, not legal advice.
What changes when you use an API
- No account, no click-through. You never sign up on Douyin or Weibo for the job, never accept their user agreement for it, and never send requests to their sites. The questions of bot detection and account bans move to the vendor.
- robots.txt is not addressed to you. The Robots Exclusion Protocol tells crawlers which paths a site prefers they skip. You are not crawling, but the site's stated preference is still useful context when you judge how a use will look.
What does not change
- Personal data law. Once you store posts and usernames, you are the one handling personal information under PIPL, GDPR or local law.
- Copyright. Post text, images and video belong to their authors. Analysis is one thing; republishing is another.
- The platform's interests. Chinese courts have heard unfair-competition cases brought by platforms over third parties' use of their data, such as Weibo against Maimai. Uses that substitute for the platform, or harm it, invite that kind of dispute.
- The vendor's terms. okdata's terms are still a draft, but their outline is public: public data only, no use that breaks the law where you operate or the rights of the people in the data, and no reselling the raw feed as a competing API.
Keep less than you collect
Many questions are answered by aggregates, which hold no personal data at all. Counting how many Weibo posts mention a product each day needs only IDs and timestamps:
import collections, os, requests
URL = "https://api.okdata.online/v1/weibo/posts/search"
H = {"Authorization": f"Bearer {os.environ['OKDATA_API_KEY']}"}
per_day, cursor = collections.Counter(), None
for _ in range(10):
params = {"query": "新品发布", "sort": "latest", "fields": "id,published_at",
**({"cursor": cursor} if cursor else {})}
body = requests.get(URL, headers=H, params=params, timeout=60).json()
per_day.update(p["published_at"][:10] for p in body["data"]["items"])
if not body["page"]["has_more"]:
break
cursor = body["page"]["next_cursor"]
print(sorted(per_day.items()))
Questions to ask any data vendor
- Do you collect public data only, without logged-in sessions?
- Do you store what I fetch, and for how long?
- Where is the data processed?
Douyin video search and Weibo post search here return public data only, read from the platform when you call; a response served from a short-lived cache says so in meta.cached. For a broader view of the legal picture, see is it legal to scrape public social media data?, and for aggregate-first analysis, trend research.
Updated