Answers

How do I cut the tokens when I feed social media API results to an LLM?

Short answer

Ask the API for less instead of trimming afterwards: okdata takes a list of the fields to keep and a compact view that drops empty values, and neither changes the price ($0.02 per Xiaohongshu search page either way). For a comment sentiment task, three fields per comment is usually all the model needs.

Three Douyin searches into a research session, the context is half full and the agent starts forgetting the brief. Claude Code may even move a large tool result to a file because it is too big to show. The fix is to ask for less data per item, not fewer items: a page of 20 videos where each video carries five fields is a small fraction of the same page with author avatars, image lists, hashtags and every metric.

Two parameters that shrink every response

Every okdata endpoint, from Douyin video search to Xiaohongshu note search and Douyin video comments, takes the same two options:

  • fields keeps only the paths you list, with dots for nested values: fields=id,title,metrics.likes. A path that matches nothing is reported in warnings, so typos do not fail silently.
  • view sets how much comes back by default. standard is the default, compact also drops every null field, and full adds platform extras under ext that you rarely want in a prompt.

Neither changes the price or the number of items. You pay per call, so the slim page costs the same as the fat one.

Which fields for which job

Task fields Price per page
Comment sentiment text,likes,published_at $0.002
Hot list scan rank,name,heat $0.002
Creator shortlist id,name,followers,posts,likes,bio $0.002
Content research id,url,title,text,metrics.likes,metrics.comments,author.name $0.02

See the difference yourself

This runs the same search twice, once in the default shape and once trimmed, and prints the size of each. It is two calls, $0.04 in total.

import json, os, requests

URL = "https://api.okdata.online/v1/douyin/videos/search"
H = {"Authorization": f"Bearer {os.environ['OKDATA_API_KEY']}"}

for fields in (None, "id,title,metrics.likes,author.name"):
    params = {"query": "露营装备", **({"fields": fields} if fields else {})}
    body = requests.get(URL, headers=H, params=params, timeout=60).json()
    if body["code"] != "ok":
        raise RuntimeError(body["message"])
    size = len(json.dumps(body["data"], ensure_ascii=False))
    print(fields or "standard", size, "characters", body["warnings"])

Over MCP, the agent passes the same thing inside params, for example {"query": "露营装备", "fields": "id,title,metrics.likes"}. Tell it to do so in your prompt; models rarely ask for fewer fields on their own.

Habits that save more than any parameter

  • Keep the raw data, show the model the slim copy. Save full responses to a file or database, and pass the model only what the current step needs. You can always go back to the raw data without paying again.
  • Summarize each page before fetching the next. Ten pages of comments become ten short notes, then one summary, instead of one giant prompt.
  • Transcripts: fields=text returns the plain transcript without per-segment timestamps, often half the size. Keep the segments only when you need to cite the second.
  • Cap the items, not just the fields. If the model only needs the top 20 comments by likes, sort in code and drop the rest before the prompt.

Other ways to do it

Post-processing with jq or a few lines of Python works with any API and costs nothing; it just means the full response travels to your code first. Letting the MCP client truncate long results is the worst option: it cuts from the end, so the model silently loses the last items rather than the useless fields. The full parameter list is on the requests page.

Updated