Latency by model, measured at the gateway

Benchmarks measure models in isolation. Production teams care about the number their users see: how long the request takes end to end. That number includes upstream queueing, prefill and generation, so it is noisy — but it is the honest one.

The tiers

Measured p50 latency

Only models whose median prompt is under 2,000 tokens are listed, so the comparison is not distorted by prompt size (median input shown). Latency is end-to-end at the gateway.

Three observations

1. Cheap and fast coincide at the small-model end

Qwen3.5-OCR answers in 370ms at $0.70 per 1M tokens. Hunyuan MT2-plus translates in 975ms at $0.072 per 1M. The expensive end of the catalog is not faster — it is slower, because flagship models generate longer answers and are sized for reasoning rather than turnaround. If your task is classification, extraction or translation, the cheap model is usually also the fast one.

2. Output length, not input length, drives the tail

The p90 values spread far more than p50 — from 869ms to 71s. That tail is generation: a model writing 900 tokens takes far longer than one writing 50, regardless of prompt. Streaming hides this from users but not from your request pipeline.

3. Generation tasks have a different shape entirely

Image models are 10–68 seconds at p50. That is not a latency problem to optimise; it is a job to queue. Plan for async: submit, poll, webhook — not request/response.

How to use this

• Interactive work (editor autocomplete, chat): pick from the sub-2s tier.
• Batch work (summaries, extraction): latency is irrelevant; pick on cost.
• Generation (image, video): design for async from day one.

Every model in the table is reachable through the same endpoint, so switching tiers is a per-request model id change. Prices: live feed; cost side: what real calls cost.

FAQ

What is a typical latency for a Chinese LLM API?

From 30 days of gateway traffic: OCR and translation models respond in 324-975ms at p50, mid-tier chat models in 1.1-2.6s, flagship text models in 3-16s depending on size, and image generation takes 10-68s. The median p50 across 63 models was 1.86s.

Why are the cheapest models often the fastest?

Small models are cheaper per token and need less compute per request. Qwen3.5-OCR responds in 370ms at $0.70 per 1M tokens, while a flagship 1M-context model can take 16s. Cheap and fast coincide at the small-model end.

Does latency include generation time?

Yes. This is end-to-end time as observed at the gateway: upstream queueing plus generation. It is not time-to-first-token, so models producing long outputs appear slower even when their prefill is fast.

Do larger prompts make responses slower?

Not in a simple way in this data, because prompt size correlates with which model a workload chooses. Comparing only models whose median prompt is under 2K tokens removes that confound, and there the ordering follows model class rather than prompt size.

Window: 30 days ending 2026-10-22. 63 models with at least 25 latency samples. Latency is end-to-end at the gateway (queue + prefill + generation), recorded per request; p50/p90 computed over all successful calls in the window. Prices: /china-ai-leaderboard.json.