Speed
measured live 2026-07-27 · model gemma-4-31b · point-in-time, under that evening's load · back to the matrix
What a real agentic call — 20,000 tokens of context in, 256 tokens out — costs in wall-clock, per provider. Two things dominate it: whether the provider reuses context it has already seen, and how fast it reads uncached context. The generation speed providers advertise barely matters at agentic context sizes; the spread between working providers is roughly 13×.
| Provider | Caching | Floor (ms) | Floor high‑water (ms) | Read (tok/s) | Write (tok/s) | TTFT @20K | Turn, cold | Turn, hit |
|---|---|---|---|---|---|---|---|---|
| Cerebras | Yes3/3 hits | 197 | 197 | 13.9K | 1855 | 1.6s | 1.8s | 0.7s |
| Lightning | Yes3/3 hits | 186 | 186 | 21.5K | 339 | 1.1s | 1.9s | 1.2s |
| AWS Bedrock (Chat Completions) | No0/8 hits | 385 | 406 | 12.2K | 198 | 2.0s | 3.3s | — |
| AWS Bedrock (Responses) | No0/8 hits | 363 | 371 | 6.8K | — | 3.3s | — | — |
| SambaNova | No0/8 hits | 1255 | 1681 | 8.6K | 189 | 3.6s | 4.9s | — |
| Together AI | Sometimes2/3 hits | 284 | 284 | 6.2K | 83 | 3.5s | 6.6s | 3.4s |
| DeepInfra | Sometimes1/3 hits | 418 | 15,071 | 3.7K | 11 | 5.8s | 29.7s | 24.8s |
| Parasail | Sometimes1/3 hits | 289 | 289 | 2.5K | 15 | 8.3s | 25.1s | 17.4s |
| Novita | No0/8 hits | 439 | 70,035 | 3.5–6K | 9–12 | ~4–6s | ~30s | — |
| Google (native) | testing | 638 | 638 | — | 5.3 | — | — | — |
| Google (OpenAI-compat) | testing | 593 | 593 | — | 5.3 | — | — | — |
What the columns mean
The measurement: a fresh ~19K-token agentic transcript is read cold, then again warm one second later. A unique marker makes the cold read provably unseen, so a warm answer arriving under 3× the provider's floor can only be cache reuse. TTFT = time to first token.
- Caching — is already-seen context reused? Yes: every trial. Sometimes: some trials. No: never.
- Floor — TTFT of a tiny request. Pure overhead: network, queue, scheduling.
- Floor high-water — the worst floor ever observed. How long a trivial request can pend under load.
- Read — how fast uncached context is ingested.
- Write — how fast new tokens are generated.
- TTFT @20K — estimated wait before the first token over 20,000 uncached context tokens. Floor + 20,000 ÷ read.
- Turn, cold — TTFT @20K plus 256 written tokens. One full uncached call.
- Turn, hit — the same call when the cache hits. Empty where there is no cache to hit.
Notes
- AWS Bedrock (Responses) — Write speed unmeasured: the streamed usage report on this API carried no completion-token count. The sibling AWS chat endpoint writes 198 tok/s on the same backend.
- Novita — Least trustworthy estimates on the sheet: within single runs, cold reads of the identical context ranged 9.7–32.6s and one warm call pended 40.1s while tiny requests answered in 0.4s — large calls queue when small ones don't.
- Google (native) — Read speed unmeasured: the key's per-minute token quota can't fit a cold+warm measurement pair at this context size. Write speed of 5.3 tok/s means 256 tokens take ~48 seconds.
- Google (OpenAI-compat) — Read speed unmeasured: the key's per-minute token quota can't fit a cold+warm measurement pair at this context size. Write speed of 5.3 tok/s means 256 tokens take ~48 seconds.
Speeds are point-in-time under real load, not a benchmark under ideal conditions — read speed on one provider moved from 7.0K to 12.2K tok/s between same-night runs. The caching class per provider also appears on the grade matrix.