Hosted gemma-4-31b, graded.
All cells probed live · latest probe 2026-07-27 · model gemma-4-31b · how grading works
A provider has one job: surface the model's capabilities to application developers. The model's capabilities are defined by its chat template. Everything the template defines and the provider doesn't deliver — under the best dialect they offer — is documented here, with a reproducible probe behind every grade.
| Provider | Harness | Billing | Caching | Agentic turn speed | Faithfulness |
|---|---|---|---|---|---|
| Google (native)reference generateContent — Google's own protocol | A− 1 yellow | ✓ | testing | — | reference |
| Parasail OpenAI chat/completions | A− 1 yellow | ✓ | Sometimes | 17.4–25.1s | — |
| Lightning OpenAI chat/completions | B 2 yellow | ✓ | Yes | 1.2s | 20/29matches reference |
| Cerebras OpenAI chat/completions | B− 1 red · 1 yellow | ✓ | Yes | 0.7s | 24/29matches reference |
| Together AI OpenAI chat/completions | B− 1 red · 1 yellow | ✓ | Sometimes | 3.4–6.6s | 26/29matches reference |
| Novita OpenAI chat/completions | C− 2 red · 1 yellow | ✓ | No | ~30s | — |
| SambaNova OpenAI chat/completions | C− 2 red · 1 yellow | RED2.18× overcharge | No | 4.9s | 29/29matches reference |
| DeepInfra OpenAI chat/completions | C− 2 red · 1 yellow | ✓ | Sometimes | 24.8–29.7s | 27/29matches reference |
| AWS Bedrock (Chat Completions) OpenAI chat/completions on Bedrock Mantle | D 2 red · 2 yellow | ✓footnote on card | No | 3.3s | 28/29matches reference |
| AWS Bedrock (Responses) OpenAI Responses on Bedrock Mantle | D− 2 red · 3 yellow | ✓footnote on card | No | — | 25/29matches reference |
| Google (OpenAI-compat) OpenAI chat/completions shim over generateContent | D− 3 red · 1 yellow | ✓footnote on card | testing | — | 29/29matches reference |
How grading works
Grades are computed, not opined. Every flag is a documented defect with live evidence and a stated repro; the letter is arithmetic over the flags. This section is the rubric; each finding documents the probe that produced it.
Axes
- Harness — does the provider's serving layer deliver what the chat
template defines? Reasoning carried across tool-call turns, images in tool results,
tool_choiceactually enforced, tool results paired by id, arguments delivered byte-exact, honestfinish_reason. Letter grade, see ladder below. - Billing — do reported token counts match what was sent and received? Binary: ✓ or RED. Metering errors in the customer's favor are footnoted on the provider's card, not flagged.
- Caching — does the provider reuse context it has already seen, instead of re-reading (and re-billing) it on every call? Measured by timing a cold read of a fresh ~19K-token agentic transcript against a warm repeat one second later. Three answers: Yes (reused every trial), Sometimes (some trials), No (never).
- Agentic turn speed — what that same measured call — 20,000 context tokens in, 256 out — costs in seconds under best caching conditions; where caching is Sometimes, both bounds: cache-hit (lower) to no-hit (upper). Full speed measurements on the speed page.
- Faithfulness — does the served model behave like the creator org's reference implementation? Currently reported raw: agreement with the reference provider over a 29-cell deterministic battery (10 tasks × 3 languages, temperature 0, thinking disabled). A 0–10 score replaces this once the rubric is locked.
The harness ladder
Every provider starts at A. Each red flag — a defect that breaks or corrupts a real application — subtracts a full letter. Each yellow flag — a defect that degrades an application or violates the dialect contract with a workaround — subtracts half a step. The scale floors at F: A, A−, B, B−, C, C−, D, D−, F.
The reference provider (Google native) is graded under the same rules as everyone else — it holds an A− on its own yellow flag, not an exemption.