Hosted gemma-4-31b, graded.

All cells probed live · latest probe 2026-07-27 · model gemma-4-31b · how grading works

A provider has one job: surface the model's capabilities to application developers. The model's capabilities are defined by its chat template. Everything the template defines and the provider doesn't deliver — under the best dialect they offer — is documented here, with a reproducible probe behind every grade.

How grading works

Grades are computed, not opined. Every flag is a documented defect with live evidence and a stated repro; the letter is arithmetic over the flags. This section is the rubric; each finding documents the probe that produced it.

Axes

• Harness — does the provider's serving layer deliver what the chat template defines? Reasoning carried across tool-call turns, images in tool results, tool_choice actually enforced, tool results paired by id, arguments delivered byte-exact, honest finish_reason. Letter grade, see ladder below.
• Billing — do reported token counts match what was sent and received? Binary: ✓ or RED. Metering errors in the customer's favor are footnoted on the provider's card, not flagged.
• Caching — does the provider reuse context it has already seen, instead of re-reading (and re-billing) it on every call? Measured by timing a cold read of a fresh ~19K-token agentic transcript against a warm repeat one second later. Three answers: Yes (reused every trial), Sometimes (some trials), No (never).
• 20K-token agentic turn — what one measured call — 20,000 context tokens in, 256 out — costs. Speed is that call in seconds under best caching conditions; where caching is Sometimes, both bounds: cache-hit (lower) to no-hit (upper). Full speed measurements on the speed page. Cost — the same call in cents at list price, derived the same way: where caching is Sometimes and the provider prices cached reads cheaper, both bounds. A cache hit only changes the price where a cached-read rate exists — a hit billed as uncached costs full price. TBD where no price is on record.
• Faithfulness — does the served model behave like the creator org's reference implementation? Measured as agreement with the reference provider over a 29-cell deterministic battery (10 tasks × 3 languages, temperature 0, thinking disabled). Scores publish once the 0–10 rubric is locked — the matrix says coming soon until then.

The harness ladder

Every provider starts at A. Each red flag — a defect that breaks or corrupts a real application — subtracts a full letter. Each yellow flag — a defect that degrades an application or violates the dialect contract with a workaround — subtracts half a step. The scale floors at F: A, A−, B, B−, C, C−, D, D−, F.

The reference provider (Google native) is graded under the same rules as everyone else — it holds an A− on its own yellow flag, not an exemption.