# inferencecanary — hosted gemma-4-31b, graded

A provider has one job: surface the model's capabilities to application developers.
The model's capabilities are defined by its chat template. Everything the template
defines and the provider doesn't deliver — under the best dialect they offer — is
documented here, with a reproducible probe behind every grade.

All cells probed live; latest probe 2026-07-27. Grading rubric: see "How grading works" below.
Every page on this site has a markdown twin at `<page-url>index.md`; page index at [/llms.txt](https://inferencecanary.com/llms.txt).

## The grade sheet

| Provider | Dialect | Harness | Billing | Caching | 20K-token agentic turn — speed | 20K-token agentic turn — cost | Faithfulness |
|---|---|---|---|---|---|---|---|
| [Google (native)](https://inferencecanary.com/google/gemma-4-31b/index.md) (reference) | generateContent — Google's own protocol | A− | OK | testing | — | TBD | coming soon |
| [Parasail](https://inferencecanary.com/parasail/gemma-4-31b/index.md) | OpenAI chat/completions | A− | OK | Sometimes | 17.4–25.1s | 0.13–0.31¢ | coming soon |
| [Lightning](https://inferencecanary.com/lightning/gemma-4-31b/index.md) | OpenAI chat/completions | B | OK | Yes | 1.2s | 0.29¢ | coming soon |
| [Cerebras](https://inferencecanary.com/cerebras/gemma-4-31b/index.md) | OpenAI chat/completions | B− | OK | Yes | 0.7s | 2.0¢ | coming soon |
| [Together AI](https://inferencecanary.com/togetherai/gemma-4-31b/index.md) | OpenAI chat/completions | B− | OK | Sometimes | 3.4–6.6s | 0.80¢ | coming soon |
| [Novita](https://inferencecanary.com/novita/gemma-4-31b/index.md) | OpenAI chat/completions | C− | OK | No | ~30s | 0.29¢ | coming soon |
| [SambaNova](https://inferencecanary.com/sambanova/gemma-4-31b/index.md) | OpenAI chat/completions | C− | RED (2.18× overcharge) | No | 4.9s | 0.79¢ | coming soon |
| [DeepInfra](https://inferencecanary.com/deepinfra/gemma-4-31b/index.md) | OpenAI chat/completions | C− | OK | Sometimes | 24.8–29.7s | 0.27¢ | coming soon |
| [AWS Bedrock (Chat Completions)](https://inferencecanary.com/aws/gemma-4-31b/index.md) | OpenAI chat/completions on Bedrock Mantle | D | OK | No | 3.3s | 0.29¢ | coming soon |
| [AWS Bedrock (Responses)](https://inferencecanary.com/aws-responses/gemma-4-31b/index.md) | OpenAI Responses on Bedrock Mantle | D− | OK | No | — | 0.29¢ | coming soon |
| [Google (OpenAI-compat)](https://inferencecanary.com/google-openai/gemma-4-31b/index.md) | OpenAI chat/completions shim over generateContent | D− | OK | testing | — | TBD | coming soon |


## Current findings

- **[red · billing] SambaNova bills identical requests inflated token counts — 2.18× average overcharge** (2026-07-26; affects sambanova) — details: [/sambanova-overbilling/index.md](https://inferencecanary.com/sambanova-overbilling/index.md)
- **[red · harness] Four providers drop replayed reasoning: the model never sees its own prior thinking** (2026-07-26; affects togetherai, aws, aws-responses, google-openai) — details: [/replayed-reasoning-dropped/index.md](https://inferencecanary.com/replayed-reasoning-dropped/index.md)
- **[red · harness] Google's OpenAI-compat shim has no reasoning channel; thought text leaks into content** (2026-07-26; affects google-openai) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#google-openai-no-reasoning-channel)
- **[red · harness] Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers** (2026-07-26; affects cerebras, novita, sambanova, deepinfra, aws, aws-responses, google-openai) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#tool-result-images)
- **[red · harness] SambaNova enforces tool_choice by post-hoc 400, not constrained decoding** (2026-07-26; affects sambanova) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#sambanova-tool-choice-400)
- **[red · harness] DeepInfra validates tool_choice but does not enforce it** (2026-07-26; affects deepinfra) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#deepinfra-tool-choice-unenforced)
- **[red · harness] Novita silently ignores tool_choice** (2026-07-26; affects novita) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#novita-tool-choice-ignored)
- **[yellow · harness] Tool results replayed out of order are mis-paired on most providers** (2026-07-26; affects lightning, sambanova, cerebras, togetherai, deepinfra, novita, parasail, google) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#tool-results-out-of-order)
- **[yellow · harness] AWS Bedrock never emits parallel tool calls** (2026-07-26; affects aws, aws-responses) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#bedrock-single-tool-call)
- **[yellow · harness] AWS Bedrock un-escapes tool-call arguments in transit** (2026-07-26; affects aws, aws-responses) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#bedrock-argument-corruption)
- **[yellow · harness] Google's shim mislabels streamed tool calls finish_reason: "stop"** (2026-07-26; affects google-openai) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#google-openai-finish-reason)
- **[yellow · harness] Lightning leaks template reasoning markers into content on truncation** (2026-07-26; affects lightning) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#lightning-marker-leak)
- **[yellow · harness] AWS Bedrock (Responses) rejects temperature** (2026-07-26; affects aws-responses) — details: [/findings/index.md](https://inferencecanary.com/findings/index.md#aws-responses-temperature)


## How grading works

- **Harness** (letter grade): does the serving layer deliver what the chat template
  defines — reasoning carried across tool-call turns, images in tool results,
  `tool_choice` enforced, tool results paired by id, arguments byte-exact, honest
  `finish_reason`. Every provider starts at A; each red flag (breaks or corrupts a real
  application) subtracts a full letter, each yellow flag (degrades one, or violates
  the dialect contract with a workaround) subtracts half a step. Scale: A, A−, B,
  B−, C, C−, D, D−, floor F.
- **Billing** (binary): reported token counts vs bytes actually on the wire.
  Metering errors in the customer's favor are footnoted, not flagged.
- **Caching**: does the provider reuse context it has already seen, instead of
  re-reading (and re-billing) it on every call? Measured by timing a cold read of a
  fresh ~19K-token agentic transcript against a warm repeat one second later. Three
  answers: Yes (reused every trial), Sometimes (some trials), No (never).
- **20K-token agentic turn**: what one measured call — 20,000 context tokens in,
  256 out — costs. Speed is that call in seconds under best caching conditions;
  where caching is Sometimes, both bounds: cache-hit (lower) to no-hit (upper).
  Cost is the same call in cents at list price, derived the same way: where caching
  is Sometimes and the provider prices cached reads cheaper, both bounds. A cache hit
  only changes the price where a cached-read rate exists — a hit billed as uncached
  costs full price. TBD where no price is on record.
  Full speed measurements: [https://inferencecanary.com/speed/index.md](https://inferencecanary.com/speed/index.md).
- **Faithfulness**: agreement with the creator org's reference implementation over
  a 29-cell deterministic battery (10 tasks × 3 languages, temperature 0, thinking
  disabled). Scores publish once the 0–10 rubric is locked — the matrix says coming
  soon until then.

Made by Joel Solymosi (https://custlabs.com) and Claude — tips: joel@custlabs.com