# Introducing inferencecanary

HTML version: [https://inferencecanary.com/introducing-inference-canary/](https://inferencecanary.com/introducing-inference-canary/)

Inference serving is a multi-billion-dollar business, and a lot of it is quietly broken.
**inferencecanary** is an open conformance suite and a standing public grade sheet for hosted
LLM inference — think SSL Labs, but for inference providers: the same model, served by many
vendors, probed the same way, graded in public. One-shot exposés decay; a continuously
re-probed grade sheet compounds — and providers can optimize against it, which is the point.

## A provider has one job

Surface the model's capabilities to application developers. Not some of them — all of them.
What those capabilities *are* is not a matter of opinion: every open model ships a **chat
template**, the text program that turns a conversation into the exact input the model reads.
Everything the template defines — a reasoning channel, tool calls, images in tool results, how
prior turns are rendered back — is the model's capability matrix. Anything a provider fails to
deliver from it, under the best dialect that provider offers, is a defect, and this site
documents it with a reproducible probe.

## Status codes are not evidence

The founding observation of this suite: a **200-but-blind** response — success returned,
capability silently dropped — is a distinct and *worse* failure class than an honest error.
Example: one provider accepts an image inside a tool result, returns 200, and the model never
sees the image — it answers confidently about content it never saw. So every probe here
verifies *delivery*, not acceptance: did the reasoning reach the model, did the image reach its
eyes, did the constraint actually constrain. Where perception is the question, the probe asks
the model to name what it saw.

## What gets graded

- **Harness** — does the serving layer deliver what the chat template defines? Reasoning
  carried across tool-call turns, images in tool results, `tool_choice` actually enforced, tool
  results paired by id, arguments delivered byte-exact. Letter grade from a published ladder:
  every provider starts at A, each red flag subtracts a letter, each yellow half a step.
- **Billing** — do reported token counts match the bytes that actually crossed the wire?
  Binary: OK or RED. Metering errors in the customer's favor are footnotes, not flags.
- **Caching** — the axis agentic economics actually depend on and nobody advertises: does the
  provider reuse context it has already seen across tool-call turns, and what are the read and
  write speeds around it? Graded *Yes* / *Sometimes* / *No* by timing cold and
  warm reads of the same agentic transcript; the full speed table:
  [https://inferencecanary.com/speed/index.md](https://inferencecanary.com/speed/index.md).
- **Faithfulness** — does the served model behave like the creator org's reference
  implementation, or has it been quietly altered? Measured as agreement with the reference
  provider on a deterministic battery, reported raw until the rubric is locked.

## The rules the graders play by

Grades are arithmetic over documented flags — computed, not opined. Every graded provider is an
endpoint+dialect pair, because a vendor's shim and native protocol routinely behave
differently. Every probe speaks the provider's best dialect, so no one is graded on a protocol
they don't claim. The reference provider is graded under the same rules as everyone else — it
holds its grade on its own flag, not an exemption. Where a claim is strong, the repro is a
standalone script anyone can run with their own key. And defects are reported to providers in
public: the disclosure and its outcome are part of each finding's record.

## Where to go

- [The grade matrix](https://inferencecanary.com/index.md) — every provider × every axis, one screen.
- [The gemma-4-31b hub](https://inferencecanary.com/gemma-4-31b/index.md) — the model's capability matrix and every provider serving it.
- [Findings](https://inferencecanary.com/findings/index.md) — the dated defect record, with evidence and disclosures.
- [Changes](https://inferencecanary.com/changes/index.md) — measured verdicts that moved between probe dates.

Made by Joel Solymosi (https://custlabs.com) and Claude. Have a tip, a correction, or a
provider response? joel@custlabs.com.