Introducing inferencecanary

the what and the why · the grade matrix

Inference serving is a multi-billion-dollar business, and a lot of it is quietly broken. inferencecanary is an open conformance suite and a standing public grade sheet for hosted LLM inference — think SSL Labs, but for inference providers: the same model, served by many vendors, probed the same way, graded in public. One-shot exposés decay; a continuously re-probed grade sheet compounds — and providers can optimize against it, which is the point.

A provider has one job

Surface the model's capabilities to application developers. Not some of them — all of them. What those capabilities are is not a matter of opinion: every open model ships a chat template, the text program that turns a conversation into the exact input the model reads. Everything the template defines — a reasoning channel, tool calls, images in tool results, how prior turns are rendered back — is the model's capability matrix. Anything a provider fails to deliver from it, under the best dialect that provider offers, is a defect, and this site documents it with a reproducible probe.

Status codes are not evidence

The founding observation of this suite: a 200-but-blind response — success returned, capability silently dropped — is a distinct and worse failure class than an honest error. Example: one provider accepts an image inside a tool result, returns 200, and the model never sees the image — it answers confidently about content it never saw. So every probe here verifies delivery, not acceptance: did the reasoning reach the model, did the image reach its eyes, did the constraint actually constrain. Where perception is the question, the probe asks the model to name what it saw.

What gets graded

• Harness — does the serving layer deliver what the chat template defines? Reasoning carried across tool-call turns, images in tool results, tool_choice actually enforced, tool results paired by id, arguments delivered byte-exact. Letter grade from a published ladder: every provider starts at A, each red flag subtracts a letter, each yellow half a step.
• Billing — do reported token counts match the bytes that actually crossed the wire? Binary: ✓ or RED. Metering errors in the customer's favor are footnotes, not flags.
• Caching — the axis agentic economics actually depend on and nobody advertises: does the provider reuse context it has already seen across tool-call turns, and what are the read and write speeds around it? Graded Yes / Sometimes / No by timing cold and warm reads of the same agentic transcript; the full speed table lives on the speed page.
• Faithfulness — does the served model behave like the creator org's reference implementation, or has it been quietly altered? Measured as agreement with the reference provider on a deterministic battery, reported raw until the rubric is locked.

The rules the graders play by

Grades are arithmetic over documented flags — computed, not opined. Every graded provider is an endpoint+dialect pair, because a vendor's shim and native protocol routinely behave differently. Every probe speaks the provider's best dialect, so no one is graded on a protocol they don't claim. The reference provider is graded under the same rules as everyone else — it holds its grade on its own flag, not an exemption. Where a claim is strong, the repro is a standalone script anyone can run with their own key. And defects are reported to providers in public: the disclosure and its outcome are part of each finding's record.

Where to go

• The grade matrix — every provider × every axis, one screen.
• The gemma-4-31b hub — the model's capability matrix and every provider serving it.
• Findings — the dated defect record, with evidence and disclosures.
• Changes — measured verdicts that moved between probe dates.

Made by Joel Solymosi and Claude. Have a tip, a correction, or a provider response? joel@custlabs.com.