About

inferencecanary is an open conformance suite and public grade sheet for hosted LLM inference. A provider has one job: surface the model's capabilities to application developers. The model's capabilities are defined by its chat template — everything the template defines is the model's capability matrix, and everything a provider fails to deliver from it, under the best dialect they offer, is documented here.

Method

Every graded provider is an endpoint+dialect pair — the same vendor appears once per API it speaks, because the shim and the native protocol routinely behave differently. Every probe speaks each provider's best dialect, parameterized the way that provider documents. Every flag is backed by a live, deterministic probe with the request and response preserved; where a claim is strong, the repro is a standalone single-loop script anyone can run with their own key. Status codes are not trusted: a 200 that silently drops capability is a distinct — and worse — failure class than an honest error, so probes verify delivery, not acceptance.

Grades are arithmetic over flags, computed from a published ladder — not editorial judgment calls. The reference provider is graded under the same rules as everyone else. Findings are point-in-time: every cell carries the date it was last probed, and grades change when providers fix things (or break them).

Disclosure

Defects are reported to providers in public, and the disclosure — with its outcome — is part of each finding's record. Providers who fix a defect get the fix probed and the grade updated; a response that does not address the defect is noted as such.

Who

Made by Joel Solymosi and Claude. Have a tip, a correction, or a provider response? joel@custlabs.com.