# About inferencecanary

inferencecanary is an open conformance suite and public grade sheet for hosted LLM
inference. A provider has one job: surface the model's capabilities to application
developers. The model's capabilities are defined by its chat template — everything the
template defines is the model's capability matrix, and everything a provider fails to
deliver from it, under the best dialect they offer, is documented here.

## Method

Every graded provider is an endpoint+dialect pair — the same vendor appears once per API
it speaks, because the shim and the native protocol routinely behave differently. Every
probe speaks each provider's best dialect, parameterized the way that provider documents. Every
flag is backed by a live, deterministic probe with the request and response preserved;
where a claim is strong, the repro is a standalone single-loop script anyone can run
with their own key. Status codes are not trusted: a 200 that silently drops capability
is a distinct — and worse — failure class than an honest error, so probes verify
delivery, not acceptance.

Grades are arithmetic over flags, computed from a published ladder — not editorial
judgment calls. The reference provider is graded under the same rules as everyone else.
Findings are point-in-time: every cell carries the date it was last probed, and grades
change when providers fix things (or break them).

## Disclosure

Defects are reported to providers in public, and the disclosure — with its outcome —
is part of each finding's record. Providers who fix a defect get the fix probed and
the grade updated; a response that does not address the defect is noted as such.

## Who

Made by Joel Solymosi (https://custlabs.com) and Claude.
Tips, corrections, provider responses: joel@custlabs.com