Findings
Dated defects with live evidence and a stated repro. An observation enters this list only when it would matter to someone — a developer choosing a provider, or a provider fixing it. red breaks or corrupts a real application; yellow degrades one or violates the dialect contract with a workaround. Where we have reported a defect to the provider, the disclosure and its outcome are part of the record. The deepest findings carry full writeup pages with complete repro payloads.
SambaNova bills identical requests inflated token counts — 2.18× average overcharge
The same 35-token request body, sent 40 times in sequence, was billed ten distinct values between 27 and 2478 tokens. Across three independent batteries, 9,176 tokens were charged for 4,200 actually sent — 2.18× on average, carried in total_tokens, the field an invoice sees. A closed pair of co-fired requests (74 and 1,197 tokens) was billed 1,197 on both members: the totals do not conserve, so this is extra billing, not counts shuffled between requests. A standalone single-loop repro is in the suite.
Four providers drop replayed reasoning: the model never sees its own prior thinking
The gemma-4 chat template renders reasoning replayed on model turns, and the model card mandates preserving thinking across tool-call turns — the reference provider reproduces this live. These four providers delete it before templating: on a continuation after a tool call, the model re-derives or invents instead of recalling, and no error is raised. AWS Bedrock's Responses dialect is the sharpest case: it defines an explicit input item for replayed reasoning, accepts it without error, and never renders it to the model.
Google's OpenAI-compat shim has no reasoning channel; thought text leaks into content
The shim never populates reasoning or reasoning_content. With thinking on — its default when no knob is sent — the model's thought text arrives inside literal <thought> tags in the answer field, with no structured way to separate them. Google's native protocol serves the same model with a proper thought channel; the shim discards it.
Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model limits: four reject the request (400/422), SambaNova returns a deterministic 500, and AWS Bedrock (Responses) is the worst class — it returns 200 and the model never sees the image, answering confidently about content it never saw. Google's own shim rejects a capability Google's native protocol serves.
SambaNova enforces tool_choice by post-hoc 400, not constrained decoding
The knob is accepted, generation runs unconstrained, and when the output violates the constraint the request fails with a provider-level 400. Whether a request succeeds depends on what the model happened to emit: forced-tool workflows that pass in testing throw unrecoverable runtime errors in production, with no tool call in the error body to repair from.
DeepInfra validates tool_choice but does not enforce it
Invalid values are rejected with 400/422 — every surface signal says the capability exists — but a valid constraint has zero effect on decoding: "required" returns prose with zero calls, and forcing a tool by name yields a call to a different tool. An application relying on forced tool calls silently gets prose.
Novita silently ignores tool_choice
"required" and forced-specific constraints are accepted and have no effect on decoding; only "none" is honored. The application consequence matches DeepInfra's: workflows that depend on forced tool calls silently receive prose.
Tool results replayed out of order are mis-paired on most providers
The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the template renders by name, and these providers render positionally, dropping their own dialect's id contract. An async harness that appends results in completion order silently swaps payloads. One provider pairs correctly, proving the fix is implementable provider-side.
AWS Bedrock never emits parallel tool calls
parallel_tool_calls: true is accepted on both Bedrock APIs, but only a single call ever comes back; every other provider emits two for the same two-lookup prompt. A model capability suppressed at the serving layer; the workaround is sequential calling at double the turns and latency.
AWS Bedrock un-escapes tool-call arguments in transit
At temperature 0, nine providers agree the model returns a tricky string argument with literal escape sequences. Both Bedrock APIs instead return real control characters — one extra unescape pass applied to the model's own output — plus a stray one-word content beside the call. Any argument whose bytes matter (embedded JSON, code, regexes) is corrupted.
Google's shim mislabels streamed tool calls finish_reason: "stop"
The same deterministic tool prompt yields finish_reason "tool_calls" non-streaming but "stop" streaming. A harness that dispatches on finish_reason behaves differently depending on transport.
Lightning leaks template reasoning markers into content on truncation
At tight max_tokens, literal template markers appear in content; unconstrained budgets come back clean. Output is polluted exactly when the response is cut short.
AWS Bedrock (Responses) rejects temperature
temperature returns 400 unsupported_parameter, so deterministic sampling is unreachable on this provider: every run is stochastic, degrading reproducibility for applications and evals alike.