Google (OpenAI-compat)

OpenAI chat/completions shim over generateContent · serves gemma-4-31b-it as gemma-4-31b · probed live 2026-07-26 · back to the matrix

Google's OpenAI-compatibility shim, serving the same model as the native provider — graded on what the shim delivers.

Harness

D−

A − 3 red − 1 yellow, per the ladder; flags below.

Billing

✓

Reported usage consistent with bytes on the wire. Footnote: completion_tokens under-counts its own delivered content — under-bills in the customer's favor.

Caching

testing

Context reuse across tool-call turns is still being probed on this provider.

Faithfulness

29/29

Agreement with the reference provider over the 29-cell deterministic battery (temperature 0, thinking disabled). 0–10 score arrives with the locked rubric.

Flags

red harness Four providers drop replayed reasoning: the model never sees its own prior thinking
The gemma-4 chat template renders reasoning replayed on model turns, and the model card mandates preserving thinking across tool-call turns — the reference provider reproduces this live. These four providers delete it...
red harness Google's OpenAI-compat shim has no reasoning channel; thought text leaks into content
The shim never populates reasoning or reasoning_content. With thinking on — its default when no knob is sent — the model's thought text arrives inside literal <thought> tags in the answer field, with no structured way to...
red harness Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model...
yellow harness Google's shim mislabels streamed tool calls finish_reason: "stop"
The same deterministic tool prompt yields finish_reason "tool_calls" non-streaming but "stop" streaming. A harness that dispatches on finish_reason behaves differently depending on transport.

Every flag links to its finding — evidence, repro, and disclosure records live there; the deepest findings have full writeup pages.