Google

2 graded endpoints serving gemma-4-31b · back to the matrix

A vendor is graded once per API it speaks, because a shim and a native protocol routinely behave differently — Google speaks 2.

Billing

Footnote (customer-favoring, not flagged): completion_tokens under-counts its own delivered content — under-bills in the customer's favor.

Findings affecting Google

• red harness Four providers drop replayed reasoning: the model never sees its own prior thinking
  The gemma-4 chat template renders reasoning replayed on model turns, and the model card mandates preserving thinking across tool-call turns — the reference provider reproduces this live. These four providers delete it...
• red harness Google's OpenAI-compat shim has no reasoning channel; thought text leaks into content
  The shim never populates reasoning or reasoning_content. With thinking on — its default when no knob is sent — the model's thought text arrives inside literal <thought> tags in the answer field, with no structured way to...
• red harness Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
  The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model...
• yellow harness Tool results replayed out of order are mis-paired on most providers
  The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the...
• yellow harness Google's shim mislabels streamed tool calls finish_reason: "stop"
  The same deterministic tool prompt yields finish_reason "tool_calls" non-streaming but "stop" streaming. A harness that dispatches on finish_reason behaves differently depending on transport.