Four providers drop gemma-4's replayed reasoning

red · harness · probed live 2026-07-26 · affects Together AI, AWS Bedrock (Chat Completions), AWS Bedrock (Responses), Google (OpenAI-compat)

Replayed reasoning is the model's own prior thinking, sent back to it on later turns of the same conversation. In an agent loop it works like this: the model reasons, calls a tool, and the application replays the whole conversation — including that reasoning — together with the tool's result, so the model can pick up where it left off. Four providers delete the reasoning from the replay before the model ever sees it. The model then re-derives or invents instead of recalling — and no error is raised.

The model's own template says it should be there

A model's chat template is the text program, shipped with the model, that turns a conversation into the exact input the model reads. gemma-4's template renders reasoning replayed on model turns after the last user message, and the gemma-4 model card mandates preserving thinking across tool-call turns. Carrying replayed reasoning is the model's own reference behavior — Google's native API, the reference provider, reproduces it live. A provider that deletes it is deviating from the template of the model it serves.

How it was measured: the 99 memory test

A canary — the number 99, wrapped in an unrelated aside — is planted only in the reasoning field of a replayed model turn, across a fabricated tool call. Then the model is asked what the number was. If the replayed reasoning reaches the model, it recalls 99; if the provider deleted it, the model invents a number. Recall versus invention is the verdict — a probe on delivery, not on status codes.

Who drops it

• Together AI — replayed reasoning deleted before templating; the canary is never recalled.
• AWS Bedrock (Chat Completions) — renders live reasoning but deletes replayed reasoning; same verdict on every rep.
• AWS Bedrock (Responses) — the sharpest case: the Responses dialect defines an explicit input item for replayed reasoning ({"type": "reasoning", …}), accepts it without error, and never renders it to the model. A documented field that does nothing.
• Google (OpenAI-compat) — drops it, while Google's own native protocol — same model, same company — carries it.

The other seven providers carry replayed reasoning. That includes every remaining OpenAI-compat endpoint — so this is a provider serving-layer choice, not a dialect limitation and not a model limit.

Why it matters

Multi-step tool workflows silently lose the model's working state at every turn. The model plans in its reasoning, acts, and then — on these four providers — returns to a conversation where the plan no longer exists. It answers anyway, re-deriving or inventing, with no error for the application to catch. Longer agent runs degrade in a way that looks like model weakness and is actually the serving layer discarding the model's memory.

Probe: the cot-carry probe in the suite (src/probes/cot-carry.ts), temp 0, evidence JSONs with full request/response preserved. Template ground truth: the executed chat_template.jinja from the upstream gemma-4-31b release.