# Four providers drop gemma-4's replayed reasoning

red · harness · probed live 2026-07-26 · affects Together AI, AWS Bedrock (Chat Completions),
AWS Bedrock (Responses), Google (OpenAI-compat) ·
HTML version: [https://inferencecanary.com/replayed-reasoning-dropped/](https://inferencecanary.com/replayed-reasoning-dropped/)

**Replayed reasoning** is the model's own prior thinking, sent back to it on later turns of the
same conversation. In an agent loop it works like this: the model reasons, calls a tool, and
the application replays the whole conversation — including that reasoning — together with the
tool's result, so the model can pick up where it left off. Four providers delete the reasoning
from the replay before the model ever sees it. The model then re-derives or invents instead of
recalling — and no error is raised.

## The model's own template says it should be there

A model's **chat template** is the text program, shipped with the model, that turns a
conversation into the exact input the model reads. gemma-4's template renders reasoning
replayed on model turns after the last user message, and the gemma-4 model card mandates
preserving thinking across tool-call turns. Carrying replayed reasoning is the model's own
reference behavior — Google's native API, the reference provider, reproduces it live. A
provider that deletes it is deviating from the template of the model it serves.

## How it was measured: the 99 memory test

A canary — the number 99, wrapped in an unrelated aside — is planted *only* in the reasoning
field of a replayed model turn, across a fabricated tool call. Then the model is asked what the
number was. If the replayed reasoning reaches the model, it recalls 99; if the provider deleted
it, the model invents a number. Recall versus invention is the verdict — a probe on delivery,
not on status codes.

## Who drops it

- **Together AI** ([grade card](https://inferencecanary.com/togetherai/gemma-4-31b/index.md)) —
  replayed reasoning deleted before templating; the canary is never recalled.
- **AWS Bedrock (Chat Completions)** ([grade card](https://inferencecanary.com/aws/gemma-4-31b/index.md)) —
  renders *live* reasoning but deletes *replayed* reasoning; same verdict on every rep.
- **AWS Bedrock (Responses)** ([grade card](https://inferencecanary.com/aws-responses/gemma-4-31b/index.md)) —
  the sharpest case: the Responses dialect defines an explicit input item for replayed
  reasoning (`{"type": "reasoning", …}`), accepts it without error, and never renders it to the
  model. A documented field that does nothing.
- **Google (OpenAI-compat)** ([grade card](https://inferencecanary.com/google-openai/gemma-4-31b/index.md)) —
  drops it, while Google's own native protocol — same model, same company — carries it.

The other seven providers carry replayed reasoning. That includes every remaining OpenAI-compat
endpoint — so this is a provider serving-layer choice, not a dialect limitation and not a model
limit.

## Why it matters

Multi-step tool workflows silently lose the model's working state at every turn. The model
plans in its reasoning, acts, and then — on these four providers — returns to a conversation
where the plan no longer exists. It answers anyway, re-deriving or inventing, with no error for
the application to catch. Longer agent runs degrade in a way that looks like model weakness and
is actually the serving layer discarding the model's memory.

## Run it yourself

One request body, `temperature: 0`, fully self-contained: a replayed tool chain whose prior
turn's reasoning — planted under both common field spellings at once, so no provider is helped
by a spelling mismatch — says the number is 99. The completion resumes after the tool result.
**Answers 99** → the provider delivers replayed reasoning. **Invents a number** (it reliably
picks 42) → dropped. Set `model` to your provider's id for gemma-4-31b.

```json
{
  "model": "google/gemma-4-31B-it",
  "temperature": 0,
  "max_tokens": 400,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "send_message",
        "description": "Send a chat message to the user.",
        "parameters": {
          "type": "object",
          "properties": { "text": { "type": "string" } },
          "required": ["text"]
        }
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "Think of a number, then send \"okay\" to me with the send_message tool, then tell me the number you thought of."
    },
    {
      "role": "assistant",
      "content": null,
      "reasoning": "I thought of number 99. Now on to sending okay.",
      "reasoning_content": "I thought of number 99. Now on to sending okay.",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": { "name": "send_message", "arguments": "{\"text\":\"okay\"}" }
        }
      ]
    },
    { "role": "tool", "tool_call_id": "call_1", "content": "sent" }
  ]
}
```

Save it ([download](https://inferencecanary.com/assets/repro/reasoning-replay.json)) and POST it to the
provider's Chat Completions endpoint with your key:

```
curl "$BASE_URL/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @reasoning-replay.json
```

### The same test for AWS Bedrock's Responses API

The chat body above can't reach the Responses dialect, so the sharpest case needs its own:
here the replayed reasoning rides the input item the dialect itself defines for it
(`{"type": "reasoning", …}`). No `temperature` field — this endpoint rejects the parameter,
which is why the verdict takes an occasional extra rep. POST it to `$BASE_URL/responses`
([download](https://inferencecanary.com/assets/repro/reasoning-replay.responses.json)):

```json
{
  "model": "google.gemma-4-31b",
  "max_output_tokens": 400,
  "store": false,
  "reasoning": { "effort": "high" },
  "tools": [
    {
      "type": "function",
      "name": "send_message",
      "description": "Send a chat message to the user.",
      "parameters": {
        "type": "object",
        "properties": { "text": { "type": "string" } },
        "required": ["text"]
      }
    }
  ],
  "input": [
    {
      "role": "user",
      "content": "Think of a number, then send \"okay\" to me with the send_message tool, then tell me the number you thought of."
    },
    {
      "type": "reasoning",
      "id": "rs_replay",
      "summary": [{ "type": "summary_text", "text": "I thought of number 99. Now on to sending okay." }],
      "content": [{ "type": "reasoning_text", "text": "I thought of number 99. Now on to sending okay." }]
    },
    { "type": "function_call", "call_id": "call_1", "name": "send_message", "arguments": "{\"text\":\"okay\"}" },
    { "type": "function_call_output", "call_id": "call_1", "output": "sent" }
  ]
}
```

Probed live at temperature 0, full request/response evidence preserved. Template ground truth:
the executed `chat_template.jinja` from the upstream gemma-4-31b release.