AWS Bedrock (Responses)
OpenAI Responses on Bedrock Mantle · serves google.gemma-4-31b as gemma-4-31b ·
probed live 2026-07-27 · back to the matrix
Billing
✓
Reported usage consistent with bytes on the wire. Footnote: Reports reasoning_tokens: 0 beside delivered reasoning — under-counts in the customer's favor.
Caching
No
0/8 cold→warm trials hit — every call re-reads (and re-bills) the full context. Speed numbers →
Faithfulness
25/29
Agreement with the reference provider over the 29-cell deterministic battery
(temperature 0, thinking disabled). 0–10 score arrives with the locked rubric.
Flags
-
red harness
Four providers drop replayed reasoning: the model never sees its own prior thinking
The gemma-4 chat template renders reasoning replayed on model turns, and the model card mandates preserving thinking across tool-call turns — the reference provider reproduces this live. These four providers delete it... -
red harness
Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model... -
yellow harness
AWS Bedrock never emits parallel tool calls
parallel_tool_calls: true is accepted on both Bedrock APIs, but only a single call ever comes back; every other provider emits two for the same two-lookup prompt. A model capability suppressed at the serving layer; the... -
yellow harness
AWS Bedrock un-escapes tool-call arguments in transit
At temperature 0, nine providers agree the model returns a tricky string argument with literal escape sequences. Both Bedrock APIs instead return real control characters — one extra unescape pass applied to the model's... -
yellow harness
AWS Bedrock (Responses) rejects temperature
temperature returns 400 unsupported_parameter, so deterministic sampling is unreachable on this provider: every run is stochastic, degrading reproducibility for applications and evals alike.
Every flag links to its finding — evidence, repro, and disclosure records live there; the deepest findings have full writeup pages.