SambaNova

OpenAI chat/completions · serves gemma-4-31B-it as gemma-4-31b · probed live 2026-07-27 · back to the matrix

Harness

C−

A − 2 red − 1 yellow, per the ladder; flags below.

Billing

RED

Identical requests billed inflated token counts — 2.18× average overcharge across three 40-call batteries.

Caching

0/8 cold→warm trials hit — every call re-reads (and re-bills) the full context. Speed numbers →

Faithfulness

29/29

Agreement with the reference provider over the 29-cell deterministic battery (temperature 0, thinking disabled). 0–10 score arrives with the locked rubric.

Flags

red billing SambaNova bills identical requests inflated token counts — 2.18× average overcharge
The same 35-token request body, sent 40 times in sequence, was billed ten distinct values between 27 and 2478 tokens. Across three independent batteries, 9,176 tokens were charged for 4,200 actually sent — 2.18× on...
red harness Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model...
red harness SambaNova enforces tool_choice by post-hoc 400, not constrained decoding
The knob is accepted, generation runs unconstrained, and when the output violates the constraint the request fails with a provider-level 400. Whether a request succeeds depends on what the model happened to emit:...
yellow harness Tool results replayed out of order are mis-paired on most providers
The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the...

Every flag links to its finding — evidence, repro, and disclosure records live there; the deepest findings have full writeup pages.