SambaNova

1 graded endpoint serving gemma-4-31b · back to the matrix

Billing

red Identical requests billed inflated token counts — 2.18× average overcharge across three 40-call batteries.

Findings affecting SambaNova

• red billing SambaNova bills identical requests inflated token counts — 2.18× average overcharge
  The same 35-token request body, sent 40 times in sequence, was billed ten distinct values between 27 and 2478 tokens. Across three independent batteries, 9,176 tokens were charged for 4,200 actually sent — 2.18× on...
• red harness Images in tool results: rejected, 500'd, or silently blinded on 7 of 11 providers
  The model reads images returned by tools — the reference provider delivers them, perception-judged, and three OpenAI-compat providers prove the standard shape works. Seven providers fail on schema choice, not model...
• red harness SambaNova enforces tool_choice by post-hoc 400, not constrained decoding
  The knob is accepted, generation runs unconstrained, and when the output violates the constraint the request fails with a provider-level 400. Whether a request succeeds depends on what the model happened to emit:...
• yellow harness Tool results replayed out of order are mis-paired on most providers
  The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the...