Lightning

OpenAI chat/completions · serves lightning-ai/gemma-4-31B-it as gemma-4-31b · probed live 2026-07-27 · back to the matrix

Harness

A − 0 red − 2 yellow, per the ladder; flags below.

Billing

✓

Reported usage consistent with bytes on the wire.

Caching

Yes

Reused already-seen context on 3/3 cold→warm trials. Speed numbers →

Faithfulness

20/29

Agreement with the reference provider over the 29-cell deterministic battery (temperature 0, thinking disabled). 0–10 score arrives with the locked rubric.

Flags

yellow harness Tool results replayed out of order are mis-paired on most providers
The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the...
yellow harness Lightning leaks template reasoning markers into content on truncation
At tight max_tokens, literal template markers appear in content; unconstrained budgets come back clean. Output is polluted exactly when the response is cut short.

Every flag links to its finding — evidence, repro, and disclosure records live there; the deepest findings have full writeup pages.