Deciding Which Tests Need Determinism and Which Don't

Temperature 0 gets applied to an entire suite because it fixed one flaky test. That is a reasonable reflex and a bad default: two of the five things a model test can be asserting become weaker under determinism, and one becomes meaningless.

The question that decides it

Not “is this test flaky?” but “what is this test asserting about?” Almost every test that touches a model is really asserting about one of three subjects: your code, the interface between your code and the model, or the model’s behaviour. The amount of determinism you want is a direct function of which.

If the subject is your code, the model should not be in the loop at all — determinism is total because there is nothing random left. If the subject is the interface, you want reproducibility so that a contract violation can be re-triggered. If the subject is the model’s behaviour, you want variance, because behaviour is a distribution and looking at one point of it tells you almost nothing about the distribution.

Stated that way the answer usually falls out immediately, and the uncomfortable cases are the tests that turn out to be asserting about two subjects at once. Those are not tests with a determinism problem; they are two tests that need splitting.

Five classes of model test

• Plumbing. Authentication, retry on 429, timeout handling, streaming chunk assembly, usage accounting, error translation. The model is irrelevant; these want a recorded or synthetic response and no network at all. Determinism is total and free. Every one of these tests that calls a live model is money spent on nothing — see testing without the model.
• Contract. The response validates against the schema, the declared tool fires, the enum value is in the enum, the JSON parses. The assertion is an invariant, so exact reproducibility adds nothing to whether it passes — but it adds a lot to whether a failure can be investigated. Temperature 0 here, with property assertions.
• Regression. A specific past failure, pinned so it cannot come back. This class wants everything: pinned model, temperature 0, a seed if the provider honours one, and the original input verbatim. It is the only class where maximum determinism is unambiguously right, because the whole point is to re-run one historical event.
• Quality. Accuracy on a labelled set, rubric scores, win rates. This is an eval, not a test, and it wants production settings — including production temperature — several samples per case, and a reported number rather than a pass or fail. A golden dataset is the input to it.
• Robustness. Does the pipeline survive whatever the model does? Malformed JSON, an unexpected tool, an over-long answer, a refusal, a truncation. This class needs variance by construction: a deterministic run produces one output shape and therefore tests one branch.

Where determinism hides the bug

The quality and robustness classes are the two where temperature 0 is not merely unnecessary but actively misleading, and the mechanism is the same in both: greedy decoding samples the mode of the distribution, and a defect that lives in the tail is invisible at the mode.

In the quality class this shows up as a score that is both higher and more stable than production. Greedy output is the model’s most confident continuation, so accuracy measured at temperature 0 is an upper bound on accuracy at the temperature you actually ship. Track that number over months and you have a well-maintained chart of a quantity nobody experiences. Worse, it will not move when the thing your users complain about gets worse, because what got worse was the spread.

In the robustness class the effect is starker. A JSON parser that happens to cope with the greedy formatting and breaks on the fenced variant the model emits one time in forty will pass a temperature-0 suite forever. So will a downstream regex tuned to the greedy phrasing, a truncation handler that greedy output never reaches because greedy answers run short, and a tool-dispatch branch for a tool the model only selects when it is less certain. Every one of these is a real production incident that a deterministic suite is structurally incapable of finding.

There is a third, quieter case: a prompt that only works at temperature 0. Teams tune a prompt against a deterministic suite until it passes, ship it at temperature 0.7, and discover the instruction was being followed by one token’s margin. The suite never had a way to see the margin.

A decision you can write down

For each test, answer three questions in order and stop at the first that applies.

1. Could this test pass with the model replaced by a recorded response? If yes, it is plumbing. Replace the model, disable the network for that tier, and stop thinking about determinism — you have removed the source.
2. Is the assertion an invariant that any acceptable answer satisfies? If yes, it is contract or regression. Temperature 0, pinned model, seed where available, property assertions, blocking on every commit.
3. Is the assertion about how often something is true, or about what happens when the model behaves unusually? If yes, it is quality or robustness. Production temperature, multiple samples per case, non-blocking, on a schedule, reported as a number or a rate.

The three answers map cleanly onto the two-tier split described in splitting tests into a mocked tier and a live tier, which is not a coincidence: the determinism decision and the spending decision are the same decision seen from two sides.

Making the choice visible in the suite

Whatever you decide, encode it as a marker rather than as a per-test parameter, so that the class of a test is visible at the top of the file and selectable from the command line. In pytest that is a marker registered in configuration and selected with -m; the equivalent in a JavaScript runner is a tag or a separate project entry. The important property is that somebody can run exactly the contract tests, or exactly the robustness tests, without reading them.

The second thing worth encoding is the temperature itself, in the marker rather than in the body: a fixture that reads the marker and sets temperature accordingly means no test can silently disagree with its own classification. A robustness test that quietly runs at temperature 0 because somebody copied a fixture is exactly the failure this page is about, and it is invisible in review unless the setting and the label are the same thing.