Combining Low Temperature With Property Assertions for Stable Tests

These are usually presented as two answers to one question — lower the temperature, or stop asserting on exact output. They are answers to two different questions, and a suite that picks one is leaving a specific, nameable gap open.

Two techniques, two different jobs

Lowering temperature narrows the distribution of outputs the model is allowed to produce. Its job is reproducibility: making the same input tend to the same output so that a failure can be re-triggered and inspected. It says nothing about whether the output is correct.

A property assertion narrows what you demand of an output. Its job is tolerance: making the test pass for any of the many outputs that are genuinely acceptable, so that acceptable variation is not reported as a defect. It says nothing about whether the failure you just saw can be reproduced.

Reproducibility and tolerance are orthogonal. You can have a reproducible test with an intolerant assertion (an exact-string match at temperature 0), a tolerant test with no reproducibility (a schema check at temperature 1), both, or neither. Only one of those four is a suite you want.

What low temperature alone leaves open

Three gaps, and the third is the one that does real damage.

• It is not actually reproducible. Greedy decoding removes the sampler, not the batching, the hardware, the serving version or the backend configuration. A suite whose assertions depend on exact output at temperature 0 will break on a change you did not make, at a time you did not choose.
• It over-specifies the contract. An exact-match assertion says the answer must be this sentence. Almost never is that the real requirement; the real requirement is that the answer classifies correctly, or cites a real source, or fits the schema. A test that fails because a synonym appeared is a false alarm, and a suite that produces false alarms gets muted.
• It tests one path and hides the others. This is the expensive one. Temperature 0 exercises exactly one trajectory through the output space. Your production traffic, at whatever temperature you actually ship, explores many. A parser that only handles the greedy formatting, a downstream regex that only matches the greedy phrasing, a retry branch that greedy output never triggers — all of these pass a temperature-0 suite and fail in production. The suite is green because it is only ever looking at one sample.

What properties alone leave open

Assert only invariants and run at production temperature, and you get a suite with the opposite pathology.

• Failures do not reproduce. A property that holds 29 times in 30 fails as an unexplained red build. The developer re-runs it, it passes, and the finding is lost. This is the mechanism by which real intermittent bugs become permanently invisible: the evidence is destroyed by the retry.
• The failure rate is unmeasurable in CI. One run tells you the property held or did not. Estimating how often it holds needs many runs, which is an eval, not a test — a different tier with a different budget. See splitting tests into a mocked tier and a live tier.
• Properties can be vacuously satisfied. “The response is valid JSON” is satisfied by {}. A property test needs its own negative case — feed it an input you know should fail and confirm it does — or you have written an assertion that cannot fail.

What the combination gives you

Run the blocking tier at temperature 0 with property assertions, and each technique covers the other’s gap. Low temperature makes a failure likely to recur when you re-run it, which is what makes debugging possible. Properties make the assertion survive the variation that temperature 0 does not remove, which is what stops the vendor’s next infrastructure change from failing your build.

The third gap — that temperature 0 only ever samples one path — is not closed by properties, and it is not closed by anything inside the blocking tier. It needs a separate, non-blocking tier that runs the same property assertions at production temperature, several times per case, and reports rather than gates. That tier is where you find the parser that only handles greedy formatting. Its results belong on a dashboard, not on a pull request.

Written out, the arrangement is three lines: temperature 0 plus properties, blocking, on every commit. Production temperature plus the same properties, non-blocking, on a schedule. Exact-output assertions nowhere at all, except against a recorded fixture where the model is not in the loop.

One practical note on getting there from an existing suite. The conversion is rarely a rewrite; it is usually asking, of each exact-match assertion, what would have gone wrong in production if that string had been different. The answer names the property, and often it turns out there is no answer at all, in which case the assertion was asserting nothing and can go.

Properties that are worth asserting

The useful ones share a shape: they are computable from the output and the input together, without a second model and without a human.

• Schema conformance. Validate against the actual schema object, not a hand-written field check. If the schema is the contract, the validator is the assertion.
• Closed-set membership. A label is one of the labels you defined. A cited document id is one of the ids you supplied. A currency code is in the list.
• Containment. Every number, name or quotation in the output appears in the input. This is a substring or set check and it is the strongest cheap defence against fabrication.
• Absence. No secret, no system-prompt fragment, no redacted identifier, no forbidden term. Absence assertions are worth writing even when they seem paranoid, because their failure mode is a disclosure rather than a wrong answer.
• Shape and bounds. Exactly three items, at most 200 words, a score between 1 and 5, a date that parses and falls in a plausible range.
• Tool invariants. The expected tool fired, once, with arguments that validate. See testing structured output for the schema side of this.
• Metamorphic relations. Two outputs relate as they should, without either being specified — the technique that makes non-deterministic behaviour testable at all.