Setting Temperature to Zero for Reproducible Tests

Temperature 0 is the first thing anyone does to a flaky model test, and it is the right first thing. It is also routinely oversold. It removes one source of variance out of several, and the useful version of this setup step is the one that tells you which ones are left.

What the parameter actually changes

The model returns a distribution over the next token. Temperature is applied to the logits before that distribution is sampled from: dividing by a number below 1 sharpens the distribution, dividing by a number above 1 flattens it. At 0 the operation degenerates — implementations special-case it to greedy decoding, which means “take the highest-scoring token” with no sampling at all.

So temperature 0 removes the random draw. That is the entire mechanism, and it is why the parameter is worth setting in a test: the random number generator is no longer in the loop, so one whole class of run-to-run difference is gone. It is also why the parameter cannot do more than that. Everything upstream of the draw — which logits the model produced — is untouched, and that is where the remaining variance lives.

One consequence people trip over: at temperature 0, top_p and top_k stop mattering, because truncating a distribution you are taking the argmax of changes nothing. Setting all three is not more deterministic than setting one. Anthropic’s API documentation goes further and advises against tuning temperature and top_p together at all.

Set it in one place, not on every call

The failure mode with per-call parameters is not that somebody sets them wrong. It is that somebody adds a new call site six months later and does not set them at all, and the suite becomes intermittently flaky in one file. Put the parameter in the factory that builds the request, and let the tests call the factory.

# tests/conftest.py
import os
import pytest
from openai import OpenAI

# One place. Every test that talks to a model goes through this.
TEST_MODEL = os.environ["TEST_MODEL"]          # a pinned, dated string
TEST_PARAMS = {"temperature": 0, "max_tokens": 512}

@pytest.fixture(scope="session")
def client():
    return OpenAI()

@pytest.fixture(scope="session")
def complete(client):
    def _complete(messages, **overrides):
        return client.chat.completions.create(
            model=TEST_MODEL,
            messages=messages,
            **{**TEST_PARAMS, **overrides},
        )
    return _complete

The overrides escape hatch matters. Some tests want temperature 1 on purpose — see which tests need determinism and which do not — and they should have to say so explicitly rather than by omission. A grep for temperature= in the test tree should return the fixture and the deliberate exceptions, nothing else.

Set max_tokens in the same place, for a reason that is about stability rather than cost. An unbounded ceiling means one case can return an answer twenty times longer than the others, which changes the runtime of the suite unpredictably and gives a truncation-handling bug nowhere to show itself. A bound that is generous but finite makes the suite’s worst case knowable.

The layers that default it elsewhere

Your request passes through more code than you wrote, and several layers have opinions. The provider is the least of them: OpenAI’s Chat Completions reference and Anthropic’s Messages reference both document a default temperature of 1 at the time of writing, so not setting it means sampling, not greedy decoding.

• Framework wrappers. Orchestration libraries construct a client object with its own default temperature in the constructor and forward it on every call. If you build the chain object once in a fixture and set temperature on the individual invocation, whichever one the library gives precedence to wins, and that is a library detail, not a rule.
• Structured-output and JSON modes. These constrain which tokens are legal at each step, which interacts with sampling but does not replace it. A schema-constrained call at temperature 1 still varies within the schema.
• Reasoning models. Several models in this class either ignore the temperature parameter or reject it outright. A test that sets temperature 0 and believes it got greedy decoding is wrong if the endpoint silently dropped the field. Assert on the echoed request or read the error, do not assume.
• Gateways and proxies. Anything that rewrites the request body can normalise or drop parameters. The cheap check is to send the same prompt twice with the same seed and diff the two responses.

The four variance sources that remain

This is the part the setup step usually omits, and it is the reason a suite pinned to temperature 0 still goes red occasionally.

• Batching. A served model does not process your request alone. vLLM’s reproducibility documentation is explicit that the same request can be batched differently depending on what else is in flight, and that floating-point reduction order changes with batch composition, which can change which token has the highest logit. vLLM ships a separate batch-invariance feature precisely because ordinary serving does not have that property.
• Hardware and version. The same vLLM documentation limits its reproducibility guarantee to the same hardware and the same version. A hosted provider changes both without telling you.
• Backend configuration. OpenAI surfaces this as system_fingerprint, an identifier for the current combination of weights, infrastructure and configuration, which it says may change a few times a year. See testing against a fixed system_fingerprint.
• The model itself. An alias like a bare family name resolves to whatever snapshot is current. This is the largest of the four and the easiest to fix — pin a dated model string.

Wiring it into a suite

1. Put temperature, max_tokens and the model string in one session-scoped fixture, as above. Read the model from an environment variable so a live tier can override it.
2. Add a smoke test that calls the fixture twice with the same input and asserts the two responses are byte-identical. Mark it as informational rather than blocking: it tells you whether your stack is currently reproducible, and it is allowed to be false.
3. Assert the echoed response.model equals the pinned string. This catches a gateway or a provider quietly resolving your request somewhere else.
4. For every assertion that compares output text, ask whether it would survive a one-token difference. If not, replace it with a property — low temperature plus property assertions is the combination that holds.
5. Record the temperature you used alongside any stored expectation. A golden file recorded at temperature 1 and replayed at 0 is a mismatch nobody will diagnose quickly.