Non-Determinism: Why Temperature 0 Isn't Deterministic

You set temperature to 0, sent the same prompt twice, and got two different answers. Nothing is broken and the sampler is not lying to you. The determinism you were promised is conditional on something that a production inference server does not provide.

What temperature 0 actually does

Temperature divides the logits before the softmax. As it approaches zero the distribution approaches a point mass on the highest-scoring token, and implementations short-circuit to argmax. So greedy decoding is a deterministic function of the logits: same logits, same token, every time.

The guarantee therefore rests entirely on the logits being bit-for-bit identical between two runs. They are not. And because decoding is autoregressive, one differing token at position 30 conditions everything after it — a single flipped argmax where the top two tokens were nearly tied produces a completely different second half. The divergence is not proportional to the numerical error; it is a threshold effect, amplified by the loop.

Floating point is not associative

The first-order explanation, and the one everybody stops at. In floating-point arithmetic (a + b) + c is not necessarily a + (b + c), because each addition rounds. A matrix multiplication is a very large number of additions, and its result depends on the order they were performed in.

On a GPU the order depends on how the work was split across thread blocks, which reduction algorithm the kernel chose, and whether any partial sums were accumulated with atomics — atomic accumulation completes in whatever order the blocks finish, which varies run to run. At low precision the rounding is coarser and the effect is larger, which is why quantised serving diverges more readily.

This is all true. It is also, on its own, an incomplete explanation, and the incompleteness matters because it points at the wrong fix.

The real culprit: batch invariance

The clearest published account of this is Thinking Machines Lab’s Defeating Nondeterminism in LLM Inference (2025). Its central observation: on a single GPU, run the same forward pass repeatedly with the same inputs and you generally get the same bits — the kernels are run-to-run deterministic in practice. The nondeterminism you observe in a served model comes from somewhere else.

It comes from batching. An inference server batches concurrent requests together, and the batch composition changes from moment to moment with load. Standard attention and matrix-multiply kernels are not batch-invariant: they select tiling and reduction strategies based on the shape of the batch, so the same request’s arithmetic is performed in a different order depending on how many other requests happened to be in flight and how long their prompts were.

The consequence is worth stating plainly, because it is unintuitive and it is the actual mechanism: your output depends on other people’s traffic. Not on their content — there is no leakage — but on the numerics of the reduction that your tokens went through alongside theirs. Server load, at the millisecond your request arrived, is an input to your result. The paper’s fix is to write batch-invariant kernels, which makes results reproducible at some throughput cost; that is a serving-side choice, not one you can make from the client.

Related, and in the same family: mixture-of-experts routing is computed per batch, and with capacity limits a token’s expert assignment can depend on which other tokens are competing for the same expert. Speculative decoding adds another path — the accepted-token boundary depends on draft-model timing, and the verification arithmetic differs slightly from a plain forward pass.

Everything else that differs between two calls

• Different hardware. A provider’s fleet is not homogeneous. Two GPU generations produce different results for the same kernel, and you do not choose which one you land on.
• Different serving builds. A rolling deploy means some replicas run last week’s kernels. Nothing in the API tells you which.
• Different quantisation. The same open-weights model served at different precisions is, numerically, a different model. Where several providers serve one model, this is the largest source of cross-provider disagreement.
• Prompt caching. A cached prefix may have had its keys and values computed under a different batch than a fresh prefill would use, so a cache hit and a cache miss can diverge.
• Silent model updates. An unpinned alias can point somewhere new — a different question, covered on model degradation over time.

Seed parameters help only within what a provider can control. OpenAI’s seed is documented as best-effort and is paired with a system_fingerprint that changes when the backend configuration changes — which is an admission, in the API surface, that determinism is not being promised.

Measuring your own divergence

Before designing around it, find out how much of it you actually have. The useful statistic is not “are they identical” but where they first differ and whether the difference matters.

import hashlib
from collections import Counter

def divergence(prompt, n=20):
    outs = [call_model(prompt, temperature=0, max_tokens=400) for _ in range(n)]
    hashes = Counter(hashlib.sha256(o.encode()).hexdigest()[:12] for o in outs)

    # First token index at which any two runs disagree.
    first_div = None
    toks = [o.split() for o in outs]
    for i in range(min(len(t) for t in toks)):
        if len({t[i] for t in toks}) > 1:
            first_div = i
            break

    return {"distinct_outputs": len(hashes),
            "modal_share": hashes.most_common(1)[0][1] / n,
            "first_divergent_token": first_div,
            "semantically_equal": all_same_after_parse(outs)}

Run it at different times of day. A divergence rate that climbs during your provider’s busy hours is the batching effect showing itself, and it is the observation that usually convinces a sceptical colleague. Run the same test against a second provider serving the same open-weights model and the outputs will typically differ far more than two calls to one provider do, because you are now comparing quantisations rather than reduction orders.

Designing so it does not matter

• Never assert on exact strings. Parse, then assert on the parsed value. A test that compares a model’s prose to a golden file is a test that will fail on a Tuesday for no reason.
• Cache on the input, not the output. If two identical requests must return the same thing — and for user-facing consistency they often must — put a cache in front. This is the only actual determinism available to you.
• Pin the model version and the provider. It removes the largest sources without removing the batching effect.
• Use structured output. Constrained decoding narrows the space in which a flipped argmax can send the generation somewhere else, and makes the remaining variation parseable.
• Treat instability as information. A prompt whose output varies at temperature 0 is a prompt sitting near a decision boundary — the top two tokens are nearly tied. That is a genuine uncertainty signal about that input, and it is worth logging rather than suppressing.