Determinism in Self-Hosted Llama: Why the Same Weights Can Still Drift

You are running the weights yourself. Nothing is being A/B tested behind your back, no silent model update happened, temperature is 0 and the seed is fixed — and the output still changed. The cause is underneath the model, in how the arithmetic was scheduled, and it is not a bug in anything.

What temperature 0 actually buys you

Setting temperature to 0 makes the sampler deterministic: instead of drawing from the distribution, take the highest-scoring token. Given identical logits, that is a pure function and it will return the same token every time.

“Given identical logits” is the entire problem. Greedy decoding is deterministic conditional on the forward pass producing bit-identical numbers, and the forward pass does not guarantee that. When two candidate tokens have logits that differ in the seventh decimal place, a difference of one unit in the last place flips the argmax — and then that token is appended, and every subsequent step conditions on a different sequence. One flipped tie near the start produces a completely different paragraph. The divergence is not gradual.

So temperature 0 removes one source of variation and leaves the rest fully exposed. It is closer to a magnifying glass than to a fix.

Floating point is not associative

The root cause is arithmetic. In real numbers, (a + b) + c = a + (b + c). In floating point it is not, because each addition rounds. Sum a thousand values in a different order and you get a slightly different result — not wrong, just different in the last bits.

A transformer forward pass is enormous numbers of summations: matrix multiplications, attention score reductions, layer-norm means. On a GPU these are computed in parallel and combined in a reduction tree, and the shape of that tree depends on how the work was split across threads and blocks. Change the split, change the order, change the last bits of the result.

None of this is exotic and none of it is specific to language models. PyTorch documents the general situation and what limited controls exist in its reproducibility notes, which state plainly that bitwise reproducibility is not guaranteed across releases, platforms or even different invocations on the same hardware for some operations. The reason it is more visible in LLM inference than elsewhere is the argmax: most numerical code tolerates a difference in the seventh decimal, and a token choice does not.

Your batch contains other people’s requests

This is the mechanism most people have not considered, and on a shared server it is usually the dominant one.

A serving stack like vLLM or TGI does continuous batching: it runs many requests through the model together, adding and removing them as they arrive and finish. Your request’s tokens are computed inside a matrix multiplication whose dimensions depend on how many other requests happened to be in flight at that moment.

Kernels are not batch-invariant. A GEMM with a batch dimension of 4 takes a different code path, tiling and reduction order than the same GEMM with a batch dimension of 37 — and produces results that differ in the last bits. Your logits therefore depend on your neighbours’ traffic. That is why the same prompt at 3am and at peak can return different answers on the same server, with no configuration change anywhere and nothing you can point at in your own request.

This diagnosis was set out in detail in Thinking Machines’ September 2025 article “Defeating Nondeterminism in LLM Inference”, which argues that batch invariance in the kernels, rather than seeding, is what determinism in a serving stack actually requires. The practical reading for now is that unless your stack advertises batch-invariant kernels, batch composition is a live input to your output.

Kernels, parallelism and caches

• Kernel autotuning. Libraries select an implementation based on shapes and on measured performance at runtime. A different selection is a different reduction order. Setting torch.backends.cudnn.benchmark = False removes one instance of this.
• Tensor parallelism. Splitting a layer across GPUs adds an all-reduce whose combination order can vary, and changing the degree of parallelism changes results even on identical hardware. A model served on 2 GPUs and on 4 will not agree bit for bit.
• Atomics. Some kernels accumulate with atomicAdd, which combines in whatever order threads arrive — genuinely nondeterministic on the same hardware with the same input.
• Prefix caching. With a cache hit, part of the KV state is reused from a previous computation performed under different batch conditions; without one it is recomputed now. The two paths do not produce identical numbers, so cache state becomes an input.
• Quantisation and dtype. A 4-bit or 8-bit checkpoint has far less numerical headroom, so ties are closer together and flip more readily. Lower precision makes every effect above worse.
• Hardware and driver. Different GPU generations, CUDA versions and kernel libraries produce different results for the same operation. Reproducibility does not survive a hardware migration.

Separating drift from an actual change

Everything above is numerical noise, and it looks identical from the outside to something having genuinely changed. The distinction matters, because one of those is a fact of life and the other is a regression somebody introduced. The two behave differently and you can tell them apart without instrumentation.

• Noise is intermittent and unbiased. Send the same prompt twenty times and you get a scatter of outputs, most of them equally good, with no direction to the variation. It appears and disappears with load.
• A real change is consistent and directional. Every request after some point in time behaves the new way, and usually the new way is worse in a describable respect — shorter, differently formatted, refusing things it did not refuse. A step function in a metric, not a widening band.

If it looks like a step function, the candidates are, in the order they are worth checking: the template rendering changed (a framework upgrade, a new date in an injected line, a modified chat_template in a re-pulled checkpoint); the sampling parameters changed, often because a default moved rather than because anyone edited a config; the checkpoint itself moved, if you are resolving it by tag rather than by hash; or the quantisation changed, which is the one people most often forget is part of the model.

The cheapest defence against all four is to log a small fingerprint alongside each response — the resolved checkpoint revision, the framework version, the quantisation, and a hash of the fully rendered prompt. Three of those four are constants you already have, and the fourth catches the template class of problem the moment it happens rather than a week later.

What you can actually pin down

Full bitwise determinism is achievable only by controlling all of the above, which in practice means batch size 1 on fixed hardware with a fixed software stack and caching disabled — a configuration that costs most of your throughput. That is a real option for a compliance workload and a bad one for a product. The realistic position is to reduce variance and stop depending on the absence of it.

1. Pin what is cheap to pin: the checkpoint by content hash, the framework version, the quantisation, the dtype, the tensor-parallel degree, the GPU model. Record all of it with every output you intend to reason about later. Weights alone do not identify a system — see pinning a Llama checkpoint by hash.
2. Set temperature to 0 and fix the seed anyway. It eliminates the largest single source and costs nothing.
3. For a workload that must be reproducible, serve it with batch size 1 on a dedicated instance with prefix caching off, and accept the throughput cost as the price of the property.
4. Test on distributions rather than on strings. An assertion that an output equals an exact string is a test of your scheduler. Assert on what the output must contain, on a parsed structure, or on a threshold over many samples.