Why a Local Model’s Output Isn’t Identical Run to Run at Temperature Zero

You set temperature to 0, pinned the seed, sent the same prompt twice, and got two answers that agree for a hundred tokens and then diverge. Nothing is misconfigured. The cause is below the sampler, in how the kernels add numbers up.

First, rule out the sampler

Four settings can produce this symptom for ordinary reasons, and they are worth eliminating before accepting the harder explanation.

• Temperature is not actually zero. Some stacks clamp a requested 0 to a small positive number rather than switching to argmax, which leaves you sampling from an extremely peaked distribution — almost always the same token, occasionally not. Setting top_k to 1 forces argmax regardless of how temperature is interpreted, and is the cleanest way to be sure.
• A penalty is active. llama.cpp’s documented sampler chain is penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature. Penalties run before temperature, so a repetition penalty modifies the logits themselves and can change which token is the maximum even under greedy decoding. The documented default for --repeat-penalty is 1.00, meaning disabled, but front-ends frequently set 1.1 without saying so.
• The seed is not pinned. Documented default is -1, meaning a random seed per run. Under true greedy decoding the seed is irrelevant, which makes it a useful test: if pinning the seed changes anything at all, you were sampling.
• The prompt is not identical. A system prompt containing a timestamp, a conversation id, or trailing whitespace that a client strips inconsistently is a different prompt. Compare the token counts the server reports, not the strings you typed.

If all four are clean and the output still varies, the sampler is not your problem.

Greedy is deterministic; the logits are not

Greedy decoding takes the argmax of the logit vector. Given the same logits it returns the same token every time; there is no randomness in it. So the variation has to be upstream, in the numbers themselves.

Floating-point addition is not associative. In IEEE 754, every addition rounds its result to the nearest representable value, so (a + b) + c and a + (b + c) can differ in the last bits. Every layer of a transformer performs reductions — sums over thousands of terms in each dot product, in each normalisation, in each attention weighting — and the value each reduction produces depends on the order the partial sums were combined.

The differences are minute: a handful of units in the last place. They matter because argmax is a comparison. For most tokens the top two logits are far apart and no rounding difference could reorder them. Occasionally they are nearly tied, the tiny difference flips the comparison, a different token is chosen, and from that point the two sequences are conditioned on different text and diverge completely. That is why the outputs agree for a long prefix and then separate entirely rather than differing slightly throughout.

The real cause: kernels are not batch invariant

The usual explanation stops at “floating point plus GPU concurrency”, and that explanation is incomplete in a way that matters for fixing it. Thinking Machines Lab’s September 2025 analysis, Defeating Nondeterminism in LLM Inference, makes the point directly: running the same matrix multiplication on the same data repeatedly gives bitwise identical results. The forward pass of an LLM contains essentially no atomic adds, so there is no race between threads accumulating into one location, and each kernel is run-to-run deterministic.

What is not fixed is the reduction strategy. High-performance kernels split a reduction across cores to keep the machine busy, and how many pieces they split it into is chosen from the shape of the work — the batch size, the sequence length, the number of tokens in flight. Split-K matrix multiplication, split-reduction normalisation and FlashDecoding-style split-KV attention all do this. Change the shape and the kernel picks a different split; a different split is a different summation order; a different summation order is a different last bit. The property the kernels lack has a name: batch invariance, the guarantee that one request’s result does not depend on what else was in the batch with it.

The scale of the effect is not theoretical. That analysis reports that sampling 1,000 completions from Qwen3-8B at temperature 0 produced 80 unique completions, with the most common appearing 78 times, and the first divergence occurring at token 103. With batch-invariant kernels substituted, all 1,000 completions were identical. The same team notes that the SGLang project subsequently integrated those kernels, with the reported overhead of the batch-invariant path around 34%.

What changes the batch shape on your own machine

On a hosted endpoint the batch shape varies because other people are using it. On your own machine you are the only user, which is why this surprises people locally — but the shape still moves, for reasons you control:

• Micro-batch size. llama.cpp documents -b, --batch-size with a default of 2048 and -ub, --ubatch-size with a default of 512. Prompt processing is chunked at those sizes, so a 500-token prompt and a 600-token prompt are not merely longer and shorter — they are processed in a different number of chunks, with different reduction shapes.
• Concurrent requests. With -np greater than 1, a second request arriving mid-generation joins the same forward pass. Your request’s logits now depend on somebody else’s timing. See how server slots work.
• Prefix caching. The second identical request may reuse a cached prefix and prefill only the new tokens. Fewer tokens through prefill is a different shape, and therefore potentially a different result from the run that computed the whole prompt.
• Layer placement. Changing -ngl moves layers between the CPU and GPU backends. Those are entirely different kernels with different accumulation precision, so a partially offloaded model is not numerically the same model as a fully offloaded one.
• Attention implementation. Turning flash attention on or off replaces the attention reduction with a different algorithm. So does a driver update that changes which kernel a library’s heuristics select.
• Different hardware. A different GPU has a different core count and different kernel choices. Determinism is never portable; the most you can have is reproducibility on one machine with one build.

What you can actually do

1. Pin everything that selects a kernel. Same binary, same build flags, same GPU, same -ngl, -b, -ub, same flash-attention setting, same context size. Run with -np 1 and send one request at a time so nothing shares a batch with you. Disable prefix-cache reuse for the comparison. This gives reproducibility on that machine, which is usually what was actually wanted.

2. Force greedy explicitly. "temperature": 0, "top_k": 1 and every penalty at its disabled value, so that no sampler stage can contribute.

3. Use batch-invariant kernels if bit-identical output is a requirement. This is a real option in serving stacks that have integrated them, and it costs throughput. It is worth it when the output is an audit artefact and not otherwise.

4. Otherwise, design around it. Do not use a model response as a cache key or a hash input. Do not assert exact strings in tests — assert on parsed structure, on a schema, or on a property the answer must have. Where an exact output must be reproducible later, store it rather than plan to regenerate it.

The general phenomenon is not local-specific, and the hosted-side treatment lives at temperature-zero nondeterminism. What is specific to running locally is the last section: the shape changes are yours to control, which means reproducibility on one box is genuinely achievable here in a way it is not through somebody else’s load balancer.