Phi-4's Output Determinism at Temperature 0

Setting temperature to 0 on Phi-4 does not make it deterministic. It makes it deterministic given identical arithmetic, and identical arithmetic is a much stronger condition than running the same model twice.

What temperature 0 actually does

The model produces a logit for every entry in its roughly 100K-token vocabulary. Temperature divides those logits before the softmax: higher temperature flattens the distribution, lower temperature sharpens it. Zero is a division by zero, so no runtime implements it literally. Every one of them special-cases it to greedy decoding — take the argmax, ignore top-p and top-k entirely.

The implementations differ in how they get there, which matters when you are comparing two stacks:

transformers   do_sample=False               -> greedy; temperature is ignored
               do_sample=True, temperature=0 -> warns, effectively greedy
vLLM           SamplingParams(temperature=0) -> greedy path, top_p/top_k unused
llama.cpp      --temp 0                      -> greedy
Ollama         "options": {"temperature": 0} -> greedy

A common source of confusion: in transformers, passing temperature=0 while leaving do_sample at its default has historically produced a warning and behaviour that depends on version. Set do_sample=False explicitly and the intent is unambiguous.

After that, there is no random number generator in the loop. A seed changes nothing, because nothing is being sampled. If your output still varies, the variation is upstream of the sampler.

What it does not remove

Argmax over the logits is exactly reproducible only if the logits are bit-identical. Producing bit-identical logits from a large floating-point computation is genuinely hard, for reasons that have nothing to do with the model:

• Floating-point addition is not associative. A matrix multiplication sums thousands of products, and the order of that summation is chosen by the kernel. Change the order and you change the last bits of the result. This is not a bug and cannot be fixed by the model.
• Batching changes the reduction order. Under continuous batching in vLLM or TGI, your request is processed alongside whoever else arrived. Different batch composition selects different kernel tilings and different split-K strategies, so the same prompt at 3am and at peak can produce different logits.
• Attention backends differ. FlashAttention, xformers and the plain SDPA math path compute the same quantity by different routes with different rounding. The backend is often selected automatically from the hardware and the sequence length.
• Hardware and dtype differ. bf16 has fewer mantissa bits than fp16 across a different exponent range; TF32 accumulation on Ampere and later is not the same as fp32. Two GPU generations running the same code produce different numbers.
• Quantisation is a different model. A Q4_K_M GGUF is not Phi-4 at lower precision in a way that preserves argmax. It is a different set of weights that mostly agrees.

One thing Phi-4 does not suffer from is worth naming: it is a dense 14B model, so there is no expert router whose choices depend on what else is in the batch. Mixture-of-experts models add exactly that source of batch-dependent variation on top of everything above, which is why Phi-3.5-MoE is harder to pin down than Phi-4 is.

Why one flipped token changes everything

The numerical differences above are tiny — a change in the last bits of a logit. They would be irrelevant if the model made one decision. It makes one per token, and the decisions compound.

Consider a step where the top two candidates are separated by a logit gap of 0.0001. A reduction-order difference perturbs both by more than that, and the argmax flips. The chosen token is appended to the context, so every subsequent forward pass sees a different sequence — not slightly different logits, a genuinely different input. From that point the two runs diverge completely.

So the observed behaviour is bimodal rather than gradual: runs are usually identical, and occasionally completely different from some point onwards. That is exactly what you would expect from a system whose only source of variation is rare tie-breaking, and it is why “it was reproducible when I tested it” is weak evidence. Near-ties are more common on the outputs you care about — a hedged answer, a borderline classification, a choice between two equally plausible tool calls.

What has to match for runs to repeat

If you need reproducibility, you are freezing an environment, not setting a parameter. In rough order of how often each one is the culprit:

1. The checkpoint revision. Not the repository name — the commit. Config and tokenizer files change under main. See pinning a Phi checkpoint.
2. Batch size 1. The single largest source of variation in a served deployment. Reproducibility and throughput are in direct conflict here; a reproducibility harness should not share a server with production traffic.
3. Framework and kernel versions. The runtime, the CUDA version and the attention library, pinned together. Pin the attention backend explicitly rather than letting it be selected.
4. dtype and quantisation. Stated explicitly rather than left to torch_dtype="auto", which resolves differently on different hardware.
5. The same GPU model. Not merely the same manufacturer or the same memory size.
6. The rendered prompt. Byte-identical after the chat template is applied, including trailing whitespace. Compare with repr or a hash, not by eye.

Where a seed does and does not help

A seed initialises the pseudo-random number generator that the sampler draws from. At temperature 0 there is no draw, so the seed is inert — setting one does not make greedy decoding more reproducible, and removing one does not make it less so. This confuses people because both knobs are described as being about determinism, and they operate on different halves of the problem.

Where the seed earns its place is the opposite configuration: sampling on, temperature above zero, and reproducibility still wanted. That is a real and often better setup, because greedy decoding is not free of quality cost — it is prone to repetition and to committing early to a locally-attractive continuation. Sampling at a low temperature with a fixed seed gives you varied, higher-quality output that you can replay, subject to exactly the same numerical caveats as above:

# transformers
from transformers import set_seed
set_seed(11)
out = model.generate(**inputs, do_sample=True, temperature=0.3, top_p=0.9,
                     max_new_tokens=512)

# vLLM
SamplingParams(temperature=0.3, top_p=0.9, seed=11, max_tokens=512)

Two limits on this. The seed reproduces the draw, not the logits, so every hardware and batching caveat above still applies — the seed removes one source of variation and leaves the rest. And in a server that processes many requests, a global seed set once is consumed by whatever order requests happen to arrive in; per-request seeding, as in the vLLM line above, is the only form that means anything under concurrency.

Testing against a model that wobbles

The engineering conclusion is not to chase bit-exactness. It is to stop writing assertions that depend on it. A test that compares generated text to a stored string is testing your kernel selection, and it will fail on an unrelated infrastructure change with a diff that tells you nothing.

Assert on what the output has to be true of instead: that it parses as JSON and validates against the schema, that the classification field is one of five permitted values, that the cited identifier appears in the source document, that the answer is under the length you promised a user. Where you genuinely need distributional confidence, run the same input several times and assert on agreement rate rather than on identity — which also gives you a number that degrades visibly when a model change makes the output less stable, instead of a boolean that flips.