Why "Deterministic" Output Is Not Deterministic After a Migration

The seed is set, the temperature is zero, the prompt is identical, and the new provider returns a different answer on the second call. Nothing is broken. The seed never controlled the part that is varying.

The symptom, precisely

It is worth being exact about which of three situations you are in, because they have different causes:

• Different from the old provider, stable on the new one. Two different models produce two different answers. There was never any reason to expect otherwise; a seed is not a cross-model identifier and the same integer means nothing in common between two samplers.
• Unstable on the new provider, run to run. The interesting case, and what the rest of this page is about.
• Stable for a week, then different. Usually a provider-side change — a new build, a routed-to variant, a silently updated alias. The library covers that pattern in silent model updates.

Distinguish them before debugging: run the same request twenty times on the new provider in one minute and count distinct outputs. If the answer is one, you are in the first case and the work is a fixture update. If it is more than one, keep reading.

What a seed is scoped to

A seed initialises the pseudo-random generator that the sampler uses to draw a token from a probability distribution. Give it the same distribution and the same seed and it draws the same token, every time. It is a completely reliable mechanism within that scope.

The scope is the problem. Nothing about the seed reaches the computation that produces the distribution. If the logits at step 40 differ in the fifth decimal place between two runs, the distributions differ, and the sampler — faithfully, deterministically — may draw a different token from them. From there the two sequences diverge completely, because token 41 is conditioned on token 40.

This also explains why a seed appears to work perfectly on short outputs and to fail on long ones. Divergence requires one step where two nearly-tied candidates swap order. The probability that no such step occurs falls as the number of steps rises, so a ten-token answer reproduces almost always and a thousand-token answer reproduces rarely, with the same seed and the same everything else.

Why the distribution itself moves

Two mechanisms make the forward pass produce slightly different numbers for identical input on production inference infrastructure, and neither is under your control.

Floating-point arithmetic is not associative. Adding the same set of numbers in a different order gives results that differ in the last bits. GPU kernels sum in whatever order the parallel reduction happens to schedule, and that order can depend on the shape of the work — which is to say, on what else is being computed at the same time.

Requests are served in batches, and your batch changes. A serving system groups concurrent requests to keep the hardware busy. The batch your request lands in depends on who else called in that instant, and the batch shape can change kernel selection and reduction order. Your input is identical; the arithmetic performed on it is not bit-identical. On a mixture-of-experts architecture the coupling can be stronger still, because routing decisions are made over the batch.

This is a property of shared, high-throughput serving rather than a defect, and it is why the hedged wording in every provider’s seed documentation is honest rather than lawyerly. The library argues the full case in temperature zero and nondeterminism; the point to carry here is that it applies to seeded requests exactly as much as to unseeded ones, because it operates before the sampler.

What your migration changed as well

Before accepting the above as the explanation, rule out the changes you made yourself. A migration that moves providers usually changes several inputs at once, and each of these produces run-to-run instability that looks identical to the batching effect:

• The seed stopped being sent. The most common cause by a distance. The field was renamed, or moved inside a generation config object, and the provider ignores unknown top-level keys. Log the outbound request body once and read it.
• Temperature is no longer zero. Defaults differ between providers, and code that relied on the old default is now sampling at whatever the new one is. Set it explicitly.
• The prompt is assembled differently. An adapter that moves the system message into a top-level parameter, joins message content with a different separator, or serialises tool schemas from an unordered dictionary produces a different input on every process start.
• Something in the prompt is not constant. A retrieved chunk set that depends on index state, a current date, a randomly-ordered few-shot block. If a RAG pipeline sits in front, the provider-swap page for RAG templates covers where those creep in.
• You are routed across model versions. An alias that resolves to more than one build, or a fallback that quietly served a different model on some requests, gives you two distributions and no seed can bridge them.

Check these in order and log the full outbound request alongside the response for a hundred calls. If the request bytes are identical across calls and the outputs are not, the cause is provider-side and no amount of further debugging on your side will change it.

Making the failing tests correct instead

When the cause is provider-side, the fix is in what you assert. A test that compares an output string to a stored string was always testing the provider’s infrastructure as much as your code; the migration simply revealed it. Three replacements, in the order they usually apply:

1. Assert on structure and extracted values. Constrain the output with a schema and assert on the parsed fields. A test that checks result.category == "refund" is stable under every mechanism on this page; one that checks the sentence around it is not.
2. Replay recorded responses for everything that is not about the model. Most of a test suite exercises your parsing, routing and error handling. Those tests should never call a provider. Record once, replay always, and keep the small number of tests that genuinely need a live model separate and tolerant.
3. For the tests that must be live, assert a distribution. Run the case a fixed number of times and require that the extracted answer is correct in at least k of n. That is a real property of the system, it fails when quality actually regresses, and it does not fail because a reduction order changed.

Doing this costs a day or two and it is the only durable outcome. The alternative — hunting for the configuration that restores byte equality — is a search for something that no provider has agreed to give you. The mapping of the parameter itself, and how to measure what it is worth on a given provider, is in the seed parameter field mapping.