What Actually Changes When You Bump a Model's Minor Version

A same-family bump — one snapshot of a model to the next — keeps the API shape and changes the function behind it. Knowing which of those two halves a given behaviour lives in is what turns “re-test everything” into a short list.

What a snapshot actually pins

A dated snapshot pins the weights and the serving configuration for that model. It does not pin the API: the request and response schema belongs to the endpoint and its version, not to the model, which is why a new parameter can appear and become available to a snapshot you pinned months ago. It also does not pin anything about the system around the model — moderation classifiers, safety filters and routing infrastructure are updated independently, which is why identical requests to an identical snapshot can start returning a finish_reason of content_filter without the model having changed at all.

So the honest statement is narrower than “pinning a snapshot makes behaviour stable”. Pinning makes the model stable and leaves several other moving parts in the path. What a bump does is move the one part pinning was holding still, all at once.

The output distribution moves

The mechanism here is worth being precise about, because it explains why the failures are so uneven. A model returns a probability distribution over next tokens; a new snapshot returns a different distribution for the same context. For a prompt where the intended answer was overwhelmingly likely, a small shift changes nothing observable. For a prompt sitting near a boundary — a classification where two labels were nearly tied, a formatting instruction the old snapshot followed about eighty percent of the time — a small shift flips it.

That is why post-bump reports cluster the way they do: the bulk of traffic looks identical and one specific prompt breaks completely. The prompts that break are the ones that were already marginal, and they were marginal before the bump too; the bump only revealed it. A prompt whose behaviour changes on a version bump is usually a prompt that was already flaky at temperature above zero, which is a separate and fixable problem — see prompt sensitivity.

Two corollaries. First, temperature: 0 does not protect you: it picks the top token of whatever distribution the new snapshot produces, so it makes each version internally consistent and does nothing to make the two versions agree. Second, a seed that reproduced an output on the old snapshot will not reproduce it on the new one, and the system fingerprint changing is the signal that this has happened.

Token efficiency, and why the bill moves too

Two distinct things can move your token counts on a bump, and they need separating because only one of them is under your control.

The first is verbosity. A new snapshot tuned to be more thorough produces longer answers for the same prompt, and output tokens are the expensive side of the bill. Nothing about your code changed and your cost per request rises. This shows up immediately in your own metrics if you record output token counts per route, and is invisible if you only record request counts.

The second is tokenisation, which moves only when the family changes its tokeniser — rarer than a snapshot bump, and much more disruptive when it happens, because it changes the token count of text you did not touch. If your application enforces a context budget by counting tokens locally with a library pinned to the old tokeniser, that count silently stops matching what the provider bills and what the provider truncates against. The general treatment is on the tokeniser mismatch bug and token count mismatch. The migration-specific advice is simply: when the family changes, re-derive your budget from the provider’s reported usage figures rather than from your local count, and treat the local count as an estimate until the two agree again.

Reasoning-capable snapshots add a third line. Tokens spent thinking are billed as output but are not returned to you as content, so a bump that increases reasoning effort raises cost with no visible change in the response. If your accounting derives cost from the length of the text you received rather than from the usage object, that increase is invisible.

Defaults, refusals and tool formatting

• Parameter defaults are per-model and undocumented in places. The default maximum output length, in particular, differs between models and has changed between snapshots. If you never set an explicit cap, a bump can change how long your answers are allowed to be in either direction. Set it explicitly; a request that relies on a default is a request whose behaviour is owned by somebody else. The relationship between that cap and the context window is covered on context window versus max tokens.
• Parameter validity can narrow. Newer snapshots have dropped support for sampling parameters that older ones in the same family accepted, returning an unsupported-parameter error rather than ignoring the field. A request body that has been valid for a year can become a 400 on a bump.
• The refusal boundary moves in both directions. Safety tuning is part of what a snapshot is. Content the old version handled can be refused and content the old version refused can be handled. The class most often hit accidentally is domain material that reads as sensitive out of context — medical, legal, security. If refusals matter to you, they need a fixture set of their own, because they do not show up as errors.
• Tool-call formatting tightens or loosens. Whether the model emits multiple tool calls in one turn, how strictly it adheres to a JSON Schema, and how it behaves when no tool applies are all model behaviour rather than API contract. A schema contract test is the check that catches this.

What to re-test and what you may assume

You may assume the transport is unchanged: the endpoint, the authentication, the request schema, the response schema, the streaming event names and the set of possible finish_reason or stop_reason values are properties of the API, not of the snapshot. Code that parses responses does not need revisiting for a same-family bump.

What needs re-testing is everything downstream of the text: any parser that reads structure out of a response, any classification whose labels feed a branch, any prompt with a formatting instruction, any refusal-sensitive path, and your cost per request. That is a list you can turn into a fixture set once and re-run on every bump — building that harness is the next page. Run both snapshots concurrently while you do it: the old string keeps resolving until its retirement date, and having both available at once is the whole reason to pin snapshots rather than track an alias.