Regression Testing After a Provider Silently Updates a Model

The suite was green on Thursday, nobody merged anything, and on Friday eleven cases are red. This is a real category of event, it has a small number of specific causes, and the investigation is a process of elimination that takes about twenty minutes if you logged the right fields and is close to impossible if you did not.

Six ways the model changes without a deploy

“Pin the model ID” is the standard advice and it is good advice, but it closes only the first of these.

• An alias that resolves elsewhere. Whether a model string is a pointer or a pin depends on the provider and the generation. Anthropic’s models overview states that every Claude model ID is a pinned snapshot, that dated IDs are fixed to that release, and that from the Claude 4.6 generation onward the dateless IDs are pinned snapshots too — while for earlier generations the alias entries are convenience pointers resolving to a dated ID. So the same-looking string carries different guarantees depending on which model it names. Look it up for the exact strings in your configuration.
• No model specified at all. An SDK or framework default, upgraded with a dependency bump you did not read. The request has no model field and the library picks.
• The serving stack behind a fixed ID. The reason OpenAI returns system_fingerprint, documented as representing “the backend configuration that the model runs with” and intended for use with seed to see when backend changes have been made that might affect determinism. The provider is telling you in the response schema that a pinned ID is not a pinned computation.
• Endpoint or region routing. Cloud deployments of the same model distinguish global endpoints, which route dynamically for availability, from regional ones. Which capacity served you can vary between runs without anything in your code changing.
• Provider-side content and safety layers. Filters and system-level instructions sit outside the model and are updated on the provider’s schedule. These usually show up as refusals on inputs that worked, which is why a refusal directory in a failure-mode layout is worth having.
• Quantisation or hardware changes. Most relevant with third-party hosts and self-hosted weights, where the same weights served at a different precision give measurably different outputs with no version string to record it.

Establishing it was not you

The first question a red build on an unchanged tree raises is whether the tree really is unchanged, and the honest answer is often no. Compare against the last green run, field by field. This is a five-minute check when the run header carries the values and an afternoon of archaeology when it does not.

• Prompt hash. The content hash of every template in play, per the tagging convention. Catches an edit to a shared fragment that nobody thought of as touching this prompt.
• Resolved model ID. Not the string in your config — the ID the provider reports in the response, which is the only one that reflects an alias resolution.
• Backend fingerprint. Where the provider returns one. A changed fingerprint between two otherwise identical runs is close to a direct answer.
• SDK and dependency versions. A lockfile diff. Client libraries change default parameters and default endpoints.
• Sampling parameters and tool schemas. Temperature, top-p, max tokens, and the serialised tool definitions. A tool description edit is a prompt edit.
• Retrieval corpus and fixtures. If context comes from an index, the index is an input. A reindex is a change even when no code moved.

If every one of those is byte-identical to the last green run, the remaining variables are the provider and the sampler. That is a real conclusion, and it is only available because those fields were recorded — which is the strongest practical argument for putting them in the run header and the logs.

Separating a change from sampling noise

Now distinguish the two remaining candidates, because they look identical in a single run and demand opposite responses.

Re-run only the failing cases, k times each, at the same parameters. Twenty repetitions is a reasonable default; the arithmetic for why k must be that large sits in why a passing suite misses regressions. Then read the counts.

A case failing one or two times in twenty was always doing that. The mode existed before today; you got unlucky on Friday and lucky on the hundred runs before it. Nothing changed on the provider’s side, and the fix is repetition on that case rather than an incident.

A case failing twenty times in twenty, which has been green on every run for months, is a change. Deterministic failure on an input that was deterministically passing is not something sampling produces. That is the signal, and it is worth acting on.

The middle ground — eight in twenty, on a case with no failure history — is genuinely ambiguous from one session, and the correct answer is to compare against the historical rate rather than to reason harder about today. Which is the argument for storing pass counts per case per run rather than only the final verdict: without a history, the middle ground stays ambiguous forever.

What to do when it is real

In order, and the second step is the one most often skipped.

Scope it first. Which failure modes, how many cases, and is the pattern concentrated or spread. Fourteen format failures across every fixture is a changed output contract. Three refusals on inputs mentioning a particular topic is a changed safety layer. One fabrication failure is probably not this event at all.

Then ask whether the assertion or the behaviour is what is wrong, because a provider change that makes the model better will break cases that pinned prose, and those cases were wrong when they were written. If the failing assertion is an equality against stored text and the new output is fine, you have found a drifted baseline rather than a regression. Fix the assertion. Only when a real invariant is violated — a schema, a total, an identifier that was never supplied — is this a regression.

If it is, pin to the previous version to buy time, if one is still served, and check the deprecation schedule at the same moment. Pinning is a delay rather than a cure: dated snapshots retire, and a system that only works on a retired ID has a deadline nobody has written down. Use the bought time to fix the prompt, then re-run the comparison properly as a paired run against both IDs.

Finally, keep the case. A behaviour that changed once under you is a behaviour the provider is willing to change, which makes it a failure mode worth a permanent case and possibly a directory of its own.

The control run that makes this cheap

Everything above depends on one piece of infrastructure, and it is the single most valuable thing in this cluster: a scheduled run of the deterministic tier against a frozen tree.

A suite that only runs on your commits cannot ever distinguish “we broke it” from “it changed”, because it only executes at moments when you changed something. Every red build is confounded with your own diff by construction. Add a nightly run against the default branch with no code change between executions, and the confound disappears: red on an unchanged tree has exactly one class of cause.

It is cheap. The tier is the deterministic one, so no judge calls; at the shape derived in the sizing arithmetic a nightly full run is a few dollars a night. Record the resolved model ID, the backend fingerprint and the per-case pass counts on every execution, and keep them. Six months of that history turns the ambiguous middle ground above into a lookup, and turns “something changed” into a date.