Silent Model Updates: Detecting a Behaviour Change

“They changed the model” is the first explanation offered for any unexplained quality drop, and it is right often enough to be worth actually testing. There are two versions of the claim, they need different detectors, and only one of them is cheap.

Aliases, snapshots, and what you actually pinned

Providers generally publish two kinds of model identifier. A snapshot id names a frozen build and is dated — gpt-4o-2024-08-06 and claude-sonnet-4-5-20250929 are the shape. An alias — the undated name, or one ending in something like -latest — points at whichever snapshot the provider currently considers current, and that pointer moves.

If you call an alias, you have opted into a rolling upgrade with no release notes on your side and no rollback. That is a defensible choice for a chat toy and a poor one for anything with an eval suite behind it. The rule worth adopting: production calls a snapshot id, and moving to a new snapshot is a deploy that goes through the same canary process as any other change.

Two caveats, because pinning is not total insulation. Snapshots get deprecated and eventually retired, so a pin is a commitment to a migration on the provider’s schedule — track their deprecation pages. And serving-side changes (quantisation, routing, speculative decoding, safety-filter updates) can alter behaviour without any identifier moving at all. Pinning removes the largest source of surprise, not every source.

The cheap detector: diff the response model

Most providers echo back which model actually served the request, and the OpenTelemetry GenAI conventions give it a name: gen_ai.response.model, distinct from gen_ai.request.model for exactly this reason. Storing both — as requested_model and served_model in the request log — makes the identifier case a one-line detector.

-- Any pair we have not seen before in the last 30 days.
with seen as (
  select distinct requested_model, served_model
  from llm_request
  where started_at between now() - interval '30 days' and now() - interval '1 day'
)
select r.requested_model, r.served_model, min(r.started_at) as first_seen, count(*)
from llm_request r
left join seen s
  on s.requested_model = r.requested_model and s.served_model = r.served_model
where r.started_at >= now() - interval '1 day'
  and s.requested_model is null
group by 1, 2
order by first_seen;

Run it hourly, alert on any row. It has a near-zero false-positive rate, it costs nothing, and it catches the case that accounts for most real instances of “the model changed”. Keep the result as an append-only table of known pairs rather than recomputing the thirty-day window each time, so that a pair which appeared once during an incident and never again is still on the record months later when somebody is reconstructing a timeline. Where a provider also returns a backend configuration fingerprint on the response — OpenAI’s system_fingerprint is the documented example — store and diff that too; it moves on serving changes that leave the model id alone.

A canary suite, and what it must control for

The identifier detector cannot see behaviour changes under a stable id. For those you need a fixed set of prompts, run on a schedule, with the outputs compared over time. The design is simple; the traps are all in what you control for.

• Freeze everything except the provider. Same prompt text, same parameters, same seed where supported, same SDK version, same tool definitions. If your canary prompts live in the same registry as production prompts, an ordinary prompt edit will look like a provider change.
• No dates, no randomness, no retrieval. A canary prompt that injects today’s date changes every day by construction. Retrieval makes the corpus a second moving part.
• Score on something stable. Exact-match on structured tasks, numeric answers for arithmetic, schema validity, answer-set membership for classification. Free prose scored by another model gives you two moving parts and no way to tell which moved.
• Cover the behaviours you depend on, not a benchmark. Instruction adherence, tool-call formatting, refusal boundaries, output length under a “be brief” instruction, non-English handling if you have it. Thirty to a hundred prompts is plenty.
• Run it against a pinned snapshot and the alias. The divergence between those two is the most interpretable number the suite produces.

The tolerance band

Here is the part that determines whether the suite is useful or just noisy. Greedy decoding is not bit-reproducible in practice — batching, reduced-precision arithmetic and expert routing in sparse models all introduce run-to-run variation, so identical inputs at temperature 0 can yield different outputs from an unchanged model. A canary that alerts on “the output differs from yesterday” will alert constantly and teach everyone to ignore it.

So establish the band empirically before you use it as a detector:

• Run the suite k times (k = 10 is a reasonable start) against a single pinned snapshot within a short window, when you have no reason to think anything changed.
• For each metric — pass rate, mean output length, exact-match rate — record the spread across those runs. That spread is your noise floor.
• Set the alert threshold outside the observed spread, and require the deviation to persist across two consecutive scheduled runs before it fires. A single excursion is a sample, not a signal.
• Re-establish the floor whenever you change the suite, the parameters, or the snapshot. The floor belongs to a configuration, not to a provider.

This is a control design, not a claim about how well it works. What it buys you is a defensible answer to “is this a real change?” — one that distinguishes a shift from the variation your own suite produces on a model that definitely did not change.

Budget for the suite honestly, because an expensive canary gets turned off. A hundred prompts run four times a day against two model identifiers is eight hundred calls a day of mostly short outputs — small, but not free, and worth putting on the cost dashboard under its own feature so it never gets mistaken for production spend. If that number is uncomfortable, reduce the frequency before reducing the prompt count: a suite that runs daily and covers the behaviours you depend on is far more useful than an hourly one that covers three of them.

What to do when it fires

Have the response written down before you need it, because the useful window is short.