Model Degradation Over Time: Real or Perceived?

Every few months a wave of “it got worse” posts appears, followed by a wave of “it is your imagination” replies. Both sides are arguing without data, because almost nobody froze a test set before the thing they are arguing about happened. This page is about not being in that position.

The recurring argument

The claim is hard to evaluate because so many things are changing at once: the model behind an alias, the provider’s serving stack, your own prompts, your own context, the difficulty of the work you are throwing at it, and your expectations. Perceived degradation is genuinely common — novelty wears off, the easy wins were taken first, and today’s baseline is last quarter’s ceiling. Actual degradation also genuinely happens. Nothing about the shape of the complaint distinguishes them.

The study everyone cites

Chen, Zaharia and Zou’s How Is ChatGPT’s Behavior Changing over Time? (2023) compared the March and June 2023 snapshots of GPT-3.5 and GPT-4 across several tasks: prime identification, code generation, visual reasoning and answering sensitive questions. The headline finding was that behaviour changed substantially over three months in both directions, with the widely quoted case being GPT-4’s accuracy on the prime-identification task falling from around 84% to around 51%. The code-generation result — a large drop in directly executable output — was also widely shared.

It is a valuable paper, and it established the thing that matters most: model behaviour behind a stable API name is not stable, and drift is measurable.

The critique, which is also correct

The paper drew serious methodological objections almost immediately, and a page that cites the result without them is not being honest.

• The prime task was unbalanced. The evaluation set consisted of numbers that were in fact prime. A model that shifted towards answering “yes” would score near-perfectly and a model that shifted towards “no” would score near-zero, without any change in its ability to test primality. Measured on composite numbers as well, the picture changes considerably. This is the objection that stuck.
• Much of the code result was formatting. The June model wrapped code in Markdown fences more often, which made output fail a directly-executable check without making the code worse. That is behaviour drift, which is real and can break your integration, but it is not capability loss and calling it that is misleading.
• Behaviour drift and capability drift need separate metrics. This is the durable lesson. Format compliance, verbosity, refusal rate and answer-bias are all things that can move without accuracy moving at all — and they are things that break production systems, so measure them, just do not call them the same thing.

What genuinely changes

• The alias repoints. An unversioned model name is a pointer, and vendors move it. Pinning a dated snapshot removes this entirely and is the single highest-value habit in this list.
• The system layer changes. Provider-side system prompts, safety classifiers and routing sit between you and the weights, and they are updated without release notes.
• The serving configuration changes. Quantisation, kernel versions, hardware generation, speculative decoding. For open-weights models served by several providers this is the largest variable, and it is the subject of non-determinism at temperature zero.
• Snapshots are retired. Your pin eventually stops working, and the migration is a real change you should treat as one.
• Your prompt grew. Six months of accreted instructions is a different prompt, usually longer, often internally contradictory. This is the cause people discover last and it is frequently the actual one.
• Your inputs got harder. As a feature succeeds, users bring it harder cases. The model did not change; the distribution did.

The harness that settles it for you

No method settles the general argument, because the general argument is about a workload nobody froze. This one settles it for yours, and it has to exist before the change you want to detect.

• Freeze 200–500 real inputs with expected outputs. Version it, keep it in the repository, and never quietly edit it — an edited evaluation set makes every historical number incomparable.
• Record the full identity of every run. Model id including the snapshot date, provider, the fingerprint field if there is one, all sampling parameters, your prompt’s git hash, and the timestamp. A result without this is not evidence of anything.
• Run it on a schedule, not on suspicion. Weekly, on the pinned version. Suspicion-triggered runs have no baseline, which is the exact problem you are trying to avoid.
• Report capability and behaviour separately. Accuracy, and alongside it: mean output length, schema-compliance rate, refusal rate, tool-call rate, latency. The critique above is the reason.
• Use an interval, and know your resolution. With 300 items a difference of a couple of points is noise. Run a two-proportion test against the previous week, and remember that the prompt-format spread from the sensitivity page is a lower bound on what you can resolve at all.
• Sample the failures by hand every month. The aggregate tells you something changed; twenty read failures tell you what. Nothing replaces this step.

When it fires, check the cheap explanations first: a repointed alias, a prompt change in your own history, an input-mix shift. In practice those account for most confirmed regressions, and the model is exonerated more often than the forum posts suggest — but with a frozen set you will know rather than argue.