Testing for Catastrophic Forgetting After a Fine-Tuning Run

A fine-tune is a gradient update, and a gradient update computed entirely on invoice classification does not know that the model is also expected to speak Portuguese, refuse a dangerous request and return valid JSON. Those capabilities are not protected by anything. The only reason you would notice them degrading is that you measured them first.

Why the weights move somewhere you did not look

Fine-tuning minimises loss on your dataset. Every parameter that can be adjusted to reduce that loss will be, and nothing in the objective expresses a preference for leaving unrelated behaviours intact. If your dataset is 4,000 examples of terse category labels, the update that best fits it is one that makes terse category labels more likely in general — including in contexts where you wanted a paragraph.

The distribution of your data is the thing doing the damage, more than its size. A narrow dataset is a strong statement that this is what output looks like now. That is why the symptom is so often stylistic rather than factual: the model has not lost knowledge so much as lost the range of behaviours it used to select between. Format compliance, verbosity, refusal willingness and language choice are all decisions made near the output distribution, which is precisely where a narrow fine-tune pushes hardest.

Two structural facts follow. First, parameter-efficient methods that constrain how much of the model can move — adapters trained at a low rank, with a low learning rate and few epochs — reduce the blast radius, because there is simply less capacity available to overwrite. Second, mixing general-purpose examples into the training set works for the same reason a regression suite works: it puts the old behaviour back into the objective. Neither is free, and both trade against how strongly the model adopts the new behaviour.

What belongs in the held-out set

The set must contain nothing that resembles your fine-tuning data, and it must cover capabilities you rely on implicitly. That second word is the hard one: implicit dependencies are the ones no test covers, because nobody thought of them as features. A workable starting taxonomy:

• Instruction following at a shape you never trained. “Answer in exactly three bullet points”, “reply with one word”. Narrow tuning degrades this early.
• Structured output on an unrelated schema. Not your production schema — a different one, so you are testing the general ability rather than the trained one.
• Tool selection. A handful of cases with tools that were absent from training, asserting the right tool by name. This degrades quietly and is expensive when it does.
• Refusal and safety behaviour. Both directions: cases that must be refused and benign cases that must not be. Narrow tuning can make a model both more compliant and more skittish, and the second is easy to miss.
• Languages you support. If your training data is English and your users are not, this is the row most likely to move — the same territory as multilingual output consistency.
• Long-context recall. A fact placed early in a long input and asked about at the end. Cheap to write, and sensitive.
• Arithmetic and simple reasoning chains. Not to benchmark the model, but as a canary that generic capability has not shifted.

Keep it small enough to run every time — a few dozen cases per category. This set exists to detect a direction of movement, not to rank models, and a set nobody runs because it costs too much detects nothing.

The before-and-after that actually compares

Run the set against the base model before you start training, and store the per-case results as a committed artifact with the sampling configuration attached. Run it against the fine-tune afterwards through the identical harness. Then compare per case, not on an average, for the same reason described in regression testing against the base suite: an unchanged mean hides equal numbers of losses and gains, and the losses are the entire subject here.

Because these cases are open-ended, many of them cannot be graded by string equality. Grade on the property, not the prose: did the answer have three bullets, did it parse against the schema, did it name the expected tool, did it refuse, was it in the input language, did it contain the fact planted at position zero. Every one of those is a predicate that survives rewording, which is the only kind of assertion worth putting in this set.

Sample each case more than once. A capability that has weakened often shows up first as increased variance rather than as a clean flip — the model still gets it right most of the time. Three samples per case, scored as a fraction, makes a drop from three-of-three to two-of-three visible; a single sample turns it into coin-flip noise.

What forgetting looks like when it starts

It rarely announces itself as a wrong answer. The early signals are shape-shaped: answers converge on the length and register of your training data regardless of what was asked; the model starts emitting your production schema for prompts that requested a different one; instructions that were previously followed to the letter get approximated. Refusals become either blanket or absent. Non-English prompts start receiving English answers — the model has learned that output looks like your training data, and your training data is in one language.

All of these pass a test that asserts the response was non-empty and parsed. That is why the held-out set has to assert on properties that would be false under a shape collapse, and why the pass/fail of the fine-tuning objective is not evidence of anything about them.

What to change when the set goes red

Treat the result as a signal about the training run, not as a gate to override. The levers, in rough order of how often they help: fewer epochs, since most degradation of this kind appears as the run continues past the point where the target behaviour is already learned; a lower learning rate; a lower adapter rank if you are using a parameter-efficient method; and mixing general examples into the training set so the old behaviour is represented in the objective.

There is also the option nobody likes hearing, which is that the behaviour did not need a fine-tune. If the target was formatting or tone, and the cost is measurable damage to instruction following and language coverage, a prompt and a validator often buy the same behaviour with no weights moved and no baseline to maintain. Fine-tune when you need behaviour that prompting cannot reach, and keep this set running either way — a base model updated in place by its provider can move these same rows without any training run at all.