Catastrophic Forgetting: What Fine-Tuning Breaks

A fine-tune optimises one objective on one distribution. Everything the model could do that is not in that objective is, from the optimiser’s point of view, unconstrained — and unconstrained things drift.

The mechanism

The term is older than language models. McCloskey and Cohen coined it in 1989 for connectionist networks that lost previously learned associations when trained on new ones, and the cause has not changed: gradient descent moves the parameters that reduce the current loss, and nothing in the update expresses a preference for keeping the parameters that mattered for tasks not in the current batch.

Two things make it worse in a fine-tuning setting than in pretraining. Your dataset is narrow, so the gradient consistently points in one direction rather than averaging over a broad distribution. And your dataset is small, so you run several epochs over the same examples, repeatedly pushing in that same direction.

The result is not that the model forgets facts in the sense of losing stored information. It is that the model’s output distribution collapses toward your data — it becomes strongly biased toward the shapes, lengths and register you trained on, and applies them to requests that wanted something else. A model fine-tuned to always return JSON will return JSON when asked to explain something in prose.

What the literature reports

• The alignment tax, Ouyang et al. 2022. The InstructGPT paper reports performance regressions on several public NLP datasets after the RLHF stage, names the effect, and describes mixing pretraining gradients into the RL updates (PPO-ptx) as the mitigation. This is the most widely cited concrete instance of the phenomenon in a production-scale model.
• Continual instruction tuning, Luo et al. 2023 (arXiv 2308.08747). An empirical study of catastrophic forgetting in LLMs during continual instruction tuning, examining how forgetting behaves across model scales and tuning methods. The paper is worth reading directly rather than through a summary — the interaction between scale and forgetting is not a one-line result.
• Elastic weight consolidation, Kirkpatrick et al. 2017. The canonical mitigation from the wider continual-learning literature: penalise movement in parameters estimated to be important for previous tasks, using the Fisher information as the importance estimate. Rarely used at LLM fine-tuning scale, but it is the reference point every cheaper mitigation is an approximation of.

What this literature does not give you is a number for your fine-tune. Forgetting depends on your data, your rank, your learning rate and your epoch count, and the published studies vary on all four. The transferable part is the mechanism and the mitigations; the magnitude is something you measure.

The shapes it takes

The retention suite

The procedure is unglamorous and takes an afternoon to build. It is the difference between finding this in evaluation and finding it in a support ticket.

• Freeze a set of 100–200 prompts that have nothing to do with your task, before you train. General questions, multi-turn conversations, other languages, prompts in output formats you are not training on, and a handful of requests that should be refused.
• Record the base model’s answers to all of them, once, and store them alongside the suite. This is your baseline and it never changes for a given base checkpoint.
• Run the same suite against every candidate adapter, and diff. You are not looking for a score, you are looking for behaviour that changed — which is why you keep the base outputs rather than a number.
• Automate the parts that are checkable. Did it answer in the requested language? Did it respect “in one sentence”? Did it refuse what the base refused? Length, language identification and refusal detection are all cheap programmatic checks and they catch most of the table above.
• Run it at several checkpoints, not just the last. Retention usually degrades monotonically with training steps while task performance plateaus early. The best checkpoint is frequently not the final one, and you can only know that if you evaluated more than one.

Mitigations, ranked by what they cost

• Train less. Fewer epochs, lower learning rate, earlier stopping. Free, immediate, and the single most effective intervention. Most visible forgetting comes from runs that trained well past the point where the task metric stopped improving.
• Use LoRA rather than full fine-tuning, at a modest rank. A low-rank update constrained to attention and MLP projections has less capacity to move the model, which is a limitation for capability and a feature for retention.
• Mix in general data. The cheap approximation of PPO-ptx. Add 5–20% of general instruction data — a public instruction set is fine — to every batch. It costs a little task performance and buys a lot of retention, and it is the standard recipe in the continual-pretraining literature for the same reason.
• Keep the base and route. Serve the adapter for the requests it was trained for and the base for everything else. This makes forgetting irrelevant by construction rather than fighting it, and with adapter-based serving the marginal cost is small.
• Explicit regularisation toward the base. A KL penalty against the reference model’s output distribution, or an EWC-style parameter penalty. Effective, and rarely worth the implementation cost when the four options above are available.