Fine-Tuning Failures: The Modes and How to Tell Them Apart

The dangerous fine-tuning failures are not the ones that raise an exception. They are the ones where the run completes, the loss curve looks plausible and the model is quietly wrong — and the way to catch those is to know what each failure mode looks like from the outside.

Failures that crash

Out of memory partway through, not at the start

A run that survives a thousand steps and then raises CUDA out of memory did not gradually leak. It met the longest sequence in the dataset. Memory for activations and for the logits tensor scales with sequence length, and the tail of a real length distribution is long.

Diagnostic: plot your token-length histogram and check whether the crash step corresponds to a batch containing the maximum. Sort the dataset by length and run the longest batch first — if it fails immediately, that was it. Fix: cap max_seq_length, use length-grouped batching, reduce the micro-batch and raise gradient accumulation, or use a chunked cross-entropy that never materialises the full fp32 logits. The arithmetic is in the QLoRA memory budget.

“element 0 of tensors does not require grad”

A RuntimeError familiar to anyone who has combined gradient checkpointing with a frozen quantised base. The base weights require no gradient, so the checkpointed segment’s inputs do not either, and the recomputation has nothing to build a graph from.

Fix: the standard remedy documented in the PEFT library is to call model.enable_input_require_grads() before training, which makes the embedding outputs require gradients so the checkpointed blocks have a graph. Utility functions that prepare a quantised model for training generally do this for you, which is why the error appears mostly in hand-rolled setups.

Loss becomes NaN

Overwhelmingly this is fp16 on a model pretrained in bf16. The two formats have the same width and very different dynamic range; bf16 keeps fp32’s exponent range, fp16 does not, and activations that were unremarkable in training overflow.

Diagnostic: check the base model’s stated training precision, then check what your config set. Fix: bf16. If the hardware cannot do bf16, lower the learning rate, enable gradient clipping, and inspect the batch at the step before the NaN for a degenerate example.

Failures that are silent

These are the expensive ones. The run completes, the numbers look normal, and nothing is learned.

Failures that look like quality problems

• Memorisation reported as success. Held-out performance is excellent because near-duplicates leaked across the split. Diagnostic: near-duplicate detection between train and test at a Jaccard threshold around 0.8, and a split on a grouping key rather than on rows. Log-derived datasets are especially prone to this because production traffic is highly templated.
• Improvement that is entirely length. The judge preferred the fine-tune because its answers got longer. Diagnostic: compare the output length distributions of the two candidates before reading the win rate at all.
• Improvement inside the confidence interval. A 100-example evaluation has a roughly ten-point confidence interval, so a five-point win is indistinguishable from nothing. Diagnostic: compute the interval before you interpret the number. The arithmetic is in the evaluation page.
• General ability regression. The task improved and everything else got worse — the alignment tax reported in Ouyang et al. (2022) for RLHF, and the general phenomenon studied by Luo et al. (2023) for continual instruction tuning. Diagnostic: a frozen retention suite run against the base once and diffed against every candidate. Without it, this failure reaches users.
• Overfitting past the useful checkpoint. Training loss falls, held-out loss rises, and the shipped checkpoint is the last one rather than the best one. Diagnostic: evaluate several checkpoints, not only the final. Retention typically degrades monotonically while task performance plateaus early, so the best checkpoint is frequently in the middle.
• Contradictory targets. Multiple annotators solved the same input differently, so the gradient points in two directions and the model averages them into a style that is neither. Diagnostic: sample twenty inputs that appear more than once with different targets. If your annotators disagree, the model cannot do better than their disagreement.

Failures specific to preference tuning

• Reward hacking. The policy finds regions where the learned reward model is wrong and exploits them — a phrase, a structure, a length that scores highly and means nothing. Diagnostic: reward rising while KL divergence from the reference rises faster is the signature. Watch both as first-class metrics, not just reward.
• Length exploitation in DPO. Preference datasets usually prefer the longer response, and direct preference methods have no counterweight, so trained models get verbose. Park et al. (2024, arXiv 2403.19159) analyse this specifically and propose a length-regularised objective. Diagnostic: plot mean output length against training step.
• Off-policy drift. DPO trains on a fixed set of pairs, so as the policy moves it stops resembling the model those pairs came from and the data stops describing its current mistakes. Diagnostic: the implicit reward margin keeps improving while human or judge evaluation does not.
• KL coefficient set wrong in either direction. Too low and the policy hacks the reward; too high and it cannot move, and you have spent four models’ worth of memory reproducing the SFT checkpoint. Diagnostic: if the KL stays near zero for the whole run, nothing is being learned.

The triage order

When a fine-tune disappoints, check in this order. It is ordered by how cheap the check is divided by how often it is the answer, and the first three account for most of it.

• Is the adapter loaded, and is it the right one? A single known-different prompt answers this in seconds.
• Do the training and serving prompts render identically? Print both with special tokens visible and diff the strings.
• Was anything actually supervised? Trainable parameter count at startup, and one decoded label sequence.
• Does your evaluation have the resolving power to detect the effect you expected? Compute the confidence interval.
• Is the training data self-consistent? Twenty duplicate inputs with different targets.
• Did the split leak? Near-duplicate check across train and test.
• Only now, hyperparameters. Epochs first, then target modules, then rank, then learning rate — one at a time, seed fixed.

The ordering is the point. Almost every disappointing fine-tune is resolved somewhere in the first three items, and almost every team starts at the last one.