Evaluating a Fine-Tuned Model Against the Base

The usual fine-tuning evaluation is fifty examples, eyeballed, and a conclusion. The arithmetic below shows that such an evaluation cannot distinguish a real five-point improvement from noise — which is why it always concludes the fine-tune worked.

Four suites, not one

A fine-tune changes the whole model, so an evaluation restricted to the task it was trained on cannot tell you whether shipping it is safe. You need four separate sets, and they answer four different questions.

The fourth suite deserves particular attention because it is the one that catches the catastrophic-but-boring failures. A model that improved on your rubric and dropped from 99.4% to 96% valid JSON has made your error rate six times worse, and no quality judgement will surface that.

How many examples you need

Suppose your evaluation is a win rate: for each example, the fine-tuned output either beats the base or does not. That is a proportion, and the uncertainty on a proportion is standard:

standard error of a proportion
  SE = sqrt( p(1−p) / n )        worst case at p = 0.5

n =   50   SE = sqrt(0.25/50)   = 0.0707  →  95% CI = ±13.9 points
n =  100   SE = sqrt(0.25/100)  = 0.0500  →  95% CI = ±9.8  points
n =  400   SE = sqrt(0.25/400)  = 0.0250  →  95% CI = ±4.9  points
n = 1600   SE = sqrt(0.25/1600) = 0.0125  →  95% CI = ±2.5  points

to resolve a difference of ±d points at 95% confidence
  n ≈ 0.25 · (1.96 / d)²
     d = 0.10 (10 pts) →   n ≈   96
     d = 0.05 ( 5 pts) →   n ≈  384
     d = 0.025(2.5 pts)→   n ≈ 1537

Read the middle line. A 100-example evaluation has a confidence interval roughly ten points wide. If your fine-tune produced a genuine five-point improvement — a good outcome — a 100-example evaluation cannot distinguish it from zero, and it cannot distinguish it from a five-point regression either. The measurement is compatible with all three stories.

This is the single most useful number in this cluster, because it explains a phenomenon everyone has experienced: the fine-tune that evaluated beautifully and did nothing in production. The evaluation was not wrong. It had no resolving power, and a measurement with no resolving power returns whatever the person reading it expected.

There is a second statistical trap sitting on top of the first, and it gets worse the more diligent you are. Four suites, several metrics each, evaluated across three or four candidate checkpoints, is easily thirty or forty comparisons — and at a 5% false-positive rate you should expect one or two to look significant by chance alone. If you then pick the checkpoint that won the most comparisons, you have selected for noise. The defence is not a correction factor; it is deciding in advance which single metric on which single suite is the decision metric, and treating everything else as diagnostic information rather than evidence.

Using a model as judge

For open-ended output there is no automatic metric, so a strong model scores the pair. This works, and it has three well-known biases you must control rather than hope about.

• Position bias. Judges favour one position in an A/B presentation. The control is mandatory and cheap: run every comparison twice with the order swapped and count a win only when both orderings agree. Disagreements become ties, which is exactly what they are.
• Length bias. Judges favour longer answers independently of content. Log the length distribution of both sides. If the fine-tuned model’s outputs are systematically longer, a chunk of the win rate is length, and you should re-run the judgement with a rubric that explicitly penalises unnecessary length.
• Self-preference. Do not judge with a model from the same family as either candidate where you can avoid it, and never with the model you distilled from — it will recognise and reward its own style.

Validate the judge before trusting it. Have a human label 50 pairs, check the judge’s agreement with them, and only then run it over the full set. A judge that agrees with your annotators 65% of the time is not measuring your model, it is measuring itself.

Pair the comparison

One statistical improvement is nearly free and is worth more than doubling the sample size. Do not evaluate the two models on different examples and compare averages. Evaluate both on the same examples and compare per-example.

The variance in an unpaired comparison is dominated by example difficulty — some prompts are simply harder — and that variance is shared between the two models, so pairing removes it. What is left is the difference you care about. Concretely, count the examples where the fine-tune won and the base won, discard the ties, and test whether the split differs from 50/50; a paired bootstrap over the per-example differences gives you a confidence interval without assuming anything about the distribution.

Also hold the sampling temperature fixed and low, and use the same seed for both. Otherwise you are partly measuring sampler variance, and a fine-tune that changed nothing will still produce a non-zero win rate.

The release gate

Write the gate down before you look at any results, because a threshold chosen after seeing the numbers is not a threshold.

• Task suite improves by more than the confidence interval of the measurement — not by more than zero.
• Retention suite shows no regression on the programmatic checks, and human review of any prompt whose behaviour changed category.
• Contract checks are equal or better in absolute terms. Valid-output rate is a hard gate, not a trade-off.
• The base is still deployable behind a flag, and somebody has actually tested the rollback path. A fine-tuned model you cannot revert is a fine-tuned model you will keep past the point where you should.