Regression Testing a Fine-Tuned Model Against Its Base Model's Test Suite

Fine-tune evaluation almost always measures the new behaviour: does the model now produce your house format, your classification labels, your tone. That answers half the question. The other half is whether everything the base model could already do still works, and you already have a suite for that.

The base suite is a floor, not a benchmark

Your existing regression suite encodes what production depends on today: the JSON shapes, the tool selections, the refusals, the length budgets, the things that broke once and got a test. None of that was written with a fine-tune in mind, which is exactly why it is the right instrument. It is a description of the contract your application has with whatever model is behind it, and swapping the model is precisely the event that contract exists for.

Framing matters here. This is not a comparison of two models to pick a winner. The fine-tune is going to ship; the question is whether it costs you anything you were relying on. So the suite is a floor — a set of cases that must still pass — and the new target-behaviour eval is a separate ceiling test that lives beside it. Keeping them separate stops a strong result on the new task from paying for a regression on the old one.

Gate on the set difference, not the average

The mistake that makes this whole exercise useless is gating on an aggregate. “Base scored 0.91, fine-tune scored 0.92, ship it” is compatible with thirty cases regressing and thirty-one new ones passing. Aggregates over a heterogeneous suite are not comparable in the way people assume, because the cases are not interchangeable: the thirty that broke may all be the payment path.

Record per-case outcomes and compute two sets. Regressions are cases that passed on base and fail on the fine-tune. Recoveries are the reverse. Gate on the first being empty, or on an explicitly approved allowlist, and report the second as information.

# tests/finetune/floor_test.py
import json
import pytest

BASELINE = json.load(open("baselines/base-model.json"))   # {case_id: "pass"|"fail"}
ALLOWED  = set(json.load(open("baselines/allowed-regressions.json")))

def test_no_new_failures(finetune_results):
    regressions = {
        case_id
        for case_id, outcome in finetune_results.items()
        if BASELINE.get(case_id) == "pass" and outcome == "fail"
    } - ALLOWED

    assert not regressions, (
        "cases that passed on the base model and fail on the fine-tune:\n"
        + "\n".join(sorted(regressions))
    )

The allowlist file is the part that makes this survivable. There will be a case that legitimately changes — the fine-tune was specifically taught to answer differently — and without an escape hatch the gate gets disabled wholesale the first time that happens. An allowlist entry is a line in a reviewed file with a case id, so the exception is visible in a diff and can be counted. A disabled gate is neither.

Making the comparison fair

Several things must be held identical or the set difference measures your harness rather than the model.

• Same inputs, same assertions, same order. Obvious, and violated whenever the suite regenerates inputs from a template with a random seed. Freeze the case file.
• Same sampling configuration. Temperature, top-p, max tokens, stop sequences, seed if the provider offers one. A fine-tune evaluated at temperature 0 against a base recorded at 0.7 will show a set difference in both directions that is entirely sampling.
• Same number of samples per case. If the base baseline recorded one sample per case and the fine-tune run takes best-of-three, the fine-tune wins for free.
• Not necessarily the same prompt. This is the one exception and it is deliberate. Fine-tuning frequently exists to move instructions out of the prompt, so the fair comparison is each model with the prompt it is intended to ship with, against the same inputs and the same assertions. Holding the prompt constant instead measures the fine-tune’s ability to follow instructions it was trained not to need.

Record the whole configuration alongside the results. A baseline without its sampling parameters is not a baseline, because six weeks later nobody can reproduce it and the gate becomes advisory.

The baseline is an artifact, not a rerun

Do not compute the base result live in the same job. It doubles the cost of every run, and it makes the gate non-deterministic on both sides — a flaky base result manufactures phantom regressions and hides real ones. Run the base suite once, commit the per-case outcomes as a JSON artifact, and record next to it the model id, the date, the sampling config and the suite’s own commit hash.

Refresh that artifact deliberately: when the suite changes, when the base model version changes, and on a schedule if the provider updates models in place. Do not refresh it because the gate went red. Silently rebaselining on a failure is the same act as deleting the test, and it is much easier to do by accident — a make target called update-baseline gets run by whoever is unblocking the build at six o’clock. Make it require a flag and put the artifact behind code review.

Wiring it into the release

1. Freeze the case file and the sampling config; commit both.
2. Run the suite against the base model and commit baselines/base-model.json with per-case pass or fail, plus the configuration used.
3. After each fine-tuning run, execute the identical suite against the new model id and write the same shape of result.
4. Compute regressions and recoveries. Fail the job on any regression not in the allowlist, and print both sets.
5. Run the new-behaviour eval separately, and require both to pass. Two gates, two reasons to fail, two owners.
6. Add a general-capability set that was never in the fine-tuning data — the subject of testing for catastrophic forgetting — because your production suite covers your product and a fine-tune can degrade well outside it.

One reporting note. Print the regression set as case ids and one-line descriptions, not as a count. “Four regressions” gets argued about; four case names, one of which is refund-policy-must-not-quote-amount, ends the argument.

One structural caution about the suite itself. Because the base model’s regression suite was written against the base model, some of its cases will encode behaviour that was never a requirement — an assertion on a specific phrasing that happened to be what the model produced when the test was written. Those cases will fail against any new model, fine-tune or not, and they will consume the first day of every migration. Use the first run of this gate to find them and fix the assertions rather than allowlisting the cases: a test that only passes against one model is not a floor, it is a snapshot, and it will block the next model swap as well. The distinction to apply is whether a human reading the failure would call the new output wrong, or merely different.