Running the Same Regression Suite Against Two Model Versions

A regression suite is the cheapest migration gate you will ever build, because it already encodes every behaviour you decided you cared about. It answers one question well — does the candidate still hold every contract the incumbent holds — and it cannot answer the question people ask it, which is whether the new model is better.

A floor, not a ranking

Be precise about the claim before running anything. Your suite is a set of assertions chosen because they broke once, or because somebody thought they might. It is not a sample of your traffic and it is not balanced. Aggregate pass rate over it is therefore not a quality measure, and “candidate scored 94 per cent, incumbent scored 91 per cent” is a sentence with no defensible meaning.

What the suite does answer is a floor question: is there any contract the incumbent satisfies that the candidate does not. That is a well-posed question, it is decidable from the data you are about to collect, and it is exactly what a migration gate needs. Ranking is a different exercise on a different dataset — a golden set sampled to resemble production, scored against a rubric — and conflating the two is how a migration gets blessed by a number nobody can defend.

Making the model an input

The suite has to take the model ID from one place, read once, and printed in the run header. If any test hardcodes a model — and in most suites at least one does, usually a case somebody debugged against a cheaper model and never reverted — the comparison silently covers less than it claims.

// tests/regression/model.ts
export const MODEL = process.env.REGRESSION_MODEL;
if (!MODEL) throw new Error("REGRESSION_MODEL is required");

// per-model capability differences, declared rather than discovered
export const CAPS: Record<string, { maxOutput: number; strictTools: boolean }> = {
  "incumbent-model-id": { maxOutput: 4096,  strictTools: false },
  "candidate-model-id": { maxOutput: 16384, strictTools: true  },
};

The second half matters as much as the first. Models differ in maximum output tokens, in whether they accept a given sampling parameter, in how strictly they validate a tool schema, and in what reasoning or effort settings they expose. A candidate that fails forty cases because you sent it a parameter it does not support has told you nothing about quality, and that run will cost you an afternoon of misdirected investigation. Declare the differences up front so the harness adapts and the assertions stay about behaviour.

Run paired, in one job

Run both versions over the same inputs, in the same job, interleaved or back to back within the same window. The reason is confounding: provider latency and capacity vary by hour and by day, and so occasionally does behaviour under load. An incumbent run from Tuesday morning compared against a candidate run from Friday night differs in more than the model, and you will not be able to tell which difference produced the result.

Then handle the stochasticity, which is the part that separates a useful comparison from a coin flip. Each case under each model is a Bernoulli draw. A single paired run on a hundred cases where the candidate fails three the incumbent passed is entirely consistent with the two models being identical and the sampler being the sampler.

So repeat, on the modes where the model is stochastic, and compare pass counts rather than pass or fail. Where a provider offers a determinism aid, use it and treat it as an aid rather than a guarantee: OpenAI documents seed as a beta feature where the system “will make a best effort to sample deterministically” for repeated requests with the same seed and parameters, alongside system_fingerprint to detect backend changes that affect it. Best effort is not reproducibility, and no seed makes two different model versions comparable in one draw.

The right statistical frame for the paired result is the set of cases where the two disagree. McNemar’s test is the standard tool for paired binary outcomes: it ignores the cases where both models agree and asks whether the discordant pairs are lopsided enough to be unlikely under the hypothesis that the two are equivalent. You do not need to run the test to benefit from the framing — look at the discordant cases, not at the two totals.

Four buckets, and which one is the gate

Every case lands in one of four cells, and each carries a different action.

• Both pass. Says nothing about either model, which is most of the suite and is fine. This is the mass the disagreement analysis is designed to ignore.
• Both fail. A pre-existing failure or a bug in the harness. If you did not know these were failing, your suite was red and somebody had stopped looking.
• Incumbent passes, candidate fails. The gate. On the deterministic tier — schema, tool name, redaction — this set must be empty, and any member is a blocker until it is either fixed prompt-side or consciously accepted with a written reason. On the stochastic modes, compare counts across repetitions before calling it.
• Candidate passes, incumbent fails. The most informative bucket and the one people skim. These cases were failing before the migration and you were shipping anyway — often marked expected-fail, skipped, or quietly red for months. The candidate did not fix your product; it revealed how much of your suite was already not green.

Collect the non-content deltas in the same run, because they are free once the requests have been made and they are where migrations surprise people. Output tokens per case, latency per case, and cost per case at each model’s published rate. A candidate that passes every assertion while producing forty per cent more output tokens has passed the gate and changed your bill, and that is a decision for somebody to make deliberately rather than discover next month.

From a green comparison to a cutover

A green comparison is necessary and not sufficient. Your suite covers the inputs you thought of, and the migration will meet the ones you did not — which is the general problem of a passing suite missing a regression, concentrated into a single day.

So treat the gate as permission to start a canary, not as permission to switch. Route a small fraction of live traffic to the candidate, compare the same non-content metrics on real inputs, and keep the incumbent ID pinned and reachable so a rollback is a configuration change rather than a deployment. Canary releasing a model covers that half.

Keep the paired job afterwards, on a schedule, rather than deleting it when the migration lands. It is the same machinery you need to notice a provider changing a model underneath you, and a comparison harness that exists is worth considerably more than a spreadsheet somebody produced once and closed.