Migrating a Prompt A/B Testing Framework's Metric Definitions

Your helpfulness score went up four points on the new model. Before anybody celebrates, work out how much of that is the model and how much is the ruler.

The metric is an instrument, and it has drifted

An A/B framework for prompts almost always ends in a judged score: another model reads the output against a rubric and returns a number. That number is not a measurement of the output in the way a latency figure is. It is the opinion of a second model, expressed through a rubric written by someone who was looking at a particular model’s outputs when they wrote it.

Rubrics accumulate the shape of the outputs they were calibrated on. “Does the response directly address the question” sounds model-independent until you notice it was written while reading a model that opened with a one-line summary, so a judge given that rubric rewards a leading summary and penalises a model that builds to its conclusion. “Is the response appropriately detailed” encodes whatever length the author was used to. “Does it avoid unnecessary hedging” encodes one model’s hedging register as the baseline for “unnecessary”.

None of that is a flaw in the rubric. A rubric has to be specific to be useful. It is a reason the number is not portable, and the migration is the moment that stops being an abstraction.

The length prior, which is the big one

Of all the biases in a judged metric, response length is the one that moves most on a migration and the one that explains most of the unexplained delta. Judges reward longer answers on open-ended rubrics — more of the rubric’s ground appears to be covered, and more text gives the judge more to credit. Output verbosity is also one of the most visibly different things between model families, and one of the most heavily tuned between versions of the same family.

Which means a migration can move a helpfulness score by several points in either direction with no change in the usefulness of a single answer. If the new model is more concise, you will see a quality regression that is a formatting difference. If it is more expansive, you will ship a win you did not get.

The diagnostic is cheap. Record output length alongside every judged score, on both arms, and plot score against length. If the two arms sit on the same curve and differ only in where they sit along it, the model did not get better — it got longer, and your rubric has a length prior you should now control for explicitly. This is worth doing even outside a migration, but a migration is when it stops being optional.

Separating metric drift from model drift

The procedure that separates the two is re-scoring history. It works because the old model’s outputs are a fixed quantity: whatever score they get today, their quality has not changed since you generated them.

1. Take a held-out set of the old model’s outputs that already carries scores from before the migration. A few hundred examples spanning the tasks you care about is enough; it does not need to be large, it needs to be representative.
2. Re-score them now, with the judge configuration you intend to use going forward. Same rubric, same judge model, same temperature, same everything.
3. Compare the two sets of scores on identical text. Any difference is pure instrument drift — a judge model that was silently updated, a rubric someone edited, a harness change. That difference is your offset and it applies to every comparison you make.
4. Only now score the new model’s outputs. Subtract the offset. The remainder is the part attributable to the model.
5. Repeat the re-score whenever the judge changes, and pin the judge model to an explicit version string so you find out when it does. A judge that migrates alongside the target destroys the only fixed point you have.

The last point deserves emphasis, because it is the mistake that makes a migration unanalysable. If you upgrade the judge and the target together, there is no arithmetic that recovers which one moved the number. Migrate the judge on a different week, with its own re-baseline, and never during a cutover.

The harness itself has migration bugs

Beyond the metric, the machinery around it usually contains two assumptions that a migration invalidates.

• Pinned sampling parameters that the new model rejects. A/B harnesses pin temperature to zero for comparability, and some current models no longer accept sampling parameters at all — Anthropic documents temperature, top_p and top_k as deprecated on Claude Opus 4.7 and later, returning a 400 when set to a non-default value. Anthropic’s model deprecations page lists the parameter deprecations alongside the model ones. A harness that sets it unconditionally does not produce a bad score; it produces no score, and if the harness swallows errors, an empty arm reads as a neutral result.
• Variance assumptions carried across. The sample size that gave you a significant result was chosen against the old model’s output variance. A model with different variance needs a different n for the same power. Re-estimate the variance from a pilot run before trusting any significance claim made with the old sample size.

Both of these are the reason to re-run the harness against the old model first: it exercises the machinery under the new configuration while the expected answer is still known.

Which metrics need redefining and which do not

Not everything needs re-calibrating, and treating everything as suspect is its own way to lose a week.

Metrics with a ground truth carry over unchanged. Exact match, field extraction accuracy, schema validity, whether a tool was called, whether a required fact survived a summary — these compare against something outside the model and are not affected by whose outputs the rubric was written against. Keep them exactly as they are; they are your anchor during the migration and the reason to have as many of them as possible.

Metrics with a judge in the loop all need the re-baseline above. So do any threshold you set on a continuous score, because a threshold is a calibration decision even when the metric underneath is sound. And so does any metric with a formatting component, which is a larger set than it looks: a formatting consistency score is entirely a statement about one model’s conventions. Migrating the test dataset covers the inputs; this page covers the ruler applied to the outputs, and it is worth being explicit about which of the two you are changing at any moment.