Regression Testing for Prompts and Models

A CI gate on a language model application has to answer a question ordinary CI never faces: the same input produced a different output, and the score moved by a point and a half. Is that a regression, or is that Tuesday?

Why the usual gate does not work

The reflex is assert pass_rate >= 0.85. It fails in both directions. Set the threshold near the current level and the build breaks on noise until people start re-running it until it passes, which is the same as having no gate. Set it low enough to be quiet and it will not catch a real four-point regression.

The second reflex is to remove the noise by setting temperature to zero. This helps and it is not determinism. Greedy decoding fixes the sampler, but the logits themselves are not bit-identical run to run on production inference stacks: floating-point reductions depend on how work is split across the hardware, and the split depends on batch composition, which depends on who else is sending traffic at that moment. On a mixture-of-experts model, batch composition can also affect expert routing under capacity limits. The practical consequence: at temperature zero most outputs are stable and a small fraction still differ, usually at exactly the token where two candidates were nearly tied — which is where the meaning changes.

So the gate has to be built for a system that has a distribution of outcomes rather than an outcome. That means two tiers with two different rules, not one threshold.

Two tiers, two different rules

Keeping these separate is what makes the gate legible. When tier 1 fails, somebody broke something and the diff says what. When tier 2 fails, a distribution moved and the conversation is about evidence. Merging them into one score means every failure needs an investigation to find out which kind it was.

Tier 1: contracts

Contract assertions are the part of an LLM test suite that behaves like normal software, and most teams have far fewer of them than they could. Everything checkable belongs here:

• Structural. Output parses as JSON; validates against the schema; every tool call names a tool that exists with arguments of the right types; required fields are present and non-empty.
• Referential. Every citation id appears in the retrieved set. Every product code mentioned exists in the catalogue. Every URL is on an allowed domain. These catch the failure class users describe as “it made something up”, and they catch it deterministically.
• Prohibitions. No system-prompt text in the output. No email address or card-shaped number in a context that should not contain one. No promise of a refund amount from a system that is not allowed to make one.
• Budget. Token count under the cap, tool-call count under the loop limit, latency under the p95 budget on the CI provider. A prompt change that silently doubles output length is a regression even when quality is flat.

Run contracts with k samples per item — three is usually enough — and require all k to pass. A contract that passes two times out of three is not passing; it is a violation with a 33% incidence rate, and that is precisely the kind of thing single-sample testing hides.

Tier 2: a non-inferiority gate

For quality, the right question is not “is the new score above a line”. It is “can I rule out that the new system is worse than the baseline by more than I am willing to accept?” That is a non-inferiority test, and the gate is on the lower bound of a confidence interval on the paired difference.

Pick a margin δ — the amount of quality you would knowingly trade for whatever the change buys you. Two percentage points is a common choice for a cost-motivated model swap; zero is right for a refactor that should change nothing. Then:

import json, random, sys

def paired_bootstrap_ci(base, cand, iters=10000, alpha=0.05, seed=0):
    """base, cand: per-item scores in [0,1], SAME items in the SAME order.
    Resamples items (not calls) so the CI reflects item-level uncertainty."""
    assert len(base) == len(cand) and base
    rng = random.Random(seed)
    n = len(base)
    diffs = []
    idx = range(n)
    for _ in range(iters):
        pick = [rng.randrange(n) for _ in idx]
        diffs.append(sum(cand[i] - base[i] for i in pick) / n)
    diffs.sort()
    lo = diffs[int(alpha / 2 * iters)]
    hi = diffs[int((1 - alpha / 2) * iters) - 1]
    return sum(c - b for b, c in zip(base, cand)) / n, lo, hi


def gate(base, cand, margin=0.02):
    point, lo, hi = paired_bootstrap_ci(base, cand)
    verdict = "PASS" if lo > -margin else "FAIL"
    print(f"delta {point:+.4f}  95% CI [{lo:+.4f}, {hi:+.4f}]  "
          f"margin -{margin:.3f}  -> {verdict}")
    if lo > 0:
        print("  note: CI excludes zero -- this is an improvement, not just "
              "non-inferiority")
    if hi - lo > 4 * margin:
        print("  note: interval is wide relative to the margin; this run "
              "cannot resolve a difference of the size you care about")
    return verdict == "PASS"


if __name__ == "__main__":
    run = json.load(open(sys.argv[1]))
    ok = gate(run["baseline"], run["candidate"], margin=float(sys.argv[2]))
    sys.exit(0 if ok else 1)

Three properties make this gate behave sensibly where a threshold does not. It is paired: both systems are scored on the same items, so item difficulty cancels and the test has far more power than comparing two independent pass rates. It is conservative in the right direction: a noisy run produces a wide interval whose lower bound is below the margin, so an underpowered run fails rather than passing by accident — which is the correct default, because “we could not tell” should not ship. And it distinguishes the two good outcomes: non-inferior (lower bound above the margin) from genuinely better (lower bound above zero).

The third printed note is the one that saves programmes. If the interval is four times wider than the margin, the honest reading is that the eval set is too small for this decision, and the sample-size arithmetic tells you how many items it would take.

Making it cheap enough for every PR

A four-hundred-item eval at three samples each is 1,200 calls per run, which nobody will tolerate on every push. Four measures make it tractable.

• Cache on a content hash. Key on sha256(model_id + params + rendered_prompt + sample_index) and store the completion. A pull request that touches one prompt template re-runs only the items that template renders into; the rest are cache hits at zero cost. This alone usually takes a full run down to a few percent of its calls.
• Two stages. A 60-item smoke tier on every push, stratified to cover every failure class, plus the full set on merge to main and before release. The smoke tier is a tripwire, not a measurement — gate it on contracts only, and leave the statistical gate to the full run where the sample size supports it.
• Pin exact model versions. A floating alias that resolves to whatever is current means your baseline changes underneath you and you will attribute the shift to your own diff. Pin the dated version string, and treat bumping it as its own pull request that runs the full gate.
• Persist per-item scores, not just the total. The bootstrap needs the vector, the debugging needs the vector, and the total is recoverable from it. Storing only the aggregate is the most common irreversible mistake in eval infrastructure.

Flakes, quarantine and honesty

Some items will be genuinely borderline: the model gets them right about half the time and no amount of prompt work changes that. The temptation is to delete them. The better move is to quarantine them — excluded from the gate, still run, still reported — with a required expiry date and an owner. A quarantine list with no expiry becomes the place failures go to be forgotten.

And name the flake rate explicitly. Track, per item, the proportion of the k samples that passed, and report the count of items whose rate is strictly between 0 and 1. That number is a health metric in its own right: a system whose items are mostly 0.0 or 1.0 is a system with a clear decision boundary, and one with a large middle band is telling you the prompt is underspecified — which is a finding a pass rate alone will never surface.