When a Passing Regression Suite Still Missed a Regression

The suite was green on the commit that broke production. That is not a contradiction and it is not usually a bug in the suite. It follows from what a suite of examples can claim, and the claim is much narrower than people read it as.

What a green suite actually claims

A green run says: at these particular inputs, on this particular sample, these particular assertions held. Three qualifiers, each of which is a place a real failure fits through. The input space is infinite and you enumerated a hundred points of it. The output is stochastic and you drew once. The assertions are a proper subset of what “working” means, usually a small one.

For deterministic code the first qualifier is the only one that matters, which is why the intuition people bring from unit testing misleads them here. With a model, the second qualifier is doing more damage than the first, and almost nobody accounts for it.

Five gaps that produce a green miss

• The suite is one case forty times. Cases accumulate by copy-paste from the last one, so forty fixtures exercise the same path with different names in them. Coverage is not the case count; it is the count of distinct assertion outcomes the suite can distinguish. A quick audit: if you deleted every case but one per failure mode, which real bugs would newly get through? If the answer is “none”, the other thirty-nine were paying rent for nothing.
• The assertion is weaker than the requirement. You asserted the invoice JSON parses. The requirement was that the total equals the sum of the line items. The model returned well-formed JSON with a total that was wrong by twelve pence, the schema check passed, and the suite reported success on a case that was, by the standard anyone cares about, a failure. Schema validation is necessary and people stop there because it is easy.
• The failure is intermittent and you drew once. Covered in full below; it is the largest of the five.
• Production inputs do not look like fixtures. Fixtures are clean, short, in one language, and written by the person who wrote the prompt. Real inputs are five thousand tokens of pasted email with a signature block, a second language midway through, a null where the schema promised a string, and often the model’s own previous output fed back in. The mode that broke was never a case because nobody would have thought to write it. Replaying real traffic is the only reliable source of inputs you would not have imagined.
• The regression is outside what any assertion looks at. Latency, output token count, tool-call count, cost per request. A prompt edit that adds a paragraph of instructions can leave every content assertion green while raising average output length by forty per cent, which is a regression that arrives as an invoice.

Why sampling once is the usual culprit

Take a failure mode that occurs on some fraction of generations for a given input — the model omits a required field one time in twenty. Call that rate p. Your suite runs that case once per CI run. Whether it goes red is a single draw with probability p.

At p = 0.05, the case is green 95 per cent of the time. The suite runs on every push, so it is green on essentially every push before the release, and the mode ships. Meanwhile in production the same prompt serves ten thousand requests a day, which is five hundred failures a day. The suite is not wrong; it drew once and got the common outcome, which is what drawing once does.

The arithmetic for what it would take to see it, assuming independent draws:

P(at least one failure in n runs) = 1 - (1 - p)^n

p = 0.10, want 90% detection:
  0.9^n <= 0.10
  n >= ln(0.10) / ln(0.9) = 21.9   ->  n = 22

p = 0.05, want 90% detection:
  n >= ln(0.10) / ln(0.95) = 44.9  ->  n = 45

Twenty-two repetitions of one case to catch a one-in-ten mode with reasonable confidence. Forty-five for one in twenty. Note what that implies about a suite built the usual way: forty cases run once each also gives you forty draws, and if all forty were vulnerable to the mode it would go red 1 − 0.95⁴⁰ = 87 per cent of the time. But they are not all vulnerable — only the one case with the triggering shape is, and it is drawn once. Breadth across different inputs does not buy depth on one input.

So repeat the risky case rather than adding neighbours of it, and assert on the pass count rather than on the run:

const N = 20;
test(`[format] omits no required field across ${N} draws`, async () => {
  const results = await Promise.all(
    Array.from({ length: N }, () => runRefund(CASE.input)),
  );
  const ok = results.filter((r) => Decision.safeParse(r).success).length;
  expect(ok).toBe(N);
}, 120_000);

This is expensive, which is the honest reason it is rare. Twenty draws per case does not fit on the gating tier for more than a handful of cases. Put the repeated draws on the scheduled tier, restricted to the modes where the model is genuinely stochastic, and keep the single-draw version on the gate.

Spending the budget differently

The five gaps are not equally cheap to close, and closing them competes for the same triage budget. In rough order of return:

Strengthen assertions before adding cases. Turning “parses as JSON” into “parses, and the total equals the sum of the lines, and every identifier appears in the input” costs one afternoon and applies to every existing fixture at once. Adding forty fixtures costs forty afternoons of maintenance and closes a narrower gap.

Then get real inputs in. Sample production requests, redact them, and promote the strange ones into fixtures — the ones with unusual length, mixed scripts, embedded newlines, or empty optional fields. The distribution of what breaks a prompt is not the distribution of what you would write.

Then add non-content assertions to cases you already have: an upper bound on output tokens, an expected tool-call count, a wall-clock ceiling. These cost nothing per case because the request was already made, and they close the fifth gap entirely.

Then, last, add repetition where the mode is stochastic. It is the most expensive per unit of coverage, which is why it goes last, but for the modes it addresses nothing else works at all.

After a miss, the case is not the fix

When something escapes, the reflex is to add the exact input that broke as a new case. Do that — there is a procedure for turning the ticket into a case — but understand that it fixes exactly one point in an infinite space and will never fire again, because that is the one input you have now handled.

The part that pays is the second question: which of the five gaps let it through, and what does closing that gap look like across the whole suite. If the assertion was too weak, the fix is to strengthen that assertion everywhere, not to add a fixture. If the input shape was absent, the fix is a fixture family covering that shape, plus a note in your intake process. If it was stochastic, the fix is repetition on that mode. If it was invisible to every assertion you own, the fix is a new assertion — and quite possibly a new failure-mode directory that was empty because nobody had named the mode yet.