Running a Flaky Test N Times and Requiring K Passes

Some assertions are irreducibly probabilistic: the model gets it right most of the time and you want the test to fail when “most” drops. A retry cannot express that. Running the case N times and requiring K passes can, and the choice of N and K is arithmetic rather than taste.

Retry measures nothing; repetition measures

A retry is a stopping rule: run until it passes or you run out. That makes the observed outcome a function of the retry budget rather than of the underlying pass rate, and it means a retry configuration cannot tell you a 95%-correct prompt from a 55%-correct one — both go green. The arithmetic of exactly how much that hides is in retrying a flaky test the right number of times.

N-of-K is a different thing. Every one of the N attempts runs regardless of outcome, the pass count is the measurement, and K is a threshold on that measurement. The test is now a hypothesis test on the per-attempt pass rate, and you can compute exactly what it will and will not detect.

The binomial arithmetic

Assume each attempt passes independently with probability q. Then the number of passes in N attempts is binomial, and the test goes green with probability:

P(pass) = sum over i = K..N of  C(N, i) * q**i * (1 - q)**(N - i)

from math import comb

def p_green(q: float, n: int, k: int) -> float:
    return sum(comb(n, i) * q**i * (1 - q)**(n - i) for i in range(k, n + 1))

Two numbers matter and they pull in opposite directions. The false-red rate is 1 - p_green(q_healthy, N, K): how often a perfectly healthy prompt fails the suite, which is your contribution to the flake budget in setting an acceptable flake rate. The false-green rate is p_green(q_regressed, N, K): how often a genuinely degraded prompt still passes. Raising K cuts the second and inflates the first. You cannot minimise both without raising N.

Choosing N and K by discrimination

Pick the two rates you are discriminating between before you touch N and K. Both are assumptions and both must be stated: say a healthy prompt passes 90% of attempts, and a regression you want to catch drops that to 60%. Then compute the grid.

q_healthy   = 0.90     # assumed, not measured
q_regressed = 0.60     # the degradation you want to catch

N  K   P(green | 0.90)   P(green | 0.60)   false-red   false-green
1  1     0.9000            0.6000            0.100       0.600
3  2     0.9720            0.6480            0.028       0.648
3  3     0.7290            0.2160            0.271       0.216
5  4     0.9185            0.3370            0.082       0.337
5  5     0.5905            0.0778            0.409       0.078
7  6     0.8503            0.1586            0.150       0.159
9  7     0.9470            0.2318            0.053       0.232
9  8     0.7748            0.0705            0.225       0.070

Read N=3, K=2 first, because it is the one people reach for. Its false-red rate is a tolerable 2.8%, and its false-green rate is 64.8% — a prompt that has degraded from 90% to 60% still sails through nearly two runs in three. Three attempts and a majority vote is essentially a retry with extra cost. N=5, K=4 is the first row where both columns are defensible: 8.2% false red, 33.7% false green. Even there, a single run misses the regression a third of the time, which is the honest and slightly uncomfortable finding of this whole calculation.

The general shape: distinguishing two rates that are 30 percentage points apart, with both error rates under 10%, needs somewhere around nine to fifteen attempts. If the gap you care about is 10 points rather than 30, the required N runs into the hundreds, and at that point you do not have a CI test — you have an evaluation, and it belongs on a schedule with a golden dataset rather than on every push.

One structural improvement is free: spend the N across N different inputs rather than N repeats of one input. The arithmetic is identical if the per-case pass probability is the same, and you get coverage of the input space for the same number of calls. Repeating one input is only right when the specific input is what you are protecting.

Implementing it

1. Write the single-attempt check as a plain function returning a boolean, with no assertion inside it. The assertion belongs at the aggregate level; an assertion inside the loop would abort the run and destroy the count.
2. Run it N times, collecting outcomes rather than stopping at the first failure.
3. Assert on the count, and put both the count and the failing cases in the failure message, so a red build tells you 3/5 rather than “assertion failed”.
4. Run the attempts concurrently if your rate limit allows, and bound that concurrency explicitly — an N-of-K test is exactly the shape that trips a per-minute limit.

# pytest
N, K = 5, 4

def attempt_passes(client, case) -> bool:
    try:
        parsed = client.extract(case.transcript)
        return parsed.order_id == case.expected_order_id
    except (ValidationError, ValueError):
        return False

def test_extraction_meets_threshold(client, cases):
    outcomes = [attempt_passes(client, c) for c in cases[:N]]
    passed = sum(outcomes)
    assert passed >= K, (
        f"{passed}/{N} attempts passed, threshold is {K}/{N}. "
        f"failing cases: {[c.id for c, ok in zip(cases, outcomes) if not ok]}"
    )

// vitest
import { test, expect } from "vitest";

const N = 5;
const K = 4;

test("extraction meets threshold", async () => {
  const outcomes = await Promise.all(
    cases.slice(0, N).map(async (c) => {
      try {
        const parsed = await extract(c.transcript);
        return parsed.orderId === c.expectedOrderId;
      } catch {
        return false;
      }
    }),
  );
  const passed = outcomes.filter(Boolean).length;
  expect(passed, `${passed}/${N} passed, threshold ${K}/${N}`).toBeGreaterThanOrEqual(K);
}, 120_000);

The failure message carries the count and the identity of the failing cases, and that is not a nicety. An N-of-K test that reports only “assertion failed” forces whoever picks it up to re-run the whole thing to learn what happened, at N times the cost, and they will do that before reading the code. Put the ratio in the message and the failing case ids beside it, and most triage becomes reading one line of CI output.

Set the timeout explicitly, as the last argument above. N sequential model calls will exceed the default per-test timeout of most runners, and a timeout failure looks exactly like a threshold failure in the report while meaning something completely different.

What it costs and when not to

The cost multiplier is N, applied to both money and wall-clock time, and it lands on every commit. Compute it before adopting the pattern: N attempts times your per-request cost times the number of runs per week is a number you should be willing to say out loud. If it is uncomfortable, the honest response is fewer such tests rather than a smaller N, because a small N is a test that does not discriminate.

• Do not use it to rescue an exact-text assertion. If the flakiness comes from wording, N-of-K is paying five times over to average out a question you should have stopped asking. Rewrite the assertion as a property first, and see whether N can go back to 1.
• Do not use it on a non-idempotent test. N attempts against a system that retains state are not N independent draws, and every number on this page assumed independence.
• Record the pass count, not just the verdict. The count is a time series — a test drifting from 5/5 to 4/5 to 3/5 over a month is the earliest signal of prompt decay you will get, and it is invisible if you store only green or red. A flake dashboard should carry this column.