Retrying a Flaky LLM Test the Right Number of Times

Retrying a failed test is the cheapest thing in this whole cluster to turn on and the easiest to turn up too far. The number you pick is a statement about how much genuine failure you are willing to make invisible, and the arithmetic that says so takes one line.

What a retry actually does

A retry does not reduce flakiness. It changes what the suite reports when flakiness occurs. The underlying test still fails at whatever rate it failed at before; the retry converts most of those failures into a green build and, if your runner is honest about it, a “flaky” annotation. Playwright reports exactly this category — a test that failed on the first run and passed when retried is reported as flaky rather than as passed — and the JUnit XML dialect that Maven Surefire and several other runners emit has a <flakyFailure> element for the same purpose, distinct from <failure>.

That annotation is the entire value. A retry configuration that produces a green build with no record of the retry has deleted your only signal, and you will discover the deletion months later when somebody asks how long a test has been flaky and the answer is unknowable. Before tuning the count, make sure the count of retries is being written somewhere — a minimal flake dashboard reads precisely this field.

The arithmetic, in one line

Assume a test fails independently on each attempt with probability p. Independence is an assumption, not a fact, and it is the one this whole calculation rests on; the section after next says when it breaks. Under it, the test is reported as failed only if all N attempts fail, so the observed failure rate is p^N.

observed_failure_rate = p ** N

# assumed per-attempt failure rate p = 0.05 (one run in twenty)
N = 1  ->  0.05        1 build in 20 goes red
N = 2  ->  0.0025      1 build in 400
N = 3  ->  0.000125    1 build in 8,000
N = 4  ->  0.00000625  1 build in 160,000

Independence is doing real work in that formula and it is worth checking rather than assuming. It holds when the failure is a fresh sampling draw or a transient transport error. It fails badly when the cause persists across attempts — a rate limit whose window has not reset, a provider degradation lasting minutes, a cassette that is missing and will still be missing on the retry. In those cases the second attempt is nearly certain to fail too, the observed rate is close to p rather than p^N, and the retries buy you nothing but a longer build and a larger bill.

The interesting thing is how fast the returns stop. Going from one attempt to two removes 95% of the spurious red builds. Going from three to four removes a further 0.0001 percentage points, which for any suite that runs fewer than several thousand times a day is a change you will never observe. Almost every argument about whether the retry count should be three or five is an argument about a difference nobody in the room will ever see.

The same arithmetic hides a regression

Now run it with a different p. Suppose your prompt change broke something and the test now genuinely fails half the time — p = 0.5, a serious defect that would be caught instantly by a human running the thing twice.

# assumed per-attempt failure rate p = 0.5 (a real, severe regression)
N = 1  ->  0.5     every other build red
N = 2  ->  0.25    1 build in 4
N = 3  ->  0.125   1 build in 8
N = 5  ->  0.031   1 build in 32

With three attempts, a defect that halves your success rate shows up in roughly one build in eight. On a busy repository that reads as intermittent noise, gets attributed to the provider, and survives review. With five attempts it is one in thirty-two, which is indistinguishable from the background rate of genuinely flaky infrastructure. The retry count did not just hide noise; it hid the thing the test was written to catch.

This is the real cost function. Retries do not distinguish between causes of failure, so any count large enough to make a 5% flake invisible is already large enough to make a 50% regression look intermittent. There is no setting that suppresses one and not the other, and the fix is not a better number — it is to stop using the retry as the measurement and start counting flips, which is what running N times and requiring K passes does instead.

Retries are not free here

In a browser suite a retry costs seconds. In a suite that calls a model, every retry is another billed request, and the assertion failure that triggered it usually happened after the full completion was generated — so you pay for output tokens on every attempt.

Work it symbolically with your own numbers. If a suite makes R model requests per run, a fraction f of them fail and are retried up to N times, and the average request costs c currency units end to end, then the expected cost per suite run is approximately R * c * (1 + f * (N - 1)) in the worst case where retried requests always exhaust their attempts, and lower in practice. The multiplier only bites when f is large — which is exactly the situation where you should be fixing the test rather than paying to rerun it. The general treatment of retry economics is in the cost of retries.

Choosing the number

• One retry, for most suites. It removes the large majority of spurious reds at a 5% assumed flake rate and leaves a genuine 50% regression failing a quarter of the time, which is visible.
• Restrict what is retried. pytest-rerunfailures accepts --only-rerun with a regex or exception class, so you can retry a transport error or a 429 and never retry an AssertionError. That distinction is worth more than any change to the count: an assertion failure is your code disagreeing with itself, and it deserves zero retries.
• Retry the request, not the test, where you can. A 429 or a 503 belongs in the client’s retry policy with backoff, not in the test runner. If the runner is retrying transport errors, your production client is probably not handling them either.
• Never retry a non-idempotent test. If the test writes a row, sends a message or spends from a quota, the second attempt runs against a world the first attempt changed, and the independence assumption that this entire page rests on is gone.