Retrying a Flaky Eval Job in CI Without Hiding a Real Failure

“Just hit re-run” is a reasonable response to a flaky unit test and a bad one for an eval suite, because the two things are failing for different reasons and only one of them is fixed by trying again.

Three things called flaky

The word covers three distinct failures with three different correct responses, and treating them as one is how a retry policy ends up hiding regressions.

• Transport failure. A 429, a 503, a connection reset, a read timeout. The request never produced a result. Retrying is not merely acceptable here, it is the correct semantics — and it belongs in the HTTP client with backoff, not in the test runner and certainly not in the CI job.
• Sampling variance. The request succeeded and the model returned something different. A case that passes 70% of the time is not flaky infrastructure; it is a case whose assertion is too tight for the sampling settings, or a genuinely borderline behaviour. Retrying converts a measurement into a search for a result you like.
• A real regression that happens to be intermittent. A prompt change that drops a case from 99% to 60% shows up as flakiness first. This is the one a retry policy is most likely to bury, because it looks exactly like the second category from inside a single run.

The practical consequence is that a retry policy has to be conditional on the error, not on the fact of failure. Both major runners support this now. In pytest, pytest-rerunfailures takes --only-rerun with a regular expression, repeatable to accumulate patterns, alongside --reruns, --reruns-delay and --reruns-delay-backoff-factor. In Vitest, retry accepts an object of { count, delay, condition } where condition is a regular expression matched against the error, which arrived in 4.1.0; before that it was a bare number and could not discriminate.

# Retry only what the network broke.
pytest eval/ \
  --reruns 2 --reruns-delay 1 --reruns-delay-backoff-factor 2 \
  --only-rerun 'APIConnectionError' \
  --only-rerun 'RateLimitError' \
  --only-rerun 'APITimeoutError'

// vitest.config.ts
export default defineConfig({
  test: {
    retry: {
      count: 2,
      delay: 1000,
      condition: /ECONNRESET|ETIMEDOUT|rate_limit|overloaded/i,
    },
  },
});

An unconditional --reruns 3 is the policy that hides regressions, and it does so in a specific way: a case that has dropped to a 60% pass rate passes at least once in three attempts about 94% of the time. The regression is real, it is large, and your gate will almost never see it.

Why rerunning the suite is different

Rerunning a single case re-samples that case. Rerunning the whole job re-samples every case at once, and when the gate is a threshold over an aggregate that changes what a green result means.

Consider a suite of 200 cases gated at “at least 92% must pass”. Suppose the current true per-case success rate is 93%. The expected count is 186 passes against a bar of 184, so the gate is nominally satisfied — but the observed count varies from run to run, and a meaningful fraction of runs will land at 183 or below purely by sampling. Under that regime, “re-run until green” does not remove noise; it selects the favourable tail of it. Every rerun makes the reported pass rate a little more optimistic than the system it measures.

A deterministic test suite does not behave like this. Rerunning it either produces the same answer or reveals a genuinely non-deterministic test, which is itself a bug worth fixing. That asymmetry is the reason a retry policy copied from a unit-test pipeline misbehaves here.

Two responses are honest. Either set the threshold with the run-to-run spread in mind, so the bar sits below the range you are willing to accept rather than at the mean, or make the metric itself less noisy — gate on things that are close to deterministic (schema validity, which tool was selected, whether a refusal occurred, whether a redaction held) and keep the judged-quality score as a tracked trend rather than a blocking check. The second is usually better and is the same argument eval blind spots makes from the other direction.

Bounding retries at the case

Where retrying is legitimate, bound it in three ways at once.

• By count. Two is almost always enough. If two retries do not clear a transport error, the provider is having an incident and you want to know that rather than to wait it out.
• By error class, as above. This is the bound that does the actual work.
• By total retry budget for the run. Per-case limits compose badly: 200 cases at two retries each is a worst case of 600 requests, which is a spend problem and a rate-limit problem before it is a correctness problem. Count retries across the run and fail the job when the total crosses a threshold, even if every case eventually passed.

That last one is the rule that keeps a retry policy honest. A run that needed 140 retries to go green is not a green run; it is an incident report that happened to exit zero.

One thing to keep out of the retry path: any request that has side effects. If a case exercises a tool that writes somewhere, retrying it duplicates the write unless the call carries an idempotency key. This is why retries belong close to the request, where you can tell a read-only completion from one that triggered a tool.

Retries at the job level

GitHub Actions has no retry: key on a job. The available mechanisms are re-running a workflow from the interface or with gh run rerun <run-id> --failed, which re-runs only the failed jobs of a run, and step-level retry loops you write yourself.

A whole-job retry is the right tool for exactly one class of problem: the runner itself failed. A lost network route, a runner reclaimed mid-run, a package registry that timed out during install. These produce failures before your suite starts producing results, and they are distinguishable — the job has no test report at all. If you want to automate anything, automate that distinction: a step that retries the setup portion, and no automatic retry of the eval step.

Resist the pattern of a workflow that re-dispatches itself on failure. It removes the signal from exactly the place a human would have looked and, because each attempt is a fresh sample, it is the suite-level version of the selection problem above, running unattended.

Making retries visible

A retry that leaves no trace is indistinguishable from a pass, which is the whole problem. Emit the numbers into the run summary and into the history you keep across runs:

1. Count retries by error class during the run and write the totals to $GITHUB_STEP_SUMMARY, so they appear on the run page without anyone opening a log.
2. Record retry count alongside the score in your per-commit history, so a slow rise in retries is visible as a trend rather than discovered during an outage. See storing eval history.
3. Report per-case pass rates over the last N runs rather than per-case pass or fail. A case at 0.7 is the thing you want to find, and it is invisible in any single run’s output.

The last one is the highest-value change on this page. Most flaky-eval pain comes from treating a probabilistic case as a boolean; once the suite records rates, the difference between transport flake, a borderline assertion and a regression usually becomes obvious without any retry policy at all.