Skip to content
Multigrid

Flaky tests against a model

Telling a flaky test from a real regression, and what to do with the ones you cannot make deterministic.

A test suite that calls a model is flaky by construction. Sampling is random unless you turn it off, and turning it off does not make the provider’s serving stack, its routing, its rate limiter or its model version hold still. So the usual instinct — find the source of nondeterminism and remove it — runs out early here, and what is left is a suite that sometimes goes red for no reason anyone changed.

These pages are about the part after that. How to write an assertion that does not depend on the sampler, how to tell one bad run from a real regression without arguing about it in a pull request, how to price the retries that hide the difference, and what to do with the tests you genuinely cannot make deterministic. Where a page needs a flake rate, it defines the statistic and gives you the query that computes it from your own CI history, because yours is the only one that means anything.

Quarantining a Flaky LLM Test Without Deleting It

A marker convention that keeps a known-flaky model test running and visible while removing its power to block a merge.

9 min read

Retrying a Flaky LLM Test the Right Number of Times

What a retry count actually buys, and the arithmetic showing how quickly it starts hiding a genuine regression rather than noise.

9 min read

Telling a Flaky Test Apart From a Real Regression

A short decision procedure for the judgment call teams make under pressure, using reruns on two commits and evidence from outside your repository.

9 min read

Setting an Acceptable Flake Rate for a Prompt Test Suite

Why a published per-test tolerance is worth more than an argument, and the suite-level arithmetic that shows why the number has to be small.

9 min read

Why the Same Prompt Test Passes Locally and Fails in CI

Six specific causes of a prompt test that is green on your machine and red in the pipeline, and the check that tells you which one you have.

10 min read

Deduplicating Flaky Failures Across a Large Prompt Test Suite

Grouping failures by a normalised root-cause signature instead of by test name, so one provider incident stops looking like forty unrelated bugs.

10 min read

A Flakiness Score for Ranking Which Prompt Tests to Fix First

A derived score that ranks a backlog of unreliable tests by how much trouble each one actually causes, and survives small sample counts.

9 min read

Reducing Flakiness by Testing Properties Instead of Exact Text

The families of assertion that hold across every valid sampling of a model, and what you give up by moving to them.

10 min read

Running a Flaky Test N Times and Requiring K Passes

The majority-vote pattern for a nondeterministic assertion, with the binomial arithmetic for choosing N and K and an implementation in two runners.

10 min read

Isolating Flakiness Caused by the Provider From Flakiness Caused by Your Code

A two-replay diagnostic at the HTTP boundary that tells you which side of the API to debug, and the third category both replays miss.

10 min read

A Dashboard for Tracking Flaky Prompt Tests Over Time

The one table and three queries that turn CI artifacts you already produce into a usable view of which tests are unstable.

9 min read

Bisecting a Prompt Change That Introduced Flakiness

How to run git bisect against a stochastic test without converging on the wrong commit, including the probe arithmetic and the exit codes that matter.

10 min read

Other topics