Quarantining a Flaky LLM Test Without Deleting It

The pressure to delete a flaky test arrives about ten minutes after the third unrelated pull request goes red because of it. Deleting it is a real loss: that test encoded something somebody understood about the system. Quarantine is the alternative, and it is more than a skip decorator — it is a second job, an owner and a date.

Why skipping is the wrong tool

@pytest.mark.skip and test.skip remove the test from execution entirely. That solves the merge-blocking problem and creates a worse one: the test stops producing data. You now have no idea whether it fails one run in fifty or one run in two, whether it started failing because of your prompt change or because the provider rolled a model, or whether the thing it protects broke completely three weeks ago. A skipped test is indistinguishable from a deleted one after the first sprint, which is why skip lists grow and never shrink.

Quarantine keeps the test running on every commit and keeps its result recorded. What it removes is one specific thing: the test’s ability to fail the check that gates a merge. That is a wiring decision, not a code decision, and keeping it in the CI configuration rather than in the test file is what makes it reversible.

There is a second reason to keep it running, and it is the one that justifies the extra minutes of CI time. A quarantined test is the only instrument you have pointed at the thing it covers. If the underlying feature breaks completely while the test is quarantined, the quarantine job goes from intermittently red to permanently red, and that transition is detectable — but only if the job actually executes. A skip produces the same output whether the feature works perfectly or has been deleted, which is why a skip list is not a lighter form of quarantine but a different and worse thing.

One marker, registered

Use a single marker name across the whole repository, and register it so a typo is an error rather than a silently unmatched selector. In pytest, unregistered markers raise a warning by default and an error under --strict-markers; put the registration in your config:

# pyproject.toml
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
  "quarantined(owner, since, reason): known-flaky; runs, but does not gate a merge",
]

Then mark the test, and carry the metadata in the marker rather than in a comment, because a comment cannot be queried:

import pytest

@pytest.mark.quarantined(
    owner="search-team",
    since="2026-07-14",
    reason="tool_choice=auto occasionally returns no tool call on long inputs",
)
def test_router_emits_a_tool_call(client):
    result = client.route("refund my order 4471")
    assert result.tool_calls, "expected at least one tool call"
    assert result.tool_calls[0].name == "lookup_order"

Note what the assertion is: a tool name and a call count, not a sentence. That is deliberate, and it is the subject of asserting on properties instead of exact text. A test worth quarantining should already be asserting on something structural; if it is asserting on prose, quarantine is treating a symptom.

Splitting the suite into two jobs

The whole mechanism is two invocations with complementary selectors. The first gates the merge and never sees a quarantined test. The second runs the quarantined ones, records the result, and is allowed to fail.

1. Run the gating job with the quarantined tests deselected: pytest -m "not quarantined" --junitxml=reports/gate.xml. This is the job your branch protection rule points at.
2. Run the quarantine job separately with retries and its own report: pytest -m quarantined --reruns 3 --junitxml=reports/quarantine.xml. The --reruns flag comes from the pytest-rerunfailures plugin, which also accepts --reruns-delay and --only-rerun to restrict reruns to a named exception.
3. Mark that second job non-blocking in your CI configuration rather than by swallowing the exit code in the shell. GitHub Actions has continue-on-error: true on the step; GitLab CI has allow_failure: true on the job. Piping to || true works but throws away the exit status, which you want for the dashboard.
4. Upload both JUnit XML files as build artifacts. They are the input to a flake dashboard and there is no reason to collect them later.

The same convention in Vitest

Vitest has no marker system, so the convention is a name prefix or a directory, matched by --exclude and --include. Both accept glob patterns, and both can be set in vitest.config.ts or on the command line. Recent versions also expose per-test options as an object argument — test(name, options, fn) — which is where retry lives per test rather than per project.

// quarantine by path: tests under __quarantine__ are excluded from the gate
// package.json
{
  "scripts": {
    "test:gate": "vitest run --exclude '**/__quarantine__/**'",
    "test:quarantine": "vitest run --dir src --include '**/__quarantine__/**' --reporter=junit --outputFile=reports/quarantine.xml"
  }
}

A path convention beats a name convention here because it survives renaming and because it is visible in a directory listing. Somebody reviewing the repository can see how large the quarantine is without running anything, which is exactly the pressure you want.

Keep the quarantined file importing the same fixtures and helpers as its neighbours rather than forking a copy. The most common way a quarantined test dies is that the code it calls is refactored, the quarantine job starts failing to import, and nobody notices because that job is allowed to fail. Guard against it by asserting in the quarantine job that the expected number of tests was collected: pytest exits with code 5 when it collects nothing, and treating that specific code as a hard failure costs one line and catches the silent-rot case.

Quarantine needs an exit

The failure mode of quarantine is that it becomes a graveyard with better branding. Three rules stop that, and all three are enforceable by a script rather than by good intentions:

• Every quarantined test has a named owner. Not a person — a team. People leave; the marker outlives them.
• Every quarantined test has a since date, and a job fails the build if any marker is older than your agreed window. The point of the failure is to force a decision: fix it, rewrite the assertion, or delete it deliberately with a commit message that says why.
• The quarantine has a size cap. A repository that can hold at most fifteen quarantined tests forces triage; one with no cap will find its equilibrium somewhere far worse. Rank the ones you fix by a flakiness score rather than by file order.

One thing quarantine must never absorb: a test that is failing every single run. That is not flakiness, it is a regression wearing a marker, and it is the most expensive mistake available here because the quarantine job is allowed to fail and nobody reads it. Before marking anything, run it enough times to establish that it does sometimes pass — the procedure in telling a flaky test from a real regression is the check, and it takes about two minutes.