When to Stop Dual-Running and Cut Over Fully

“We will dual-run for a month” is a duration. Durations end when someone gets bored or the invoice arrives, which means the decision is made by fatigue. The alternative is a condition, written down before the run starts, in terms the run itself produces.

A condition, not a duration

A duration answers the wrong question. The question is not how long you have watched but how much you have seen, and those come apart badly at low traffic and at low split percentages. A month at 1% of traffic on a quiet service can be a few hundred requests through the new provider — less evidence than an afternoon at 50%. Two teams can run for the same month and be in completely different epistemic positions.

So state the exit in samples and rates. The rest of this page derives how many samples, from your own numbers, using two standard results that are worth knowing because they come up constantly and both are short enough to derive rather than look up.

What zero failures actually proves

The happy case: you have shadowed or split some traffic and seen no failures at all on the new side. How good is that evidence? If the true failure rate is p and samples are independent, the probability of seeing zero failures in n samples is (1 − p)^n. Set that equal to 0.05 — the point at which a rate would have been unlucky to hide from you — and solve:

(1 − p)^n = 0.05
n · ln(1 − p) = ln 0.05 ≈ −3.0
for small p,  ln(1 − p) ≈ −p
so            n · p ≈ 3
              p ≈ 3 / n

This is the rule of three, and it converts a comfortable feeling into a bound. Zero failures in 100 requests is consistent with a true failure rate of up to about 3%. Zero in 300 is consistent with up to 1%. To be confident the rate is below one in a thousand you need about 3,000 clean samples, and below one in ten thousand about 30,000.

Say those numbers out loud next to your traffic. “We saw no problems in 200 requests” is compatible with one and a half per cent of your users hitting a failure, which for most products is a very bad week. The rule of three does not tell you to run longer; it tells you what the run you did is worth, which is usually less than it felt.

Sizing a parity test on error rate

The realistic case is not zero on both sides but some rate on each, and the question is whether they differ. The standard sample size for comparing two proportions, at significance α and power 1 − β, is:

n per arm ≈ (z_{α/2} + z_β)² · 2 · p̄(1 − p̄) / δ²

with the conventional constants
  z for α = 0.05 two-sided : 1.96
  z for 80% power          : 0.84
  (1.96 + 0.84)²           = 7.84

n per arm ≈ 15.7 · p̄(1 − p̄) / δ²

worked, with your own numbers substituted:
  p̄ = 0.02   (assumption: current failure rate, 2%)
  δ  = 0.01   (assumption: the smallest difference you care about, 1pp)

  n ≈ 15.7 × 0.02 × 0.98 / 0.0001
    ≈ 15.7 × 0.0196 / 0.0001
    ≈ 3,077 requests per arm

Roughly three thousand on each side. Now the duration falls out of your traffic instead of out of intuition: at 10,000 requests a day split evenly, that is under a day. At a 5% canary on the same traffic it is about six days for the new arm to accumulate 3,000. At 500 requests a day it is twelve days at an even split, and a 5% canary would need four months — which is the real finding, because it says the canary percentage, not patience, is what is stopping you deciding.

One refinement usually applies. You rarely need to prove the two are equal; you need to prove the new one is not worse by more than δ, which is a one-sided non-inferiority question. Replace 1.96 with 1.645 and the constant becomes (1.645 + 0.84)² = 6.18, giving n ≈ 12.4 · p̄(1 − p̄) / δ² and about 2,420 per arm in the worked example — roughly a fifth fewer.

The parameter that deserves the argument is δ, and it is not a statistical choice. It is the answer to “how much worse are we willing to be and not notice”, which is a product decision. Notice the square in the denominator: halving δ quadruples the sample. If someone proposes detecting a tenth of a percentage point, the arithmetic says 300,000 requests per arm, and that number is the argument.

The signals slower than your sample

Here is the part the sample-size calculation cannot fix, and it is the real reason dual runs overrun. Some failures do not appear in a request-level counter at any sample size. A support ticket takes days. Churn takes weeks. A monthly report that is subtly wrong takes a month plus however long until somebody reconciles it. If your exit condition depends on a signal with a two-week latency, no amount of traffic shortens the run.

The answer is not to wait. It is to convert each slow signal into a fast proxy, before the run starts, and to accept the proxy as the decision input. If the slow signal is “tickets about wrong citations”, the fast proxy is a per-request check that the cited chunk contains the asserted claim. If it is “users abandon the flow”, the fast proxy is a per-request structural check on whatever makes the flow unusable — an unparseable response, a missing field, a refusal. If a slow signal cannot be proxied at all, say so explicitly and accept that you are cutting over without evidence on that dimension, which is a legitimate decision as long as it is a decision rather than an oversight.

The proxies have to exist before the run, because building one afterwards means restarting the clock on the only measurement that would have let you stop.

Why an open-ended dual run is actively harmful

The default assumption is that dual-running is merely expensive, so running a bit longer is a bit more cost for a bit more confidence. That understates it in four ways.

• Both paths must keep working. Every change to the prompt, the tool schema or the response parser has to be made and tested twice, and the second copy is the one that gets a lighter review.
• The old path rots invisibly. It is still load-bearing but nobody is looking at it, which is the exact condition under which a fallback fails at the moment it is called.
• Ownership evaporates. A migration that has been “nearly done” for two months stops being anyone’s current work, and confidence does not accumulate while nobody is reading the numbers.
• The deadline arrives anyway. If the migration exists because the old model is being retired, an open-ended dual run ends on the retirement date with the fallback still wired to something that no longer answers. That is the worst possible ending and it is reached by doing nothing.

Confidence does not grow indefinitely with time either. After the sample sizes above are met, additional traffic through a stable system tells you very little that is new, because you are re-sampling the same request distribution. What would tell you something new is different traffic — a month-end batch, a seasonal shape, a new tenant — and if one of those is genuinely outside what you have exercised, name it as a specific gate rather than extending the run in the hope of catching it.

Writing the exit condition down

Before the first shadow request, put five things in the document, and treat them as the definition of done.

1. The structural gates, which must be exactly zero. Schema validation failures, tool-call mismatches, unparseable responses — measured against the A/A floor from the diffing harness. These are not statistical; a response that does not validate is a bug and there is nothing to test.
2. The sample size and δ. Computed as above, with the assumed p̄ written next to it so the number can be revised when the real rate turns out different.
3. The fast proxies for every slow signal, each with a threshold, and an explicit list of the dimensions you have chosen not to measure.
4. A named decider. One person who reads the numbers and calls it. A committee produces an extension by default, because no individual is accountable for the cost of waiting.
5. An abandon date. The date at which failing to meet the condition means stopping the migration rather than extending the run. This is the clause nobody writes and it is the one that actually ends things — without it, every review meeting has exactly one available outcome, which is to keep going.

When the condition is met, cut fully and remove the old path in the same sprint. A dual run left switched on after the decision is not a safety net; it is an untested code path with a plausible-sounding name, and the next person to read the service will assume it works. If you want a genuine fallback afterwards, that is a separate design with its own tests — not the residue of a migration nobody finished cleaning up.