Sequential Testing for a Prompt Change Instead of a Fixed-Horizon A/B Test

The sample size you computed before starting the experiment is only valid if you look at the result once. Everybody looks more than once. The gap between those two sentences is where most prompt experiments quietly lose their error guarantee, and closing it needs a different test rather than more discipline.

What peeking actually does

A fixed-horizon test makes one promise: if there is no real difference, the probability of declaring one is at most α, usually 5%. That promise is about a single decision made at a single predetermined sample size. Evaluate the same test repeatedly as data accumulates, and stop the first time it crosses the threshold, and you are no longer running that test. You are running a procedure that takes the maximum of many correlated statistics, and the maximum crosses a fixed line far more often than any single value does.

The limiting case is a theorem rather than a simulation result. Under continuous monitoring, the standardised difference between two arms behaves like a random walk, and by the law of the iterated logarithm that walk will exceed any fixed multiple of its standard error eventually with probability one. Run a null experiment long enough, check it often enough, and you will find significance every time. The finite version — you peek ten times, not continuously — sits somewhere between 5% and 100% depending on how the peeks are spaced, and there is no closed form for it, which is precisely why it is not something to reason about informally.

Prompt experiments are unusually exposed to this because the result is on a dashboard, the change is cheap to ship, and somebody is anxious about it. The dashboard is the peeking. The problem is not that engineers lack discipline; it is that a test which forbids looking is the wrong instrument for a decision people need to be able to make early.

Wald’s sequential probability ratio test

The SPRT, published by Abraham Wald in Sequential Tests of Statistical Hypotheses (1945), takes the opposite starting point: it has no fixed sample size at all. After each observation you update a running log-likelihood ratio between two specific hypotheses, and you stop when it leaves a band.

After each observation x:
    S  +=  log( P(x | H1) / P(x | H0) )

Stop and accept H1 if  S >= log( (1 - beta) / alpha )
Stop and accept H0 if  S <= log( beta / (1 - alpha) )
Otherwise, keep going.

With alpha = beta = 0.05:
    upper = log(0.95 / 0.05) = log 19  =  2.944
    lower = log(0.05 / 0.95)           = -2.944

Two properties make this the right shape for a canary. It is optimal in a precise sense: Wald and Wolfowitz proved in 1948 that among all tests with the same two error probabilities, the SPRT minimises the expected number of observations under both hypotheses. And it can accept the null, which a fixed-horizon test cannot — a prompt experiment that concludes “no meaningful difference, stop” after 300 requests has saved you the other 450 and told you something actionable.

The cost is the thing people trip over. The SPRT tests one specific alternative against one specific null. You do not say “detect any regression”; you say “distinguish a 2% failure rate from a 4% failure rate”, and the test’s behaviour against a 3% failure rate is not what the α and β you set describe. It also has no upper bound on run length in principle, so a real implementation needs a truncation rule, and a truncated SPRT’s error rates are slightly worse than the nominal ones.

Group sequential boundaries

Clinical trials solved the same problem differently, and the machinery transfers cleanly. Instead of continuous monitoring, you fix a small number of interim analyses in advance and use a stricter threshold at each one, chosen so that the total probability of a false rejection across all the looks is still α. The idea is that α is a budget that gets spent across looks rather than a per-look level.

Two boundary families dominate. The one published by Pocock in Biometrika in 1977 uses the same nominal significance level at every look — for five equally spaced analyses at a two-sided α of 0.05, the published constant nominal level is about 0.016. The one published by O’Brien and Fleming in Biometrics in 1979 is very strict early and nearly the unadjusted level at the final look, so it almost never stops early but costs very little sample size if it runs to completion.

Pocock, 5 looks, two-sided alpha = 0.05
    nominal level at every look:  ~0.016
    stops early readily; final look is stricter than 0.05

O'Brien-Fleming, 5 looks, two-sided alpha = 0.05
    nominal levels rise across looks, ending near 0.05
    rarely stops early; maximum sample size close to fixed-horizon

For a prompt canary the choice maps onto what you fear. If the expensive outcome is shipping a regression, you want to stop early on bad news, which argues for Pocock-like boundaries or an SPRT with an asymmetric α and β. If the expensive outcome is abandoning a genuine improvement because of an unlucky first hour, O’Brien-Fleming is the conservative choice.

Always-valid inference

The third family gives up the idea of a look schedule altogether. An always-valid p-value is constructed so that its guarantee holds simultaneously at every sample size — you may look continuously, stop whenever you like for any reason, and the type I error is still bounded by α. Johari, Pekelis, Koomen and Walsh set this out for the A/B testing case in Always Valid Inference: Continuous Monitoring of A/B Tests, building on the mixture sequential probability ratio test, and it is the approach that several commercial experimentation platforms adopted afterwards.

What you pay is width. An always-valid confidence interval at a given sample size is wider than a fixed-horizon one at the same sample size, because it is buying you the right to have stopped at any earlier point. If you genuinely will look exactly once, it is strictly worse. The moment you might look twice, it is strictly better, because the alternative is a fixed-horizon interval whose stated coverage is fiction.

The practical appeal for a prompt rollout is that it needs no pre-registered alternative and no look schedule, which are the two things a small team will not maintain. It composes well with a dashboard: the interval is simply always correct to read.

Choosing between them

• You have one clear alternative and want the fewest requests. SPRT. “Distinguish a 2% schema-failure rate from 4%” is exactly its shape, and a canary with a fixed budget of exposure is exactly the situation its optimality is about.
• You have a review cadence anyway. Group sequential. If somebody looks at the canary at 09:00, 13:00 and 17:00, that is a three-look schedule and you may as well get a valid one.
• People will watch a graph. Always-valid. It is the only one of the three whose guarantee survives arbitrary human attention, and arbitrary human attention is the actual operating condition.
• The metric is a mean, not a rate. All three extend to means, but the variance has to be estimated rather than implied by the rate, and heavy-tailed metrics — latency, cost per request, tokens — break the normal approximation the boundaries assume. Cap or winsorise before testing, and say that you did.

One thing none of them fixes: stopping early on a good result systematically overstates the effect size, because you stopped at a moment the noise was in your favour. The decision to ship is sound; the number you quote for how much better it is will be optimistic. Report the interval rather than the point estimate, and if the size of the win matters for a downstream decision, keep measuring after you ship. The sample size arithmetic behind all of this is worked in choosing a canary percentage, and it is worth knowing what the fixed-horizon number is even when you intend to stop before reaching it.