A/B Testing Machine Learning Systems

Testing a model is not testing a button. The unit you randomise may influence the units around it, the effect changes over the first two weeks in both directions, and the model itself keeps learning from the traffic it is being tested on. Each of those breaks a standard A/B test in a way that is invisible in the p-value.

What is different about testing a model

• The treatment is not fixed. A recommender in the treatment arm trains on the interactions it generated yesterday. By week three it is a different system from the one you launched, and the measured effect is an average over a moving target.
• Units interact. In a marketplace, a better ranking for treated buyers consumes inventory that control buyers would have bought. The control group is damaged by the treatment, which inflates the measured difference — sometimes to the point where a neutral change looks like a large win.
• The effect is heterogeneous by construction. A model that helps the top decile and hurts the tail can show zero average effect. The average is the answer to a question nobody asked, and the uplift framing is what recovers the useful version.
• The metric that moves is rarely the metric that matters. Click-through rises and downstream satisfaction falls. This is not hypothetical; it is the default behaviour of any system optimised against an engagement proxy.

Commit to the metric before the split

Write this down before any traffic is assigned, and store it in the repository next to the experiment configuration.

1. One primary metric. One. It decides ship or not ship. If you cannot pick one you do not have a hypothesis, you have a hope.
2. The minimum detectable effect. The smallest change that would be worth shipping — a business number, not a statistical one. It determines the sample size, and choosing it after seeing the data is the most common form of experiment fraud that is not perceived as fraud.
3. Guardrail metrics. Things that must not get worse: latency, error rate, revenue per session, complaint rate. A win on the primary metric with a guardrail breach is not a win.
4. The duration, in whole weeks. Weekday and weekend populations differ; stopping on a Wednesday because significance arrived is how a weekly cycle gets reported as an effect.
5. The stopping rule. Fixed horizon, or an explicitly sequential design. “Check daily and stop when p < 0.05” is neither, and it produces false positives at several times the nominal rate.

Sample size, derived

For a two-arm test on a proportion, at 5% two-sided significance and 80% power, the required sample per arm follows from the two z-values:

n per arm = 2 * (z_alpha/2 + z_beta)^2 * p(1-p) / d^2

  z at 2.5% one tail = 1.960
  z at 20% one tail  = 0.842
  (1.960 + 0.842)^2  = 2.802^2 = 7.85
  x 2               = 15.7  ~  16          <-- the "rule of 16"

n per arm = 16 * p(1-p) / d^2

Baseline conversion 4%, want to detect a 0.4pp absolute lift:

  n = 16 * 0.04 * 0.96 / 0.004^2
    = 16 * 0.0384 / 0.000016
    = 0.6144 / 0.000016
    = 38,400 per arm     (76,800 total)

At 5,000 sessions a day that is about 16 days. Halve the effect you
want to detect and the requirement quadruples: 153,600 per arm, or
about two months.

The d² in the denominator is the number to internalise. Detecting small effects is not slightly harder, it is quadratically harder, and this is why most model improvements — which are small — cannot be validated by a two-week test on a mid-sized product. Knowing that before you start is worth more than any analysis afterwards.

Variance reduction, and what it buys

If you have pre-experiment data for the same units, you can subtract the part of each unit’s outcome that was predictable before the experiment began. The technique is usually called CUPED, and the effect on required sample size is exact.

Adjusted metric for unit i:

  Y'_i = Y_i - theta * (X_i - mean(X))

where X is a pre-period covariate (the same metric, measured before
the experiment) and theta = Cov(Y, X) / Var(X).

Var(Y') = Var(Y) * (1 - rho^2)      rho = correlation(Y, X)

  rho = 0.3   ->  variance x 0.91   ->   9% fewer units needed
  rho = 0.5   ->  variance x 0.75   ->  25% fewer
  rho = 0.6   ->  variance x 0.64   ->  36% fewer
  rho = 0.8   ->  variance x 0.36   ->  64% fewer

The adjustment does not change the expected treatment effect --
X is measured before assignment, so it cannot be affected by it.
It only removes noise that was already there.

The pre-period value of the same metric is usually the best covariate and it is free. On a metric with meaningful per-user persistence — spend, sessions, orders — a correlation of 0.5 to 0.7 is ordinary, and a third off the required sample is the difference between a test that fits in a quarter and one that does not.

Interference: when units are not independent

The standard test assumes one unit’s treatment does not affect another unit’s outcome. Model deployments break this constantly.

The remedies all trade statistical power for validity, which is why they are unpopular and necessary.

• Cluster randomisation. Randomise cities, warehouses, or accounts rather than users, so interference happens inside a cluster rather than across arms. The effective sample size is the number of clusters, not the number of users — twenty cities is twenty observations, and no amount of traffic changes that.
• Switchback tests. Alternate the whole system between treatment and control on a time schedule. Standard for logistics and marketplaces where geographic clusters are too few. Needs careful handling of carryover between periods.
• A long-run holdback. Keep 1–5% of traffic permanently on the old system. It is the only way to measure the cumulative effect of a year of shipped changes, each of which individually measured as a small win.

Novelty and primacy

Two opposite time-varying effects, and both are real.

Novelty: users engage with a change because it is different. A redesigned recommendation row gets clicked because it looks new. The effect decays over one to three weeks and can decay to zero or below.

Primacy: users are worse at a changed interface until they relearn it. The measured effect starts negative and improves. Ship decisions made on week one reject genuinely good changes for this reason.

The diagnostic is the same for both: plot the treatment effect by day since the user’s first exposure — not by calendar day, which confounds it with the ramp-up of the experiment. A flat line means neither is present. A decaying line means you are measuring novelty and the honest estimate is the plateau, not the average. If it has not plateaued by the end of the test, the test was too short and no statistic recovers what it needed.

Reading the result honestly

1. Check the split first. If the arms differ in size by more than randomisation explains, something is filtering users differently and every downstream number is suspect. This one check catches more broken experiments than all the others together.
2. Check the pre-period. The arms should show no difference on the primary metric before the experiment started. If they do, the randomisation is not random.
3. Report the confidence interval, not the point estimate. “+2.1%” is not a result. “+2.1%, 95% CI [+0.3%, +3.9%]” is, and it makes visible how much of the range is below the minimum detectable effect you committed to.
4. Do not slice until significant. Twenty segments at 5% gives one false positive on average, and it will be the segment with a story attached. Pre-register the segments you intend to look at, or correct for the number you looked at — the statistical errors page catalogues the rest of this family.
5. A null result is a result. The model that does not move the metric has saved you its maintenance cost, and that finding is worth writing up. Most model changes do not move a business metric, and a team that only publishes wins learns nothing.

One last thing to check before the test: is the model calibrated, and is the threshold set from costs? A model that is better on AUC but scored at the same threshold can easily perform worse in production because the score distribution shifted underneath the cut-off — the threshold page works that case.