What a Provider Migration Does to A/B Test Validity

You are four days into a two-week experiment on prompt B when infrastructure moves the whole application onto a different inference provider. The dashboard keeps counting. The question is whether the number at the bottom of it still means anything, and the answer is not “it is a bit noisier now”.

An arm has to name one thing

Every A/B test rests on an assumption that is rarely written down because it is normally free: that the label variant = B picks out a single, well-defined treatment. Causal inference calls this consistency, or the no-multiple-versions-of-treatment condition. The estimate you compute is a difference between two averages, and it answers a question of the form “what happens if everyone gets B” only if there is one B to get.

A provider change breaks that quietly. After the cutover, requests labelled B are served by a different model, with a different tokenizer, different sampling defaults, a different refusal boundary and different tail latency. Those are not perturbations of B; they are a second treatment wearing B’s label. Your dataset now contains B₁ and B₂, and the average of them estimates the effect of a treatment nobody will ever ship, because after the migration nobody serves B₁ again.

This is why the instinct to “let it run, the noise averages out” is wrong in kind rather than in degree. Noise averages out. A mixture of two different treatments averages to the mixture, and the mixture weights are set by whatever the deployment calendar happened to be.

Three ways the change lands

What you can salvage depends entirely on how the change hit the arms, and this is the first thing to establish before touching the analysis.

• One arm only. The new provider serves B but the control still runs on the old one, perhaps because the control is pinned to a model the new provider does not host. This is the worst case and it is not repairable: provider and variant are now perfectly collinear, so no amount of data separates “prompt B is better” from “the new provider is better”. There is no statistical fix for a confounder that has no independent variation.
• Both arms, at the same instant. Both variants move together at the cutover. Within each period the comparison is still clean, because both arms see the same provider. What you have is two valid experiments in sequence, not one long one, and the useful question becomes whether they agree.
• Both arms, gradually. A percentage ramp moves traffic over days. This is the case that looks safest and is the easiest to analyse wrongly, because at any moment both arms are mixtures, and the mixture proportion is only equal across arms if the ramp was applied independently of variant. Verify that from the routing logs rather than assuming it: a ramp keyed on user id and an assignment keyed on user id can correlate for reasons nobody intended.

Why pooling is worse than noisy

Take the second case, the friendly one, and show why you still cannot simply pool. Suppose the metric is task success rate, and suppose the new provider lifts both arms equally. Write the period-level rates as fractions and the arm sizes as counts:

period 1 (old provider)      A: 400/1000 = 0.400   B: 440/1000 = 0.440
period 2 (new provider)      A: 270/ 500 = 0.540   B: 592/1040 = 0.569

pooled A = (400 + 270) / 1500 = 0.4467
pooled B = (440 + 592) / 2040 = 0.5059

within-period difference      period 1: +0.040   period 2: +0.029
pooled difference:            +0.059

The pooled difference is larger than the difference in either period. Nothing anomalous happened; the arms simply had different shares of the high-rate second period, because traffic to the two arms was not identical after the cutover. That is Simpson’s paradox arriving through the side door, and it can point the other way just as easily — the same arithmetic with the shares swapped produces a pooled difference smaller than both periods, or of the opposite sign.

The repair for this specific case is standard and cheap: stratify by period. Compute the difference within period 1 and within period 2, and combine them with weights that do not depend on the arm split — equal weights, or weights proportional to period size. What you must not do is add the numerators and denominators, which is what every dashboard does by default.

Sticky assignment makes the damage permanent

Everything above assumes assignment is per-request and the metric is short-horizon: one call in, one outcome out. If assignment is sticky per user, and it usually is, the picture is worse in a way that no re-slicing recovers.

A user who spent four days on B₁ and then moved to B₂ carries the first exposure into the second period. Their learned expectations, their saved outputs, their support ticket, their decision to stop using the feature — all of that was produced partly by a treatment that no longer exists. Restricting the analysis to post-cutover data does not clean them, because the unit is the user and the user is contaminated. The set of users who are genuinely clean is the set that first appeared after the cutover, and that set is self-selected: new users differ from returning users on almost every metric worth measuring.

This is the same identity problem that makes variant stickiness worth testing on its own. Stickiness is what makes long-horizon metrics measurable; it is also what makes a mid-flight treatment change unfixable.

Pause, restart, or carry on

A rule that survives contact with a real release calendar, in order of the questions to ask:

1. Did the change hit one arm or both? One arm: the experiment is dead. Stop analysing it, record why, and do not let the partial result inform the decision, because a confounded estimate is not a weak signal — it is a number with an unknown sign.
2. Is assignment per-request or per-user? Per-request with a same-session metric: you can discard the pre-cutover data and continue accumulating, at the cost of the days you threw away. Recompute the required sample size from scratch; do not reuse the original stopping rule, because that rule was set for a sample you no longer have.
3. Per-user, or metric with memory? Restart with fresh units. Exposed users are burnt for this experiment. If the population is small enough that you cannot afford to exclude them, the honest move is to accept the experiment as inconclusive rather than to analyse it anyway with a caveat nobody will read.
4. Can you delay the cutover instead? Compare the days remaining in the test against the days the migration slips. Tests usually lose this argument and should, but the comparison is worth making explicitly rather than by default, because a restarted two-week test costs two weeks plus the four days already spent.
5. Before the next one, put the provider in the assignment record. Log the provider, model string and routing decision alongside the variant on every request. Then the analysis can stratify without an archaeology exercise, and a cutover you were not told about shows up as a covariate rather than as an unexplained step in the metric.

The general point outlives the migration. Any change that alters what an arm means — a model version bump, a routing rule, a retry policy that silently sends failures elsewhere — has the same effect as a provider swap, and the ones that happen without an announcement are the dangerous kind, because you never learn to stratify on a change you did not observe. Treat the experiment platform as something that has to know about the serving stack, not as a layer above it.