A/B Testing Prompts in Production

Offline evals answer “is this output better”. A live A/B answers “does anything change for the people using it”. The second question is more expensive, so it is worth knowing before you start whether your traffic can answer it at all.

When an A/B is the right instrument

Run the offline eval first, always. It is faster, cheaper, has no user impact and catches format regressions. Reserve live testing for the things it uniquely measures:

• Behavioural outcomes — did the user retry, escalate, convert, abandon.
• Real input distribution, including the inputs nobody put in the eval set.
• Operational effects — latency, output length, cost per session, cache hit rate.
• Anything where your offline metric is a proxy you do not fully trust.

Two constraints before launch, neither of them statistical. A prompt experiment ships a behavioural change to real people, so the losing arm should be a variant you would be willing to serve for the whole test — not a deliberately weakened control. And the arm has to be visible to support: when somebody reports a bad answer, the first question is which prompt they were on, and an experiment that cannot answer that is one nobody will let you run twice.

Picking a metric you can compute

The metric has to be computable per request without a human, or the experiment will end when someone gets bored of grading. Usable primary metrics, roughly in order of how directly they capture value: task completion where you can observe it; escalation or handoff rate; user-initiated retry or rephrase rate; explicit feedback, accepting that it is sparse and biased; and parse or validation success, which is weak as a primary metric but excellent as a guardrail.

Declare one primary metric and two or three guardrails — cost per request, p95 latency, refusal rate — before launch, and write them down. Choosing the winner after seeing which metric moved is the most common way a prompt test produces a confident false conclusion.

Model-graded metrics are the tempting shortcut and they need validating first. If a judge model scores both arms, check it against human labels on a sample, and check it for the known biases — position, length, and a preference for the house style of whichever family it belongs to. A judge that mildly prefers longer answers will hand the win to whichever prompt is more verbose, which is not the question you asked.

Splitting traffic

Split on a stable unit, not per request. A user who sees prompt A on one turn and prompt B on the next gives you an incoherent experience and a contaminated measurement.

def arm(experiment: str, unit_id: str) -> str:
    h = sha256(f"{experiment}:{unit_id}".encode()).hexdigest()
    return "B" if int(h[:8], 16) % 100 < 50 else "A"

# stable across restarts and deploys — no assignment table to keep
# salted per experiment, so the same users are not always in B
# log with every generation:
#   experiment, arm, prompt_version, prompt_sha, model, params

Use the conversation id when a session is the natural unit and the account id when behaviour is account-level. And log the arm alongside the prompt fingerprint on every generation — without that join, the analysis is a reconstruction and you will not fully trust it.

Sizing it before you run it

This is the step that gets skipped and the one that determines whether the experiment can succeed. For a binary metric, the standard approximation for 80% power at the conventional 5% significance level is:

n per arm  ≈  16 · p(1 − p) / δ²

baseline p = 0.80,  so p(1−p) = 0.16

  δ = 1pp   ->  16 · 0.16 / 0.0001  =  25,600 per arm
  δ = 3pp   ->  16 · 0.16 / 0.0009  =   2,844 per arm
  δ = 5pp   ->  16 · 0.16 / 0.0025  =   1,024 per arm
  δ = 10pp  ->  16 · 0.16 / 0.01    =     256 per arm

Read the table honestly against your traffic. At a thousand relevant requests a week, a three-point improvement takes about six weeks per arm to detect — and a one-point improvement is out of reach forever. That is not a reason to skip the experiment; it is a reason to test changes big enough to see, and to accept small changes on offline evidence instead.

Two adjustments. If the same user contributes many requests, the effective sample size is closer to the number of users than the number of requests, so analyse at the user level or your confidence intervals will be far too narrow. And for latency and cost, compare distributions rather than means — a p95 with a bootstrap interval says something a mean cannot, because one cold start drags a mean past every request anyone experienced.

The other half of sizing is duration. Run for whole weeks rather than to a fixed request count: traffic composition changes by day of week, and a test that ends on a Wednesday afternoon has sampled a particular kind of user. Two full weeks is the usual minimum for anything user-facing, and for most teams it binds before the sample size does.

Five ways prompt tests go wrong

• Peeking. Checking daily and stopping when it looks significant inflates the false-positive rate well beyond the nominal 5%. Fix the horizon in advance, or use a sequential test designed for continuous monitoring.
• The arms differ by more than the prompt. A new prompt shipped with a new model, a new temperature or a longer max_tokens is not a prompt test. Diff the whole request.
• Cache asymmetry. Arm A’s prefix has been warm for months; arm B’s is new. Early latency and cost differences may be entirely the cache, so exclude a warm-up window before comparing operational metrics.
• Sampling noise mistaken for effect. At non-zero temperature, the same prompt disagrees with itself. An A/A test — both arms identical — is the cheapest way to see how much of your metric is noise, and it is worth running once before you trust any result.
• Segment mining after the fact. “It won for mobile users in Germany” after twelve segment cuts is one expected false positive. Pre-register the segments you care about.

When the result comes back flat — and most prompt A/Bs do — resist the reflex to slice until something appears. A null result on a properly sized test is informative: it says the change is smaller than the effect you can afford to detect, which is a good argument for spending next week on retrieval, routing or the model itself rather than on wording.