Calculating How Many Users a 5% Prompt Rollout Actually Reaches

“Ship it to 5%” is a decision about how many people see a change and how fast you would find out if it were bad. Both are arithmetic, and both are usually skipped.

Percent of what

The first thing to fix is the denominator, because a rollout percentage can mean at least three different things and they produce different numbers.

• 5% of users — the assignment is a function of a stable user identifier. Each selected user sees the variant on every request. This is the only one of the three that supports a per-user metric such as retention or session length.
• 5% of sessions — assignment keyed on a session identifier. A user can be in the variant today and out of it tomorrow.
• 5% of requests — assignment rolled per call. Maximum statistical power per exposure, and the worst experience: the same user gets two different behaviours in one conversation. That failure is the subject of variant stickiness, and it is a bug rather than a design in almost every LLM product.

For a prompt change, user-level is nearly always right, and the rest of this page assumes it. Note the immediate consequence: 5% of users is not 5% of requests unless usage is uniform, and usage is never uniform. If your top decile of users generates half your traffic, then a 5%-of-users rollout can carry anywhere from about 1% to about 20% of requests depending on which users the hash happens to select.

The worked table

Let D be daily active users and p the rollout fraction. The expected number of exposed users per day is D × p. The arithmetic is trivial; the point of writing it out is that the answers at the small end are smaller than people expect.

D (daily active users)   p = 1%      p = 5%      p = 10%
        1,000              10          50         100
       10,000             100         500       1,000
      100,000           1,000       5,000      10,000
    1,000,000          10,000      50,000     100,000

At 1,000 daily active users, a 5% rollout is fifty people. If your quality signal is a thumbs-down and roughly one user in two hundred leaves one, the expected number of thumbs-down from the variant is fifty divided by two hundred, which is a quarter of one per day. You will wait a week for the first data point, and its absence will tell you nothing. That is the calculation that decides whether 5% is a canary or a placebo, and the input to it — your feedback rate — is a number you already have.

Users, sessions and requests

Users are the assignment unit; exposures are what you actually get to measure. Chain the two ratios you know:

exposed users/day        = D × p
exposed sessions/day     = D × p × s        (s = sessions per user per day)
exposed requests/day     = D × p × s × r    (r = model calls per session)

Worked, with D = 40,000, p = 0.05, s = 1.4, r = 6:
  users     = 40,000 × 0.05           = 2,000
  sessions  = 2,000 × 1.4             = 2,800
  requests  = 2,800 × 6               = 16,800 model calls per day

The values of s and r above are stated as assumptions, not as facts about anyone’s product; replace them with your own medians. The reason to carry the chain all the way to requests is cost: 16,800 calls a day at whatever your per-call cost is under the new prompt is the daily price of the experiment, and if the new prompt is longer, the difference in input tokens multiplied by 16,800 is the incremental bill. That number belongs in the rollout decision, and it is the same arithmetic as bounding a loop by cost, applied to a population instead of a single run.

How long until you would notice

The honest version of “is the variant worse” is a two-proportion comparison, and the sample size needed scales with how small a difference you care about. A standard approximation for detecting a difference between two rates, at the conventional 95% confidence and 80% power, needs roughly

n per arm  ≈  16 × p̄ × (1 − p̄) / d²

  p̄ = the pooled rate you are comparing
  d  = the absolute difference you want to detect

Worked, for an error rate of 2% and a detectable difference of 1 point:
  p̄ = 0.02, d = 0.01
  n ≈ 16 × 0.02 × 0.98 / 0.0001 ≈ 3,136 observations per arm

The factor of 16 is the usual approximation for those confidence and power levels; it is a rule of thumb, and any statistics text will give the exact form. Applied to the worked example above — 16,800 variant requests per day — 3,136 observations arrive within a few hours, so a one-point regression in a 2% error rate is detectable the same day. At 1,000 daily active users and 5%, the same detection takes weeks, and the correct response is to raise the percentage rather than to pretend the wait is a safety measure.

Two caveats that change the number more than the formula does. If the unit of assignment is the user but the unit of measurement is the request, the observations are not independent and the effective sample size is smaller than the raw count — often much smaller when a few users dominate. And if you are watching several metrics at once, the chance of one crossing a threshold by luck rises with the number you watch.

Bucket granularity and skew

The last piece of arithmetic is the one that makes a 5% rollout not be 5%. Most assignment code hashes an identifier and takes a modulus. With 100 buckets, the finest rollout you can express is 1%, and 0.5% is not representable at all — it silently rounds to something. With 10,000 buckets, 5% is buckets 0 through 499 and the resolution is 0.01%.

At small user counts the realised share also varies. With 1,000 users assigned independently at 5%, the expected count is 50 and the standard deviation is the square root of 1000 × 0.05 × 0.95, about 6.9 — so seeing 40 or 62 exposed users is unremarkable. Do not treat a realised 4.1% as a bug; do treat a realised 0.4% as one, because that is a hash or a modulus problem rather than noise.

One last asymmetry is worth folding into the decision. The cost of a rollout that is too small is time: you wait longer for a signal, and the change ships later. The cost of one that is too large is the product of the harm per affected user and the number of users affected before you notice — and that second number is not the daily exposure figure, it is the exposure accumulated over your detection latency. If your dashboard is reviewed once a day, a 10% rollout at 100,000 daily active users has exposed 10,000 people by the time anyone looks. Halving the percentage halves that; halving the review interval halves it too, and is usually cheaper. Compute both numbers before arguing about the percentage, because the one people adjust is rarely the one doing the damage.