Sampling Rate for Human Review of LLM Output in Production

“We review one percent of output” is the policy almost everyone writes, and it is the wrong shape. Sample size for estimating a rate depends on the precision you want and hardly at all on how much traffic you have, so a percentage policy over-spends at high volume and under-delivers at low volume, and never lands on the precision anybody actually needs.

Decide what you are estimating

Human review of production output serves two entirely different purposes, and conflating them is why the sizing question feels unanswerable.

Measurement asks: what fraction of our output is unacceptable? The answer is a number with an interval around it, it feeds trend lines and release decisions, and it requires a uniform random sample. Any selection rule that favours suspicious outputs destroys it.

Defect-finding asks: what is going wrong, so we can fix it? Here uniform sampling is close to the worst possible use of a reviewer’s time, because if the defect rate is 1% they spend ninety-nine reads on fine output to find one problem. This is what triage is for, and it is the subject of routing low-confidence outputs to a reviewer.

Both are worth doing and they need separate queues with separate budgets. Everything below sizes the first one, because it is the one that has an arithmetic answer.

The sample size

You are estimating a proportion. The standard formula for a margin of error e at confidence 1 − α is:

n = z^2 * p(1-p) / e^2

At 95% confidence, z = 1.96, so z^2 = 3.8416.

p unknown, use p = 0.5 (the worst case, maximising p(1-p) at 0.25):

  e = 0.10  ->  3.8416 * 0.25 / 0.01    =    96.0  ->    97
  e = 0.05  ->  3.8416 * 0.25 / 0.0025  =   384.2  ->   385
  e = 0.03  ->  3.8416 * 0.25 / 0.0009  = 1,067.1  ->  1,068
  e = 0.02  ->  3.8416 * 0.25 / 0.0004  = 2,401.0  ->  2,401

If you already have a rough idea of the rate, the requirement drops sharply, because p(1−p) is small near the ends of the scale. This is the single biggest lever available and it costs nothing:

p = 0.05 (you expect about 5% unacceptable):

  e = 0.05  ->  3.8416 * 0.0475 / 0.0025 =    73.0  ->    73
  e = 0.02  ->  3.8416 * 0.0475 / 0.0004 =   456.2  ->   457
  e = 0.01  ->  3.8416 * 0.0475 / 0.0001 = 1,824.8  ->  1,825

Note what “margin of error” means before choosing one. At p = 0.05 and e = 0.05 your interval is 0% to 10%, which is useless for detecting a change. If the point of the sample is to notice the rate moving from 5% to 7%, you need the interval narrow enough that those two are distinguishable, which is a detection question, not an estimation one, and it wants the two-proportion formula in choosing a canary percentage instead. Estimation says how well you know the number now; detection says how small a change you can see between two periods. Sizing for estimation and then reporting week-over-week movement is a very common way to generate imaginary trends.

Why traffic volume barely matters

The formula above has no term for how many outputs exist. That is not an approximation for large populations only — it is the main result, and it is the part that overturns the usual policy. The correction for a finite population is:

n_adjusted = n / (1 + (n - 1) / N)

n = 385, population N per week:

  N =     2,000  ->  385 / (1 + 384/2000)     = 323
  N =    20,000  ->  385 / (1 + 384/20000)    = 378
  N =   200,000  ->  385 / (1 + 384/200000)   = 384
  N = 2,000,000  ->  385 / (1 + 384/2000000)  = 385

A hundredfold increase in traffic moves the required sample from 378 to 384. Meanwhile a 1% policy moves the review workload from 200 a week to 20,000 a week — a hundredfold increase in cost for a 1.6% improvement in precision that nobody asked for. In the other direction, at 2,000 requests a week the same 1% policy yields 20 reviews, which supports a margin of error of about ±22% and is therefore indistinguishable from not measuring.

So the policy should be a count per period, not a rate. “385 uniformly sampled outputs reviewed per week, giving us the overall unacceptable rate to within five points at 95% confidence” is a sentence that survives a tenfold change in traffic without anybody renegotiating headcount.

When you see no failures at all

A reviewer works through 300 sampled outputs and finds nothing wrong. The point estimate is zero, and the standard interval formula gives zero width, which is obviously wrong. The clean answer is the rule of three, popularised by Hanley and Lippman-Hand in JAMA in 1983 under the title “If nothing goes wrong, is everything all right?”, and it derives in two lines:

If the true rate is p, the chance of seeing zero failures in n draws is
    (1 - p)^n

Set that to 0.05 (the 95% upper bound) and solve:
    n * ln(1 - p) = ln(0.05) = -2.996
    for small p, ln(1 - p) ~ -p
    p ~ 2.996 / n  ~  3 / n

  n =   100  ->  upper bound 3.0%
  n =   300  ->  upper bound 1.0%
  n = 1,000  ->  upper bound 0.3%
  n = 3,000  ->  upper bound 0.1%

This is the most useful line in the whole subject for anybody trying to justify a review budget, because it converts the reviewer’s time directly into a claim. If somebody wants assurance that the unacceptable rate is under one in a thousand, that is three thousand uniformly sampled reviews with zero findings, and no amount of cleverness reduces it. Most people discover on seeing this that the assurance they were asking for costs more than they want to pay, which is a productive conversation to have early.

Spending the budget where it buys most

A single global rate is rarely the useful output. Break the sample into strata and size each one, because a rate averaged across a product that does five things tells you about none of them.

stratum          share    reviews/wk    margin of error at p=0.05
summarisation      70%          150            +/- 3.5 points
extraction         20%          150            +/- 3.5 points
open chat           8%          100            +/- 4.3 points
non-English         2%          100            +/- 4.3 points
                              -----
                                500

Weighted overall rate = sum(share_i * rate_i), which is unbiased
even though the sample is not proportional to traffic.

Five hundred reviews a week, deliberately over-sampling the 2% that is non-English, and every stratum has a usable number. Proportional sampling would have put ten reviews a week on non-English, which supports no conclusion at all — and non-English behaviour is exactly where prompt changes break, as prompts that work in English and fail elsewhere documents.

Writing the policy down

• State a count and a precision, not a percentage. The count is what the reviewers do; the precision is what it buys. If you cannot say what the precision is, you are not measuring.
• Sample uniformly and sample before filtering. Draw from all output, including requests that errored or were truncated. A sample that excludes the failures is measuring the successes.
• Keep the measurement queue and the triage queue separate, with separate people if you can, because a reviewer who has just worked a queue of known-bad outputs is calibrated differently for an hour afterwards.
• Fix the rubric before the sample. A rate is only comparable across weeks if “unacceptable” means the same thing each week; see evaluation rubrics.
• Budget for double-coding, and treat it as part of the sample. A rate produced by one reviewer applying a rubric is only as reproducible as that reviewer’s reading of it, and the way you find out is to have a second reviewer independently judge a subset. Ten percent double-coded is the usual starting point, and it is not free: if the budget is 385 reviews, double coding 10% costs 39 extra judgements, so either the sample drops to about 350 or the budget rises. Report the agreement rate alongside the defect rate. If two reviewers agree on only three quarters of items, the rubric is the problem and no sample size fixes it, because you are measuring two different quantities and averaging them.
• Re-derive when you learn the rate. The first fortnight’s sample is sized on p = 0.5 because you know nothing. Once you know the rate is near 3%, the same precision costs a third as many reviews, and that saving should be taken rather than absorbed.