Online Evaluation: Grading Production Traffic

An offline eval measures a distribution you chose. Production is a distribution that changes without telling you — new customers, a seasonal shift in question types, a marketing campaign that brings in people who ask something nobody anticipated. Online evaluation is how you find out.

What online evaluation catches that offline cannot

• Input drift. Your eval set is a photograph of last quarter’s traffic. When the mix moves, offline scores stay flat while real quality falls, and nothing in the offline pipeline can notice.
• Provider-side change. A silently updated model behind a floating alias, a routing change, a quantisation change on a hosted endpoint. Offline runs are cached and infrequent; online grading sees it the same day.
• The long tail. The failure that happens to 0.3% of requests is worth about a sixth of an item in a 50-item eval and is worth several hundred graded requests a week in production.
• Real inputs. Users paste things nobody would think to write into an eval file — half a spreadsheet, a screenshot description, three questions at once, a language you do not support.

Two layers: guardrails and metrics

These have different economics and should not share a pipeline.

Most teams build the second and skip the first, which is backwards. Guardrails are cheaper, catch the more serious failures, and can actually prevent them. Build the free layer first and it will also tell you which requests are worth grading.

The sample size is not a percentage

“We grade 1% of traffic” is the standard answer and it is arbitrary in both directions: at 200 requests a day it grades two, at twenty million a month it grades two hundred thousand and burns a budget for precision nobody needed. Sampling rate should be derived from the interval width you want on the metric.

For a proportion, half-width w at 95% confidence:

    n = z^2 * p(1-p) / w^2      z = 1.96

Target: +/- 2 points on a metric currently around 0.85, per week.

    n = 3.8416 * 0.85 * 0.15 / 0.02^2
      = 3.8416 * 0.1275 / 0.0004
      = 1,225 graded requests per week

Tighter, +/- 1 point:
    n = 3.8416 * 0.1275 / 0.0001 = 4,899   -- four times the cost for
                                              half the interval

Note what this does not contain: your traffic volume. The precision of a proportion depends on the number sampled, not on the fraction of the population it represents. So the sampling rate is just 1225 / weekly_requests — 6% at 20,000 requests a week, 0.25% at 500,000, 0.03% at four million. Large services are usually over-sampling by an order of magnitude, and small ones are under-sampling so badly that their weekly chart is pure noise.

Two adjustments. If you report per-segment metrics, each segment needs its own n — that is where the budget actually goes, and it is the argument for having three segments rather than fifteen. And the quadratic in w is the thing to internalise before promising anyone a dashboard that resolves one-point changes.

Oversampling without breaking the estimate

Uniform sampling wastes most of the budget on requests that are fine. You want to oversample the suspicious ones — a request the user retried, an unusually long or short output, a low retrieval score, a guardrail near-miss, a new customer, a rare language. But an oversampled set no longer estimates the population.

The fix is standard survey methodology and it is four lines. Assign each request an inclusion probability by stratum, sample with that probability, and weight each graded result by its inverse — the Horvitz-Thompson estimator:

RATES = {          # inclusion probability by stratum
    "retried":        0.50,   # user hit regenerate -- probably bad
    "guardrail_warn": 0.40,   # passed, but only just
    "new_customer":   0.20,
    "rare_language":  0.20,
    "default":        0.004,  # everything else
}

def should_sample(req, rng):
    p = RATES.get(stratum_of(req), RATES["default"])
    return (rng.random() < p), p     # STORE p WITH THE RESULT

def population_mean(rows):
    """rows: [{'score': float, 'p': inclusion probability}]"""
    num = sum(r["score"] / r["p"] for r in rows)
    den = sum(1.0        / r["p"] for r in rows)
    return num / den                 # Hajek estimator: ratio form,
                                     # stabler than dividing by N

Now you get both readings from one budget: the weighted mean is an estimate of overall production quality, and the raw unweighted score within the retried stratum is a magnifying glass on the traffic you suspect. Storing p alongside every graded row is the non-negotiable part — a result set without inclusion probabilities can never be un-biased afterwards, and this is the mistake that quietly invalidates a quarter of dashboard.

One caution: heavy oversampling of a small stratum makes the weighted estimate high-variance, because a handful of rows carry enormous weights. Keep the ratio between the largest and smallest inclusion probability within about two orders of magnitude, and report the effective sample size rather than the row count.

Where the grading runs

• Never in the request path. A judge call in the response path doubles latency and adds a second dependency that can fail. Log the request and response, publish an event, grade from a worker.
• Grade from a stored artefact, not by replaying. Re-running the request to grade it measures a different response than the user got. Store the actual output, the retrieved context, the model version and the parameters.
• Use a cheaper judge than you would offline, and calibrate it. The same rubric on a smaller model, validated against your labelled set. This is where the cost of continuous grading is won or lost.
• Alert on the metric, not on individual failures. A single bad graded response is expected. A week-over-week shift outside the interval is the signal, and it needs the interval to be computed, which is why the sample size above matters.
• Feed the failures back. Every graded failure is a candidate eval item with real provenance. This is the mechanism that keeps an offline set from going stale, and the loop only closes if somebody owns the triage.

One constraint that is easy to design past and expensive to retrofit: grading production traffic means sending real customer content to a judge, which is a data-handling decision and often a contractual one. Settle three things before the pipeline exists — whether the graded sample may leave your infrastructure, how long the stored artefacts are retained, and whether the sampler must exclude requests from customers whose contracts forbid it. A redaction pass before grading usually satisfies all three, at the cost of some judge accuracy on items where the redacted entity was the point.