Comparing Shadow Traffic Outputs Without a Human Reading Every One

Ten thousand shadow pairs a day and two people who can look at fifty of them. The question is not how to compare two outputs — it is which fifty to put in front of the humans, and the arithmetic that answers that is unforgiving in a way most filter designs ignore.

Three layers, cheapest first

Organise the comparison as a cascade. Each layer is more expensive per pair than the last and sees fewer pairs, and the ordering is what makes the whole thing affordable.

• Deterministic checks on fields and structure. Free, run on every pair, and produce a hard verdict rather than a score.
• Cheap similarity signals — length deltas, token overlap, embedding distance. Cents per thousand pairs. These produce a score, not a verdict.
• Model judgement, on the small residual the first two layers surface. This is a full inference call per pair and often on an expensive model, so it should never see more than a few percent of traffic.

The mistake that makes shadow comparison unaffordable is running layer three on everything, on the theory that a judge is the only thing that understands the output. It is also the layer with the worst reliability per dollar, which the library covers separately under eval blind spots.

Hard checks that need no model

These are verdicts. If one trips, the pair is a defect and does not need ranking.

• Parse and schema validity. If the contract is JSON, run the actual validator you use in production against both outputs. A shadow output that fails where the baseline passes is a regression, full stop, and this catches a large share of real prompt breakage on structured endpoints. See testing structured output.
• Tool-name set equality. Compare the set of tools called, and their order if order matters. This is exact, deterministic and needs no interpretation of prose.
• Required-field presence. If the schema is loose, check the fields the downstream consumer actually reads, which is usually a shorter list than the schema.
• Finish reason mismatch. Baseline stopped normally, shadow hit the token ceiling. Two enum values.
• Empty or whitespace-only output. A count.
• Leaked content. Regexes for the system prompt’s distinctive phrases, for API-key shapes, and for anything else that should never appear in output. Cheap, and it is the one check whose failures are urgent rather than merely informative.

What must not be on this list is an exact or near-exact match on the assistant’s prose. Two runs of the same prompt at nonzero temperature differ; two different prompts differ more; and an assertion on the sentence is the single most common way an LLM test suite becomes noise. Assert on the structure the prose carries, never on the prose.

Soft signals and where they lie

These produce a number per pair. Treat every one as a ranking signal rather than a threshold.

• Length delta. Compute in tokens, not characters, and use a relative measure — the absolute delta between a 40-token and a 4,000-token answer are not comparable events. This is the best single cheap signal, because almost every prompt regression that matters changes length in one direction.
• Embedding cosine distance. Cheap and useful for “these two answers are about different things”. Nearly useless for “this one is wrong”, because a confidently wrong answer sits close to the right one in embedding space. Use it to catch topic drift, not error.
• Numeric-token disagreement. Extract numbers from both outputs and compare the multisets. Where the task involves figures, this catches a specific and expensive failure that similarity metrics smooth over entirely.
• Refusal indicators. Not a phrase list. Short output plus no tool call plus a normal finish reason, combined with the provider’s own refusal flag where it exists.
• Language identification. If baseline answered in the user’s language and shadow did not, that is a defect a similarity score will happily rank as a mild difference.

One design rule holds all of these together: every signal should be computable on a single output as well as on a pair. Signals that only exist as a difference cannot be used once the change ships and there is no longer a baseline to diff against, so you end up building the monitoring twice.

Why specificity is the number that matters

Here is the arithmetic that decides whether the filter is usable. Assume 10,000 shadow pairs a day and assume that 1% of them contain a genuine regression — both are inputs, and the second one is a guess you should replace with your own once you have reviewed a random sample. That is 100 real defects and 9,900 clean pairs.

filter A: sensitivity 0.90, specificity 0.95
  true positives  = 100  * 0.90 =  90
  false positives = 9900 * 0.05 = 495
  flagged = 585      precision = 90 / 585 = 15.4%

filter B: sensitivity 0.70, specificity 0.995
  true positives  = 100  * 0.70 =  70
  false positives = 9900 * 0.005 = 49.5
  flagged = 119.5    precision = 70 / 119.5 = 58.6%

Filter B misses more defects and is far more useful. It flags 120 pairs against 585, and a reviewer working through B’s queue finds something real six times in ten rather than three times in twenty. With a review capacity of fifty pairs a day, filter A’s queue is twelve times oversubscribed — which means the reviewers see an arbitrary subset of it, and the effective sensitivity of the whole system is not 0.90 but 0.90 × (50/585) = 7.7%.

That is the general result and it is worth stating plainly: when the base rate is low, the false positives come from the enormous clean population, so specificity dominates precision. Tuning a prefilter for recall is the natural instinct and it produces a queue nobody can work. The same arithmetic governs every screening problem and is why the triage design for routing low-confidence outputs is built the way it is.

Rank to a budget, do not threshold

The way out is to stop making a binary decision. Combine the soft signals into one score, sort the day’s pairs by it, and give the reviewers the top fifty. Nothing about this requires calibration, nothing overflows, and the reviewers’ time is fully used every day regardless of how the traffic mix moved.

The metric changes accordingly. Precision and recall are properties of a threshold; what you care about is recall at fifty — the fraction of the day’s real defects that landed in the top fifty. You can measure it honestly by taking a small uniform random sample, reviewing it separately, and counting how many of the defects it contains were also in the ranked top fifty. That random sample is doing double duty, because it is also the only unbiased estimate of your overall defect rate — see sampling rate for human review for how large it has to be.

Two practical notes on the scoring. Normalise each signal before combining — a raw cosine distance and a token delta are on different scales and summing them lets whichever has the larger variance decide everything. And keep the hard-check failures out of the ranking entirely: they go to the front of the queue as verdicts, because a schema failure does not need a human to decide whether it is bad, only to decide what to do about it. That queue is the subject of building a review queue for failed cases.