Replaying Production Traffic Against a New Model

Replay differs from shadowing in one useful way: it is offline. You are not racing production, so you can run the same corpus against four candidates, re-run it after a prompt edit, and diff any pair. The corpus is the asset; everything else is a script.

Building the corpus

A uniform random sample of production requests is a bad corpus, because production is dominated by easy requests and the interesting behaviour lives in the tails. Stratify deliberately, and write the strata down so the corpus can be rebuilt.

-- Stratified corpus: N per (feature × input-length decile), plus
-- everything that went wrong, plus everything a user complained about.
with base as (
  select request_id, feature, input_hash, input_ref,
         ntile(10) over (partition by feature order by input_tokens) as len_decile,
         error_type, finish_reason, schema_valid, thumbs_down,
         row_number() over (
           partition by feature, ntile(10) over (
             partition by feature order by input_tokens)
           order by random()
         ) as rn
  from llm_request
  where environment = 'prod'
    and started_at between $1 and $2
    and tenant_id not in (select tenant_id from tenant where replay_opt_out)
)
select distinct on (input_hash) *
from base
where rn <= 40                          -- 40 per decile per feature
   or error_type is not null            -- every failure
   or finish_reason = 'length'          -- every truncation
   or schema_valid = false              -- every malformed output
   or thumbs_down                       -- every complaint
order by input_hash, rn;

Three details matter more than the numbers. distinct on (input_hash) deduplicates: production is full of identical requests, and a corpus that is 40% one repeated health-check prompt measures nothing. The opt-out join is not optional — some tenants have contractual restrictions on how their data may be processed, and a replay is processing. And the corpus should be frozen and versioned once built, because a corpus that changes between runs makes every comparison uninterpretable.

Size it by what the comparison needs, not by what is available. Around two thousand requests is enough to detect a couple of percentage points of movement in a rate-type signal, is affordable to run repeatedly, and is small enough that a human can read the interesting tail. Ten times that gives you tighter intervals on numbers you were not going to act on at that precision, and makes the run a project rather than a step.

Rebuild the corpus on a schedule — quarterly is a reasonable default — because production drifts. A corpus assembled before you shipped file uploads does not contain any requests with attachments, and it will keep confidently reporting that your candidate handles your traffic well. Version it, keep the old ones, and note which corpus version each result came from, or two results a year apart will get compared as though they measured the same thing.

Making the replay hermetic

A replay is only meaningful if the only thing that changed is the model. Four sources of drift have to be pinned shut:

For an agent loop, hermetic replay means recording the whole trajectory — every tool call and response in order — and replaying against it. When the candidate makes a call that has no recording, that divergence is a result, not an error: record it, stop the trajectory there, and count it. “The candidate took a different path” is one of the more informative outcomes a replay produces.

The diff ladder

Comparing thousands of output pairs by hand is not a plan. Run a ladder of checks, cheapest first, and only escalate the pairs that survive each rung. In practice most pairs are settled before the expensive rungs.

• Rung 0 — identical. Byte equality after normalisation. Settled, no further work.
• Rung 1 — structurally equal. For JSON output, parse both and compare semantically: key order and whitespace do not count, numeric tolerance is explicit. For tool-calling, compare the sequence of tool names and the normalised arguments.
• Rung 2 — mechanical checks. Schema validity, the presence of required fields, citation ids resolving, length ratio inside a band, language match, refusal detection. Each is a boolean and each can flip a pair to “worse” without any judgement.
• Rung 3 — a rubric judge. Only on what remains. Give the judge the input, both outputs in randomised order, and a rubric; ask for better / same / worse with a reason. Randomising the order matters: judge models have well-documented position biases, and an unrandomised comparison measures the ordering as much as the outputs.
• Rung 4 — a human. On a sample of the disagreements and all of the “much worse” verdicts. This is where the actual decision gets made, and the point of the ladder is that it is fifty pairs rather than five thousand.

The economics of the ladder are the reason to build it in that order. Rung 3 is the expensive rung — a judge reads the input and both outputs, so it costs several times what generating the candidate output cost in the first place — and rung 4 is the scarce one, because it consumes a person’s attention. Everything cheaper exists to keep the population arriving at those two rungs small. On structured outputs it is common for rungs 0 to 2 to settle the large majority of pairs; on free prose they settle far fewer, which is a reason to make your outputs structured wherever the product allows it.

The report

The output of a replay run is not a score. It is a table plus a reading list.

Replay corpus v4 (2,840 requests, 2026-07-28 → 2026-08-01)
baseline: model-A-2026-03-11   candidate: model-B-2026-06-02

                                 baseline   candidate    delta
  schema valid                     98.2%       96.9%     -1.3pp
  citation resolves                94.1%       95.8%     +1.7pp
  refusal                           1.1%        2.4%     +1.3pp
  truncated (finish=length)         0.6%        1.9%     +1.3pp
  mean output tokens                 412         587      +42%
  p95 latency (ms)                  4,210       3,480     -17%
  cost per request (USD)          0.00214     0.00191     -11%

  judged (n=310 after rungs 0-2):  better 96 · same 141 · worse 73

  Read these: 20 worst-judged pairs, all 54 new schema failures,
  all 37 new refusals.  → replay/v4/regressions.html

Two things in that table are worth noticing as a pattern. The output token increase of 42% against a cost decrease of 11% means the candidate’s lower headline price is being substantially eaten by verbosity — the comparison you want is always cost per request on your own corpus, never price per million tokens. And a rise in truncation alongside longer outputs usually means the candidate needs a higher max_tokens, which is a configuration change to make before concluding anything about quality.

Budget the run before you start it

Replay costs real money, twice: once for the candidate, and again for the judge, which is often the larger of the two because it reads two full outputs per pair. Compute it up front from the corpus:

select count(*)                                  as requests,
       sum(input_tokens)                         as input_tokens,
       round(avg(output_tokens))                 as avg_output,
       -- candidate cost, at rates you fill in per model
       round(sum(input_tokens) / 1e6 * :in_rate
           + count(*) * avg(output_tokens) / 1e6 * :out_rate, 2) as est_candidate_usd
from llm_request
where request_id in (select request_id from replay_corpus where version = 4);

Then decide the corpus size against that number rather than the other way round. A 2,000-request corpus that you can afford to run weekly is worth far more than a 50,000-request corpus you run once and never repeat — the value of a replay corpus comes from running it against every candidate, every prompt change and every provider incident, and that only happens if a run is cheap enough to be routine.

One more constraint people hit late: rate limits. Two thousand requests fired as fast as your client allows will be throttled, and a replay that half-fails on 429s produces a result set biased toward whatever succeeded. Run the replay with concurrency low enough to stay inside quota, treat throttled requests as retryable rather than as failures, and record which requests never completed so the report can say so instead of quietly shrinking the denominator.