Shadow Traffic: Testing a New Model on Real Requests

Shadow traffic answers a question no offline eval can: how would the candidate model handle the requests your users actually send, in their real distribution, including the strange ones. It is also the easiest way to double your inference bill and leak data into a system nobody reviewed.

What shadowing is, and what it is not

Shadowing means sending a copy of a production request to a second model and discarding the result — the user never sees it, never waits for it, and is unaffected if it fails. It is not an A/B test: nobody is served by the candidate, so you learn nothing about user behaviour or downstream outcomes. What you learn is the distribution of the candidate’s outputs, latency and cost on real inputs.

That makes it the step between an offline eval on a curated set and a canary that actually serves users. Offline evals tell you about the cases you thought of. Shadowing tells you about the cases you did not — the 40,000-token prompt, the request in a language you did not test, the one that trips a refusal.

Five rules that keep it safe

• Never on the critical path. The shadow call is fire-and-forget after the response has been returned. If it shares a connection pool, a rate limiter or a semaphore with production, it is on the critical path whether you meant it to be or not.
• Bounded queue, drop on full. Under load the candidate will be slower than production sometimes. A queue that grows is a memory leak with a deadline; dropping shadow requests is free and correct.
• Hard budget. Shadowing at 100% doubles spend on that feature. Sample, and cap the sampled spend explicitly, with the cap enforced in code rather than in a runbook.
• Stub every side effect. See below. This is the one that causes incidents.
• Same redaction as production logging. The shadow path is a second copy of user content going somewhere new, and it deserves the same scrutiny as the first.

The mirroring middleware

type ShadowConfig = {
  enabled: boolean;
  candidateModel: string;
  sampleBps: number;        // basis points of eligible requests
  maxQueueDepth: number;
  hourlyBudgetUsd: number;
};

const queue: Array<() => Promise<void>> = [];
let inFlight = 0;
let spentThisHour = 0;
const MAX_CONCURRENT = 4;

export function maybeShadow(req: LlmRequest, primary: LlmResponse, cfg: ShadowConfig) {
  if (!cfg.enabled) return;
  if (hash32(req.requestId) % 10000 >= cfg.sampleBps) return;
  if (spentThisHour >= cfg.hourlyBudgetUsd) { metrics.shadowSkipped.add(1, { reason: "budget" }); return; }
  if (queue.length >= cfg.maxQueueDepth) { metrics.shadowSkipped.add(1, { reason: "queue" }); return; }

  queue.push(async () => {
    const started = Date.now();
    try {
      // Same resolved body. Tools replaced with recorded responses.
      const shadow = await callModel({
        ...req.body,
        model: cfg.candidateModel,
        tools: req.body.tools ? freezeToolsToRecorded(req) : undefined,
      }, { timeoutMs: 30_000, tag: "shadow" });

      spentThisHour += shadow.costUsd;
      await recordComparison({
        requestId:     req.requestId,     // joins back to the production row
        traceId:       req.traceId,
        primaryModel:  primary.servedModel,
        shadowModel:   shadow.servedModel,
        primaryTokens: primary.outputTokens,
        shadowTokens:  shadow.outputTokens,
        primaryMs:     primary.durationMs,
        shadowMs:      Date.now() - started,
        shadowCostUsd: shadow.costUsd,
        shadowValid:   validateSchema(shadow.text),
        shadowRefused: looksLikeRefusal(shadow.text),
        shadowRef:     await putRedacted(shadow.text),
      });
    } catch (err) {
      // A shadow failure is data, not an incident.
      metrics.shadowError.add(1, { error_type: err.constructor.name });
    }
  });
  pump();
}

function pump() {
  while (inFlight < MAX_CONCURRENT && queue.length > 0) {
    const job = queue.shift()!;
    inFlight++;
    void job().finally(() => { inFlight--; pump(); });
  }
}

Note what is deliberately absent: no await reaching the caller, no shared retry policy with production, no throw that can escape. The worst thing this code can do on its worst day is stop shadowing.

Sampling on a hash of the request id rather than a random draw is a small choice with a useful property: the same request is either shadowed or not, deterministically, so a rerun of the same traffic selects the same subset. It also lets you widen the sample later without losing comparability, because the requests already in the sample stay in it as sampleBps rises.

The concurrency limit is set independently of production’s because it is protecting something different. Production concurrency protects user latency; shadow concurrency protects your rate limit headroom, which is shared between the two. A shadow path that consumes your quota during a traffic peak causes 429s on the requests that matter, which is the one way this design can still hurt users despite everything above.

Side effects: the thing that ruins it

A model that only produces text is safe to shadow. A model with tools is not. If the candidate decides to call send_email, create_ticket or refund_order, and your tool executor is the real one, you have just performed a production action on behalf of a test.

There are two workable approaches, and a third that only looks workable.

• Record and replay. Capture the tool calls and responses from the production run, and serve the candidate the recorded response when it asks for the same call. Comparable to how HTTP fixtures work in tests. It works well when the candidate makes the same calls and degrades to the next option when it does not.
• Refuse and record. Return a well-formed error to any tool call the candidate makes that has no recording, and count it. “The candidate wanted to call a tool the production model did not” is itself one of the more interesting findings you can get from shadowing.
• A read-only tool executor sounds safe and usually is not, because “read-only” is a property nobody audited across every tool, and one of them writes an audit log, increments a counter or costs money per call.

The second-copy problem

Shadowing sends user content to a provider that may not be in your current data-processing agreements, in a region you have not checked, under a retention policy nobody read. That is a compliance decision, not an engineering one, and it is much easier to get right before you turn it on than to unwind afterwards.

The practical checklist: confirm the candidate provider is covered by the same agreements and sub-processor disclosures as the incumbent; exclude tenants whose contracts restrict processing; apply the same capture-time redaction to anything you store from the shadow path; and give shadow comparison rows a short TTL, because their value is entirely in the week you are running the comparison.

Reading the results

The comparison table joins back to the production row on request_id, so the questions are paired rather than distributional — which is the whole advantage over comparing two separate benchmark runs.

select count(*)                                                as n,
       round(avg((not shadow_valid)::int)::numeric, 4)         as shadow_schema_fail,
       round(avg(shadow_refused::int)::numeric, 4)             as shadow_refusal,
       round(avg(shadow_ms - primary_ms))                      as mean_latency_delta_ms,
       percentile_cont(0.95) within group (order by shadow_ms) as shadow_p95_ms,
       round(sum(shadow_cost_usd) / nullif(count(*), 0), 6)    as shadow_cost_per_req,
       round(avg(shadow_tokens::numeric / nullif(primary_tokens, 0)), 3) as verbosity_ratio
from shadow_comparison
where created_at >= now() - interval '7 days';

Three of those columns tend to decide the migration on their own. A verbosity ratio above 1 means the candidate’s headline per-token price understates what you will pay. A refusal rate materially above the incumbent’s means user-visible behaviour change regardless of benchmark scores. And the p95 rather than the mean latency is what your users will experience.

What this cannot tell you is whether the candidate’s answers are better. Nothing here grades correctness; for that you need a rubric over a sampled subset, or a canary that serves real users and measures what they do. Shadowing derisks the migration. It does not decide it.

Know when to stop, too. Shadowing is a temporary instrument: it costs money continuously, it holds a second copy of user content, and its findings converge after a few days of representative traffic. Give each shadow run an end date when you start it, and wire the enable flag so that it defaults back to off rather than staying on because nobody remembered. A shadow path left running for a quarter is a recurring expense that has stopped producing information.