Running a Shadow Deployment Without Doubling Your Bill

Mirroring every request to a second prompt costs roughly what your inference bill costs, which is why most teams read about shadow deployment and then do not do it. The sample size a comparison actually needs is usually a few hundred requests, and a few hundred requests is not a doubling of anything.

Why shadow at all

A shadow deployment sends a copy of live traffic to the new prompt and throws the output away. Nobody sees it, so the blast radius is zero, and that is the property worth paying for. A canary trades user harm for a decision; a shadow trades money for the same decision. Where a prompt change is risky enough that you do not want any user to see it before you have evidence, shadow is the only instrument that gives you evidence on real inputs.

It also removes a confound that canaries have. Both arms see the exact same request, so you can compare paired outputs rather than two samples from possibly different populations — and a paired comparison needs meaningfully fewer observations for the same power, because the input variation cancels.

What it cannot tell you is anything downstream of the user. No conversion, no thumbs-down, no follow-up turn, no interaction with the rest of the session, because the shadow response never entered the conversation. Shadow answers “is the output different, and worse by some automatic measure”. The rest still needs a canary.

Turning a sample size into a sample rate

Start from the sample size, not from a percentage. Suppose the signal is schema-validation failure, with a baseline of 2%, and you want to detect a doubling to 4% at the conventional 5% significance level with 80% power. Because the baseline arm here is your entire live traffic rather than a matched slice, the comparison is effectively one-sample against a well-established rate:

n_shadow = (z_a/2 + z_b)^2 * p2(1-p2) / (p2 - p1)^2
         = 7.849 * (0.04 * 0.96) / (0.02)^2
         = 7.849 * 0.0384 / 0.0004
         = 754 requests

Now convert that into a rate and a time, for a service handling 200,000 requests a day:

sample rate   shadow req/day   time to 754        cost multiplier
     100%         200,000        5.4 minutes           2.00x
      10%          20,000        54 minutes            1.10x
       1%           2,000        9 hours               1.01x
     0.1%             200        3.8 days              1.001x

One percent of traffic answers the question overnight and adds one percent to the bill. Mirroring everything answers it five minutes sooner per run and doubles the bill permanently, because nobody turns a mirror off. The only case for a high rate is when you are hunting rare failures rather than testing a rate — if the thing you are worried about occurs in one request in ten thousand, 754 samples will not contain one, and no amount of statistical care fixes that. Size the sample for the rarest event you care about, not the commonest.

Where the extra cost actually is

“Shadow costs 1× extra” is the estimate, and there are three reasons the real figure is higher. Each is worth checking before you set the rate.

The shadow request breaks your prompt cache

This is the big one and it is easy to miss. Prompt caching keys on an exact prefix. The entire point of the shadow run is that its system prompt differs from the live one, so the shadow request cannot hit the live request’s cache entry — it creates its own. Anthropic documents cache writes as charged at 1.25× the base input rate and cache reads at 0.1×, so a workload where live traffic reads a warm cache and shadow traffic writes a cold one is not paying 1× extra on input, it is paying something closer to 12× the input a live request costs. At a low sample rate, the shadow prefix may never warm at all, because entries expire on a short idle timeout.

Shadow output is not free just because nobody reads it

You are billed for generated tokens whether or not a user sees them, and a shadow prompt is frequently the more verbose one, since verbosity is a common side effect of adding instructions. The shadow arm can easily cost more per request than the live arm it is copying.

The comparison itself costs money

If you compare outputs with an LLM judge, that is a third inference call per sampled request, often on a larger model. Budget it explicitly. The cheap automatic prefilters in comparing shadow outputs without a human exist partly to keep the judge off most pairs.

Stratify instead of sampling uniformly

Uniform sampling spends your budget in proportion to how common a request type is, which is exactly backwards: the common path is the one you already have the most evidence about, and the tail is where prompt changes break things. Stratified sampling fixes this and usually improves the answer while cutting the cost.

Concretely: bucket incoming requests by something you already know before the call — intent label, locale, whether retrieval returned anything, whether the conversation has tool calls in it — and set a per-bucket rate.

bucket                 share of traffic   sample rate   shadow req/day
general chat                     88%             0.2%             352
tool-using turns                  8%               5%             800
non-English                     3.5%              20%           1,400
retrieval returned nothing      0.5%             100%           1,000
                                                              -------
                                                                3,552
uniform 1% of 200,000 would be:                                 2,000

Slightly more requests, and now every bucket has enough observations to say something, including the one that was 0.5% of traffic and would have contributed ten samples under uniform sampling. When you report an aggregate you have to weight back by the true shares, which is one multiplication and is the standard cost of stratification.

Cutting the per-request cost

• Cap max_tokens on the shadow call — but only if none of your comparison metrics is length-sensitive. Capping at 256 when the live call allows 2,048 saves most of the output cost and destroys your ability to compare truncation rates or output length, which are two of the more useful signals. Usually not worth it.
• Do not stream the shadow call. Streaming exists to improve perceived latency for a human, there is no human, and non-streaming responses are simpler to record and hash.
• Run the shadow call asynchronously, off the request path. Enqueue the recorded request and run it from a worker. This means a slow or failing shadow call cannot touch live latency, and it lets you drop shadow work under load rather than shedding real traffic.
• Record inputs once and replay them. If you have a request log, you may not need live mirroring at all — sampling yesterday’s requests gives you the same input distribution with no coupling to production. That is traffic replay, and it has one real disadvantage: replayed requests carry real user data into a fixture store, which is the subject of scrubbing PII from fixtures.
• Set a hard spend cap on the shadow path. A mirror with a bug is an unbounded bill, and it is the kind of bug that shows up on an invoice rather than in a dashboard.