Dual-Running Two Providers During a Cutover

Running the old and new provider at once is two separate techniques that get called one name. Doing them in the wrong order is why dual runs produce a pile of data and no decision.

Two shapes, two different questions

Shadowing serves the user from the old provider and additionally sends the same request to the new one, discarding the new response and recording it. The user is never exposed to the new provider. It answers structural questions: does the response parse, does it satisfy the schema, does it select the right tool, does the request shape even work.

Splitting serves a fraction of real traffic from the new provider. The user is exposed. It answers the questions shadowing cannot: does the answer help, does the conversation stay coherent over several turns, does the tail latency hold under real concurrency, do people complain.

The order is shadow first, split second, and the reason is economic. Every defect shadowing can find is a defect that would have reached a user under a split, and shadowing finds them at zero user risk. Going straight to a split means spending user trust on discovering that your tool schema is rejected — something a shadow run would have told you on the first hundred requests. Conversely, extending shadowing in the hope that it will eventually answer a question about user satisfaction is a category error; it cannot, because nobody ever saw the output.

Shadowing without touching the user’s latency

The shadow call must be incapable of affecting the request it shadows. That means it is dispatched after the primary response has been committed to the user, on its own budget, with its own timeout, and with every failure swallowed into a metric rather than an exception. A shadow that can fail a request is worse than no shadow, because you have taken on the new provider’s availability without any of the benefit.

async def handle(req):
    primary = await old_provider.complete(req)   # what the user gets
    schedule_background(shadow(req, primary))    # fire and forget
    return primary

async def shadow(req, primary):
    started = monotonic()
    try:
        async with timeout(SHADOW_TIMEOUT):      # its own budget
            candidate = await new_provider.complete(strip_side_effects(req))
    except Exception as e:
        metrics.increment("shadow.error", tags={"kind": type(e).__name__})
        return
    finally:
        metrics.timing("shadow.latency", monotonic() - started)

    store_pair(
        request_id = req.id,
        prompt_hash = hash_prompt(req),          # the join key, later
        primary = primary,
        candidate = candidate,
    )

The prompt_hash is the field people leave out and regret. Weeks later you will want to group pairs by the prompt template and version that produced them, and without a stable hash recorded at the time there is no way to reconstruct the grouping. Store both raw responses in full, both usage blocks, and both stop reasons, because you cannot re-derive them and re-running the pair costs money and no longer reproduces the same traffic.

Splitting live traffic by a stable key

When you do start exposing users, the assignment must be stable per user, tenant or conversation — never random per request. Randomising per request means a user’s follow-up question is answered by a different model than their first, which breaks multi-turn coherence, defeats prompt caching, and produces bug reports that cannot be reproduced because the reporter never hit the same path twice.

def use_new_provider(tenant_id: str, pct: int) -> bool:
    # stable across requests, uniform across tenants, no shared state
    h = int(blake2b(tenant_id.encode(), digest_size=8).hexdigest(), 16)
    return (h % 100) < pct

Ramp in held steps — 1, 5, 25, 50, 100 is a reasonable ladder — and make the length of each hold a function of your slowest detector rather than of your confidence. If a 500 shows up in seconds but a support ticket takes two days, a four-hour hold has only tested for the first class of problem. This is the same discipline as a model canary release, and the percentage should be a runtime configuration value so that dropping to zero is immediate.

Record the assignment on every request. Without a field saying which provider served it, your dashboards mix the two populations and every metric becomes a weighted average of two things you were trying to tell apart. Split every quality and latency metric by that field from the first day of the ramp, not from the day someone asks.

What it costs, and how to bound it

Shadowing at 100% doubles the token bill for shadowed traffic and adds the new provider’s full request volume on top of your existing one. Splitting does not increase the bill much, since each request is served once, but it does move spend between two invoices in a proportion that changes daily.

Bound the shadow cost by sampling, and sample stratified rather than uniformly. A uniform 5% sample of production traffic is 95% the same three request shapes and almost none of the rare ones — and the rare shapes are where migrations break, because they are the long prompts, the unusual tool sequences and the non-English inputs. Bucket requests by shape (template id, presence of tools, input length band, language) and sample a fixed number from each bucket per hour. You will shadow a small fraction of traffic and a large fraction of the behaviour space.

The three traps

• Side effects. This is the one that causes real damage. If the model call drives tool execution, a shadow run must not execute the tools — otherwise every shadowed request sends a duplicate email, writes a second row, or charges a card twice. Compare the tool call the model proposed, not the result of running it: stub the tool layer for the shadow path, return canned results, and treat the comparison as being about the arguments. If the loop is multi-turn and each turn depends on real tool output, shadow only the first turn, or replay recorded tool results.
• Server-side state. Any feature where the provider holds the conversation — a thread, an assistant, a stored response id, a server-side cache handle — does not exist on the other side and cannot be shadowed as-is. Either restrict the dual run to stateless calls where you send the full transcript yourself, or reconstruct the transcript from your own logs before dispatching the shadow. Discovering this halfway through is a rewrite of the harness.
• The new provider’s cold rate limit. A new account starts on a low tier. Shadowing production traffic into it produces a wall of 429s, and those failures will be read as reliability problems with the new provider when they are your provisioning problem. Request the limit increase before the shadow starts, ramp the shadow percentage the same way you would ramp a split, and make sure your backoff on 429 is in the shadow path too — but capped, because a shadow that retries forever competes with your real traffic for the same connection pool.

One smaller trap worth a sentence: prompt caching. The shadow’s traffic pattern does not warm the same caches as production, so the new provider’s cost and latency during a shadow run are both worse than its steady-state numbers would be. Do not use shadow-run latency as your estimate of post-cutover latency; measure that during the split, when the traffic is real. What to do with all the pairs you have collected is the diffing harness.