What Happens to Cache Hit Rates Right After a Provider Migration

Caches are per-provider and per-model. The moment you change either, every entry you had is unreachable, and the first request on every distinct prefix is a miss that also pays a write premium. The important question is not whether the hit rate drops — it must — but how long it should take to come back, and whether yours is coming back at all.

The drop that is supposed to happen

At the instant of cutover the new provider has never seen any of your prefixes. Every request in the first minutes is a cache miss. Worse than a miss, in cost terms: on an explicit-breakpoint API, writing an entry costs more than an uncached request would have, because the provider is doing the prefill and storing the result. So the first window after a migration is not just the absence of a discount, it is a surcharge.

This is why the honest expectation for the first hour is: cost per request above the old baseline, not merely equal to it. Teams that have not modelled this see the spike, assume the migration was a mistake, and roll back before the cache has had time to fill.

What steady-state hit rate is a function of

Once things settle, the hit rate is determined by three quantities and nothing else:

• P — the number of distinct cacheable prefixes your traffic produces. One shared system prompt is P = 1. A per-tenant system prompt across 400 tenants is P = 400.
• λ — the request rate per prefix, in requests per second. Note: per prefix, not total. A thousand requests a minute spread across 400 tenants is a very different situation from a thousand against one prefix.
• T — the cache TTL in seconds, refreshed on each read where the provider does that.

A prefix stays warm as long as the gap between consecutive requests touching it is shorter than T. So the governing quantity is λT: the expected number of requests arriving during one TTL window on a single prefix. If λT is comfortably above one, that prefix is warm essentially all of the time and its hit rate approaches one. If λT is below one, the entry usually expires before the next request arrives, and the prefix is effectively uncacheable no matter what you do to the prompt.

That single ratio explains most “caching does not work for us” reports, and it explains why a migration between providers with different TTLs can genuinely change the answer. Halving T halves λT. A workload sitting at λT just above one on a long TTL falls below it on a short one, and the hit rate does not recover — not because anything is broken, but because the arithmetic changed.

The warm-up window itself is the time for each prefix to be touched once. For a prefix with rate λ, that is on the order of 1/λ seconds; across P prefixes with mixed rates, the overall hit rate approaches its steady value as the long-tail prefixes get their first touch. For a single shared system prompt under any real traffic this is seconds. For a per-tenant prefix where the median tenant sends one request an hour, full warm-up is measured in hours to days, and the steady-state hit rate will be low regardless.

The break-even arithmetic

To answer “when does the migration pay for itself” you need three multipliers, all relative to the normal input price of the same tokens. Take them from your provider’s pricing page and substitute:

• W — cost of writing a cache entry, as a multiple of normal input price for those tokens.
• R — cost of reading a cached entry, same basis.
• N — the number of reads a single written entry serves before it expires.

For a prefix of k tokens, one write plus N reads costs k(W + NR). The same N+1 requests with no cache at all cost k(N + 1). Caching wins when W + NR < N + 1, which rearranges to N > (W − 1) / (1 − R).

That inequality is the whole thing. Read it carefully: the break-even read count depends only on the two multipliers, not on the size of the prefix. Prefix size determines how much you save, not whether you save. And because R is typically very small, the denominator is close to one, so the break-even is approximately W − 1 reads.

Applying this to the migration window: the total extra cost of the cold start is roughly P × k × (W − 1) — the write premium, paid once per distinct prefix. For a single shared prefix that is a rounding error. For a large P it is a real number, and it is worth computing before the cutover so that nobody has to argue about the spike afterwards.

Warm-up or structural break?

After an hour or two, the two situations are easy to tell apart if you are looking at the right number. Plot cached input tokens as a fraction of total input tokens, per minute.

• A warm-up rises. It starts at zero, climbs steeply as the high-traffic prefixes fill, then flattens as the long tail gets its first touch. If the curve is going up at all, wait.
• A structural break is flat at or near zero and stays there. The cache is never being read because it is never matching. Something in the prefix differs between consecutive requests, or the prefix is below the minimum cacheable length, or on an explicit-breakpoint API there is no breakpoint at all.
• A partial break is a flat line at a value well below the old one. Usually this means the cacheable prefix got shorter — the breakpoint landed earlier than it used to, or content that was previously inside the cached region is now outside it.

For the flat-at-zero case there is a decisive test that takes two minutes: send the identical request twice in a row from a script and read the cache fields on the second response. If the second reports zero cached tokens, the problem is not traffic patterns and not warm-up — the prefix is not matching itself, which means something non-deterministic is being rendered into it. Diff the two serialised request bodies byte for byte. The culprit is nearly always a timestamp, a UUID, or an unsorted serialisation. The structural fixes are covered in porting a caching setup between providers.

Shortening the window

If a cold start is genuinely expensive for your P, you can pay it deliberately rather than during peak traffic.

• Pre-warm before cutover. Send one request per distinct prefix against the new provider ahead of the switch, with the smallest possible output. The write happens on your schedule instead of your users’.
• Do not fan out on a cold prefix. An entry becomes readable only once the first response has begun; N parallel requests on a cold prefix all miss and all pay the write. Send one, wait for it to start returning, then release the rest.
• Cut over gradually. Moving 10% of traffic warms the same prefixes as moving 100%, at a tenth of the exposure if something is wrong.
• Reduce P before you migrate. If per-tenant prefixes are the reason your hit rate is low, factor the shared portion to the front and put the tenant-specific part after the breakpoint. That change is worth more than any provider choice.