What “Context Caching” Migration Actually Requires

Caching is the feature most likely to survive a migration in name and die in fact. Nothing errors when it stops working; the requests succeed, the answers are correct, and the bill goes up.

Two families of caching

Providers implement roughly two designs, and which one you are moving between decides how much work the migration is.

Implicit prefix caching

The provider hashes the leading portion of your request and reuses previously computed state if it matches. There is no API surface: no field to set, nothing to create or delete, and nothing to get wrong except the shape of your prompt. OpenAI’s documentation describes automatic caching of this kind, matched on prefixes above a minimum length, with the hit reported back in the usage object. The library covers the mechanism in automatic prompt caching.

Explicit caching

You mark what to cache. Anthropic’s documentation describes per-block cache_control markers with a limited number of breakpoints per request, where a breakpoint caches everything before it. Google’s Gemini API documents a different explicit shape again: a cached-content resource you create, give a TTL, and then reference by handle on subsequent requests.

The migration directions are asymmetric. Explicit to implicit is mostly deletion — remove the markers, keep the prompt structure, and the benefit continues if the prefix is long and stable enough. Implicit to explicit is the hard direction, because a decision the old provider made for you (where the reusable part ends) now has to be made by you, in a fixed and small number of places, and made correctly or you get nothing.

What a stable prefix really means

Every design in both families requires the cached region to be byte-identical across requests. Not semantically equivalent — identical. The practical consequence is a rule about ordering: every part of the prompt that varies per request must come after every part that does not, with no exceptions, because a single differing byte early in the prompt invalidates everything after it.

The things that break it are almost always accidents rather than decisions:

• A timestamp or date in the system prompt. The single most common cause. “Today is 12 August 2026, 09:41” at the top of the system prompt gives you a cache hit rate of zero, and rounding it to the day gives you one that resets at midnight.
• A tenant or user identifier interpolated into the preamble. Move it to the end, or into the user turn.
• Tool definitions serialised in nondeterministic order. Tool schemas are usually part of the prefix, and if they are built from a map with unstable iteration order or serialised by a JSON encoder that does not fix key order, the bytes differ between processes even though the content does not. Sort keys and fix the order explicitly.
• Retrieved documents in retrieval order. RAG context is variable by nature; the mistake is putting it before the static instructions rather than after them.
• Few-shot examples sampled per request. Randomised example selection and caching are directly opposed. Pick one.

A useful pre-migration exercise costs an hour: capture a few hundred real outbound request bodies, compute the length of the longest common prefix across them, and compare it to the target provider’s minimum cacheable length. If the common prefix is shorter than the floor, no amount of configuration will produce a hit, and the work is in restructuring the prompt rather than in the caching API.

Floors, breakpoints and TTLs

Three numbers govern whether a structurally correct setup actually caches, and all three differ by provider and by model.

• The minimum cacheable length. Caching does not engage below a floor measured in tokens. At the time of writing, OpenAI documents automatic caching for prompts above roughly a thousand tokens, and Anthropic documents minimum cacheable prefix lengths that differ by model, with smaller models requiring a longer prefix than larger ones. A prompt that was comfortably above one provider’s floor can sit below another’s.
• The breakpoint budget. Explicit designs limit how many cache markers one request may carry. When you arrive from an implicit provider you have no markers at all and must choose placements: typically one after the tool definitions and one after the static instruction block, leaving the variable material uncached.
• The lifetime. Short-lived caches — of the order of minutes, often refreshed on each hit — against explicitly created caches with a TTL you set. This is the number that decides whether your traffic pattern benefits at all.

The TTL interacts with your arrival rate in a way worth writing down. If requests sharing a prefix arrive independently at rate lambda and the cache lives for T after the last use, the probability that a given request finds a warm cache is the probability that the previous one arrived within T. Under a Poisson arrival assumption — which is an assumption, and is wrong for bursty batch traffic — that is:

P(hit) = 1 - exp(-lambda * T)

lambda = 1/minute,  T = 5 minutes   ->  1 - exp(-5)     = 0.993
lambda = 1/minute,  T = 1 minute    ->  1 - exp(-1)     = 0.632
lambda = 1/hour,    T = 5 minutes   ->  1 - exp(-1/12)  = 0.080

The third line is the one that catches people. A low-traffic tenant gets almost no benefit from a short-lived cache no matter how perfect its prompt structure, which means caching economics are per-tenant, not per-application. If your traffic is unevenly distributed across tenants, model the busy ones and the quiet ones separately.

The economics change shape

Implicit designs typically discount cached input and charge nothing extra to populate the cache; explicit designs typically charge a premium to write and a steep discount to read. Those are different decision problems, and only the second one has a break-even.

Let P be the size of the cacheable prefix in tokens, n the number of requests that use it inside one cache lifetime, w the write multiplier and r the read multiplier, both expressed relative to the ordinary input price. Then:

uncached cost   = n * P
cached cost     = w * P + (n - 1) * r * P

cached < uncached  when  w + (n - 1) * r  <  n
                          n * (1 - r)     >  w - r
                          n               >  (w - r) / (1 - r)

with w = 1.25 and r = 0.1  (the multipliers Anthropic documents
at the time of writing for a write and a read):
                          n > 1.15 / 0.9 = 1.28

so the second read inside the lifetime already pays.

That result is the useful one, and it is robust: for any read multiplier well below one and any write multiplier near one, the break-even is under two reads. The risk in explicit caching is therefore not that the premium is too high — it is that you write caches that are never read, because the TTL expires first or the prefix is not as stable as you believed. Every wasted write costs w times the ordinary price for nothing.

Which is why the hit-rate formula above and the break-even formula have to be used together. Multiply them: the expected cost per request under caching is P * (r * p_hit + w * (1 - p_hit)), and with w above 1 there is a hit rate below which caching is worse than not caching at all. Solve for it and you get p_hit > (w - 1) / (w - r), which for the multipliers above is about 0.22. Below roughly a fifth of requests hitting, an explicit cache is costing you money.

Proving a hit after the migration

Because nothing errors, the only evidence is in the usage object, and the field names differ. OpenAI reports cached input tokens under usage.prompt_tokens_details.cached_tokens. Anthropic reports two separate fields, usage.cache_creation_input_tokens and usage.cache_read_input_tokens, which is more informative — you can see writes and reads independently, and the wasted-write problem above becomes directly measurable as writes without matching reads.

Make this an assertion rather than a dashboard. Add a smoke test that sends the same prompt twice in quick succession and requires the second response to report a non-zero cached count; run it against every provider binding after every prompt change. A prompt edit that moves a variable token above the breakpoint is a normal-looking commit that silently multiplies your input bill, and this test is the only thing that catches it on the day.

Two more things to check while you are there. Caches are scoped by model string, so a model version bump invalidates everything and your first minutes on a new version will look expensive — expected, not a regression. And caches are scoped to your organisation or project, so in a multi-tenant system a shared prefix is shared across tenants by default, which is efficient but is a decision your data-handling review should have made deliberately rather than inherited.