Prompt Caching: How the Major Implementations Differ

Every implementation of prompt caching does the same physical thing — keeps the KV entries for a prefix so the next request does not have to recompute them. What differs is who decides what gets cached, what silently destroys it, and how the saving is billed. Those three are worth more attention than the discount percentage.

Dating this page. Every figure below was read from the named vendor’s public documentation on 3 August 2026 and is quoted as documentation, not as measurement. These are the fastest-moving numbers in this entire library — minimum token counts, TTLs and multipliers all changed more than once in the preceding two years. Treat the mechanisms as durable and re-check every number against the source before you rely on it.

One mechanism, three control models

The underlying trick is the KV cache again. Prefill computes a key and value vector per token per layer; those depend only on the tokens up to that point, so if the next request starts with the identical prefix, the identical vectors would be produced. Keeping them turns a recomputation into a memory read. That is why the saving lands entirely on input tokens and entirely on time-to-first-token — it cannot make generation faster.

Because it is a prefix cache, it is exact and positional. A single changed token anywhere in the prefix changes every subsequent key and value, so there is no partial credit and no fuzzy matching. If you want fuzzy matching you want a different technique — semantic caching — with an entirely different risk profile.

The control models divide into three:

• Explicit breakpoints. You mark where the cacheable prefix ends. More work, and full control over what is written and when. Anthropic’s cache_control is the reference implementation of this style.
• Automatic. The provider hashes prefixes and caches eligible ones with no code change. Nothing to integrate, and correspondingly little control — with the practical consequence that you can lose the cache without doing anything visible.
• Hybrid. Automatic by default with optional explicit breakpoints, which is where OpenAI’s newer models landed, and implicit-plus-explicit as Google documents for Gemini.

What the docs said on 3 August 2026

The structural reading matters more than the cells. Explicit caching costs you a code change and a premium on the write, and gives you certainty about what is cached and for how long. Automatic caching is free to adopt and gives you a hit rate you observe rather than control. Neither is better; they suit different amounts of engineering attention.

Invalidation is what decides your hit rate

This is the part that turns a promised discount into an actual one, and it is where documentation repays reading closely. Anthropic documents an explicit hierarchy — tools, then system, then messages — where a change at one level invalidates that level and every level after it. Change a tool definition and you have invalidated everything, including the long system prompt that had nothing to do with it.

The same documentation lists a set of non-obvious invalidators: toggling web search or citations, changing thinking or effort settings, adding or removing images, and changing the speed setting all invalidate at least the message blocks. The general lesson generalises past any one vendor — anything that changes what the model sees, or how the request is configured, can change the hash — and the practical consequences are the same everywhere:

• Never put a timestamp near the front of a prompt. A date-and-time in the system message is the single most common cause of a 0% hit rate. It invalidates every request, forever, and looks entirely innocent in review.
• Order by volatility. Most stable first: tool definitions, then the system prompt, then retrieved documents, then the conversation, then the user’s new turn. Any deviation puts volatile content inside the prefix you were hoping to cache.
• Watch out for non-deterministic serialisation. A JSON blob whose key order varies between runs produces a different prefix each time even though the content is identical. Sort keys.
• Routing can lose the cache. The cache lives on a specific serving instance or shard. Requests distributed across replicas may miss simply because they landed elsewhere, which is what a cache key or session-affinity mechanism is for.

The break-even arithmetic

With an explicit cache that charges a write premium, whether to cache at all is a calculation rather than a preference. Using the multipliers Anthropic documents, and writing N for the number of requests that share the prefix within the TTL, in units of the base input price:

uncached      = N * 1.0
cached (5m)   = 1.25 + (N - 1) * 0.1
cached (1h)   = 2.00 + (N - 1) * 0.1

5-minute cache pays off when   1.25 + 0.1(N-1) < N
                               1.15 < 0.9 N        ->  N > 1.28  ->  2 requests
1-hour cache pays off when     2.00 + 0.1(N-1) < N
                               1.90 < 0.9 N        ->  N > 2.11  ->  3 requests

At N = 20 on a 10k-token prefix, in base-input units:
  uncached      = 20.0
  cached (5m)   = 1.25 + 1.9 = 3.15    -> 84% off the prefix
  cached (1h)   = 2.00 + 1.9 = 3.90    -> 81% off the prefix

Two readings. The break-even is astonishingly low — a second request inside the window already pays for the write — so for any repeated prefix the answer is simply yes. And the choice between TTLs is not about the discount, which is nearly identical at volume; it is about whether your traffic has gaps longer than five minutes. A steady stream refreshes a short cache for free; bursty or per-user traffic does not, and pays the write premium repeatedly unless you buy the longer window.

Designing a prompt to be cacheable

The design rule is one sentence: a prompt is a stable prefix followed by a volatile suffix, and your job is to make the boundary as late as possible. Concretely, that means hoisting anything shared into the front, resisting the urge to personalise the system message with a name or a date, keeping retrieved documents in a fixed order across turns of the same conversation, and appending new context rather than reflowing the whole prompt.

Then verify rather than assume. Every implementation reports cache hit sizes in the usage block of the response; log that number alongside the request and treat a falling hit rate as a regression with a cause. Prompt caching is the rare optimisation that is free, large, and silently broken by a one-line change, which is exactly the combination that requires a metric rather than an intention.