Cached Tokens: The Cheapest Tokens You’ll Ever Buy
5 min read · updated August 3, 2026
Prompt caching is the rare optimisation with no quality cost: the same model, the same output distribution, a fraction of the price. The catch is that the hit rate is entirely determined by how you assemble your prompt, and most prompts are assembled in a way that guarantees a miss.
What is being cached
Not the response. What is cached is the key/value tensors produced during prefill for a prefix of your prompt. When a later request begins with exactly the same tokens, the server reuses those tensors instead of recomputing them, and skips straight to the part of the prompt that differs.
Three properties fall directly out of that and they explain every rule that follows:
- It is a prefix match, from token zero. Not a substring match, not a semantic match. One changed character at position 5 invalidates everything after it.
- It is per tokenizer and per model. KV tensors are model-specific. Routing the same prompt to a different model is a cold cache, which matters if you use automatic failover.
- It expires. The entry occupies real accelerator memory, so lifetimes are minutes by default, typically extended each time the entry is used.
The shape of the pricing
The structure is stable across vendors even though the numbers move, so learn the structure. Three rates exist where a normal model has one: a write multiplier for the request that populates the cache, a read multiplier for requests that hit it, and on some platforms a storage charge per token-hour for explicitly managed caches.
As of mid-2026, the published documentation of the major providers described these shapes — verify against the pricing page before you build a budget on them, because this is the most frequently revised part of any price list:
| Shape | Description |
|---|---|
| explicit breakpoints | Anthropic’s cache_control markers let you nominate where the cacheable prefix ends. Documented as a write premium above the base input rate and a read rate an order of magnitude below it, with a short default lifetime refreshed on use and a longer paid option. |
| automatic prefix | OpenAI caches eligible prefixes above a minimum length with no API changes at all, and reports the count back as prompt_tokens_details.cached_tokens. Nothing to configure, and correspondingly nothing to control except prompt order. |
| managed cache object | Google’s context caching creates a named cache resource with an explicit TTL, billed for storage as well as for discounted reads. Suits a large static corpus reused over hours. |
Break-even, in two lines
Let w be the write multiplier and r the read multiplier, both relative to the ordinary input rate, and let N be the number of requests that share a prefix before it expires. Caching pays when
w + (N - 1) * r < N (cached cost < uncached cost) => N > (w - r) / (1 - r) with w = 1.25, r = 0.1: N > 1.28
So with the multipliers commonly published, the second request against a prefix already pays for the write. That is the number worth internalising, because it means the question is never “is caching worth it” — it is “does my prefix survive long enough to be hit twice”.
For a realistic estimate, add a hit rate h. The effective input multiplier across all requests is h × r + (1 - h) × w for the cacheable portion, which at r = 0.1 and w = 1.25 is below 1.0 for any h above about 22%. Below that, caching costs you money — which is exactly the situation a per-user prompt prefix produces.
A worked month
Assumptions, all of which you replace: a 10,000-token prefix of system prompt, tool schemas and few-shot examples; 100 requests an hour, around the clock; a base input rate of $3.00 per million; the multipliers above; and a lifetime long enough that continuous traffic needs roughly one write per hour.
prefix cost, uncached: 10,000 / 1e6 * $3.00 = $0.0300 / request prefix cost, cache read: 0.1x = $0.0030 / request prefix cost, cache write: 1.25x = $0.0375 / write per day, 2,400 requests, 24 writes: uncached 2,400 * $0.0300 = $72.00 cached 24 * $0.0375 + 2,376 * $0.0030 = $8.03 30 days: $2,160 -> $241 (about 89% off the prefix line)
Note what is not in that saving: the per-request variable content and the entire output line, neither of which is cacheable. If your prefix is 10,000 tokens and your variable content is 30,000, the headline saving shrinks accordingly. Compute it on the split you actually have.
What silently destroys your hit rate
These are all real, all common, and all invisible unless you read the usage fields:
- A timestamp in the system prompt. “Today’s date is …” at the top of the prompt gives you a 0% hit rate, permanently. Move it to the end, after the cacheable block.
- Non-deterministic JSON. Tool schemas serialised from a dict with unstable key order, or floats formatted differently between runs, produce a different byte string each time. Serialise with sorted keys.
- User-specific content first. Any per-user preamble before the shared block means each user has their own cache entry, and the shared block is never shared.
- A prefix below the minimum. Providers set a floor — typically in the high hundreds or low thousands of tokens — below which nothing is cached. A 400-token system prompt may simply be ineligible.
- Retrieved documents above the instructions. The most common RAG layout puts the variable part first. Invert it: static instructions and schemas, then documents, then the question.
- Cross-model failover. A retry that lands on a different model is a cold cache and a write charge.
- Low traffic. If requests arrive less often than the TTL, every request is a write. This is the case where caching is actively more expensive.
- A/B tests on the prompt. Two variants halve each variant’s effective traffic against the cache; four quarter it.