Prompt Caching: Where the Savings Actually Come From

Prompt caching is the rare optimisation that costs almost nothing to adopt and can move a large fraction of a bill. It is also the one most often turned off by accident, by a single line at the top of a system prompt that nobody thought of as a cost decision.

What is actually being reused

When a model reads a prompt, it computes a key and a value vector for every token — the attention state, often called the KV cache. That computation is the prefill, and for a long prompt it is most of the work the request does before generation starts. If the next request begins with exactly the same tokens, that state is identical, so a provider can keep it and skip recomputing it.

Two things follow immediately, and they explain nearly every question people have about the feature. First, this is a compute saving, not a storage trick, so it reduces latency as well as price — time to first token drops on a hit. Second, the reused state is attention state and not text, so nothing about the semantics of the prompt matters. Two prompts that mean the same thing but differ by one token share nothing.

Prefix, not fuzzy match

The cache matches a prefix: the longest run of tokens from the very start of the request that is identical to something already cached. Not a substring in the middle. Not a similar passage. Not the same paragraph in a different position.

Which means one changed token invalidates everything after it. Put the current timestamp in the first line of your system prompt and the entire prompt is uncached on every request, forever, silently, with no error and no warning — the bill is simply higher than it should be. The same is true of a user’s name, a request id, a randomised greeting, a “today is” line, or a tools array that your code builds by iterating a hash map with non-deterministic order.

That last one deserves naming, because it is invisible in review: if the serialisation of your tool definitions is not byte-stable across processes, your cache hit rate depends on which server handled the request. Sort the keys.

The cost of a cached request

Split the input into a stable prefix of T_p tokens and a variable tail of T_v tokens. Let r be the cached-read price as a multiple of the normal input price, and w the cache-write price as a multiple of it. Both r and w vary by provider and are the two numbers to look up before anything else — r is generally well below one, and w is either one (no premium) or somewhat above it.

miss (first request, writes the cache):
  cost = ( w * T_p * P_in  +  T_v * P_in  +  T_out * P_out ) / 1e6

hit:
  cost = ( r * T_p * P_in  +  T_v * P_in  +  T_out * P_out ) / 1e6

Worked, with assumed values P_in = $1.00/M, P_out = $5.00/M, r = 0.1, w = 1.25, a 6,000 token prefix (system prompt plus tool definitions plus a few-shot block), a 400 token tail and a 300 token answer:

no caching : (6000 + 400)*1.00/1e6 + 300*5.00/1e6 = $0.00790
cache miss : (1.25*6000 + 400)*1.00/1e6 + 0.00150   = $0.00940
cache hit  : (0.10*6000 + 400)*1.00/1e6 + 0.00150   = $0.00250

hit vs no caching: 68% cheaper per request

Note the shape of that result. The saving is bounded by how much of the request is prefix: with a 6,000 token prefix out of 6,700 tokens you can remove nearly all of the input line, but the output line is untouched and sets a floor. If your prompt were 400 tokens of prefix and 6,000 of variable content, caching would be worth almost nothing. Check the ratio before estimating the benefit.

How many hits pay back a write

Where a write premium exists, the first request is more expensive than it would have been without caching. So a prefix that is used once and never again is a loss. The break-even is the number of requests n sharing a prefix within its lifetime, one write and n − 1 hits, that recovers the premium:

extra paid on the write  = (w - 1) * T_p * P_in
saved on each hit        = (1 - r) * T_p * P_in

(w - 1) * T_p * P_in  <  (n - 1) * (1 - r) * T_p * P_in

           n  >  1 + (w - 1) / (1 - r)

T_p and P_in cancel: the threshold is a pure ratio.

With w = 1.25 and r = 0.1: n > 1 + 0.25/0.9 = 1.28. Two requests. That is the entire answer to “is caching worth enabling” for almost every workload: if any two requests within the cache lifetime share the prefix, it has already paid for itself. Where w = 1 the threshold is n > 1 and there is no decision to make at all.

The formula only turns hostile if a provider charges a large write premium against a read multiplier close to one. Compute it rather than assuming; it is two divisions.

Ordering a prompt for hits

The rule is one sentence — stable first, volatile last — and the work is in noticing what is volatile.

GOOD                              BAD
1  static system instructions     1  "Current time: 14:32:07"
2  tool definitions (sorted)      2  "User: Amara (plan: pro)"
3  few-shot examples              3  static system instructions
4  stable per-tenant policy       4  tool definitions
5  retrieved documents            5  few-shot examples
6  conversation history           6  the actual question
7  the current turn

The right-hand column caches nothing. It is not an exaggerated example; a timestamp at the top of a prompt is a normal thing to write and its cost is invisible.

• Move volatile context into the user turn. If the model needs the current date or the caller’s name, they belong at the end, not in the shared preamble.
• Group by tenant, not by request. If each customer has their own policy block, the cache key is per-customer, so the hit rate depends on request concurrency per customer rather than overall. A thousand customers with one request each will not hit.
• Keep retrieval after the static block. Retrieved chunks are the most volatile large thing in a RAG prompt; putting them before the few-shot examples throws away the examples’ cacheability too.
• Append, never rewrite, conversation history. A conversation that only grows at the end reuses the entire previous prefix. Re-summarising the history each turn, or reordering it, invalidates all of it — which makes summarisation a trade against caching rather than a free win.
• Instrument the hit rate. The usage object reports cached token counts. If the ratio of cached input to total input is not roughly what your prompt structure predicts, something upstream is inserting a volatile byte and you will not find it by reading the prompt.

TTL, and the traffic it assumes

Cached prefixes expire, on the order of minutes for the default tiers most providers offer, with longer retention sometimes available at a higher write price. The consequence is that caching rewards request density per prefix, not request volume.

Model it directly. If a given prefix is used at rate λ requests per second and the TTL is T seconds, then in steady state the expected fraction of requests that hit is roughly 1 − 1/(λT + 1) for Poisson-ish arrivals — one write per idle gap, hits for everything else. At λ = 0.1/s and T = 300s that is 1 − 1/31 = 97%. At λ = 0.002/s — one request every eight minutes — it is 1 − 1/1.6 = 38%, and the write premium starts to bite.

This is why caching is spectacular for a busy shared system prompt and unremarkable for a long-tail per-user prefix, and it is also why “keep the cache warm” schemes exist: a heartbeat request against the prefix costs one cheap call per TTL and converts a sparse, expensive pattern into a dense, cheap one. Whether that is worth it is the same arithmetic as everything else here — the heartbeat cost per hour against the writes it avoids.