Context Caching Strategy: Ordering for Cache Hits

Prompt caching is not a setting you enable. It is a property your prompt either has or does not have, and one timestamp in the wrong place removes it from every request you will ever send.

It is a prefix, and that is the whole design

The pricing and mechanics of cached tokens belong to the tokens cluster. The one structural fact that governs assembly is this: caching operates on a prefix. The provider can reuse the computed state of your request up to the first position where it differs from a previous request, and not one token further.

This is a much harsher constraint than “repeated content is cheaper.” Content that repeats but has moved is worth nothing. A 40,000-token reference block that is identical between two requests gets no discount at all if a 12-token session id was inserted ahead of it. There is no partial credit, no fuzzy match, and no recovery later in the prompt — one early difference forfeits the entire remainder.

Which turns cache optimisation into a single question with a mechanical answer: what is the longest prefix of my request that is byte-identical across the requests I actually send? Everything else follows.

Order by mutation rate

Sort blocks by how often they change, most stable first. The boundary between cached and uncached then falls as late as it possibly can.

tier 0  never changes        system prompt, persona, output contract
tier 1  changes on deploy    tool schemas, static reference material
tier 2  changes per session  user profile, account settings, compacted record
tier 3  changes per turn     conversation history (append-only: prefix stable)
tier 4  changes per request  retrieved documents, tool results, the question
                             ^ cache boundary lands here

Tier 3 is the interesting one. Conversation history changes every turn, but it changes by appending, so its prefix is stable even though its content is not: turn 12’s history is turn 11’s history plus two messages. That means history is cacheable up to the point of last turn’s end, provided nothing below tier 3 sits after it. This is the concrete reason history should precede retrieved documents — the reverse ordering makes every turn’s history uncacheable for the sake of putting retrieval slightly earlier.

Compaction is the exception that has to be planned for. When the compactor rewrites the history block, the tier-2 record changes and the prefix breaks — one expensive uncached request, then stability returns. That is fine as long as compaction is triggered at a threshold rather than continuously; a compactor that trims one message per turn produces a permanently cold cache.

Seven things that break a prefix

All of these are real, all of them are invisible in code review, and each one costs the entire cache.

• A timestamp in the system prompt. “The current date and time is 2026-08-03T11:42:07Z.” Changes every second. If the model needs the date, give it to the minute or the day, or move it to the end of the prompt.
• Non-deterministic serialisation. Tool schemas built from a dictionary whose key order is not stable, or JSON serialised with different whitespace between runs. Byte-identical means byte-identical; sort keys explicitly and pin the serialiser.
• Per-user personalisation at the top. A name, a plan tier or a locale injected into the system block gives every user a private cache. Sometimes that is acceptable — a heavy user fills their own cache — but it must be a decision, and for long-tail traffic it means no caching at all.
• A request id or trace id. Injected by middleware, never reviewed, unique by construction. The purest form of the bug.
• Shuffled few-shot examples. Randomising example order to reduce bias is a defensible technique that is completely incompatible with prefix caching. Pick one.
• Allocator drift. If your allocator produces 41,318 tokens of reference this turn and 41,290 next turn, the block differs at its tail and everything after it is uncached. Round allocations to a coarse granularity so they are stable.
• Model or parameter changes. Caches are keyed by more than text. Switching model, or in some implementations switching sampling parameters or tool definitions, is a different cache entry.

What it is worth

Let C be the cacheable prefix in tokens, V the volatile remainder, P the uncached input price, r the cache read multiplier, w the cache write multiplier where one is charged, and h the hit rate. Per request:

no cache : (C + V) * P
cached   : h*(C*P*r) + (1-h)*(C*P*w) + V*P

saving   : C * P * [ 1 - h*r - (1-h)*w ]

Break-even hit rate (when w > 1):   h  >  (w - 1) / (w - r)

The break-even is the part worth carrying around. If writing costs a premium — say w = 1.25 and r = 0.1, both assumptions, substitute your provider’s actual figures — then caching only pays above a hit rate of 0.25/1.15 ≈ 22%. Below that you are paying write premiums on prefixes nobody reuses. Where writes are not charged separately, any hit rate above zero is a gain.

The other lever in that expression is C itself, and it is linear: doubling the length of the stable prefix doubles the saving. This is exactly why the static-versus-retrieved decision leans further towards static than it did before caching existed — material carried in the prefix is charged at P × r, not P.

Hit rate is also a function of traffic, not just of layout. Caches have a lifetime, so a prefix reused every few seconds stays warm and one reused every hour usually does not. Low-volume applications should compute with h near zero regardless of how beautifully ordered the prompt is.

Verifying you are actually hitting it

Cache optimisation is uniquely prone to silent failure: the prompt looks well-ordered, the code has a comment saying the prefix is stable, and the hit rate is zero because of a header injected two layers down. Assume nothing and check.

• Log the prefix hash. Hash the first C tokens of every rendered request and log it. If the hash changes between consecutive requests that should have shared a prefix, you have found the bug without needing any provider telemetry at all.
• Diff two rendered requests. Not the inputs — the final serialised payload. The first differing byte is the answer, and it is almost never where you expected.
• Watch the reported cached-token count. The provider tells you how many tokens were served from cache. That number against C is your real hit rate, and it is the only figure worth trusting.
• Alert on it. Hit rate is a metric that silently drops to zero when somebody adds a field to the system prompt. It deserves a dashboard line and a threshold, because the failure is a cost increase with no error attached.