Caching Layers in an AI App: What to Cache Where

“Add a cache” is four different projects with four different keys, four different correctness hazards and four wildly different payoffs. Deciding which one you mean is most of the work.

Four caches, not one

They compose — you can run all four — but they are independent, and the cheapest one is not the most talked-about one.

Layer 1: exact response cache

The key is where this goes wrong, and it goes wrong in one direction: people key on the prompt text and nothing else. Everything that can change the answer belongs in the key.

const key = sha256(JSON.stringify({
  task: task.name,
  taskVersion: task.version,       // bump when the prompt template changes
  model: profile.id,               // including the version suffix, not the alias
  params: { temperature, top_p, max_tokens, seed },
  tools: toolSchemaHash,
  input,                           // the rendered user content
}));

The two omissions that cause real bugs are taskVersion and a concrete model id. Without the first, editing a prompt has no effect on anything already cached, and you will spend an afternoon convinced the change did not deploy. Without the second, a model alias that silently points at a new revision serves answers generated by the old one indefinitely.

Invalidation is mostly a non-issue if the key is right: a change to any input produces a different key, so entries expire by irrelevance rather than by deletion. Set a TTL anyway, sized to how long a stale answer is acceptable, and cache failures for a much shorter period or not at all — a cached 429 that lives for an hour is a self-inflicted outage.

Where this layer shines is not user chat, where exact repeats are rare, but everything machine-driven: batch jobs re-run over overlapping inputs, a page rendered for many users from the same source, retries and idempotent replays, and development. In fact the idempotency table from the previous page is already an exact response cache keyed by a natural key, and if you have built one you may not need the other.

Layer 2: semantic cache

Embed the request, look for a stored request within a similarity threshold, return its answer. This is the layer with the highest ceiling and the only one that can be actively wrong, because “similar” is not “equivalent”.

The failure is asymmetric and that should drive the threshold. A false miss costs one generation. A false hit returns a confidently wrong answer to a question the user did not ask, and it does so invisibly. Nothing in the response says it came from a cache. So set the threshold conservatively, and be aware that a fixed cosine threshold behaves differently across embedding models and across query lengths.

Three practical rules. Never share a semantic cache across authorisation boundaries — a hit that crosses tenants is a data leak, so the tenant is part of the lookup, not part of the score. Never use it where negation or a small numeric difference changes the answer; embeddings are notoriously weak at exactly that, and “refund a charge over fifty” and “refund a charge under fifty” are close in vector space and opposite in meaning. And log the similarity score with every hit, so you can review near-threshold hits and see whether they were right.

Layer 3: the provider’s prefix cache

Several providers offer a discount when a long prompt prefix is repeated, because the computed attention state for that prefix can be reused. You do not implement anything; you arrange your prompt so the stable part comes first and the variable part comes last.

The mechanics differ per provider and change over time — whether it is automatic or opt-in, the minimum cacheable length, how long entries live, and how the discount is expressed — so check the current documentation of the provider you use rather than a general rule. What is stable is the shape of the advice:

• Put the system prompt, tool definitions and any shared reference material at the front, byte-identical across requests.
• Keep anything per-request — user name, timestamp, request id — out of the prefix. A single injected timestamp at the top of a system prompt defeats the entire mechanism, and it is a common bug because it looks harmless.
• Treat the discount as a bonus rather than a design constraint, and confirm it in the usage figures the provider returns rather than assuming it applied.

Layer 4: derived artefacts

The highest-value cache in most applications is the one nobody calls a cache: the embeddings, classifications, extracted fields and summaries you generated once and stored in your own database. Their hit rate is effectively total, because they are read many times and written once.

The discipline they need is invalidation, and it is real invalidation rather than expiry. Store the source revision and the task version alongside every artefact. When the source document changes, the artefact is stale and you know it. When you change the prompt or move to a new model, you have a query that finds every artefact produced by the old version, which turns “regenerate everything” into a bounded, resumable backfill rather than a guess. Without those two columns, a prompt change means either reprocessing your entire corpus or living with a silent mixture of two generations of output.

Estimating the hit rate before you build it

You do not need to build a cache to estimate what it would do. Take a day of logged requests, compute the key you would have used, and count duplicates. That is a measurement of your own traffic and it takes an afternoon.

If you want a prior before you have the logs, request popularity in most consumer-facing systems is heavily skewed — a small set of items accounts for a large share of requests. Under a Zipf-like distribution with exponent s over N distinct requests, caching the top m gives a hit rate of roughly the sum of 1/k^s for k up to m, divided by the same sum up to N. With s near 1 the harmonic sum grows like ln(m), so the hit rate is approximately ln(m)/ln(N) — which is why caching the top 1% of a million distinct requests gets you a large fraction of the traffic, and why doubling the cache after that adds very little.

Every number in that paragraph is an assumption, not a finding. The exponent for your traffic is something you measure; internal tools and per-user content are often far flatter than s = 1, and a flat distribution makes an exact response cache close to worthless. Do the log analysis before you build the infrastructure — this is the rare case where the honest estimate is cheaper than the guess.