Caching in a RAG Pipeline: Three Layers Worth Having

Caching in a retrieval pipeline is not one decision. It is three, at three different places, and they differ enormously in how badly a stale entry hurts you — which is the property that should decide the design, not the hit rate.

Three layers, three risk levels

A fourth layer sits outside your control and deserves mention: the provider’s prompt cache, which discounts a repeated prefix on the generation call. It requires nothing from you except that you keep the shared part of your prompt at the front and byte-identical — which means your system prompt and few-shot examples before the retrieved chunks, never after. That ordering is free and frequently gotten wrong.

The embedding cache

The easiest win in the pipeline. Key on a hash of the model name and the exact text:

def embed_cached(text, model, cache):
    key = "emb:" + model + ":" + hashlib.sha256(
        text.encode("utf-8")).hexdigest()
    hit = cache.get(key)
    if hit is not None:
        return np.frombuffer(hit, dtype="float32")
    vec = embed_one(text, model=model)
    cache.set(key, vec.astype("float32").tobytes())   # no TTL needed
    return vec

The model name in the key is not optional; it is the entire correctness argument. Without it, a model upgrade silently mixes two incompatible vector spaces in one index, which is the failure the index-freshness page describes as unrecoverable without a rebuild.

Where this pays is ingest, not queries. Re-running a pipeline over a corpus where 98% of chunks are unchanged should cost 2% of the embedding bill, and with content-addressed chunk ids it already does. On the query side, hit rates depend on repeated queries and the saving per hit is a fraction of a cent — real, but not the reason to build it. The reason is that a full reindex stops being a budget event.

The retrieval cache and its version key

Caching “which chunks answer this query” skips the vector search and, if you cache the reranked list, the reranker call too — which is often the largest per-query line item. The difficulty is that the answer depends on the index, and the index moves.

A TTL is the lazy solution and it is wrong in both directions: too long and edits do not appear, too short and you have no cache. The right key includes an index version that increments on every write.

key = ":".join([
    "ret",
    tenant_id,                 # NEVER omit this. See the multi-tenant page.
    str(index_version),        # bump on any write; invalidates by rotation
    embedding_model,
    str(k),
    canonical(filters),        # sorted, stable serialisation
    hashlib.sha256(normalise(query).encode()).hexdigest()[:16],
])

Bumping index_version abandons the whole generation of entries rather than deleting them, and they expire naturally under an LRU policy. This is invalidation by rotation, and it is much easier to get right than targeted eviction, because you never have to work out which cached queries a given document affected — a question that has no cheap answer.

The cost is that a busy index invalidates constantly. If writes are continuous, version by time bucket instead — the hour, or the five-minute window — accepting a bounded staleness in exchange for a usable hit rate. That is a product decision about how fresh retrieval must be, and it should be made explicitly rather than falling out of a TTL somebody picked.

Two entries in that key are load-bearing and often missing. canonical(filters) must be a stable serialisation, or two identical filter sets with different dictionary ordering produce different keys and your hit rate quietly halves. And tenant_id is the difference between a cache and a data breach.

The answer cache, and semantic caching

An exact-match answer cache — same normalised question, same tenant, same index version, same model, same prompt version — is safe and worth having. Every field in that list is part of the key, and the prompt version is the one people forget: change the system prompt, and without a bump in the key you are serving answers generated under the old instructions indefinitely.

Semantic caching — returning a cached answer when a new question is merely similar to a cached one, above some cosine threshold — is a different proposition and should be approached with suspicion. The failure mode is answering question B with question A’s answer, and the pairs that trip it are exactly the ones embeddings handle worst:

• “How do I cancel my subscription?” versus “How do I cancel my trial?” — different policies, very high similarity.
• “Is X supported on v3?” versus “Is X supported on v4?” — one token apart, opposite answers.
• Anything with a negation. “Can I do X?” and “Can’t I do X?” are near-identical vectors.

If you do it anyway: set the threshold high enough to be nearly exact, refuse to serve from the cache when either question contains a number, a version or a negation, and scope entries per tenant. A safer variant is to use the semantic hit as a retrieval shortcut rather than an answer — reuse the cached chunk ids and regenerate — which keeps most of the saving and removes the class of failure entirely.

Will any of this pay?

Before building any of it, estimate the hit rate. Query traffic is typically head-heavy, and Zipf’s law is a reasonable first model: the i-th most common query has frequency proportional to 1/i. The cumulative share covered by the top m of n distinct queries is then a ratio of harmonic numbers, which is well approximated by logarithms.

coverage(m of n)  ≈  (ln m + 0.577) / (ln n + 0.577)      [Zipf, s = 1]

n = 10,000 distinct queries:
  cache the top     10  ->  ~29% of traffic
  cache the top    100  ->  ~53%
  cache the top  1,000  ->  ~76%

Two conclusions fall straight out. A small cache captures a surprisingly large share, so start small. And the curve is logarithmic, so a hundredfold larger cache is not a hundredfold better — going from 100 to 1,000 entries buys 23 points, and the next decade buys less.

Then temper it with reality. Real query distributions are flatter than s = 1 in conversational products, because users phrase things differently every time; normalising case, whitespace and trailing punctuation before hashing recovers some of that for free. And a multi-tenant product has a separate distribution per tenant, so the effective n is per tenant and the aggregate hit rate is much lower than a single-tenant model predicts.

Measure the potential before you build. Log normalised query hashes for a week and count repeats. If the repeat rate is 4%, a query cache saves 4% of your bill in the best case and is not the optimisation to spend a sprint on.

One habit makes all three layers debuggable: record which of them served each request. A single field on the trace — none, retrieval, answer — turns two otherwise impossible investigations into ordinary ones. When a user reports a stale answer, you can say immediately whether they were served from cache or from a stale index, which are different bugs with different fixes. And when your hit rate quietly collapses after a deploy, the field tells you it happened, where a cost graph only tells you six weeks later that the bill went up.