Caching Retrieval Results

A retrieval cache is easy to build and easy to build wrong, and both failure modes are silent: too strict a key and you never get a hit, too loose and you serve one customer’s documents to another. This page names the six components the key must contain, gives an invalidation scheme that needs no keyspace scan, and is honest about the part of semantic caching that has no principled answer.

What is worth caching, and what it saves

There are three cacheable stages in a retrieval pipeline and they have very different economics.

Whether the middle one is worth building at all depends on one number you can measure in an afternoon: the repeat rate of your queries.

-- From your telemetry, over the last week:
WITH q AS (
  SELECT md5(lower(btrim(query_text))) AS h, count(*) AS n
  FROM search_log
  WHERE created_at > now() - interval '7 days'
  GROUP BY 1
)
SELECT sum(n) AS total,
       sum(n) FILTER (WHERE n > 1) AS repeated,
       round(100.0 * sum(n - 1) FILTER (WHERE n > 1) / sum(n), 1) AS max_hit_rate_pct
FROM q;

That last column is the ceiling on your hit rate for an exact-match cache. If it is three per cent, a retrieval cache saves three per cent of your vector scans and is not where your effort should go. If it is forty per cent — common for a documentation search or a support assistant, where everyone asks the same twenty questions — it is the cheapest latency win available.

The six parts of the key

You cannot key on the query embedding. Floating-point vectors are never exactly equal across runs, the vector is large, and two byte-different vectors may be semantically identical. Key on the inputs instead, and all six of these belong in the hash.

key = "ret:" + sha256(canonical([
    embedding_model,    # 1
    index_version,      # 2
    tenant_id,          # 3
    top_k,              # 4
    filters,            # 5  canonicalised: sorted keys, sorted values
    normalised_query,   # 6  lowercased, whitespace-collapsed, trimmed
]))

Cache the chunk ids and scores, not the chunk text. The text may be edited or deleted between caching and use; the ids are stable and the fetch by primary key is microseconds. This also means a deleted document disappears from cached results automatically, because the join returns nothing for it — which is a considerable safety property, given what deletion that reaches the vector index says about how hard the alternative is.

import hashlib, json

def retrieval_key(model, index_version, tenant, k, filters, query):
    payload = json.dumps({
        "m": model, "v": index_version, "t": str(tenant), "k": k,
        "f": filters, "q": " ".join(query.lower().split()),
    }, sort_keys=True, separators=(",", ":"))
    return "ret:" + hashlib.sha256(payload.encode()).hexdigest()

def retrieve(query, tenant, k=10, filters=None):
    key = retrieval_key(EMBED_MODEL, index_version(), tenant, k, filters or {}, query)
    hit = r.get(key)
    if hit:
        ids = json.loads(hit)
    else:
        ids = [row[0] for row in db.execute(RETRIEVAL_SQL, ...)]
        r.set(key, json.dumps(ids), ex=3600)
    return fetch_chunks_by_id(ids)     # always fresh text, from the database

Invalidation without a scan

The obvious approach after a re-index is to find and delete the affected keys. It does not work: you cannot tell which cached queries would have matched a changed document without re-running every one of them, and SCAN over a large keyspace is slow and easy to get wrong under concurrent writes. KEYS is worse — it blocks the server.

Version the namespace instead. Every key already contains index_version, so bumping that value makes every existing key unreachable in one atomic operation, with no scan and no possibility of a partial invalidation:

# One counter, read on every request, bumped after a re-index.
def index_version():
    v = r.get("index:version")
    return int(v) if v else 1

# After a re-index completes — atomic, instantaneous, total.
r.incr("index:version")

Orphaned entries are reclaimed by their TTL and by LRU eviction, which is why the cache instance must run allkeys-lru — the reasoning in Redis for caching, queues and vectors.

Two refinements worth having. Version per tenant (index:version:{tenant}) if tenants re-index independently, so one customer’s upload does not cold-start everybody. And read the version once per request rather than per cache lookup, or you have added a Redis round trip to every retrieval to save a Redis round trip.

Semantic caching and the threshold problem

A semantic cache goes further: embed the incoming query, find the nearest cached query, and if it is within a distance threshold, return that entry’s results. It converts a three per cent hit rate into something much higher, because “how do I reset my password” and “forgot password, what now” become one entry.

It also introduces a failure mode the exact cache does not have: a wrong answer, served confidently, with no error. And the entire design rests on one number nobody can give you.

-- The cache is itself a small vector table.
CREATE TABLE semantic_cache (
  id            bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  index_version int  NOT NULL,
  query_text    text NOT NULL,
  query_vec     vector(1536) NOT NULL,
  chunk_ids     bigint[] NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  hits          int NOT NULL DEFAULT 0
);
CREATE INDEX ON semantic_cache USING hnsw (query_vec vector_cosine_ops);

-- The lookup. Note that tenant and index_version are still exact
-- predicates — only the QUERY is approximate. Never approximate
-- an authorisation boundary.
SELECT chunk_ids, query_vec <=> $1 AS distance
FROM semantic_cache
WHERE tenant_id = $2 AND index_version = $3
ORDER BY query_vec <=> $1
LIMIT 1;
-- Use the row only if distance < threshold.

The pairs that make the threshold hard are the ones that are close in embedding space and different in meaning. “How do I cancel my subscription” and “how do I cancel my order” are textually near-identical and want completely different documents. So do “export data” and “delete data”, and any pair differing by a negation — embeddings represent negation weakly, which is the same limitation that makes the absence questions in graph storage unanswerable by similarity.

Choosing the threshold, with a harness

There is no correct threshold in general; there is a correct threshold for your query distribution, and finding it takes about two hours of somebody’s time. Do this rather than copying a number from a blog post, because the number depends on your embedding model and your users’ vocabulary.

1. Sample 300 query pairs from your logs — for each of 300 real queries, its nearest neighbour among the other queries in the log, with the distance recorded.

   WITH sample AS (
     SELECT id, query_text, query_vec FROM query_log
     WHERE created_at > now() - interval '30 days'
     ORDER BY random() LIMIT 300
   )
   SELECT s.query_text AS a, n.query_text AS b, n.distance
   FROM sample s
   CROSS JOIN LATERAL (
     SELECT q.query_text, q.query_vec <=> s.query_vec AS distance
     FROM query_log q WHERE q.id <> s.id
     ORDER BY q.query_vec <=> s.query_vec LIMIT 1
   ) n
   ORDER BY n.distance;

2. Label each pair by hand. One question: would the same retrieved documents be a correct answer for both? Yes or no. This is the part that cannot be automated and it is the part that gives the exercise its value. Three hundred pairs is about ninety minutes.
3. Compute precision at each candidate threshold. For a threshold τ, precision is the fraction of pairs with distance below τ that you labelled “yes”; the hit rate is the fraction of all pairs below τ.

   for tau in [0.02, 0.04, 0.06, 0.08, 0.10, 0.15, 0.20]:
       below   = [p for p in pairs if p.distance < tau]
       correct = [p for p in below if p.label]
       print(tau,
             "hit_rate=%.1f%%" % (100 * len(below) / len(pairs)),
             "precision=%.1f%%" % (100 * len(correct) / max(len(below), 1)))

4. Choose from the precision you can defend, not the hit rate you want. Decide the acceptable rate of serving the wrong documents first — for a support assistant that might be one in a hundred, for an internal tool one in twenty — then read the threshold off the table. If no threshold reaches your precision target, the honest conclusion is that semantic caching does not work for your query distribution.
5. Re-run it when you change embedding model. The distance scale is a property of the model. A threshold tuned for one is meaningless for another, and this is the step that gets forgotten.

What must never be cached

• Anything keyed without the tenant. Stated twice on purpose. A shared cache across tenants is a cross-tenant leak that no row-level security policy can catch, because the database is never consulted.
• Results for a user with unusual permissions. If document visibility varies within a tenant — per team, per role — then the permission set is part of the key, not an afterthought. If it cannot be reduced to a small stable value, do not cache retrieval for that tenant at all.
• Anything during an incident affecting correctness. A cache extends the lifetime of a bad result well past the fix. Have a documented way to flush — bumping the version counter is that way, and it should be a one-line operation somebody can run at 3 a.m.
• The first request for a new document. After an upload, the user immediately searches for what they just uploaded. If the version bump is asynchronous and the cache still holds the pre-upload result, the product looks broken at exactly the moment the user is paying attention. Bump the version in the same transaction that marks the index complete.

Instrument the cache from the first deploy, because a cache you cannot measure is a cache you cannot reason about. Three counters are enough: hits, misses, and — the one people leave out — stale serves prevented, meaning lookups that missed because the index version had moved. That third number tells you whether your invalidation is firing, and a value of zero over a week when you know a re-index ran is a bug rather than good news.

Log the key alongside the request id for a sample of requests. When a user reports that search returned the wrong thing, the first question is whether they were served from cache, and without the key in the log that question takes an afternoon instead of a query.