Semantic Caching: Answering Before You Call

Prompt caching reuses computation for an identical prefix and is exact. Semantic caching reuses an answer for a similar question, and is therefore a probabilistic system with a false-positive rate — which means it needs calibration, not configuration.

The idea, and the immediate problem

Embed the incoming question, search a store of previous question/answer pairs, and if the nearest neighbour is closer than some threshold, return its stored answer without calling the model. A hit costs an embedding call and a vector search — typically single-digit milliseconds and a rounding error in money — against a full generation. When it works, it is the cheapest and fastest thing in the entire request path, and it consumes no rate-limit quota at all.

The problem arrives with the threshold. A cache that is too permissive answers a question that was not asked; a cache that is too strict never hits. Every published hit-rate figure is a property of the traffic it was measured on and tells you nothing about yours, which is why this page gives you the procedure rather than a number.

Similar is not equivalent

Embedding similarity approximates topical relatedness. It does not model logical equivalence, and the gap between the two is where every serious failure lives:

HIGH similarity, OPPOSITE answers
  "is aspirin safe during pregnancy"   vs  "is aspirin unsafe during pregnancy"
  "how do I enable two-factor auth"    vs  "how do I disable two-factor auth"

HIGH similarity, DIFFERENT subject
  "what is the refund policy for the Pro plan"
  "what is the refund policy for the Enterprise plan"

HIGH similarity, STALE answer
  "what is our current pricing"        asked in March, answered in March,
                                       served in August

LOW similarity, SAME answer            (a miss you would have wanted)
  "reset password"                     vs  "I can't log in and need a new
                                            passphrase"

Negation is the notorious one — most embedding models place a sentence and its negation very close together, because they share nearly all of their content words. Entity substitution is the quieter one and is worse in practice, because “Pro” and “Enterprise” differ by one token and the answers differ entirely. No threshold separates those cases, which is the first argument for scoping caches by tenant, plan, locale and anything else that changes the answer, rather than trying to make similarity carry the whole job.

Calibrating the threshold on your own traffic

The only honest way to pick a threshold is to measure the trade-off on your own question distribution. The harness is small:

// 1. Sample 500-1000 real question pairs that a naive cache would consider
//    close (nearest neighbours above a deliberately LOW threshold, e.g. 0.75).
// 2. Label each pair: would serving A's answer for B be acceptable? Human
//    judgement, or a strong model as a first pass with human audit.
// 3. Sweep the threshold and read off the curve.

type Pair = { a: string; b: string; sim: number; interchangeable: boolean };

function sweep(pairs: Pair[]) {
  const rows = [];
  for (let t = 0.80; t <= 0.995; t += 0.005) {
    const served = pairs.filter((p) => p.sim >= t);
    const good = served.filter((p) => p.interchangeable).length;
    const bad = served.length - good;

    rows.push({
      threshold: +t.toFixed(3),
      hitRate: served.length / pairs.length,        // coverage
      precision: served.length ? good / served.length : 1,
      falseHitsPer1k: (bad / pairs.length) * 1000,  // the number that matters
    });
  }
  return rows;
}

Read the output as a precision/coverage curve, not as a single score. The column to make decisions on is falseHitsPer1k: hit rate is the benefit, and false hits per thousand requests is the cost, in the units your risk assessment is actually expressed in.

Two practical notes. Relabel periodically, because your question distribution drifts and a threshold calibrated on last quarter’s traffic is calibrated on the wrong thing. And normalise before embedding — lowercase, strip punctuation, collapse whitespace — so that trivially identical questions hit the exact-match path and never reach the probabilistic one at all.

Where the threshold should sit

The threshold is the solution to an expected-cost problem, and writing it down usually settles the argument:

E[cost per request | threshold t] =

    hit(t)  * [ precision(t) * saving  -  (1 - precision(t)) * wrong_answer_cost ]
  - miss(t) * 0

  saving             = the model call you avoided. Cents, and some latency.
  wrong_answer_cost  = domain-dependent, and the whole ballgame.

  FAQ deflection, wrong answer costs a support ticket    ->  cache aggressively
  Product pricing question, wrong answer costs a refund  ->  cache narrowly
  Medical or legal guidance                              ->  do not do this
  Anything personalised or account-specific              ->  do not do this

The formula is not there to be evaluated precisely; it is there to make explicit that a threshold is a statement about how much a wrong answer costs you. Teams that skip this step invariably pick a number that feels safe, discover the hit rate is disappointing, lower it, and rediscover the trade-off through incidents.

Two implementation choices deserve more attention than they usually get. The embedding model is the whole discriminator: a model tuned for retrieval is trained to place a question near its answer, which is the wrong geometry here — you want questions near equivalent questions. Where a provider offers asymmetric query and document embeddings, embed both sides as queries. And whichever model you pick, the threshold you calibrated is bound to it; changing the embedding model invalidates the calibration and the entire stored index, so treat it as a migration rather than a config change.

The other is what a cache entry stores. The answer alone is not enough to operate the cache: you also want the original question, the timestamp, the model and prompt version that produced it, and the partition key. Without the question you cannot audit a false hit; without the version you cannot invalidate when the prompt changes; and a semantic cache that survives a prompt rewrite is quietly serving answers from a system that no longer exists.

Safer variants

• Partition the key space. Include tenant, plan, locale and document-set version in the cache key. Most catastrophic false hits are cross-partition, and partitioning removes them outright rather than trying to score them away.
• Cache retrieval, not generation. Reuse the retrieved documents for a similar query and still run the model. You keep most of the latency saving, lose most of the cost saving, and the model gets a chance to notice that the question is different.
• Verify a candidate hit with a cheap model. “Does this stored answer fully answer this question?” from a small fast model costs a fraction of the real call and catches negation and entity swaps that similarity cannot.
• Expire aggressively and invalidate on writes. A semantically correct answer that is factually stale is still wrong. Tie the cache to the version of whatever knowledge produced it.
• Log every hit with its similarity and both questions. Without that you cannot audit a complaint, and an unauditable cache is one you will eventually turn off in a panic instead of tuning.

Used inside a narrow partition on a high-volume, low-stakes surface — documentation search, FAQ deflection, autocomplete — semantic caching is one of the highest-leverage things available. Used as a general front-end to a chat product, it is a slow-acting correctness bug. The harness above is what tells you which one you are building.