Testing That Two Semantically Identical Requests Produce the Same Cache Key

A semantic cache has two ways to be wrong and they are not symmetric. A miss costs you one completion. A false hit serves one user another user’s answer. Both are properties of a pure function you can test without a model, a network or a cache.

The key function, isolated

The prerequisite is that key derivation is separable from lookup. If the key is computed inline inside the caching client, every test needs a cache; if it is a function from a request to a string, every test is two calls and an equality assertion.

// cache-key.ts
import { createHash } from "node:crypto";

export type KeyInput = {
  messages: { role: string; content: string }[];
  model: string;
  temperature: number;
  toolsHash: string;      // hash of the tool schema set
  promptVersion: string;  // from the prompt registry
  tenantId: string;
};

export function cacheKey(input: KeyInput): string {
  const canonical = JSON.stringify({
    m: input.messages.map((x) => ({ r: x.role, c: normalise(x.content) })),
    model: input.model,
    t: input.temperature,
    tools: input.toolsHash,
    v: input.promptVersion,
    tenant: input.tenantId,
  });
  return createHash("sha256").update(canonical).digest("hex");
}

export function normalise(s: string): string {
  return s.trim().replace(/\s+/g, " ").replace(/[ \t]+([.,!?;:])/g, "$1");
}

Everything the key depends on is in the argument. That is what makes the test table below possible, and it is also what makes the key auditable: when a cache serves something surprising, you can recompute the key from a logged request and see which field differed.

Pairs that must produce the same key

Write the equivalence classes as pairs, and give each pair a name that states the rule. These are the normalisations you are asserting are safe.

import { describe, it, expect } from "vitest";
import { cacheKey } from "./cache-key";

const base = {
  model: "provider/model-a", temperature: 0, toolsHash: "t1",
  promptVersion: "sum@7", tenantId: "acme",
};
const k = (content: string) => cacheKey({ ...base, messages: [{ role: "user", content }] });

describe("keys that must match", () => {
  it.each([
    ["leading and trailing whitespace", "  summarise this  ", "summarise this"],
    ["collapsed internal whitespace", "summarise\n\n  this", "summarise this"],
    ["space before punctuation", "summarise this ?", "summarise this?"],
  ])("%s", (_n, a, b) => expect(k(a)).toBe(k(b)));
});

Be conservative about what goes in this list. Every normalisation you add widens the set of requests that share an answer, and each one is a claim that the difference cannot change the output. Whitespace is safe. Case is not, in general: a model asked about “Apple” and “apple” may reasonably answer differently, and lowercasing the whole prompt to improve the hit rate is a decision somebody should make deliberately rather than inherit from a utility function.

Pairs that must never collide

This is the more important table and it is usually missing. Each row is a pair of prompts a naive normaliser might merge, with an assertion that the keys differ.

• Negation. “Is this contract enforceable” and “Is this contract not enforceable” differ by three characters and invert the question. Any normaliser that strips short words will merge them.
• Numbers and units. “Convert 5 kg” and “Convert 5 lb”, or 1,000 against 1000 against 10,000 if your separator stripping is careless.
• Names and identifiers. Two order numbers differing in one digit, two customer names differing by a middle initial. If your key is built from an embedding, these are the pairs that actually collide.
• Ordering. “A before B” and “B before A”, and any request where the message array order carries meaning. A key built from a set rather than a list loses this.

describe("keys that must differ", () => {
  it.each([
    ["negation", "is this enforceable?", "is this not enforceable?"],
    ["unit change", "convert 5 kg to lb", "convert 5 lb to kg"],
    ["one digit", "status of order 88421", "status of order 88431"],
  ])("%s", (_n, a, b) => expect(k(a)).not.toBe(k(b)));
});

Run this table against every normaliser change. Adding one innocuous rule to normalise is the commit that breaks it, and without this table the breakage surfaces as a support ticket about an answer that mentions the wrong order.

The inputs that are not the prompt

Half of the real cache bugs are not about text at all. They are fields that should be in the key and are not, and each one is a single test asserting that changing that field alone changes the key.

1. Model and version. Two models given the same prompt give different answers; caching across them is a correctness bug that looks like a routing bug.
2. System prompt version. This is the one that bites after a deploy: the user text is identical, the system prompt changed, and the cache serves answers generated under the old instructions for as long as the TTL allows. If your prompts live in a registry, the version string is already available — put it in the key and assert it.
3. Tool schema set. A changed tool description changes the model’s behaviour, so it changes the key.
4. Tenant or user scope. Assert that two tenants with a byte-identical prompt get different keys. If retrieved context is injected before the key is computed, this is already implied; if it is injected after, this test is the only thing standing between you and cross-tenant disclosure.
5. Temperature and sampling parameters. Caching a temperature-0.9 request at all is a product decision; caching it under the same key as the temperature-0 one is not.

Embedding caches and the threshold test

If the cache matches by embedding similarity rather than by hash, the key is a vector and the equality assertion becomes a threshold assertion. The two tables above still work, with a different predicate: same-class pairs must score above the threshold, different-class pairs must score below it.

Two extra properties matter here. First, the negation and one-digit pairs are exactly where embedding similarity is highest and semantic distance is largest, so they are not optional rows — they are the point of the test. Second, the embedding model is part of the key function, so pin its identifier and treat a change as a change to the cache. An embedding model that is silently updated re-keys your entire cache at once, which looks like a sudden collapse in hit rate and is the retrieval-side version of the problem in silent model updates.

There is one more property specific to threshold caches, and it has no equivalent in the hash version: the cache can be right about two requests being similar and wrong about serving one the other’s answer, because similarity of questions does not imply interchangeability of answers. “What is the refund policy” and “What was the refund policy in 2019” sit close together in embedding space and have different correct answers. The mitigation is not a higher threshold — it is a set of fields extracted from the request, dates and identifiers in particular, that must match exactly before a similarity match is allowed to serve. Test that combination directly: a pair above the threshold whose extracted dates differ must still miss.

Assert the threshold as a named constant with the margin visible in the failure. A test that says the score was 0.86 against a threshold of 0.85 tells you the cache is one rephrasing away from a false hit; one that says only “expected true” does not.