Testing Cache Invalidation After a Prompt Version Changes

You ship a prompt fix on Tuesday, the eval suite is green, and users keep reporting the old behaviour for days. The prompt changed; the cache key did not; and every request that had been seen before is still being answered from the version you replaced.

The bug has a shape: a key that is not total

A response cache is correct exactly when its key is a total function of everything that can change the answer. Every input the answer depends on but the key omits is a class of stale hit, and the omission is silent by construction — you get a hit, which is the outcome the cache is supposed to produce.

The prompt is the input people forget, because early on the prompt is a constant and hashing the user’s question feels sufficient. It stops being sufficient the first time somebody edits the prompt, which is also the first time anyone would notice, by which point the cache holds weeks of answers from the old one.

Framing this as “invalidation” is part of the trap. There is no invalidation step to get right if the key is total: a new prompt version produces new keys, the old entries are simply never looked up again, and they expire on their own TTL. Explicit purge-on-deploy is a workaround for an incomplete key, it is racy across a rolling deploy, and it throws away entries that are still valid. Fix the key.

What belongs in the key

• A hash of the rendered prompt — or of the template source plus the variable values, which is equivalent and easier to debug. Hash the canonical form: normalise line endings and strip a trailing newline, so a whitespace-only edit does not gratuitously invalidate a week of cache.
• The prompt version identifier, if you keep one in a prompt registry. Belt and braces with the hash, and much more readable in a key dump when you are trying to work out what a stale entry came from.
• The model id, exactly as sent. Including any version suffix. An alias that silently repoints is a stale-answer source your key cannot see, which is one more reason to pin.
• Every sampling parameter that affects output. Temperature, top-p, top-k, max tokens, stop sequences, seed, reasoning effort, response format. If it is in the request body and it is not the prompt, it probably belongs in the key.
• The tool definitions. A hash of the serialised tool list. Adding a tool changes what the model can do, and a cached answer from before it existed will never call it.
• Anything that scopes the answer. Tenant, locale, entitlement tier, feature flags that alter the prompt. A cache shared across tenants without a tenant term is not only stale, it is a data leak.

The reliable way to build this is to derive the key from the fully constructed request object rather than assembling it from named fields, so that a new field added to the request is in the key automatically rather than when somebody remembers.

Miss after an edit, hit without one

Both directions, and both are necessary. Only the first catches staleness, but without the second you cannot distinguish a correct cache from one that never hits at all — which passes the staleness test perfectly and costs you the entire benefit.

import { beforeEach, expect, it, vi } from "vitest";
import { cachedComplete } from "../src/cache";
import { memoryStore } from "./support/store";

const upstream = vi.fn(async () => ({ text: "answer" }));
beforeEach(() => { memoryStore.clear(); upstream.mockClear(); });

it("serves a hit when nothing changed", async () => {
  await cachedComplete({ template: "v1", vars: { q: "hi" } }, upstream);
  await cachedComplete({ template: "v1", vars: { q: "hi" } }, upstream);
  expect(upstream).toHaveBeenCalledTimes(1);
});

it("misses after the template changes, even by one word", async () => {
  await cachedComplete({ template: "v1", vars: { q: "hi" } }, upstream);
  await cachedComplete({ template: "v1-reworded", vars: { q: "hi" } }, upstream);
  expect(upstream).toHaveBeenCalledTimes(2);
});

it("misses when only the model changed", async () => {
  await cachedComplete({ template: "v1", vars: { q: "hi" }, model: "a" }, upstream);
  await cachedComplete({ template: "v1", vars: { q: "hi" }, model: "b" }, upstream);
  expect(upstream).toHaveBeenCalledTimes(2);
});

Assert on the upstream call count rather than on the returned text. The text is identical in both branches because the fake returns a constant, and a test that compares returned strings will pass whether the cache hit or missed — a test that cannot fail. The call count is the only observable that distinguishes them.

Add the whitespace case explicitly, asserting a hit when the template differs only by a trailing newline. That pins the canonicalisation and stops someone later hashing the raw bytes, which would make every editor save invalidate the cache.

The property test that finds the field you forgot

Named tests cover the fields you remembered, which is the same set you remembered when writing the key function. The bug is in the field neither of you thought about. Test the key function directly, as a property: mutate any single field of a request and the key must change.

import { cacheKey } from "../src/cache";

const base = {
  template: "support_reply@7", vars: { q: "hi", tenant: "acme" },
  model: "provider/model-2026-05-01", temperature: 0.2, maxTokens: 400,
  tools: [{ name: "lookup_order" }], locale: "en-GB",
};

it.each(Object.keys(base))("changing %s changes the key", (field) => {
  const mutated = { ...base, [field]: mutate((base as any)[field]) };
  expect(cacheKey(mutated)).not.toBe(cacheKey(base));
});

The test iterates the keys of the request object itself, so a field added to the request later is covered without anybody editing this file. It fails on the day the new field is added and the key function was not updated, which is the day you want to hear about it rather than the day a user reports an answer that ignores it.

Assert the inverse too, for the small set of fields that genuinely must not affect the key — a request id, a trace id, a timestamp. Those belong in an explicit exclusion list that the property test skips, so the exclusion is a visible decision rather than an omission. A trace id leaking into the key gives you a cache with a hit rate of zero and a storage bill, and it is a remarkably easy mistake to make when the key is derived from the whole request.

This is not the provider’s prompt cache

Keep the two ideas apart, because the word “cache” is doing two jobs. What this page describes is a response cache you own: a map from a request to a completed answer, which is why staleness is possible at all.

Provider-side prompt caching is a different mechanism. It caches the computed state of a prompt prefix so that repeated requests sharing that prefix skip recomputing it, and it returns a freshly generated answer every time. It is content-addressed by the prefix, so editing the prompt cannot produce a stale answer — the edited prefix simply misses and is recomputed. There is no invalidation problem to test.

What a prompt edit does cost you there is the discount, and the economics are worth knowing before you decide how often to edit a long system prompt: what prompt caching actually saves sets out the shape. The testable consequence is smaller than it sounds — assert that your prompt is assembled with the stable parts first and the variable parts last, so an edit to a footer does not invalidate the prefix. That is a test on the rendered prompt’s structure, not on a cache.