Testing That a Cache Doesn't Serve a Stale System Prompt After a Deploy

You ship a system prompt change, watch the deploy go green, and responses keep arriving in the old voice — with the old refusal rules, or the tool the new prompt was meant to stop it using. Ten minutes later it fixes itself, which is the worst possible outcome because it removes the pressure to find out why.

The symptom

Specifically: after a deploy, some fraction of responses behave as if the previous system prompt were still in force. The fraction usually decreases over minutes. It is not reproducible afterwards, it does not appear in staging because staging has one instance and no traffic, and the natural conclusion — “a cache somewhere is holding the old prompt” — is correct about the cause and, as usually stated, wrong about which cache.

It is not the provider’s cache

The first suspect is provider-side prompt caching, since it is the cache with the word prompt in its name and it has a short TTL that matches the “fixes itself in ten minutes” timeline. It is the wrong suspect, and knowing why saves a day.

Provider prompt caches are keyed on the content of the prefix. In Anthropic’s implementation a breakpoint is marked with cache_control, and the documentation states that changes at each level invalidate that level and all subsequent levels — modifying the system prompt invalidates the cached system prompt, and modifying tool definitions invalidates the entire cache. A request carrying a new system prompt therefore cannot hit an entry created by the old one; it is a cache miss by construction, reported as cache_creation_input_tokens rather than cache_read_input_tokens. See Anthropic’s prompt-caching documentation.

So the provider cache cannot serve yesterday’s system prompt to a request that contains today’s. What it can do is cost you money after a deploy — every cached prefix is invalidated at once, so the first requests after a prompt change all pay the cache-write premium. That is a real effect worth expecting, and it is not this bug.

The two places it really lives

A response cache keyed on the wrong thing. If your cache key is built from the user message and a prompt name or template id — `chat:v1:${hash(userMessage)}` — then editing the system prompt does not change the key. Every entry written before the deploy is still a valid hit afterwards, and it contains an answer generated under the old instructions. This is the usual culprit, it is entirely inside your own code, and it lasts exactly as long as your TTL.

The same bug appears in a prompt registry client that caches a fetched prompt by name for five minutes. The registry has the new text; the instance is still serving what it fetched before the deploy, and different instances have fetched at different times, which explains a fraction rather than all of the traffic.

The rolling deploy window. While the rollout is in progress, old and new pods are both serving. Old pods are not malfunctioning: they are correctly serving the prompt they were built with. The fraction of stale responses falls as the rollout proceeds, which is precisely the observed shape. Nothing is broken here at all, and the only bug is if the two versions must not coexist — for example if the new prompt pairs with a new tool schema and the old prompt calls a tool that no longer exists.

Distinguishing the two is easy once you know to: if every response carries the serving instance id and the prompt version it used, the first cause shows old versions from new instances and the second shows old versions only from old instances. That single log field decides it in one query.

The tests

The fix is that the cache key is derived from the content of everything that affects the answer. The test asserts that property directly rather than testing one example of it.

import { describe, it, expect } from "vitest";
import { cacheKey } from "../src/cache-key";

const base = {
  systemPrompt: "You are a support agent. Never offer refunds.",
  tools: [{ name: "lookup_order" }],
  model: "model-a",
  userMessage: "where is my order",
  params: { temperature: 0 },
};

describe("cache key derivation", () => {
  // One case per input that changes the answer. Adding an input to the
  // request without adding it here is exactly how this bug ships.
  it.each([
    ["systemPrompt", { systemPrompt: "You are a support agent. Refunds are allowed." }],
    ["tools", { tools: [{ name: "lookup_order" }, { name: "issue_refund" }] }],
    ["model", { model: "model-b" }],
    ["params", { params: { temperature: 0.7 } }],
    ["userMessage", { userMessage: "where is my parcel" }],
  ])("a different %s produces a different key", (_label, patch) => {
    expect(cacheKey({ ...base, ...patch })).not.toBe(cacheKey(base));
  });

  it("is stable for identical input", () => {
    expect(cacheKey({ ...base })).toBe(cacheKey({ ...base }));
  });

  it("does not depend on key ordering in the request object", () => {
    const reordered = { params: base.params, userMessage: base.userMessage, model: base.model,
                        tools: base.tools, systemPrompt: base.systemPrompt };
    expect(cacheKey(reordered as typeof base)).toBe(cacheKey(base));
  });

  it("a whitespace-only prompt edit still changes the key", () => {
    // Deliberate: normalising whitespace out of the key is how a
    // "harmless" reformat silently reuses answers from the old prompt.
    expect(cacheKey({ ...base, systemPrompt: base.systemPrompt + "\n" })).not.toBe(cacheKey(base));
  });
});

The first block is the important one and its shape matters: a table with one row per input means adding a new request field without adding a row is a visible omission in review. The last case is counter-intuitive and deliberate — over-eager normalisation in the key function is a common cause of exactly this bug, because a reformatted prompt is still a different prompt to the model.

Add one integration test for the deploy transition: warm the cache under prompt version A, swap the loaded prompt to version B, and assert the next request is a miss and that the stored value under A’s key is never returned to a B request. If your registry client caches, assert it re-fetches when the version pin changes rather than only when its TTL expires.

Verifying it in the deployed system

Two habits close this off permanently. Put a content hash of the rendered system prompt into the cache key and also into your response metadata and logs — not a version label a human maintains, a hash of the actual bytes sent. Then “which prompt produced this answer” is answerable from a log line rather than from archaeology, and a stale response is identifiable in one query instead of by argument.

And never determine the version by asking the model. A canary instruction like “if asked for your version, say v7” is a test of model compliance wearing a deployment test’s clothes: it fails when the model declines, when a later instruction overrides it, or when a summarisation step drops it, and each of those looks exactly like the bug you are hunting. The hash travels in metadata, where it cannot be talked out of.

For the versioning discipline this depends on, see prompt versioning; for the other way a shared cache key goes wrong under load, testing for a cache stampede.