Testing That a Prompt Cache Hit Behaves Like a Cache Miss

The usual version of this test sends the same request twice and compares the outputs. It almost always passes, and it usually passes without a cache hit having occurred at all — so it is a test of nothing.

First, prove you got a hit

Whatever you assert afterwards, the test is worthless unless it establishes that the second request was served from cache. On the Claude API the evidence is in the usage block. Anthropic’s prompt caching documentation names three fields: cache_creation_input_tokens for tokens written to the cache, cache_read_input_tokens for tokens read from it, and input_tokens for the uncached remainder after the last cache breakpoint, with the total being the sum of the three.

first  = client.messages.create(**request)   # cache write
second = client.messages.create(**request)   # expect a read

assert first.usage.cache_creation_input_tokens > 0, "nothing was cached"
assert second.usage.cache_read_input_tokens > 0,    "no cache hit"
assert second.usage.cache_creation_input_tokens == 0

Assert all three. The first line is the one people leave out, and without it a test where nothing was ever cached passes the second assertion trivially by never running. See Anthropic’s prompt caching documentation for the field definitions and the current cache-control shape, which is a cache_control block of type ephemeral, optionally carrying a longer time to live.

What equivalence can honestly mean

Provider-side prompt caching reuses the computed state of a prefix. It does not return a stored answer, and it does not fix the sampler. Two requests that both hit the cache still sample independently, so asserting that the hit and the miss produce the same string is asserting something caching never promised, and the test will flake for a reason unrelated to caching.

So assert equivalence at the level your application actually depends on. If the call returns structured output, compare the parsed objects field by field. If it selects a tool, compare the tool name. If it classifies, compare the label. Those are stable properties of a working system and they will not flake on wording. Reserve exact string comparison for the case where the request is genuinely deterministic and you have set it up to be — and even then, prefer the structural assertion, because it keeps working when somebody later raises the temperature for good reasons.

For an application-level cache — one you built, that stores the response and returns it verbatim — string equality is legitimate, because your cache really does promise to return the same bytes. Know which of the two you are testing; conflating them is why this test is usually wrong.

The cache key is the real test

The bug class this page exists for is not equivalence at all. It is a cache key that omits something that should change the answer, so a hit is served for a request that is not the same request. That is a pure function taking your request and returning a string, and it is testable exhaustively with no network.

import { describe, expect, it } from "vitest";
import { cacheKey } from "../src/cache";

const base = {
  model: "provider/model-1",
  promptTemplateVersion: "invoice@7",
  systemPrompt: "You extract invoice fields.",
  tools: [{ name: "lookup_vendor" }],
  temperature: 0,
  responseSchemaVersion: 3,
  userId: "u_1",
  input: "INVOICE 4471 ...",
};

const MUST_CHANGE_THE_KEY = [
  ["model", { model: "provider/model-2" }],
  ["template version", { promptTemplateVersion: "invoice@8" }],
  ["system prompt", { systemPrompt: "You extract invoice fields carefully." }],
  ["tool set", { tools: [] }],
  ["temperature", { temperature: 0.7 }],
  ["schema version", { responseSchemaVersion: 4 }],
  ["input", { input: "INVOICE 4472 ..." }],
] as const;

describe("cacheKey", () => {
  it.each(MUST_CHANGE_THE_KEY)("changes when %s changes", (_label, patch) => {
    expect(cacheKey({ ...base, ...patch })).not.toBe(cacheKey(base));
  });

  it("does not change for an unrelated field", () => {
    expect(cacheKey({ ...base, userId: "u_2" })).toBe(cacheKey(base));
  });
});

Every row is a real incident somebody has had. The system prompt row is the most common: a cache keyed on the user’s input alone serves yesterday’s answers after a prompt change, and the symptom is that a fix appears not to have deployed. The temperature row catches a cache that treats two different sampling configurations as one. And the last test is as important as the others in the opposite direction: including a field that should not be in the key — a request id, a timestamp, a session identifier — gives you a cache with a hit rate of zero, which costs money quietly rather than serving wrong answers loudly.

Minimums and invalidation

Two provider behaviours will make a correct test fail and send you looking for a bug that is not there. First, there is a minimum cacheable prefix length, and it differs by model — Anthropic’s documentation lists thresholds ranging from 512 tokens on some models to 4,096 on others. A prefix below the threshold for the model under test simply is not cached, so the fixture in a caching test must be long enough to qualify, and the model id in the test must be the one the threshold belongs to.

Second, more of the request participates in cache validity than the text does. Anthropic documents that changing tool definitions invalidates everything downstream, and that several request-level settings invalidate the system and message levels — adding or removing images, changing the tool choice parameter, toggling certain features. A test that varies one of those and expects a hit is asserting the opposite of the documented behaviour. Turn each of them into a deliberate negative test instead: change the tool definitions, assert cache_read_input_tokens is zero, and you have pinned the invalidation boundary your cost model depends on.

Asserting the saving

If the reason you enabled caching was cost, assert on cost, using the token fields rather than a price. Cache reads are billed at a fraction of the base input rate and cache writes at a premium over it, so the saving depends on the ratio of reads to writes across your real traffic — a workload that writes the cache almost as often as it reads it can cost more than no caching at all. The testable version is the ratio: assert that over a representative sequence of requests, cache reads exceed cache writes by whatever multiple your arithmetic requires to break even. That is a property of your access pattern, it is stable, and it fails when somebody moves a breakpoint onto content that changes every request. The arithmetic behind the break-even point is set out in working out prompt cache savings.