Reusing Prompt Caching to Cut the Cost of a Regression Suite

Prompt caching is usually pitched at chat applications with a long system prompt. A regression suite has a better shape for it than any chat application does — two hundred requests sharing an identical prefix, issued back to back — and most suites get no benefit at all because of a detail in how they are written.

Why a regression suite is the ideal shape

Provider caches key on a prefix. The saving applies to the leading tokens that are byte-identical to a recent request, and it stops at the first token that differs. That means the value depends entirely on how much of your request is shared and where the shared part sits.

A regression suite is close to the theoretical best case. Every case sends the same system prompt, the same tool schemas, the same few-shot examples and the same output-format instructions, followed by a short case-specific input. The shared part is long, the varying part is short, and the varying part comes last. In a chat application the shared prefix competes with a growing conversation; here nothing grows.

The suite also runs its cases within seconds of each other, which matters because cache entries expire quickly. A workload that would have to worry about a cold cache between users has, in a test run, a hot cache from the second case onwards.

The published multipliers and minimums

Anthropic’s prompt caching documentation states the pricing as multipliers on the base input rate: cache writes with the default five-minute lifetime at 1.25 times base, writes with the one-hour lifetime at 2 times base, and cache reads at 0.1 times base. It also sets a minimum cacheable prefix that varies by model — 1,024 tokens for several current models and higher for some smaller ones — and returns cache_creation_input_tokens and cache_read_input_tokens in the usage object. See Anthropic’s prompt caching documentation.

OpenAI’s prompt caching works without explicit breakpoints: caching applies to prefixes of at least 1,024 tokens, the routing hash is taken from the leading portion of the prompt, a cached prefix remains eligible for reuse for a documented minimum period, and cached_tokens appears in the usage details of the response. See OpenAI’s prompt caching guide.

The suite arithmetic

Take a regression suite of 200 cases with a 3,000-token shared prefix — system prompt, four tool schemas and three few-shot examples — and a 400-token case-specific suffix. Using Anthropic’s published multipliers and expressing everything in effective input tokens, so that any base price substitutes cleanly:

Without caching
  200 cases x (3,000 + 400)               = 680,000 effective input tokens

With a 5-minute cache on the shared prefix
  1 write   x 3,000 x 1.25                =   3,750
  199 reads x 3,000 x 0.1                 =  59,700
  200 suffixes x 400 x 1.0                =  80,000
                                          = 143,450 effective input tokens

  143,450 / 680,000 = 21.1% of the uncached input cost

At a placeholder base rate of $3.00 per 1M input tokens
  uncached  680,000 / 1,000,000 x $3.00   = $2.04 per run
  cached    143,450 / 1,000,000 x $3.00   = $0.43 per run

  [base rate is a placeholder — substitute your provider's current price]

A 79% reduction on the input term, and the output term is untouched because output is never cached. That last point sets the ceiling on what caching can do for you: if your cases return long answers, the output term dominates and the cache barely moves the total. Compute both terms before deciding it is worth the effort — the cost estimate page has the full formula.

The write premium is also worth noticing. One write at 1.25 times is only worth paying because 199 reads follow it. A suite of eight cases barely breaks even, and a suite of two loses money by caching. The break-even is roughly where the number of reads exceeds the extra quarter of a prefix the write cost you.

Where the provider requires explicit breakpoints rather than caching automatically, place the last one at the end of the shared block and nowhere further down. A breakpoint inside the varying part caches something that will never be read again, and you pay the write premium for it on every single case — a configuration that costs more than not caching at all.

Four ways a suite destroys its own cache

• A timestamp or a random value in the system prompt. The most common and the most complete: a current date injected into the prompt changes the prefix on every run, and if it includes a time, on every request. Every case then pays the write premium and nothing is ever read. This same habit also destroys reproducibility, so it is worth removing on both counts — inject a fixed date in tests and vary it only in the cases that are about dates.
• Per-case content placed before the shared block. A case id in the system message, a per-test correlation identifier at the top, tools ordered differently per case because they came out of a set. Anything varying that sits ahead of the shared text truncates the shared prefix to whatever precedes it, which is usually nothing. Order the request so that everything constant is first.
• Parallel workers with separate caches. Splitting the suite across workers can mean each one writes its own entry, so the write premium is paid several times and the reads are divided. With eight workers and 200 cases you pay eight writes rather than one — still a large saving, but not the one you calculated.
• Gaps longer than the retention window. A suite that interleaves model calls with slow setup, or that runs cases as part of a broader pipeline with minutes between them, can fall outside the retention window repeatedly. Group the model calls together, or use the longer cache lifetime where the provider offers one and the arithmetic supports its higher write multiplier.

Asserting on the hit rate

Caching is invisible when it fails: the requests still succeed, the tests still pass, and the only symptom is a bill that did not drop. Since the usage object reports cache activity explicitly, the fix is to assert on it — a test for your own cost assumption rather than for the model.

# Runs after the first case has warmed the prefix.
def test_prefix_is_being_cached(complete):
    complete(case_messages(CASES[0]))          # warm
    resp = complete(case_messages(CASES[1]))

    cached = resp.usage.cache_read_input_tokens   # Anthropic
    assert cached >= 1000, (
        f"expected the shared prefix to be read from cache, got {cached}; "
        "something varying is sitting ahead of it"
    )

Put that test in the live tier and let it report rather than block, or emit the cache-hit ratio for the run as a job summary alongside the token totals. Either way the number becomes visible, and a change that silently breaks the prefix — someone adding a request id to the system message — shows up the day it lands rather than in the following month’s invoice.