Migrating a Prompt-Caching Setup Between Providers

Prompt caching is the one optimisation that a provider migration can silently undo in full. The requests still succeed, the answers are still correct, and the bill goes up by a multiple because a prompt that was structured for one caching model is not structured for the other.

The one invariant both models share

Whatever the syntax, every prompt cache in production today is a prefix match. The cache key is derived from the exact bytes of the rendered prompt from position zero up to some point. A single byte changed at position N invalidates the cache for everything at or after N. There is no partial credit, no fuzzy matching, and no semantic similarity involved.

Everything else follows from that. It is why a timestamp in a system prompt is catastrophic and a timestamp in the final user turn is free. It is why a tool list serialised from a hash map without a sort order destroys caching non-deterministically. And it is why the two invocation models — explicit and automatic — are less different than they appear: both are asking the same question, and differ only in who decides where the prefix ends.

Explicit breakpoints and automatic matching

In the explicit model, you mark the end of the cacheable prefix yourself. On Anthropic’s API this is a cache_control: { "type": "ephemeral" } object attached to a content block; the prefix up to and including that block becomes a cache entry. You get a small number of breakpoints per request — four, at the time of writing — and the response reports what happened in usage.cache_creation_input_tokens and usage.cache_read_input_tokens.

In the automatic model, there is no parameter to set. The provider hashes prefixes of your request at some granularity and serves a hit if it recognises one. OpenAI’s automatic caching works this way, reporting hits in usage.prompt_tokens_details.cached_tokens, and offers an optional prompt_cache_key to improve routing affinity for requests that share a prefix.

The migration consequences run in both directions, and neither is symmetrical with the other.

• Automatic to explicit. Your prompts have no breakpoints because they never needed any. Ported unchanged, they cache nothing — not less, none. Every request is a full-price prefill. This is the single most common cause of a bill jumping after a move, and it produces no error and no warning.
• Explicit to automatic. Your breakpoints become inert, which is harmless, but the discipline they enforced silently stops being enforced. You may also lose the ability to cache a middle segment while leaving a later one uncached, because you no longer control where the prefix ends.

Render order decides what is cacheable

This is the part that actually requires restructuring the prompt, and it is where most of the work in a caching migration lives.

A request is not just a message array. Tool definitions, the system prompt and the messages are rendered into one sequence in a fixed order — on Anthropic’s API that order is tools, then system, then messages. Tools sit at position zero. That has a hard consequence: changing the tool set invalidates everything. Adding one tool, removing one, or serialising the array in a different order on each process start invalidates the system prompt cache and the entire conversation history with it.

If you are arriving from an API where tools felt like a per-request detail, this is a genuine change in how you have to build requests. Sort tool definitions deterministically. Do not build the tool list from a set. Do not add a “mode” by swapping tools in and out mid-conversation.

The same reasoning applies one level up to the system prompt. Anything interpolated into it — the current date, the user’s name, a feature flag, a session id — sits ahead of the entire conversation and invalidates all of it on every request. The fix is not to delete the dynamic content but to move it after the last breakpoint, into a later message. A fact injected at turn five invalidates nothing before turn five.

// Cache-hostile: the date sits at the front of every request.
system: `Today is ${new Date().toISOString()}.\n` + STABLE_INSTRUCTIONS

// Cache-friendly: stable prefix first, volatile content after it.
system: [
  { type: "text", text: STABLE_INSTRUCTIONS,
    cache_control: { type: "ephemeral" } },
],
messages: [
  ...history,
  { role: "user", content: `(context: today is ${today})\n${question}` },
]

Minimum sizes and TTLs

Two numeric properties differ between providers and both cause confusion because falling below them produces no error.

Minimum cacheable prefix. Below some token count, a prefix will not be cached at all — the marker is accepted, the request succeeds, and the reported cache-creation token count is simply zero. The threshold is model-dependent as well as provider-dependent, and it is not monotonic across model generations: a prompt that caches on one model in a family can fail to cache on another. If your system prompt sits near the threshold, this alone can explain a cache that “stopped working” after a model change with no code change at all.

TTL. Explicit caches typically carry a short default lifetime — five minutes on Anthropic’s API, refreshed on each read, with a longer one-hour option at a higher write cost. Automatic caches are evicted on their own schedule, generally after a few minutes of inactivity on that prefix. Either way, the operative question for your workload is: how often does a given prefix get touched? A prefix used every thirty seconds stays warm indefinitely. A prefix used twice an hour never hits, no matter how well-structured it is, and no amount of breakpoint placement will change that.

Porting a caching setup

1. Before changing anything, record the current hit rate. Sum the cached-token field over a day of traffic and divide by total input tokens. Without that number you cannot tell whether the migration hurt.
2. Render one representative request to a string on both providers and diff them. You are looking for the position at which they stop agreeing, because everything after that point is a separate cache entry.
3. Find the boundary between stable and per-request content. In most applications it is exactly one place: the end of the system prompt and tool definitions, before the first user turn.
4. Move every dynamic value that sits before that boundary to after it. Timestamps, ids, flags, per-user strings. This is the restructuring step and it is usually the only code change of substance.
5. Add the breakpoint at the boundary if the new provider uses explicit markers. For a multi-turn conversation, add a second on the last block of the most recent turn so the history accumulates as a growing cached prefix.
6. Send the same request twice and assert that the second reports a non-zero cache read. If it reports zero, something in the prefix differs between the two calls — diff the rendered bytes.
7. Re-measure the hit rate over a full day, and read what a fresh cache does to the numbers in the first hours before concluding anything from the first sample.