What a Provider Migration Does to Your Prompt Caching TTL Assumptions

Somewhere in a service that has been running for a year there is a constant called something like CACHE_WINDOW_SECONDS = 240, and a batcher that groups requests inside it. It encodes one provider’s cache lifetime. Move to a second provider and the number is not wrong in an obvious way — it is wrong in the way that costs money quietly.

What a cache TTL actually promises

Prompt caching is a prefix match. The provider stores the computed state for the first N tokens of your request and reuses it when a later request begins with byte-identical content. The TTL is the answer to one question: for how long after some event is that stored state still reusable? Every provider answers it, and every provider means a slightly different thing by “event”.

Three variables hide inside that one number, and a migration can change any of them independently. First, whether the clock is a sliding inactivity window (each hit resets it) or a fixed lifetime from creation. Second, whether you choose what gets cached or the provider decides. Third, whether you are billed for the storage duration or only for the reads and writes. A batching strategy is a bet on all three.

Three shapes the same idea takes

Anthropic’s documentation, at the time of writing, has you place the cache explicitly. A content block carries cache_control with a type of ephemeral, defaulting to a five-minute lifetime, and an optional ttl of "1h" for the longer tier. There is a limit of four breakpoints per request and a minimum cacheable prefix that varies by model. Writes cost more than uncached input (roughly 1.25× for the short tier and 2× for the hour tier) and reads cost roughly a tenth. Hits are reported as usage.cache_read_input_tokens and writes as usage.cache_creation_input_tokens. Anthropic’s prompt caching guide carries the current values.

OpenAI’s documentation describes an automatic cache instead: no breakpoints, applied to prefixes of at least 1,024 tokens. Its stated retention differs by model generation — older models keep a prefix for roughly five to ten minutes of inactivity with a one-hour ceiling, while the newer generation is documented as remaining eligible for at least thirty minutes. You do not place the cache, but you can influence which backend a request lands on with the prompt_cache_key parameter. Hits appear at usage.prompt_tokens_details.cached_tokens on Chat Completions and usage.input_tokens_details.cached_tokens on Responses — two different paths inside one vendor, which is worth noticing before you write the adapter. OpenAI’s prompt caching guide is the primary source.

Google’s Gemini API documents both an implicit cache, on by default for recent models above a per-model minimum token count (2,048 for the Gemini 2.5 pair, 4,096 for the newer entries listed at the time of writing), and an explicit cache you create as its own resource with a lifetime you set. Cached tokens are reported at usage.total_cached_tokens. The explicit path is the one that changes the shape of the problem, because an object you create and hold is billed differently from a side effect of a request. Google’s context caching documentation has the current per-model figures.

What does not survive the translation

The concept transfers. Three things do not.

• The value with no counterpart: your breakpoint. If you built a prompt with a deliberate cache boundary between a stable tool list and a volatile user turn, and the target caches automatically, that boundary becomes advisory. You still control it — by ordering content so the stable part comes first — but you can no longer name it. Prompt-assembly code that computes where to put a breakpoint has nowhere to put its answer.
• The default that differs: sliding versus fixed. A batcher built for a sliding window assumes traffic keeps the cache alive for free. Against a fixed-lifetime cache the same traffic buys nothing after expiry, and your effective hit rate falls off a cliff at a predictable moment rather than degrading with idleness.
• The field that means something subtly different: the minimum. A 700-token shared prefix caches on nothing, on either side, and the failure is silent — no error, just zeros in the usage field. If your old provider’s minimum was lower than the new one’s, prompts that were cheap become full price with no code change and no alert.

There is a fourth, and it only bites during the migration itself: caches are scoped to a model. The day you cut over, every prefix is cold. If you dual-run both providers on the same traffic you are paying two sets of cache writes for the entire overlap, which is the single largest line item in most migration cost overruns.

Keep-warm pings, and when they become a bill

The standard trick against an inactivity-based cache is a keep-warm request: a cheap call carrying the shared prefix, fired just under the eviction window, so a burst of real traffic after a quiet stretch lands on a hot cache. Against a sliding inactivity window this is nearly free — you pay one cache read plus a token or two of output.

Against an explicitly created, storage-billed cache it is not a ping at all. The cache exists because you created it, it is billed for as long as it exists, and pinging it does nothing except add requests. The equivalent operation is extending the resource’s lifetime, which is an explicit decision with an explicit cost. Porting the ping loop unchanged gives you an idle background job that produces requests and no savings, and it will not show up as an error anywhere.

The inverse trap is equally common. If the target has a longer documented retention than the source, your batcher is now grouping more aggressively than it needs to, adding latency to individual requests for a cache hit you would have got anyway.

Replacing the constant with a policy

Delete the constant. What belongs in its place is a small per-provider record that the batcher reads, so the migration becomes a config change rather than a code change:

// One record per provider. Values come from the provider's own
// documentation, not from folklore. Re-read them each quarter.
const CACHE_POLICY = {
  providerA: {
    minPrefixTokens: 1024,
    lifetimeSeconds: 300,
    clock: "sliding",        // each hit resets the timer
    placement: "explicit",   // we choose the breakpoint
    warm: "ping",            // cheap request keeps it alive
    storageBilled: false,
  },
  providerB: {
    minPrefixTokens: 1024,
    lifetimeSeconds: 1800,
    clock: "sliding",
    placement: "automatic",  // no breakpoint to set
    warm: "ping",
    storageBilled: false,
  },
  providerC: {
    minPrefixTokens: 2048,
    lifetimeSeconds: 3600,
    clock: "fixed",          // from creation, not from last hit
    placement: "explicit-resource",
    warm: "extend",          // costs money; do it deliberately
    storageBilled: true,
  },
};

// Derive the batch window rather than hard-coding it. 0.6 leaves
// headroom for scheduler jitter and clock skew between hosts.
function batchWindowMs(provider) {
  return CACHE_POLICY[provider].lifetimeSeconds * 1000 * 0.6;
}

// Refuse to promise savings you cannot get.
function willCache(provider, prefixTokens) {
  return prefixTokens >= CACHE_POLICY[provider].minPrefixTokens;
}

The willCache check is the one that pays for itself immediately. Call it during prompt assembly and log when it returns false, and the silent-zero failure mode becomes a line in your logs on the day of the cutover rather than a surprise in the next invoice.

Measuring hit rate across two providers

You cannot tune any of this without one hit-rate number that means the same thing on both sides, and the three usage shapes above do not give you one. Normalise at the adapter boundary: read the provider-specific path, emit one metric with a provider label, and never let provider-shaped field names past that boundary.

function cachedTokens(provider, usage) {
  switch (provider) {
    case "anthropic":
      return usage.cache_read_input_tokens ?? 0;
    case "openai_chat":
      return usage.prompt_tokens_details?.cached_tokens ?? 0;
    case "openai_responses":
      return usage.input_tokens_details?.cached_tokens ?? 0;
    case "google":
      return usage.total_cached_tokens ?? 0;
    default:
      return 0;
  }
}
// hit rate = cachedTokens / (cachedTokens + billableInputTokens)

Chart that per provider for a week before you change the batch window. A cache strategy tuned against a number you have not measured on the target is a guess with a cost attached. The related work on projecting cache cost across a migration takes this hit rate as its input, and the general treatment of cache savings covers what the number is worth once you have it.