Automatic Prompt Caching in the OpenAI API: The 1,024-Token Floor

There is nothing to enable. OpenAI caches long prompt prefixes automatically, discounts the cached portion, and reports what it matched — but only above a documented floor, and only for a prefix that is byte-identical from the very first token. Both halves of that sentence are things you can accidentally break.

The rule

As documented by OpenAI when automatic prompt caching was announced on 1 October 2024:

• A prompt must be at least 1,024 tokens for any of it to be cached. Below that floor, nothing caches, no matter how many times you send it. There is no partial credit at 1,000 tokens.
• Matching happens in increments of 128 tokens above the floor — so a cache hit covers 1,024, then 1,152, then 1,280, and so on. The remainder beyond the last increment is charged at the full rate.
• Only a prefix is matched. The cache keys on the longest identical run starting from the first token of the request. One character different at position 3 and the entire rest of the prompt is a miss, however much of it is unchanged.
• Cached input tokens are billed at a reduced rate and served faster, because the attention keys and values for that prefix have already been computed and can be reused rather than recalculated from scratch.

One request that caches and one that does not

These two send the same information. Only the first can ever produce a cache hit:

// QUALIFIES. Static content first, byte-identical every call.
{
  "model": "gpt-4o-2024-08-06",
  "messages": [
    { "role": "system",
      "content": "<3,000 tokens of policy, schema and few-shot examples>" },
    { "role": "user",
      "content": "Session 4c1f. 2026-08-11T09:14Z. Classify: ..." }
  ]
}

// DOES NOT QUALIFY. The first token of the prompt changes every call.
{
  "model": "gpt-4o-2024-08-06",
  "messages": [
    { "role": "system",
      "content": "Session 4c1f. Current time 2026-08-11T09:14Z.\n"
                 + "<3,000 tokens of policy, schema and few-shot examples>" },
    { "role": "user",
      "content": "Classify: ..." }
  ]
}

The second one is the mistake almost everybody makes once, and it is invisible: the request succeeds, the output is correct, and the bill is simply higher than it needed to be. A session id, a timestamp, a user name or a “today is” line at the top of the system message moves the divergence point to token 5, so the 3,000 tokens of identical policy behind it are re-charged in full on every call.

The fix is mechanical. Everything that is the same across calls goes first, in a fixed order; everything that varies goes after it. That is the whole optimisation, and it is worth doing before any other cost work because it costs nothing and changes no behaviour.

Reading the cached token count

You do not have to guess whether it worked. The usage object carries a breakdown:

"usage": {
  "prompt_tokens": 3184,
  "completion_tokens": 112,
  "total_tokens": 3296,
  "prompt_tokens_details": {
    "cached_tokens": 3072
  }
}

cached_tokens is a subset of prompt_tokens, not an addition to it — 3,072 of the 3,184 input tokens were served from cache and billed at the reduced rate, and 112 were not. Note that 3,072 is 1,024 + 16×128, which is the increment rule visible in a real response: the match was rounded down to a boundary rather than covering every identical token.

One accounting note. Cached tokens are still counted in prompt_tokens, so a prompt does not get shorter by being cached and the context window is unaffected — caching is a discount, not compression. If you are reconciling this against a conversation whose history grows every turn, the underlying token growth is the subject of counting tokens across a multi-turn conversation, and caching changes what that growth costs without changing the growth.

Log the ratio of cached_tokens to prompt_tokens per route. It is the single most actionable cost metric in the API, because unlike most cost metrics there is a specific structural change that moves it, and a route sitting at zero is usually one variable line in the wrong place.

How long a cache entry lives

OpenAI documents eviction after a period of inactivity — on the order of five to ten minutes idle, with entries cleared within about an hour regardless. Three consequences that follow directly:

• Caching rewards traffic, not repetition. A prompt sent once an hour caches nothing useful. A prompt sent every few seconds by many users caches almost perfectly. This is why the benefit lands hardest on the shared system prompt of a busy application and barely at all on a nightly batch job.
• The first request after a quiet period pays full price. Do not treat a single cold measurement as evidence the feature is not working.
• It is a rate discount, not a correctness feature. The cache stores computed attention state for a prefix, so a hit cannot change the output relative to a miss. Nothing about your results depends on whether it hit.

Caches are scoped to your organisation and are not shared between organisations, which is the answer to the obvious question about sending a proprietary system prompt through a shared cache.

Why a route reports zero cached tokens

When cached_tokens sits at zero on a prompt you believe is stable, the cause is almost always one of six things, in roughly descending order of how often it turns out to be the answer:

• The prompt is under 1,024 tokens. Check the actual count in usage.prompt_tokens rather than estimating. Prompts people describe as long are frequently 600.
• Something volatile is at the top. A timestamp, a request id, a user name, a randomised few-shot ordering. One token of difference at position 4 costs you the entire prefix.
• Tool definitions are serialised in a non-deterministic order. They are part of the prompt. A tool list built by iterating a hash map can come out differently on different processes, which produces a cache that works within one replica and never across them.
• Traffic is too sparse. Entries idle out in minutes. A route called twice an hour will never hit, and no amount of prompt restructuring changes that.
• The model id changed. The cache is per model, so an alias moving to a new snapshot invalidates everything — one more reason to pin a dated snapshot.
• Requests are spread across organisations or projects. Caches do not cross those boundaries. Splitting one workload across two projects halves the hit rate for no benefit.

The diagnostic that resolves all six in one go is to log the first 200 characters of the serialised prompt alongside cached_tokens for a few hundred requests and look for what varies. If the first 200 characters are ever different between two requests on the same route, you have found it without needing to reason about the rest.

Ordering a prompt to be cacheable

• Static first, in a fixed order. System instructions, tool and function definitions, schemas, few-shot examples, and any large document that is constant for the route. If the tool list is assembled from an object whose key order is not stable, serialise it deterministically — an unordered map is a moving prefix.
• Then the slowly-changing part. Retrieved documents for a session, conversation history. In a chat, history naturally grows by appending, which is exactly the shape the prefix cache wants: turn 12 shares its whole prefix with turn 11.
• Volatile last, always. Timestamps, request ids, user identifiers, the current question.
• Do not trim from the front. A conversation-history trimmer that drops the oldest turns to fit the window invalidates the prefix on the turn it fires and every turn after. Summarise into a block that then stays fixed instead, so the prefix stabilises again.