Caching Responses With Cloudflare AI Gateway

AI Gateway can return a stored response instead of calling the provider again, and turning it on is one header. Whether it does anything for you depends entirely on how the cache key is built, which is the part worth reading before you write the header.

The three request headers

Caching in AI Gateway is controlled per request, by headers you add to the call you were already making. Cloudflare’s documentation names three, and they are the whole interface:

• cf-aig-cache-ttl — how long, in seconds, this response may be reused. The documented minimum is 60 seconds and the documented maximum is one month.
• cf-aig-cache-key — replaces the computed key with one you choose.
• cf-aig-skip-cache — bypasses a stored response for this call. Useful on a “regenerate” button, where the user is explicitly asking for a different answer.

One response header comes back: cf-aig-cache-status, whose value is HIT or MISS. That header is the only reliable way to know what happened, and it is what the walkthrough below checks.

Setting a TTL and confirming a hit

The gateway sits in front of a provider, so the change is to the base URL and the headers, not to the request body. Using the OpenAI-compatible endpoint:

1. Make the call once with a TTL. The first response should carry cf-aig-cache-status: MISS, because nothing is stored yet.
2. Make the byte-identical call again inside the TTL window. It should carry cf-aig-cache-status: HIT and return noticeably faster, because no provider call happens.
3. Change one character of the prompt and call again. It will be a MISS, and that is the behaviour the rest of this page is about.

# 1. first call — expect cf-aig-cache-status: MISS
curl -sD - -o /dev/null -X POST \
  "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID/compat/chat/completions" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "cf-aig-cache-ttl: 3600" \
  -d '{
    "model": "openai/gpt-4.1-mini",
    "messages": [{"role": "user", "content": "Name the three states of matter."}]
  }' | grep -i cf-aig-cache-status

# 2. run the exact same command again — expect HIT

-sD - prints the response headers and discards the body, which is all you need to see. If the second call still says MISS, the two request bodies were not identical — a client library that adds a nonce, a timestamp, or a reordered JSON key is enough.

How the cache key is computed

This is the mechanism everything else follows from. Cloudflare documents the default key as a SHA-256 hash of five things concatenated: the provider, the endpoint, the model, the provider authentication header, and the full request body.

Read that list twice. The full request body means every message in the conversation, every tool definition you attached, the temperature, the seed, the system prompt — all of it. There is no semantic matching and no normalisation. Two prompts that mean the same thing hash differently. Two identical prompts sent with different sampling parameters hash differently. The same prompt sent by two users whose keys differ hashes differently, because the authentication header is in the key.

The last point is deliberate rather than accidental: it means one tenant’s answers cannot be served to another tenant that authenticates with a different provider key. It also means that if you use a single shared provider key for all your traffic, cache entries are shared across your users, which is what you want for a public knowledge-base bot and emphatically not what you want if responses contain anything user-specific.

Why your hit rate is lower than you expect

A body-hash cache rewards repetition and punishes personalisation. The workloads where it earns its place look like this:

• Classification and routing calls. A short fixed prompt with a small input, called constantly. Support-ticket triage sends the same twelve category definitions every time.
• Embedding a fixed corpus. Re-indexing the same documents after a deploy recomputes identical inputs.
• Demo and evaluation traffic. Test suites and landing-page demos replay the same handful of prompts thousands of times.

And the ones where it will not:

• Any chat with history. Every turn appends to the message array, so every turn is a new body and a new key. The hit rate of a multi-turn assistant is close to zero by construction.
• Prompts containing the current time or a request id. One injected timestamp in a system prompt makes every request unique. If you inject a date for grounding, inject the date and not the second.
• Anything you wanted to vary. A cached response is frozen. If you set a non-zero temperature to get variety and then cache for an hour, you have bought one sample and are serving it for an hour.

There is a billing consequence worth knowing: Cloudflare documents that when a response is served from cache, the recorded cost is zero even if you supplied a custom cost. That is correct — you were not charged by the provider — but it means the cost line in the gateway’s analytics drops as your hit rate rises, and you cannot read the cached-response percentage off the cost chart alone.

Overriding the key deliberately

cf-aig-cache-key exists for the case where you know two different bodies should share an answer. The obvious use is stripping the parts of a request that do not affect the output: if your prompt template embeds a request id for tracing, hash the semantic part yourself and send that as the key.

// Worker: cache on the question alone, not on the traced body
const question = "Name the three states of matter.";
const digest = await crypto.subtle.digest(
  "SHA-256",
  new TextEncoder().encode("faq:v3:" + question),
);
const cacheKey = [...new Uint8Array(digest)]
  .map((b) => b.toString(16).padStart(2, "0"))
  .join("");

const res = await fetch(gatewayUrl, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: `Bearer ${env.CF_API_TOKEN}`,
    "cf-aig-cache-ttl": "86400",
    "cf-aig-cache-key": cacheKey,
  },
  body: JSON.stringify(payload),
});
console.log(res.headers.get("cf-aig-cache-status"));

Note the faq:v3: prefix. A custom key takes ownership of invalidation away from the body hash, which means changing your prompt template no longer changes the key — you will keep serving answers generated by the old template until the TTL expires. Versioning the prefix and bumping it on every prompt change is the cheapest fix, and the mistake is easy to make because nothing breaks visibly.

For the same reason, keep the TTL short while you are still changing prompts. A one-month TTL on a prompt you edit weekly is a month of stale answers you cannot see. Read the rate-limiting page next if the reason you wanted caching was to stop one caller exhausting a shared provider key — caching helps only if that caller’s requests repeat, and a limit is the direct answer.