Rate Limiting a Model Provider With Cloudflare AI Gateway

A shared provider key has no notion of who is calling it. The provider throttles your account as a whole, so a single misbehaving job takes the whole application down with it. A gateway-side limit is where you get to say “stop” before the provider does.

The problem: one key, many callers

Provider rate limits are enforced on the credential. When a batch job, a retry storm, or a test suite runs hot, the provider returns 429s to everything using that key — including the interactive traffic you care most about. Nothing at the provider distinguishes the batch job from a user waiting on a page.

A gateway limit changes where the rejection happens. Cloudflare documents AI Gateway rate limiting as allowing a certain number of requests within a window of time, and that when the limit is exceeded the server responds with 429 Too Many Requests and the request is not processed. The important word is not processed: the request never reaches the provider, so it never consumes provider quota and never costs anything.

Configuring the limit

Three fields carry the whole configuration, and they are the same three whether you set them in the dashboard or through the API: rate_limiting_interval, rate_limiting_limit and rate_limiting_technique. Cloudflare’s API reference documents gateway creation as POST /accounts/{account_id}/ai-gateway/gateways, with rate_limiting_technique taking fixed or sliding.

curl -X POST \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "batch-jobs",
    "cache_ttl": 0,
    "cache_invalidate_on_update": false,
    "collect_logs": true,
    "rate_limiting_interval": 60,
    "rate_limiting_limit": 120,
    "rate_limiting_technique": "sliding"
  }'

Write it against the API rather than clicking through the dashboard, for the ordinary reason: a gateway created by a curl command in your repository is a gateway you can recreate, review in a diff, and apply to a second account. The dashboard equivalent lives under AI › AI Gateway › Settings, and that path is the part of this page most likely to be wrong in a year.

Fixed window versus sliding window

The two techniques differ in one behaviour that matters at the boundary. A fixed window counts against absolute time boundaries: with a 60-second interval, the counter resets on the minute. That permits a burst of up to twice the limit across a boundary — the last second of one window and the first second of the next — which a caller retrying on a fixed schedule will find without trying.

A sliding window evaluates the trailing interval continuously, so the boundary burst does not exist. Cloudflare describes it as evaluating the last ten minutes rather than a calendar block. It is the safer default when the thing you are protecting is a hard provider quota, and the reason to choose fixed instead is predictability: a fixed window gives callers a known reset instant to back off to.

Getting per-caller isolation

Here is the thing the feature page does not put in a box. The limit is a property of the gateway, not of the caller. Every request through that gateway counts against the same counter. If your goal was “one tenant cannot exhaust the key”, a single gateway with a single limit does not give you that — it gives you a ceiling on total spend and total provider load, which is a different and also useful thing.

Two configurations do produce isolation:

• A gateway per tenant or per workload. Gateways are cheap objects created by an API call, and the gateway id is part of the URL, so routing a tenant to its own gateway is a string substitution in your base URL. Batch jobs on one gateway with a low limit, interactive traffic on another with a high one, is the simplest version and solves the common case.
• A Dynamic Routing rate node. Cloudflare’s dynamic routing configuration documents a rate element with success and fallback outputs, so instead of returning a 429 you can send over-limit traffic down a different branch — typically to a cheaper model. That turns the limit from a rejection into a degradation, which is usually what a product wants.

What does not give you enforcement is cf-aig-metadata. That header tags requests with up to five custom entries — a user id, a team name — and Cloudflare documents it as improving log filtering and analysis. It is attribution, not control. Tag with it so you can find out who caused the spike; use a separate gateway or a rate node to stop them.

Handling the 429 on the client

A gateway 429 arrives at your code looking much like a provider 429, and the difference matters: a provider 429 usually means back off and try later, while a gateway 429 means you have hit a limit you set and nothing will improve until the window rolls. Retrying immediately against your own limit is pure waste.

const res = await fetch(gatewayUrl, init);

if (res.status === 429) {
  // Your own limit, not the provider's. Do not hammer it.
  const retryAfter = Number(res.headers.get("retry-after") ?? "0");
  const waitMs = retryAfter > 0 ? retryAfter * 1000 : windowMs;
  await scheduleRetry(job, waitMs);   // queue it, do not spin
  return;
}

For background work, the right destination for a rejected request is a queue rather than a sleep — see Cloudflare Queues for background model processing, which gives you a retry with a delay and a dead-letter queue for the jobs that never succeed. And if the reason one caller is generating so much traffic is that it repeats itself, caching removes the load instead of rejecting it.