Configuring a Fallback Provider in Cloudflare AI Gateway

A fallback is a second provider the gateway tries when the first one fails, configured once instead of written into every call site. It is genuinely useful and it is narrower than people assume, so this page builds one and then says exactly what it leaves uncovered.

Two configuration surfaces

Cloudflare currently documents two ways to express “try this, then that”, and which one you should use depends on how much longer you want the configuration to live.

• The Universal Endpoint. You post an ordered array of provider objects to the gateway and it walks the array. Cloudflare documents this endpoint as deprecated, with existing integrations continuing to work, and points new work at Dynamic Routing.
• Dynamic Routing. A small graph of nodes — start, model, conditional, percentage, rate, end — where a model node carries a fallback output alongside its success output.

Both are shown below, because a great deal of existing code is on the first and the migration is not automatic.

The Universal Endpoint array

The request body is an array. Each element names a provider, an endpoint, its own headers (including that provider’s authorization), and the query — the body you would have sent to that provider directly. The gateway tries element zero, and on failure moves to element one.

curl "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID" \
  -H 'Content-Type: application/json' \
  --data '[
    {
      "provider": "workers-ai",
      "endpoint": "@cf/meta/llama-3.1-8b-instruct",
      "headers": { "Authorization": "Bearer CF_TOKEN", "Content-Type": "application/json" },
      "query": { "messages": [{ "role": "user", "content": "What is Cloudflare?" }] }
    },
    {
      "provider": "openai",
      "endpoint": "chat/completions",
      "headers": { "Authorization": "Bearer OPENAI_KEY", "Content-Type": "application/json" },
      "query": { "model": "gpt-4.1-mini", "messages": [{ "role": "user", "content": "What is Cloudflare?" }] }
    }
  ]'

The critical detail is the second element’s query: it is a complete, provider-shaped request, not a copy of the first. Two providers do not agree on parameter names, on how a system prompt is expressed, or on what a tool definition looks like. You are writing the request twice, and the second copy is the one nobody notices has drifted until the first provider goes down.

The response tells you which one answered. Cloudflare documents the cf-aig-step response header as carrying the step number: 0 for the primary, 1 for the second, incrementing per fallback. Log it. Without it, a permanently failing primary looks exactly like a healthy system, because the user gets an answer either way — and you find out at the invoice.

A Dynamic Routing model node

Dynamic Routing expresses the same thing as a graph you store once and then address by name. Cloudflare’s JSON configuration documents a route as an id, a name and an elements array; a model element carries provider, model, timeout in milliseconds and retries, and has success and fallback outputs.

{
  "id": "chat-with-fallback",
  "name": "Chat with fallback",
  "elements": [
    { "id": "start", "type": "start", "outputs": { "next": { "elementId": "primary" } } },
    {
      "id": "primary",
      "type": "model",
      "properties": { "provider": "openai", "model": "gpt-4o-mini", "timeout": 20000, "retries": 1 },
      "outputs": { "success": { "elementId": "done" }, "fallback": { "elementId": "secondary" } }
    },
    {
      "id": "secondary",
      "type": "model",
      "properties": { "provider": "workers-ai", "model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast", "timeout": 20000, "retries": 0 },
      "outputs": { "success": { "elementId": "done" }, "fallback": { "elementId": "done" } }
    },
    { "id": "done", "type": "end", "outputs": {} }
  ]
}

Two properties are doing real work here. timeout is what converts a hung provider into a fallback — without it, a provider that accepts your connection and never responds is not an error, it is a wait, and no fallback fires. retries is what stops a single transient 500 from moving traffic off your primary, and it is also what multiplies your latency: one retry on a 20-second timeout means the fallback may not start for 40 seconds.

Once stored, the route is addressed like a model. Cloudflare’s OpenAI-compatible endpoint accepts a model string in the form provider/model, and dynamic routes appear under the dynamic provider — so the call site becomes one string, and the routing policy is no longer in your application at all.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: env.CF_API_TOKEN,
  baseURL: `https://gateway.ai.cloudflare.com/v1/${accountId}/${gatewayId}/compat`,
});

const res = await client.chat.completions.create({
  model: "dynamic/chat-with-fallback",
  messages: [{ role: "user", content: "What is Cloudflare?" }],
});

Forcing a failure to prove it works

An untested fallback is a comment. The cheapest way to fire one on demand is to break authentication on the primary, because it produces an unambiguous provider error immediately and costs nothing.

1. Send the Universal Endpoint request above with a deliberately invalid value in the first element’s Authorization header — Bearer not-a-real-key will do.
2. Confirm the response body is the second provider’s shape, not the first. This is the check that catches a stale second query.
3. Confirm cf-aig-step is 1. If the header is absent, you are talking to a different endpoint than you think.
4. For a timeout rather than an error, point a model node at a timeout of 1 millisecond and confirm the fallback fires on the same request that succeeded before.
5. Repeat the whole thing in CI on a schedule. Fallback configuration rots silently, and the day you need it is the worst day to find out.

What a fallback does not fix

The gateway retries a request against another provider. That is all it does, and four categories of trouble sit outside it:

• Response shape. Two providers return different JSON. The gateway’s OpenAI-compatible endpoint normalises a great deal of that, but tool-call formats, refusal signals and finish reasons still differ in ways your parser will notice. Your code has to accept either.
• Streaming. Fallback decisions are cheap before the first byte and expensive after it. Once the primary has streamed 200 tokens to your user and then dies, no gateway can un-send them; the recovery has to happen in your application.
• Double billing. A primary that fails after generating a response — a timeout on a slow but successful call — has already charged you. You pay for both steps and see one answer.
• Quality. The fallback model is a different model. Prompts tuned for one are not tuned for the other, and a fallback that fires for an hour is an hour of quietly worse output. Log cf-aig-step alongside whatever quality signal you have.