Fallback Chains: What to Do When a Model Is Down

A fallback chain is easy to write and easy to write badly. The hard part is not calling the second model; it is that the second model is a different model, and everything downstream of the call was built around the first one.

What a fallback is for

A fallback answers exactly one question: this specific attempt cannot be completed by this specific model, and there exists another model that could produce an acceptable answer. Both halves have to be true, and confusing a fallback with a retry is the most common design error.

• Retry when the same call could plausibly succeed: a transient 5xx, a connection reset, a rate limit with a retry-after. Same model, later.
• Fall back when this route is unavailable but the task is not model-specific: the provider is down, the breaker is open, the model was deprecated, the context is longer than this model accepts, or the deadline no longer allows the slow one.
• Do neither when the failure is about the request: a malformed body, an authentication failure, or a content refusal. A refusal in particular tends to reproduce on the next model, and chaining through four providers to collect four refusals is slow, expensive and no more informative than the first.

The quality cliff

The reason to think carefully rather than list three model names is that the failure a chain protects you from is loud and the failure it introduces is silent. An outage is visible: errors, alerts, a support queue. A chain converts that into a period during which everything returns 200 and the answers are worse.

How much worse is not something a page can tell you, because it is a property of your task and the models in your chain. What a page can tell you is how to find out, and there are only two honest routes. Either you hold a small evaluation set for your actual task and run every model in the chain against it before it enters the chain, or you accept that you do not know and treat every rung below the first as “probably acceptable, unverified”. The second is a defensible position for a low-stakes feature and an indefensible one for anything that touches a decision.

Whichever you choose, the operational requirement is the same: the served model must be recorded on the response and aggregated. Alert on the share of traffic served below rung one, not on individual fallbacks. A chain that fires occasionally is working. A chain that served forty per cent of yesterday’s traffic from rung two is an incident that produced no errors.

Five things that break on the second model

Structured output

If rung one is invoked with a strict schema mode and rung two only supports best-effort JSON, your parser is now facing a different contract on the fallback path — and it is facing it during an incident, which is the worst time to discover that the schema was never validated on that branch. Validate the parsed object against your own schema on every rung, independently of what the provider promised, and treat a validation failure as that rung failing rather than as a successful response.

Tool and function calling

Tool-call encodings differ between vendors, and so do the semantics: whether parallel calls are permitted, how a tool result is fed back, whether the model can emit prose and a call in the same turn. A chain that crosses vendors needs its tool loop written against a normalised representation, or the fallback path is code that has never run.

Prompt portability

Prompts are tuned to models. A system prompt refined against one model’s habits may produce noticeably different behaviour on another — different verbosity, different willingness to follow a negative instruction, different formatting defaults. The practical consequence is that a chain has a prompt per rung, not one prompt, and the per-rung prompts need version control and review just like the primary.

Context limits

Falling back from a long-context model to a shorter one turns a working request into a hard 400 unless something truncates. Decide the truncation policy per rung — drop oldest turns, summarise, or refuse — and make the rung declare its own limit so the chain can skip rungs that cannot hold the input at all rather than discovering it by failure.

Token accounting

Different tokenisers mean the same text costs a different number of tokens on each rung, and usage fields are reported under different names. Any budget check that assumes rung one’s tokenisation is wrong on rung two by an amount that varies with the content.

Ordering the chain

Sort by expected quality on your task, subject to two constraints, and keep it short.

• Diversify the failure domain. Two models from the same provider fall over together. If the point is availability, rung two should be a different company — or at minimum the same model served by a different provider, which is a genuinely independent failure domain for open-weight models.
• Keep it to two or three. Each rung costs a full attempt window. A four-deep chain against a provider having a slow day is a request that takes half a minute to fail, and the deadline page explains why that is worse than failing at rung two.
• Put the deterministic path at the bottom, not another model. If every model in the chain is unreachable, one more model is unlikely to change that. The last rung should be the thing that cannot fail.

Cost and time inversion

Two counterintuitive properties are worth stating plainly. First, a fallback is often more expensive than the primary, not less: if you chose rung one because it was the best value, everything below it is a compromise on some axis, and that axis is sometimes price. A chain that quietly triples your per-request cost during an outage is a bill you will read about later.

Second, the failed attempt is not free. A rung that timed out after eight seconds may have generated tokens you are paying for, and the request that eventually succeeds on rung two has cost you both. Account for the whole chain against the operation, not just the rung that won, or your cost-per-request metric will be systematically optimistic exactly when things are going badly.

Both arguments point the same way: bound the chain by remaining deadline and remaining spend, not by rung count.