Failover and retries

Two loops, not one

A request walks a list of candidate routes, and each candidate gets several attempts of its own before the walk moves on. The inner loop means “this provider is briefly unwell”; the outer means “this provider is no use to us”.

Failing over on the first 429 looks robust and is not: most models in the catalogue have exactly one route, and for those the outer loop has nowhere to go — the retry is the only thing standing between a blip and an error you see.

{
  "model": "meta/llama-3.3-70b-instruct",
  "models": ["openai/gpt-oss-20b", "openai/gpt-5-nano"],
  "messages": [{"role": "user", "content": "…"}]
}

The chain above gives three models. Every provider for the first is tried before the second model is considered at all. Set provider.allow_fallbacks: false to stop after the first usable route.

What counts as transient

Backoff and deadlines

• Three attempts per candidate by default, the first included.
• Full jitter. The delay is a uniform pick from zero up to a capped exponential, starting at 250 ms and ceilinged at 4 s. Fixed delays make every client retry in the same instant and re-create the overload that caused the 429.
• The provider’s Retry-After wins over our backoff, in either legal form — seconds or an HTTP date. It is the only party that knows when it will be ready.
• The whole walk has a deadline: 60 seconds non-streaming, 300 seconds streaming. If a wait would consume what is left of it, we fail over now instead. A retried request never takes longer than one that failed outright would have.

Streams

Retry and failover apply right up until the first byte reaches you. After that the status line is committed and the answer has started, so we do not switch providers and continue — the result would be two models’ output spliced into one answer with nothing in the response to say so. What you get instead is an event: error on the same stream, plus every token produced before the break, which are logged and billed.

Reading what happened

attempts on the response tells you whether anything went wrong at all. The detail — which provider, which status, which try — is on the request row and comes back from GET /generation:

curl "https://api.multigrid.ai/v1/generation?id=req_…" \
  -H "Authorization: Bearer $MULTIGRID_API_KEY"

# → "attempts": 3,
#   "routing": { "failures": [
#       { "provider": "deepinfra", "status": 503, "reason": "http_503" },
#       { "provider": "deepinfra", "status": 503, "reason": "http_503 (try 2)" }
#     ] }

The same failures drive the failover table on the Routing page, which is the only evidence that failover is doing anything rather than being a feature on a slide.