Skip to content
Multigrid

Testing Failover When Two Providers Are Down at Once

9 min read · updated August 11, 2026

Failover tests almost always stop one step early: primary fails, backup succeeds, green tick. The interesting state is the next one along, where the backup fails too, and it is interesting because it is the state in which a service either degrades cleanly or amplifies an outage it did not cause.

The branch nobody tests

Correlated provider failure is not exotic. Two providers can share a cloud region, a DNS resolver, an upstream model host, or your own egress proxy; and a fault entirely inside your service — an expired credential set, a network policy change, a bad deploy of your own HTTP client configuration — will present as every provider failing at once, because it is.

What runs in that state is usually code nobody has read. It is the bottom of a chain of except clauses, or the fall-through after a for loop over providers, and it typically does one of three unhelpful things: raises whatever the last provider’s exception happened to be, retries the whole chain forever, or returns a 200 with an empty string. All three are worse than a clean failure and all three are cheap to test.

A fixture that fails everything

Parameterise the failure mode rather than writing one test. The exhaustion path must behave identically whether the providers timed out, refused the connection, returned 500, or returned 429, and the only way to know that is to run the same assertions against each.

# test_all_providers_down.py
import httpx, pytest, respx
from myapp.llm import complete, AllProvidersFailed

PROVIDERS = [
    "https://api.primary.example/v1/chat/completions",
    "https://api.backup.example/v1/chat/completions",
]

FAILURES = {
    "read_timeout":   httpx.ReadTimeout,
    "connect_error":  httpx.ConnectError,
    "server_error":   lambda request: httpx.Response(503, json={"error": "unavailable"}),
    "rate_limited":   lambda request: httpx.Response(429, json={"error": "slow down"}),
    "bad_gateway":    lambda request: httpx.Response(502, text="<html>proxy</html>"),
}

@pytest.mark.parametrize("mode", list(FAILURES))
@respx.mock
def test_every_provider_down_raises_one_error(mode):
    effect = FAILURES[mode]
    routes = [respx.post(url).mock(side_effect=effect) for url in PROVIDERS]

    with pytest.raises(AllProvidersFailed) as excinfo:
        complete("hello")

    assert all(r.called for r in routes), "a provider was never tried"
    err = excinfo.value
    assert err.attempts == sum(r.call_count for r in routes)
    assert set(err.providers_tried) == {"primary", "backup"}
    assert mode in err.summary or err.last_error is not None

The bad_gateway case earns its place: a 502 from an intermediary returns HTML, not JSON, and code that reaches for response.json()["error"]["message"] in its error handler raises a JSONDecodeError from inside the error handler. That is how an exhaustion path stops producing your error type and starts producing a stack trace, and it is invisible until you mock a non-JSON body.

The exhaustion contract

Write down what your service promises when everything is down, then assert each clause. A workable contract has four:

  • One error type, regardless of cause. Callers should branch on “no provider could serve this”, not on which of five underlying exceptions surfaced. Carry the causes as data on the exception — providers_tried, attempts, last_error — where a log line can pick them up.
  • A bounded total attempt count. Assert the exact number. Two providers with two retries each is four requests; if your test finds seven, you have nested retries at two layers, which is the classic way a small provider blip becomes a self-inflicted load spike.
  • A bounded total time. Same technique as the single-timeout case: patch the sleep, sum the intended delays, assert the sum fits inside the caller’s deadline. Exhaustion is the worst case by definition, so this is the number that decides whether your upstream times out on you.
  • A status code your callers can act on. If this surfaces over HTTP, 503 with a Retry-After is meaningful and 500 is not. Assert the mapping in an API-level test, because the translation from exception to status is a separate piece of code from the routing.

Not making it worse

The exhaustion path is where retry amplification is born, and the arithmetic is unforgiving. Suppose your client retries twice, your gateway layer retries twice, and your caller’s SDK retries twice. Those multiply rather than add: a single user action becomes 2 × 2 × 2 = 8 attempts per provider, 16 across two providers. If a provider is failing because it is overloaded, you have just increased your own contribution to its load by sixteen times at precisely the moment it can least absorb it. Those numbers are arithmetic from the three retry counts named in this paragraph; substitute yours and the shape does not change.

Two structural defences, both testable. First, decide which single layer owns retries and assert that the others are configured to zero — for the OpenAI SDK that is max_retries=0 on the client, and a test that constructs your client and asserts the value is worth more than a comment. Second, put a circuit breaker in front of each provider so that after a threshold of consecutive failures the calls stop being made at all. Test the breaker’s open state directly: fail it past the threshold, then assert that the next call raises immediately with call_count unchanged. That last assertion is the entire value of a breaker — if the request still went out, the breaker is decoration. The general mechanism is covered in circuit breakers for AI calls.

Degraded is a product decision

Once the technical contract holds, the remaining question is not engineering: what should the user see? The options are real and different, and the test you write depends on which one you chose.

  • Fail visibly. An error with a retry affordance. Correct for anything interactive, and the only honest answer when the model’s output was the product.
  • Queue it. Accept the request, return an identifier, process when a provider returns. Correct for batch and async work, and the test asserts the job is durably enqueued rather than dropped.
  • Serve a non-model path. A keyword search instead of a semantic one, a template instead of a generated summary. The test asserts the fallback is clearly labelled — silently degrading model output to a template is how you get a support ticket six weeks later about quality having “drifted”.

Whichever you pick, assert that the request is recorded as an exhaustion event with the same identifier the user can quote. During a real incident, the question you will be asked is how many users hit this, and the answer has to come from a counter that existed beforehand.

Related