Testing That Fallback Providers Are Tried in the Configured Order

A fallback chain that has never failed over has never been tested. In the happy path the first provider answers, the test goes green, and the configured order is an untested assumption sitting in production.

Assert the attempt log

The observable you want is not the answer — every provider returns a plausible one — but the sequence of providers actually attempted. Instrument the client to append to a list on each attempt and assert on the list. If your client does not expose one, the fake transports below can record it themselves, which is often simpler and has the advantage of recording what left the process rather than what your code intended.

import pytest

CHAIN = ["alpha", "beta", "gamma"]

def test_fallback_follows_configured_order(router, fakes):
    fakes.fail("alpha", status=503)
    fakes.fail("beta", status=503)
    fakes.ok("gamma", text="hello")

    result = router.complete("hello")

    assert result.text == "hello"
    assert result.provider == "gamma"
    assert fakes.attempts == ["alpha", "beta", "gamma"]

The last assertion is the page. Without it, the test passes if the router tried gamma first, or tried all three in parallel and took the fastest — behaviour that is defensible but is not what the configuration says, and which triples your cost per request while looking healthy.

Forcing the failures

Table-drive it. The interesting part of a chain is not one scenario but the mapping from a failure pattern to an expected attempt sequence, and written as a table the gaps are visible:

CASES = [
    # id,               failures,                     expected attempts
    ("first-fails",     {"alpha": 503},               ["alpha", "beta"]),
    ("two-fail",        {"alpha": 503, "beta": 429},  ["alpha", "beta", "gamma"]),
    ("all-fail",        {"alpha": 503, "beta": 503,
                         "gamma": 503},               ["alpha", "beta", "gamma"]),
    ("bad-request",     {"alpha": 400},               ["alpha"]),
    ("auth-failure",    {"alpha": 401},               ["alpha", "beta"]),
    ("timeout",         {"alpha": "timeout"},         ["alpha", "beta"]),
]

@pytest.mark.parametrize("failures,expected", [(c[1], c[2]) for c in CASES],
                         ids=[c[0] for c in CASES])
def test_attempt_sequence(router, fakes, failures, expected):
    fakes.configure(failures)
    try:
        router.complete("hello")
    except router.AllProvidersFailed:
        pass
    assert fakes.attempts == expected

Note the all-fail row asserts the sequence and tolerates the exception. The error raised when every provider fails deserves its own assertions: it should name every provider tried and carry the last error from each, because an exhausted chain that reports only the final provider’s error sends everyone to investigate gamma when alpha was the outage.

Build the fakes so a provider is defined by what it does rather than by which client library it stands in for. A fake that returns a status code and records that it was called is enough for every row above, and it keeps the test about routing. The moment a fake starts imitating a vendor’s response schema you have written a second implementation of that vendor and will maintain it forever; if you need real response shapes, take them from recorded fixtures rather than from memory.

The errors that must not fall through

This is the row nobody writes and the one that costs money. A 400 is a statement about your request. Falling through means sending the same malformed request to three vendors, paying up to three times where rejected requests are billable, and returning the third vendor’s error message, which describes a schema the reader has never seen.

• Fall through: 429, 500, 502, 503, 504, connection errors, read timeouts. These are statements about the provider.
• Do not fall through: 400 and 422. The next provider will reject it too, more slowly.
• Fall through and alert: 401 and 403 for authentication. An expired key on one provider should not take you down, but it must be loud — a chain quietly running on its second choice for a month is how a cost surprise happens (a rotated API key breaking CI is the same failure caught earlier).
• Decide explicitly: a content-policy refusal. Some teams fall through in the hope another model complies, which is a policy decision rather than a technical one and should be written down in the test either way (falling back to a different prompt on model error).

Assert the classification directly, one test per status, so widening a retry predicate to fix an unrelated flake cannot silently move a 400 into the retryable set.

Order under concurrency

A chain implemented with a shared cursor — a module-level index of “current provider” — behaves correctly under one request and incoherently under ten. Two requests interleave, one advances the cursor and the other reads it, and the observed order becomes an accident of timing. It will never fail a sequential test.

def test_order_is_per_request(router, fakes):
    fakes.fail("alpha", status=503)
    fakes.ok("beta")

    logs = run_concurrently(router.complete, n=20)

    for log in logs:
        assert log == ["alpha", "beta"]

The related interaction is the circuit breaker. Once alpha’s breaker opens, the correct observed order becomes beta, gamma — which is right, and which breaks a test asserting a literal three-item list. Reset breaker state in a fixture so tests do not leak into each other, and write the open-breaker order as its own case rather than letting it contaminate the others (testing that a circuit breaker opens correctly).

The latency budget

A three-provider chain with a 30-second timeout each is a 90-second worst case, which is well past the point at which the user has left and the upstream gateway has given up. Test the budget, not just the order:

1. Give each fake a controllable delay and set them all to just under the per-provider timeout.
2. Assert the total elapsed time stays under the request deadline. Use a controllable clock rather than real sleeps, or the test costs 90 seconds every run and will be deleted.
3. Assert the per-attempt timeout shrinks as the budget is consumed. A correct implementation gives the third attempt only the time that is left, and a chain of fixed timeouts cannot honour a deadline at all.
4. Assert the caller receives a deadline error, not a generic failure, when the budget runs out mid-chain — those are different incidents and should look different in the logs.