Injecting a Fake Provider Timeout to Test Your Failover Path

Most failover tests mock a 500 and check that the second provider answers. That is the easy half. The failure that actually takes services down is the one where the provider accepts your connection, returns nothing, and holds it — because a code path that branches on a status code never runs when there is no status code.

A timeout is not an error response

Take the two paths seriously as different code. On an error response, your client library gets headers, a status, and usually a body; it raises something with a status attached, your handler matches on it, and you route elsewhere. On a hang there is a socket with nothing coming down it. Whether you ever leave that state depends entirely on a timeout being configured, and on the timeout being shorter than whatever is upstream of you.

Three things go wrong there in practice, and all three are testable:

• No timeout is set. The client’s default applies. The OpenAI Python SDK documents a default of ten minutes, which is not a failover trigger — it is an outage.
• The timeout is set but the retry policy eats it. The same SDK documents two automatic retries on connection errors and timeouts. A 30-second read timeout with two retries is 90 seconds before your own failover code is reached, and your caller gave up at 30.
• The timeout fires but the exception is caught too broadly. A bare handler that turns everything into “sorry, something went wrong” is indistinguishable from working failover in a test that only checks for a non-500 response.

Injecting the hang

You do not need a real slow server. The OpenAI Python SDK is built on httpx, and respx mocks httpx transports, so a documented side_effect that raises an httpx timeout exception reproduces the client-side observable behaviour exactly: the same exception type, from the same layer, at the same point in your code.

# test_failover_timeout.py
import httpx, pytest, respx
from myapp.llm import complete   # your failover wrapper

PRIMARY = "https://api.primary.example/v1/chat/completions"
BACKUP  = "https://api.backup.example/v1/chat/completions"

def ok(text):
    return httpx.Response(200, json={
        "id": "chatcmpl-test",
        "choices": [{
            "index": 0,
            "finish_reason": "stop",
            "message": {"role": "assistant", "content": text},
        }],
        "usage": {"prompt_tokens": 11, "completion_tokens": 3, "total_tokens": 14},
    })

@respx.mock
def test_read_timeout_on_primary_fails_over():
    primary = respx.post(PRIMARY).mock(side_effect=httpx.ReadTimeout)
    backup  = respx.post(BACKUP).mock(return_value=ok("from backup"))

    result = complete("hello")

    assert primary.called
    assert backup.called
    assert result.provider == "backup"
    assert result.text == "from backup"

@respx.mock
def test_connect_timeout_also_fails_over():
    respx.post(PRIMARY).mock(side_effect=httpx.ConnectTimeout)
    backup = respx.post(BACKUP).mock(return_value=ok("from backup"))
    assert complete("hello").provider == "backup"
    assert backup.called

Write both. ConnectTimeout and ReadTimeout are different exceptions arriving at different moments, and a handler written against one of them frequently misses the other. If your code catches the SDK’s own wrapper type rather than the httpx type, catch openai.APITimeoutError — the SDK documents it as the class raised when a request exceeds its timeout, and it is what your production code should be matching on.

The assertion that matters

“It returned something” is not the assertion. Three are:

1. The backup was actually called. assert backup.called. Without it, a wrapper that catches the timeout and returns a cached or canned answer passes a test about failover while never failing over.
2. The primary was called exactly the number of times your retry policy says. assert primary.call_count == 1 if you have disabled SDK-level retries in favour of your own routing. If this is 3 and you expected 1, you have discovered that two invisible retries sit between your timeout and your failover.
3. The response is attributed. Whatever field carries which provider served the request — and there should be one, because you will need it in your logs during an incident — must say backup. A failover you cannot observe after the fact is one you will argue about at three in the morning.

The budget assertion nobody writes

Failover that works but takes 95 seconds is a failure with extra steps. The way to catch it is to assert on the total elapsed time under a clock you control, so the test stays fast and deterministic.

import time

@respx.mock
def test_failover_stays_inside_the_budget(monkeypatch):
    calls = []
    def fake_sleep(seconds):
        calls.append(seconds)          # record backoff, do not actually wait
    monkeypatch.setattr(time, "sleep", fake_sleep)

    respx.post(PRIMARY).mock(side_effect=httpx.ReadTimeout)
    respx.post(BACKUP).mock(return_value=ok("from backup"))

    complete("hello", deadline_s=10)

    # Timeout budget spent on the primary, plus any backoff, must leave
    # room for the backup inside the caller's deadline.
    assert sum(calls) <= 2.0, f"backoff of {sum(calls)}s eats the deadline"

Patching sleep rather than sleeping is what makes this test worth having in the normal suite instead of a nightly one. You are asserting on the policy — how much time the code intends to spend — not on the wall clock, so it cannot go flaky on a loaded CI runner.

Derive the budget rather than picking a round number. If your user-facing deadline is 10 s and the backup provider’s p95 is 4 s, then everything before the backup call must fit in 6 s: a 4 s primary timeout plus 1 s of backoff leaves 1 s of headroom, and a 8 s primary timeout does not fit at all. Writing that sum into the test as a comment means the next person to raise the timeout finds out immediately.

The stream that starts and then stops

There is a third failure between “error” and “hang”, and it is the one that survives most test suites: the provider returns 200, sends a few chunks, and then goes quiet. Your read timeout, if it is a per-read timeout, never fires as long as bytes keep arriving; if it is a total timeout, it fires and you are now holding half an answer.

Failover here is genuinely harder, because you may already have streamed tokens to the user. The realistic options are to buffer until you are confident, which costs you the latency benefit of streaming, or to restart the stream from the backup and accept a visible rewrite. Pick one deliberately and test it: mock a stream that yields two chunks and then raises, and assert on which of the two behaviours your code produces. The wrong answer here is the third one, where the user receives a truncated answer with a finish_reason that never arrived and no indication anything went wrong.

Two configuration details decide which of those you get, and both are worth pinning in a test rather than assuming. The first is whether your read timeout is per-read or total. httpx distinguishes connect, read, write and pool timeouts, and the OpenAI SDK documents accepting an httpx.Timeout so you can set them independently; a per-read timeout resets on every chunk, so on a stream it detects a stall rather than a long answer, which is usually what you want. A single total timeout instead aborts a legitimately long generation as though the provider had failed. The second is whether you have a stall detector at all: if no chunk has arrived for ten seconds and the connection is still open, that is a failure your timeout catches only if it is set shorter than your patience.

Assert three things on the stall test, not one: that your code detected it rather than treating the truncated text as complete, that whatever the user receives is marked incomplete, and that the event is recorded distinctly from a successful completion. The third is what keeps your error rate honest — a stalled stream counted as a success is a failure mode your dashboards will actively hide.