Chaos Testing a Provider Outage

11 min read · updated August 4, 2026

You cannot make a model provider fail on demand, so the test has to happen on your side of the connection. That is a virtue rather than a compromise: a fault injected in your own client is deterministic, reproducible in CI, and can simulate failures a provider outage page would never mention — like a stream that opens, emits four tokens and then goes silent for six minutes.

Inject at the client, not at the provider

The question a chaos test answers is not “does the provider fail?” — it will — but “what does my system do when it does?”. That makes the client the right place to inject, and it brings three practical advantages: no dependence on anyone else’s schedule, exact control over which fault occurs, and the ability to run the whole suite in CI on every change to the failure path.

Three layers are available. A wrapper around your provider client is the easiest and can simulate anything expressible in the client’s own interface. A local HTTP proxy sits between your process and the network and can also corrupt bytes, stall the socket and reset connections. Network-level tooling can add latency and packet loss to an interface, which is the only layer that reproduces the genuinely nasty transport failures. Start with the wrapper; most of the value is there.

Seven fault modes worth simulating

Fault	Description
Hard 5xx	The provider returns 500 or 503 immediately. The easy case, and the one everybody already handles. Worth testing mainly to confirm the retry budget is finite.
429 with Retry-After	Rate limited. The assertion is that you honour the header rather than applying your own backoff, and that retries do not stampede when many workers are limited simultaneously.
Connect timeout	The connection never establishes. Distinct from a read timeout because nothing was consumed and a retry is unambiguously safe. Check you have separate connect and read timeouts at all — a single total timeout conflates these.
Slow stream	The response starts normally, then each chunk takes several seconds. Total time balloons but nothing errors. This is the one that exhausts connection pools and worker threads while every health check stays green, and no ordinary fault injector produces it.
Mid-stream disconnect	The socket closes after some tokens. You have a partial answer and no finish reason. The interesting question is what your application does with half an answer — retry the whole thing and pay twice, return the fragment, or fail. All three are defensible; doing it by accident is not.
Malformed payload	A chunk that is not valid JSON, or valid JSON missing the field you index into. Surprisingly common at the edges of an outage, and the classic cause of an exception in the error path itself.
Wrong-but-valid response	A 200 with an empty content field, a refusal where you expected data, or a truncated finish reason. Not a transport failure at all, and the mode most likely to reach a user unnoticed.

The last three are what distinguishes a chaos test for a model API from one for any other HTTP service. Retries and circuit breakers handle the first three; nothing handles the last three unless somebody wrote code for them on purpose.

The fault-injecting client

# faults.py — wrap any streaming chat client to inject deterministic faults.
import asyncio, json, random

class ProviderError(Exception):
    def __init__(self, status): self.status = status; super().__init__(str(status))

class FaultConfig:
    """Probabilities in [0,1]. Set a seed for reproducibility in CI."""
    def __init__(self, seed=None, **rates):
        self.rng = random.Random(seed)
        self.rates = rates            # e.g. slow_stream=0.2, disconnect=0.1

    def fires(self, name):
        return self.rng.random() < self.rates.get(name, 0.0)

class FaultyChatClient:
    def __init__(self, inner, faults: FaultConfig):
        self.inner, self.faults = inner, faults

    async def stream(self, messages, **kw):
        f = self.faults
        if f.fires("connect_timeout"):
            await asyncio.sleep(kw.get("connect_timeout", 5) + 1)
            raise asyncio.TimeoutError("injected connect timeout")
        if f.fires("http_500"):
            raise ProviderError(500)
        if f.fires("http_429"):
            err = ProviderError(429); err.retry_after = 12; raise err

        emitted = 0
        async for chunk in self.inner.stream(messages, **kw):
            if f.fires("slow_chunk"):
                await asyncio.sleep(f.rng.uniform(2.0, 8.0))
            if f.fires("malformed") and emitted > 0:
                yield "{not json"                      # your parser must survive
                continue
            if f.fires("disconnect") and emitted >= 3:
                raise ConnectionResetError("injected mid-stream disconnect")
            emitted += 1
            yield chunk

        if f.fires("empty_response") and emitted == 0:
            return                                      # 200 with no content

Two design choices make this useful rather than annoying. The seeded RNG means a failing CI run is reproducible from its seed — record the seed in the test output. And faults are configured as rates rather than as a script, so the same suite covers both “this specific fault, always” (rate 1.0) and “a degraded provider” (a mixture of low rates), which are different tests.

Assertions worth making

A chaos test with no assertion is a demonstration. These are the properties worth asserting, roughly in order of how often they are violated.

Total attempts are bounded. Assert a maximum count of provider calls per user request, counted at the wrapper. The classic bug is a retry inside a client inside a retry inside a framework, producing eight attempts where the code appears to say three. Count them; do not read the code.
The retry budget is respected in time as well as count. Assert that the whole operation completes or fails within a stated deadline. Three retries with exponential backoff can exceed a client’s timeout, at which point every retry is pure cost. Safe retries covers the budget model.
Retries include jitter. Run 200 simulated clients through a 429 and assert that their retry times are spread rather than clustered. Deterministic backoff synchronises a fleet into waves, which is how a brief rate limit becomes a sustained one.
The circuit breaker opens and, crucially, closes. Assert that after N consecutive failures calls stop being attempted, and that after the provider recovers the breaker returns to closed within a bounded time. The half-open state is where most implementations are wrong — circuit breakers for AI calls has the state machine.
Fallback preserves the output contract. If you fall back to another model, assert the response still validates against the same schema. A fallback that returns prose where the caller expects JSON has converted a provider outage into a parsing bug two services away. Fallback chains covers ordering the chain.
Cost is bounded under failure. Assert that total tokens billed across all attempts for one user request stays under a ceiling. Retrying a 4,000-token prompt five times is five times the prefill cost for one answer, and the failure mode where retries multiply spend is the expensive one.
Nothing is charged or committed twice. If a request has a side effect — a database write, a message sent, a credit deducted — assert that a mid-stream disconnect and retry produces exactly one. Idempotency is the mechanism.
Partial responses are handled deliberately. Assert your chosen behaviour explicitly, whichever it is. The failing case is a partial answer being stored as a complete one, which is invisible until somebody reads it.
Errors reaching users are honest and distinguishable. Assert that a provider outage produces a different user-visible message from a malformed request, and that neither leaks a provider name, a key fragment or a stack trace. Error UX covers the wording.

Running it as a game day

Automated tests check the code. A game day checks the humans, the dashboards and the runbook, and it finds different problems — usually that the alert fires into a channel nobody watches, or that the runbook references a dashboard that was renamed.

Announce it. Time, scope, blast radius, and how to stop it. An unannounced game day teaches people to distrust the exercise.
Write the hypothesis first. “When the primary provider returns 100% 503s, requests fall back to the secondary within 30 seconds, p95 latency rises by less than 500 ms, and no user sees an error.” A specific prediction is what makes the result informative.
Start in staging with synthetic load. The harness from load testing an AI endpoint gives you traffic to observe the fault against.
Inject one fault, watch, and time everything. Detection time, the time to the first correct human action, recovery time. Those three numbers are the output of the exercise.
Do not fix things during the run. Note them. Fixing mid-exercise stops you learning what the unfixed system does, which is the thing you will actually have at 3am.
Write down what surprised you. The surprises are the findings. A game day where nothing surprised anyone tested a fault you had already handled.

Taking it to production, carefully

Staging does not have production’s traffic, its concurrency or its cache state, so some failure modes only exist in production. It is reasonable to inject faults there, under four conditions, and unreasonable without them: a small percentage of traffic selected by a flag you can flip off in one action; internal or opted-in users only, for the first several runs; a hard stop condition agreed in advance — error rate, latency, or anyone saying stop; and every injected fault tagged in the logs so that an unrelated real incident during the window is not misattributed to the exercise.