Chaos Testing a Rate-Limit Response From a Provider

A 429 is the only provider error that is telling you exactly what to do next, and it is the one most often handled as if it were a 500. The distinction matters: a 500 means try somewhere else, a 429 means try here again shortly. Code that conflates them either burns a fallback it did not need or retries so fast that it stays rate-limited indefinitely.

What a 429 actually carries

The status code is the least informative part. What you should be reading is the headers, and the specifics vary by provider — which is itself the reason to test rather than assume:

• Retry-After — defined by the HTTP specification as either a delay in seconds or an HTTP date. Both forms are legal and a parser that only handles integers will crash on the other.
• Vendor-specific remaining-quota and reset headers, commonly on the pattern x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens and their -reset- counterparts. Note the two dimensions: you can be limited on request count while having plenty of token budget, or the reverse, and the remedy differs.
• An error body whose type or code distinguishes a transient throughput limit from a quota or billing condition that no amount of waiting will clear.

A chaos test is where you decide, in writing, what your code does with each of these — before a real limit event decides for you at a worse time.

Injecting it

respx’s documented list-valued side_effect returns successive responses on successive calls, which is exactly the shape of “rate-limited twice, then fine”.

# test_rate_limit_backoff.py
import time, httpx, respx, pytest
from myapp.llm import complete

URL = "https://api.primary.example/v1/chat/completions"

def limited(retry_after="2"):
    return httpx.Response(
        429,
        headers={
            "retry-after": retry_after,
            "x-ratelimit-remaining-requests": "0",
            "x-ratelimit-reset-requests": "2s",
        },
        json={"error": {"type": "rate_limit_exceeded",
                        "message": "Rate limit reached for requests"}},
    )

def ok():
    return httpx.Response(200, json={
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": "hi"}}],
        "usage": {"prompt_tokens": 8, "completion_tokens": 1, "total_tokens": 9},
    })

@pytest.fixture
def slept(monkeypatch):
    recorded = []
    monkeypatch.setattr(time, "sleep", lambda s: recorded.append(s))
    return recorded

@respx.mock
def test_backs_off_then_succeeds(slept):
    route = respx.post(URL).mock(side_effect=[limited(), limited(), ok()])

    result = complete("hello")

    assert result.text == "hi"
    assert route.call_count == 3
    assert len(slept) == 2, "did not sleep between attempts"
    assert all(s >= 2 for s in slept), f"ignored Retry-After: {slept}"
    assert slept == sorted(slept), "delays did not increase"

Asserting on the recorded sleeps rather than on elapsed time is what keeps this test in the fast suite. You are testing the retry policy — the delays the code intended — and a policy is a pure function of the responses it saw. Sleeping for real would make the same test slow, flaky under CI load, and no more informative.

Honouring Retry-After, and when not to

The rule is: wait at least as long as Retry-After says. Not less — retrying early is how a client stays limited, since each early attempt is another rejected request against the same window. Longer is fine and often wise once jitter is added.

Two cases need explicit handling and each deserves a test. The date form: pass retry_after="Wed, 12 Aug 2026 09:31:00 GMT" and assert your parser computes a non-negative delay rather than raising. A useful detail is that this form depends on your clock agreeing with the server’s; if the computed delay is negative or absurdly large, fall back to your own backoff rather than trusting it. The absent header: some 429s arrive without one at all. Assert that your code then uses exponential backoff from a sensible base rather than retrying immediately, which is what a naive int(headers.get("retry-after", 0)) produces.

There is also a ceiling question. If Retry-After says 60 seconds and your user-facing deadline is 10, waiting is not an option and the correct move is to fail over or fail fast. Assert that: mock a long Retry-After, assert no sleep longer than the remaining budget was attempted, and assert the fallback route was called instead.

Jitter, caps and the thundering herd

If every one of your workers receives a 429 in the same second and every one waits exactly the interval the header specified, they all retry in the same later second. The limit is hit again by the same synchronised burst, and the pattern can persist for many rounds. This is why jitter is not a refinement.

Full jitter — sleeping a uniform random amount between zero and the computed backoff — is the usual construction, adjusted here so the wait is never shorter than Retry-After:

import random

def next_delay(attempt, retry_after=None, base=0.5, cap=20.0):
    backoff = min(cap, base * (2 ** attempt))
    delay = random.uniform(0, backoff)          # full jitter
    if retry_after is not None:
        delay = max(delay, retry_after)         # never earlier than told
    return min(delay, cap)

Test it deterministically by seeding: random.seed(0), then assert the delays are distinct across simulated workers and that every one is at least Retry-After and at most cap. Asserting on distinctness is the part that catches a jitter implementation that was accidentally removed in a refactor, which otherwise looks identical in every single-worker test.

The cap matters as much as the jitter. Uncapped exponential backoff reaches minutes by the sixth attempt, and a request holding a connection and a request slot for four minutes is worse for your service than a fast failure. Cap the delay, cap the attempt count, and assert both.

The 429 you must not retry

Not every 429 is transient. A hard quota — a monthly spend cap, a suspended key, an organisation-level limit — can present with the same status code and will still be there in twenty minutes. Retrying it consumes your deadline for nothing and, if your fallback is a different key on the same account, fails identically.

Distinguish on the error body’s type or code field, not on the status. Write the test as a table: for each error type your provider documents, assert whether the code retried or gave up immediately. If the provider does not document a stable discriminator, the safe policy is to retry a small fixed number of times and then stop — and to say so in a comment, because the next person will otherwise assume the absence of a special case was an oversight.

One more asymmetry worth encoding: a 429 received before the request was processed is safe to retry, and one received after partial work is not necessarily. For a plain completion this is academic, but for anything that has already executed a tool with a side effect, the retry needs an idempotency key or it will run that side effect twice. Rate-limit handling and idempotency are the same problem viewed from two ends.

There is one more distinction worth a test, because getting it wrong wastes your entire retry budget. Vendors commonly limit on two dimensions at once: requests per minute and tokens per minute. A request-limit rejection clears on the request window and a short wait fixes it. A token-limit rejection triggered by a very large prompt will keep rejecting the same prompt no matter how long you wait, because the request will always exceed the per-window token budget on its own. The remedy there is to shrink the request, not to sleep. Mock both cases with whichever remaining-quota header your provider documents set to zero, and assert that the token case takes a different branch — splitting the work, trimming context, or failing with a message that says so — rather than the same backoff loop.