Retrying Model Calls With tenacity

The decorator is four lines. The decision it encodes is which of the nine ways a model call fails deserve a second attempt — because retrying a 400 wastes time, and retrying a timed-out POST can bill you twice for the same answer.

Which failures are worth retrying

Sort every failure into one of three buckets before writing a line of retry code. Nearly every bad retry policy is a policy that never made this distinction.

The policy, in tenacity

pip install tenacity. The names below are tenacity 8.x and 9.x; the library has been stable across those, but wait_exponential_jitter arrived later than the rest, so the version-proof choice is wait_random_exponential, which has been present for years and already includes randomisation.

# retrying.py
import logging

import httpx
from tenacity import (
    retry,
    retry_if_exception,
    stop_after_attempt,
    wait_random_exponential,
    before_sleep_log,
)

log = logging.getLogger(__name__)

RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504}


def is_retryable(exc: BaseException) -> bool:
    if isinstance(exc, httpx.HTTPStatusError):
        return exc.response.status_code in RETRYABLE_STATUS
    # connect errors, read timeouts, protocol errors, pool timeouts
    return isinstance(exc, (httpx.TimeoutException, httpx.NetworkError))


@retry(
    retry=retry_if_exception(is_retryable),
    wait=wait_random_exponential(multiplier=1, max=30),
    stop=stop_after_attempt(5),
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,
)
def call_model(client: httpx.Client, payload: dict) -> dict:
    response = client.post("/chat/completions", json=payload)
    response.raise_for_status()
    return response.json()

Four of those arguments earn their place:

• retry_if_exception(is_retryable) takes a predicate over the exception, which is what lets one function express “this status code yes, that one no”. retry_if_exception_type(httpx.HTTPStatusError) would retry the 401 as well.
• wait_random_exponential(multiplier=1, max=30) waits a random time up to min(multiplier × 2^attempt, max) seconds. The cap matters: uncapped exponential backoff reaches sleeps of several minutes by attempt eight, long after any caller has given up.
• stop_after_attempt(5) is the total number of attempts, not the number of retries. Five attempts with this wait is at most about a minute of sleeping.
• reraise=True makes the final failure raise the original httpx exception. Without it tenacity raises RetryError, and every caller upstream has to unwrap exc.last_attempt.exception() to find out what actually happened.

Honouring Retry-After

When a provider sends Retry-After it is telling you exactly when the limit resets. Backing off exponentially instead is either too early — which earns another 429 — or too late. tenacity’s wait accepts a callable over the retry state, so the header can override the default:

import random

from tenacity import RetryCallState

MAX_SLEEP = 60.0


def wait_from_header(state: RetryCallState) -> float:
    """Use Retry-After when present, otherwise capped exponential + jitter."""
    outcome = state.outcome
    if outcome is not None and outcome.failed:
        exc = outcome.exception()
        if isinstance(exc, httpx.HTTPStatusError):
            header = exc.response.headers.get("retry-after")
            if header:
                try:
                    return min(float(header), MAX_SLEEP)
                except ValueError:
                    pass          # HTTP-date form; fall through to backoff
    backoff = min(2 ** state.attempt_number, MAX_SLEEP)
    return backoff * random.uniform(0.5, 1.0)


@retry(
    retry=retry_if_exception(is_retryable),
    wait=wait_from_header,
    stop=stop_after_attempt(5),
    reraise=True,
)
def call_model(client: httpx.Client, payload: dict) -> dict:
    response = client.post("/chat/completions", json=payload)
    response.raise_for_status()
    return response.json()

Retry-After is permitted to be either a number of seconds or an HTTP date, which is why the float() is inside a try. The cap protects you from a provider that says 3600 on a request a user is waiting for.

Why jitter is not optional

Jitter is the difference between a retry policy that recovers and one that keeps an outage going. The reasoning is arithmetic, not folklore.

Suppose 200 concurrent workers all receive a 503 at the same moment, because the failure was upstream and hit all of them. With fixed exponential backoff, all 200 sleep exactly two seconds and all 200 fire again in the same millisecond. The provider sees a 200-request spike, is knocked over again, and the workers sleep four seconds and repeat the spike at four. The retries have reproduced the original burst indefinitely — this is the thundering herd, and it is self-sustaining.

With a random multiplier over [0.5, 1.0] of a two-second backoff, those 200 retries spread over a one-second window: roughly 200 requests per second at the peak instead of 200 in a millisecond, a reduction of about three orders of magnitude in instantaneous load, for one call to random.uniform. Full jitter over [0, backoff] spreads them wider still, at the cost of some retries firing sooner than the backoff intended.

Retrying something that already happened

A read timeout does not mean the request failed. It means you stopped listening. The provider may have generated 800 tokens, billed them, and been unable to deliver them — and your retry generates and bills 800 more.

• Bound the damage. Retries on a timeout should be fewer than retries on a 503 — one or two, not five. The cost of a duplicate is real and it scales with max_tokens.
• Use an idempotency key if the endpoint supports one. Where a provider accepts an Idempotency-Key header, sending the same key on the retry lets the server return the original result instead of recomputing it. Support varies; check your provider’s documentation rather than assuming. Idempotency covers the general pattern.
• Never retry a call with a side effect outside the model. If the same function also writes a row, sends an email or charges a card, the retry has to be around the model call only — not around the transaction. Safe retries is about exactly this boundary.

The async version, and the budget

The @retry decorator works on coroutine functions unchanged; tenacity detects them and awaits asyncio.sleep rather than blocking. Nothing else changes.

@retry(
    retry=retry_if_exception(is_retryable),
    wait=wait_from_header,
    stop=stop_after_attempt(4),
    reraise=True,
)
async def acall_model(client: httpx.AsyncClient, payload: dict) -> dict:
    response = await client.post("/chat/completions", json=payload)
    response.raise_for_status()
    return response.json()

One addition is worth making for anything a user is waiting on. Attempt counts are the wrong budget for a request with a deadline: five attempts plus backoff can exceed a minute, by which time the browser has given up. stop_after_delay(20) caps the total elapsed time instead, and the two compose:

from tenacity import stop_after_delay

stop = stop_after_attempt(5) | stop_after_delay(20)   # whichever comes first

Whatever the policy, log every retry with the attempt number, the status code and the sleep. A retry rate that has quietly climbed from 0.4 per cent to 6 per cent is a provider degrading, and it is invisible in a success-rate dashboard because the retries are working. Logging every model call has the decorator that records it.