Rate Limiting Yourself Before They Do

Rate limits are usually two numbers — requests per minute and tokens per minute — and you can breach either. A single token bucket handles both, in about sixty lines, and turns a 429 storm into a queue that is a few milliseconds slower than the limit allows.

Why limit yourself

Retrying a 429 works. Not causing it works better, for four concrete reasons.

• A rejected request still costs time. Round trip, backoff, round trip again. A client-side wait of 200 ms replaces a failure plus a two-second backoff.
• Retries convert one breach into several. Every retried request is added load at exactly the moment you are over the limit, which is why a rate-limit incident escalates rather than settles.
• Some limits are punitive. A provider that responds to sustained breaches by lengthening the cooldown makes a self-inflicted problem last far beyond the burst.
• A limiter is a place to put fairness. One bucket per tenant stops a single customer’s batch job consuming the whole key, which no retry policy can do.

The token bucket, derived

The whole algorithm is one line of arithmetic. A bucket holds up to capacity units and refills at rate units per second. To spend n units you wait until at least n are present, then subtract them.

available(now) = min(capacity, available(last) + rate * (now - last))

To spend n:
  if available >= n:   spend immediately
  else:                wait (n - available) / rate seconds, then spend

For a limit of 3,000 requests per minute:
  rate     = 3000 / 60 = 50 requests per second
  capacity = 50        (one second of burst)  or 3000 (a full minute of burst)

The capacity choice is the only judgement in the design, and it is a real trade. Capacity equal to one second of rate means an idle client cannot burst at all — smooth, safe, and slower to drain a queue. Capacity equal to the full window permits a client that has been idle for a minute to fire the entire minute’s allowance at once, which is exactly the burst that trips a provider’s own limiter if theirs is measured over a shorter interval. Somewhere between one and ten seconds of rate is a defensible default; start at one and raise it if throughput suffers.

Note there is no timer, no background thread and no queue in the algorithm. The bucket level is computed from the elapsed time whenever somebody asks, which is why the implementation is short and why it cannot drift.

The implementation

# limiter.py
import asyncio
import threading
import time


class TokenBucket:
    """A refilling bucket. Thread-safe; see AsyncTokenBucket for asyncio."""

    def __init__(self, rate_per_second: float, capacity: float | None = None):
        if rate_per_second <= 0:
            raise ValueError("rate must be positive")
        self.rate = float(rate_per_second)
        self.capacity = float(capacity if capacity is not None else rate_per_second)
        self._available = self.capacity
        self._updated = time.monotonic()
        self._lock = threading.Lock()

    def _refill(self) -> None:
        now = time.monotonic()
        elapsed = now - self._updated
        if elapsed > 0:
            self._available = min(self.capacity, self._available + elapsed * self.rate)
            self._updated = now

    def _wait_time(self, amount: float) -> float:
        """Reserve the amount, returning how long the caller must sleep first."""
        if amount > self.capacity:
            raise ValueError(
                f"cannot spend {amount}: bucket capacity is {self.capacity}"
            )
        with self._lock:
            self._refill()
            self._available -= amount          # may go negative: that IS the queue
            deficit = -self._available
            return max(0.0, deficit / self.rate)

    def acquire(self, amount: float = 1.0) -> None:
        delay = self._wait_time(amount)
        if delay > 0:
            time.sleep(delay)

    def refund(self, amount: float) -> None:
        """Give back units reserved but not used."""
        with self._lock:
            self._refill()
            self._available = min(self.capacity, self._available + amount)

The critical design choice is that _available is allowed to go negative. Subtracting inside the lock and sleeping outside it means each caller computes its own wait from a position in an implicit queue, so twenty threads asking at once get twenty staggered wake-ups rather than twenty simultaneous ones after a shared sleep. That thundering-herd bug is present in most short rate limiters on the internet, and it produces exactly the burst the limiter existed to prevent.

time.monotonic() rather than time.time(), for the same reason as in the logging recipe: a clock adjustment must not be able to hand you a minute of free capacity or stall the bucket for an hour.

The asyncio version is the same arithmetic with a different sleep:

class AsyncTokenBucket(TokenBucket):
    def __init__(self, rate_per_second: float, capacity: float | None = None):
        super().__init__(rate_per_second, capacity)
        self._alock = asyncio.Lock()

    async def acquire(self, amount: float = 1.0) -> None:   # type: ignore[override]
        async with self._alock:
            self._refill()
            self._available -= amount
            delay = max(0.0, -self._available / self.rate)
        if delay > 0:
            await asyncio.sleep(delay)

Two dimensions, one wait

Requests per minute and tokens per minute are two buckets. A call must satisfy both, and the wait is the larger of the two — which falls out naturally if you simply acquire from each in turn.

# governed.py
class ProviderLimits:
    def __init__(self, rpm: int, tpm: int, burst_seconds: float = 1.0):
        self.requests = TokenBucket(rpm / 60.0, capacity=rpm / 60.0 * burst_seconds)
        self.tokens = TokenBucket(tpm / 60.0, capacity=tpm / 60.0 * burst_seconds)

    def acquire(self, estimated_tokens: int) -> None:
        self.requests.acquire(1)
        self.tokens.acquire(estimated_tokens)

    def reconcile(self, estimated_tokens: int, actual_tokens: int) -> None:
        difference = estimated_tokens - actual_tokens
        if difference > 0:
            self.tokens.refund(difference)          # we over-reserved
        elif difference < 0:
            self.tokens.acquire(-difference)        # we under-reserved: pay it back


limits = ProviderLimits(rpm=3_000, tpm=1_000_000)


def governed_call(client, payload: dict, estimated_tokens: int) -> dict:
    limits.acquire(estimated_tokens)
    response = client.post("/chat/completions", json=payload)
    response.raise_for_status()
    body = response.json()
    usage = body.get("usage", {})
    actual = usage.get("total_tokens")
    if actual is not None:
        limits.reconcile(estimated_tokens, actual)
    return body

Acquiring the request slot before the token slot is deliberate: it is the cheaper of the two and it keeps the request count exact even when the token estimate is poor.

The problem with limiting on tokens

You must reserve capacity before the call, but you only learn the true token count after it. Input tokens are knowable in advance; output tokens are not, and the gap between max_tokens and what the model actually produced can be a factor of twenty.

Reserving max_tokens is safe and wasteful — a limiter that reserves 4,000 output tokens for answers that average 200 throttles you to a twentieth of your real allowance. Reserving an estimate and reconciling afterwards, as above, keeps the long-run average correct while permitting short-term overshoot. That is the right trade in almost every case, because provider limits are enforced over a window rather than instantaneously.

# a serviceable estimate before the call
def estimate_tokens(payload: dict) -> int:
    chars = sum(len(m.get("content") or "") for m in payload["messages"])
    prompt = chars // 4                       # ~4 chars/token for English prose
    completion = min(payload.get("max_tokens", 512), 512)
    return prompt + completion

The four-characters-per-token rule is a rough approximation for English and is markedly wrong for code, for other languages and for anything with heavy punctuation — tokens per word and the tokeniser language tax quantify the error. For accuracy, run the actual tokeniser; for a limiter that reconciles afterwards, the approximation is fine, and after a day of real traffic your own logs give you a better multiplier than any rule of thumb.

What the provider’s headers tell you

Configuring your limiter from a number in a dashboard means it is wrong the day the account is upgraded. Most providers report the live limit on every response, so the limiter can learn it instead.

The header names are not standardised. The widely used convention is a family beginning x-ratelimit- with -limit, -remaining and -reset suffixes, sometimes split into request and token variants; some gateways use the RateLimit family from the IETF draft instead, and some send nothing at all. Read them defensively, and print what your endpoint actually returns before writing code against a name:

# once, from a REPL, to see what you are actually given
response = client.post("/chat/completions", json=payload)
for name, value in response.headers.items():
    if "ratelimit" in name.lower() or name.lower() == "retry-after":
        print(f"{name}: {value}")

def observe_headers(headers, limits: ProviderLimits) -> None:
    """Adjust the local buckets from whatever the provider reported."""
    remaining = headers.get("x-ratelimit-remaining-requests")
    if remaining is not None:
        try:
            free = float(remaining)
        except ValueError:
            return
        # never grant more than the provider says is left
        with limits.requests._lock:
            limits.requests._refill()
            limits.requests._available = min(limits.requests._available, free)

Clamping downwards only is the safe direction, and it is the whole design rule here. A header that reports more headroom than your bucket thinks it has may be stale, may be counting a different window, or may belong to a different key on a shared gateway — raising your local allowance on the strength of it converts a working limiter into an intermittent one. Lowering is always safe.

The one header worth acting on unconditionally is Retry-After on a 429, which is a direct instruction rather than an estimate; the wait function in retrying model calls with tenacity already honours it. Log the remaining-quota values alongside your calls even if you do nothing with them — a headroom figure that is trending towards zero over a week is the earliest warning you will get before the 429s start.

More than one process

The bucket above is per process. Four uvicorn workers with rpm=3000 each will send 12,000 requests per minute, and the limiter will report that everything is fine.

• Divide the budget. The zero-infrastructure answer: give each of N processes rpm/N. Correct, and wasteful when the load is uneven — an idle worker’s share is unusable.
• One shared bucket in Redis. Implement the same arithmetic in a Lua script so the refill-and-subtract is atomic, and store available and updated as fields on one hash. The round trip costs about a millisecond, which is nothing against a model call.
• Put it in front of everything. A gateway or proxy that all processes call is the only design where adding a fifth service does not require re-dividing anybody’s budget, and it is also where a per-tenant limit naturally lives.

Whichever you choose, keep the retry policy from retrying model calls with tenacity underneath it. A limiter reduces 429s; it cannot eliminate them, because your view of the limit is always slightly out of date and the provider may be counting a window you cannot see. Rate limits explained covers what the headers tell you about that window.