Rate Limits: TPM, RPM and How to Live Inside Them

Rate limits are usually documented as a pair of numbers and understood as a speed limit. They are not a speed limit — they are a token bucket, which has a burst capacity as well as a rate, and the difference decides whether your traffic pattern fits or gets rejected at a fraction of your nominal quota.

The dimensions you are limited on

Providers differ, but the dimensions recur. Check your own provider’s documentation for which of these apply and at what values, because the numbers move and are frequently account-specific:

• RPM / RPS — requests per minute or second. Binds on workloads with many small calls: classification, embedding one item at a time, an agent making rapid tool calls.
• TPM — tokens per minute, usually input plus requested output. Binds on workloads with long prompts. This is the one that usually bites first for RAG.
• Concurrent requests — how many may be in flight at once, regardless of rate. Binds on long generations, because each one occupies its slot for the whole generation.
• Daily or monthly caps — a separate, slower bucket, often tied to spend rather than volume.

You are limited by whichever binds first, and it is common to be at 15% of your TPM while pinned against RPM. A limiter that models only one dimension will confidently let through requests that the server rejects.

Token bucket, and what its two numbers mean

A token bucket has a capacity C and a refill rate r. Tokens accrue continuously up to C; a request consumes some; if there are not enough, it waits or is rejected. Two behaviours follow directly, and they are the whole reason to model it properly:

BURST. From full, you may spend C instantly, then you are rate-limited to r.

  C = 100 requests, r = 100/min:
    idle for a minute, then 100 requests in one second  -> all admitted
    immediately after, the 101st waits 0.6 s

DRAIN. A burst of B from full takes this long to be fully admitted:

  t = max(0, (B - C) / r)

  C = 100, r = 100/min, B = 500  ->  (500-100)/100 = 4 minutes

STEADY STATE. Sustained arrival rate above r is rejected no matter how
large C is. Capacity absorbs bursts; it does not raise throughput.

The practical reading: a bucket forgives shape and does not forgive volume. Batch jobs that fire everything at once are fine if the total fits in C, and are rejected en masse if it does not. Spreading the same work over the window changes nothing about the total and everything about whether it is accepted.

Note also that a limiter enforced over a fixed window rather than as a rolling bucket has a boundary artefact: a burst at the end of one window plus a burst at the start of the next admits up to twice the nominal rate across the boundary. Do not build your own that way, and do not assume your provider has not.

The awkward part: TPM is charged on an estimate

The server has to decide whether to admit your request before it knows how many output tokens you will use, so it cannot charge the real number. The usual approach, and the one OpenAI documents in its rate-limit guidance, is to charge input tokens plus max_tokens up front, then reconcile against actual usage afterwards.

That produces the most surprising behaviour in this whole area: lowering max_tokens increases your effective throughput, without changing a single answer. If you set max_tokens: 4096 out of caution and your answers average 300 tokens, you are reserving thirteen times the quota you consume, and you will be rate-limited at roughly a thirteenth of the traffic you could have run.

Two corollaries. Set max_tokens to a real bound for the task rather than to the model maximum. And when you estimate consumption client-side, estimate the same way the server does — reservation, then reconciliation — or your limiter will drift out of agreement with the thing it is trying to stay inside of.

A client-side limiter

Two buckets, a reservation on the way in and a correction on the way out. This is the piece most implementations skip, and it is why they still get 429s:

class Bucket {
  private tokens: number;
  private last = Date.now();

  /** @param capacity burst size @param perSec sustained refill rate */
  constructor(private capacity: number, private perSec: number) {
    this.tokens = capacity;
  }

  private refill() {
    const now = Date.now();
    this.tokens = Math.min(
      this.capacity,
      this.tokens + ((now - this.last) / 1000) * this.perSec,
    );
    this.last = now;
  }

  /** Milliseconds until n tokens are available. 0 if they are available now. */
  waitFor(n: number): number {
    this.refill();
    if (n > this.capacity) throw new Error("request larger than bucket capacity");
    if (this.tokens >= n) return 0;
    return ((n - this.tokens) / this.perSec) * 1000;
  }

  take(n: number) { this.refill(); this.tokens -= n; }
  give(n: number) { this.refill(); this.tokens = Math.min(this.capacity, this.tokens + n); }
}

export class Limiter {
  private readonly rpm: Bucket;
  private readonly tpm: Bucket;
  private chain: Promise<unknown> = Promise.resolve();

  constructor(rpm: number, tpm: number) {
    // Capacity = one minute of rate, which is how most published limits behave.
    this.rpm = new Bucket(rpm, rpm / 60);
    this.tpm = new Bucket(tpm, tpm / 60);
  }

  /**
   * Serialised admission: the queue is FIFO, so a large request cannot be
   * starved forever by a stream of small ones. estimate() must match the
   * server's accounting -- prompt tokens + max_tokens.
   */
  async run<T>(estimate: number, fn: () => Promise<{ value: T; used: number }>): Promise<T> {
    const admitted = this.chain.then(async () => {
      for (;;) {
        const wait = Math.max(this.rpm.waitFor(1), this.tpm.waitFor(estimate));
        if (wait === 0) break;
        await new Promise((r) => setTimeout(r, wait + 5));   // +5ms: clock slop
      }
      this.rpm.take(1);
      this.tpm.take(estimate);
    });

    this.chain = admitted.catch(() => {});
    await admitted;

    const { value, used } = await fn();
    // Reconcile: hand back the difference between the reservation and reality.
    if (used < estimate) this.tpm.give(estimate - used);
    return value;
  }

  /** Call this when a 429 arrives anyway: drain the bucket for the stated wait. */
  penalise(retryAfterMs: number) {
    this.rpm.take(this.rpm.waitFor(0) + (retryAfterMs / 1000) * (60 / 60));
  }
}

The FIFO chain matters more than it looks. Without it, concurrent callers each check the bucket, all see room, and all take it — a classic check-then-act race that produces exactly the burst you built the limiter to prevent. Serialising admission (not execution) fixes it and costs nothing, since admission is microseconds.

One caveat that invalidates the whole thing if ignored: this limiter is per process. Run eight replicas and you have eight limiters, each confidently admitting the full quota, and you will offer the provider eight times your limit while every instance believes it is compliant. There are three honest fixes. Divide the configured limit by the replica count — simple, and wrong the moment an autoscaler changes the count. Move the buckets into Redis and do the refill arithmetic in a Lua script so check-and-take stays atomic — correct, at the cost of a round trip on the hot path. Or route all model traffic through a single service that owns the buckets, which is the same answer as the second but with the coordination expressed as a hop rather than as a lock.

Whichever you choose, keep the local limiter as well. A shared limiter protects the provider’s quota; a local one protects your own process from opening a thousand sockets while it waits for permission. They solve different problems and the cheap one should run first.

Living inside a limit you cannot raise

• Trim max_tokens first. Free throughput, no quality cost, and usually the largest single win available.
• Read the headers. Where a provider returns remaining quota and reset times, treat those as authoritative and reconcile your local buckets against them rather than trusting your own count.
• Prioritise. When the bucket is empty, something must wait. Better that it is the background enrichment job than the user who is watching a cursor blink — which needs a queue with classes, not a single limiter.
• Move deadline-free work to a batch endpoint. Batch APIs are typically metered separately from the synchronous limits, which means the nightly job stops competing with live traffic; see batch APIs.
• Spread across keys or providers only if permitted. Splitting a workload over several accounts to evade a limit is against most providers’ terms. Splitting across genuinely different providers is not, and gives you independent buckets.