Rate Limiter Config Generator

Your throughput and your provider's two limits in; token-bucket parameters, the limit that actually binds, and the queue wait at your utilisation out.

Throughput you needSustained, not peak.

req / min

Burst you need to absorbRequests that can arrive at once without waiting.

requests

Input tokens per requestYour own measured mean, prompt included.

tokens

Output tokens per request

tokens

Provider request limitFrom your account. 0 if you have none.

req / min

Provider token limitUsually the one that actually binds.

tokens / min

Client processes sharing the quotaPods, workers, lambdas — each with its own bucket.

instances

HeadroomHow far below the provider's limit you aim.

Generated code

Throughput this configuration sustains

533 req / min

Bound by the token-per-minute limit, minus 20.0% headroom. You asked for 450, which is 84.4% of it.

Tokens per request: 1,500
Requests per minute the token limit allows: 667
Requests per minute the request limit allows: 1,000
Which one binds: the token-per-minute limit
Usable after headroom: 533 req / min
Per instance: 133 req / min
Bucket refill rate: 2.222 permits / s
Bucket capacity: 15 permits
Time to refill a drained bucket: 6.75 s
Utilisation at your target: 84.4%
Mean queue wait at that utilisation: ≈ 304 ms
Tokens per minute you would consume: 675,000
Token limit used: 67.5%

ConfigRead-only. Select all and copy.

TypeScriptRead-only. Select all and copy.

/** A token bucket sized for 533 req/min across 4 process(es). */
export class TokenBucket {
  private tokens: number;
  private last = Date.now();

constructor(
    private readonly capacity = 15,
    private readonly refillPerSecond = 2.2222,
  ) {
    this.tokens = capacity;
  }

private refill() {
    const now = Date.now();
    this.tokens = Math.min(
      this.capacity,
      this.tokens + ((now - this.last) / 1000) * this.refillPerSecond,
    );
    this.last = now;
  }

/** Resolves when a permit is available. Rejects if that would take too long. */
  async take(maxWaitMs = 30_000): Promise<void> {
    const deadline = Date.now() + maxWaitMs;
    for (;;) {
      this.refill();
      if (this.tokens >= 1) {
        this.tokens -= 1;
        return;
      }
      const needed = ((1 - this.tokens) / this.refillPerSecond) * 1000;
      if (Date.now() + needed > deadline) {
        throw new Error("rate limiter: would wait past the deadline — shed this request");
      }
      await new Promise((r) => setTimeout(r, Math.min(needed, 250)));
    }
  }
}

// One bucket per process. With 4 process(es) the effective
// ceiling is 533 req/min only while load is spread evenly;
// if it is not, move the bucket into Redis and share it.
export const limiter = new TokenBucket();

What this configuration will do under load:

You are running at 84.4% of the usable rate. Queue wait rises as 1/(1-ρ), so the last 10% of capacity costs more waiting than the first 80% put together — the mean wait above is roughly 304 ms, and it doubles again between 90% and 95%.

What this assumes: the queue wait is an M/D/1 approximation — Poisson arrivals, constant service time — which is a reasonable shape for independent traffic and wrong for traffic that arrives in bursts on a schedule. Read it as “this is roughly where waiting becomes the dominant cost” rather than as a latency budget. Every provider limit is a field you type: nothing here ships anybody's limits, because they differ per account and change without notice. The per-instance split assumes load is spread evenly across your processes, which is the assumption most often wrong in practice and the reason the shared-bucket note appears above. Everything on this page runs in your browser. Nothing you paste or open is uploaded, logged or sent anywhere.

Two limits apply to almost every model API at once, and people configure for the wrong one. A request-per-minute limit is easy to think about and is usually not the one that stops you: at a couple of thousand tokens a request, a token-per-minute limit runs out first, and it runs out sooner every time someone lengthens the system prompt. The number that matters is the token limit divided by your tokens per request, and it is the first row of the table above.

Why a bucket rather than a sleep

A token bucket gives you two knobs that map onto what you actually want: a sustained rate, and a burst you can absorb without waiting. Sleeping a fixed interval between calls gives you the rate and throws away the burst — which is exactly the behaviour that makes a batch job take four times longer than it needs to when the quota was sitting unused for the previous minute.

Utilisation is not linear, and neither are your instances

Running at 95% of a limit does not cost 19% more waiting than running at 80%; queueing goes as 1/(1−ρ), so it costs several times more. And a per-process bucket cannot lend a permit to a busier process: split a quota four ways and a spike that lands on one pod is throttled while three buckets sit full. Both effects are in the numbers above so the choice between a local bucket and a shared one is made on arithmetic rather than on taste.

Rate Limiter Config Generator

Why a bucket rather than a sleep

Utilisation is not linear, and neither are your instances

The reasoning behind this

Related tools