Concurrency Control: Not Melting Your Own Rate Limit

The first version fires every request at once and collects 429s. The second version adds a fixed concurrency limit chosen by feel. The third version discovers that the limit which was perfect at midnight is far too high at noon, because the service time moved and the limit did not.

You are being limited on two axes

Inference APIs typically enforce at least two limits at once: requests per minute and tokens per minute, sometimes split into input and output. They are not interchangeable, and the same client can be comfortably inside one while repeatedly breaching the other.

Ten requests a minute, each with a 50,000-token document, is trivial for a request limit and enormous for a token limit. A thousand one-line classifications a minute is the reverse. Any client that controls only its request rate will eventually be throttled on tokens and have no idea why, because the error looks identical.

There is a third constraint that is yours rather than theirs: concurrency. How many calls you have in flight determines how many of your workers, connections and file descriptors are parked, and unbounded concurrency exhausts your own process long before it bothers the provider. Three constraints, three mechanisms:

How many in flight? Little’s Law

The concurrency number should not be a guess. Little’s Law — for a stable system, the average number of items in the system equals the arrival rate multiplied by the average time in the system — gives it to you directly. Rearranged for this purpose: the concurrency you need is your target throughput multiplied by the average call duration.

Thirty requests per second at an average of four seconds each needs about 120 in flight. Two per second at twenty seconds each needs forty. The formula is not the interesting part; the interesting part is that duration is in it, and inference duration varies with prompt length, output length, model and provider load. A fixed concurrency limit is implicitly a bet on a constant that is not constant.

Use the law twice. Once to set a sane starting value from your target rate and your observed p50 duration. Then again in the other direction: given the concurrency the provider will tolerate, your maximum throughput is that concurrency divided by the duration — which tells you immediately whether a latency regression on the provider’s side has just cut your capacity, and by how much.

A semaphore with a queue

export class Semaphore {
  private inFlight = 0;
  private waiting: { resolve: () => void; reject: (e: Error) => void; queuedAt: number }[] = [];

  constructor(
    private limit: number,
    private maxQueue = 1000,
    private readonly maxWaitMs = 30_000,
  ) {}

  async acquire(deadline: Deadline): Promise<() => void> {
    if (this.inFlight < this.limit) { this.inFlight++; return () => this.release(); }
    // A bounded queue is not optional. An unbounded one converts a slow
    // dependency into an out-of-memory crash, and every request in it is
    // already past its deadline by the time it runs.
    if (this.waiting.length >= this.maxQueue) throw new Overloaded();

    await new Promise<void>((resolve, reject) => {
      const entry = { resolve, reject, queuedAt: Date.now() };
      this.waiting.push(entry);
      const ms = deadline.remaining();
      setTimeout(() => {
        const i = this.waiting.indexOf(entry);
        if (i >= 0) { this.waiting.splice(i, 1); reject(new DeadlineExceeded()); }
      }, ms).unref?.();
    });
    return () => this.release();
  }

  private release() {
    this.inFlight--;
    // Drop anything that expired while queued rather than running dead work.
    let next = this.waiting.shift();
    while (next && Date.now() - next.queuedAt > this.maxWaitMs) {
      next.reject(new DeadlineExceeded());
      next = this.waiting.shift();
    }
    if (next) { this.inFlight++; next.resolve(); }
  }

  resize(limit: number) { this.limit = limit; }        // for the adaptive controller
}

The two lines that make this production-grade rather than illustrative are the bounded queue and the expiry check on release. Without the bound, a provider slowdown turns into unbounded memory growth. Without the expiry check, a recovering system spends its first minutes executing requests whose callers left long ago — paying for answers nobody will receive, which is the specific way this dependency makes a classical mistake worse.

A token bucket for the token limit

Concurrency alone will not keep you under a tokens-per-minute cap, because two calls can differ in token cost by a factor of a thousand. A token bucket refilling at your allowance, debited by an estimate before the call, is the mechanism.

Estimation is the awkward part. Input tokens you can estimate closely with the right tokeniser; output tokens you cannot know in advance, so debit max_tokens as a worst case and refund the difference when the response reports actual usage. Over-debiting costs you a little throughput; under-debiting costs you 429s, and 429s cost you retries, which cost you more of the limit you were trying to protect. Prefer to be pessimistic.

When you do get a 429, honour Retry-After if it is present — it is better information than any backoff you compute — and, importantly, apply the pause to the whole bucket rather than to the one unlucky request. A rate limit is a property of the account, not of the call, so every worker should slow down.

Adaptive concurrency

The fixed limit is a compromise between a value that is too low most of the time and too high during a bad hour. Congestion control solved this decades ago with additive-increase, multiplicative-decrease, and the same shape works here:

// On a clean success: creep up.        limit = min(limit + 1, ceiling)
// On a 429 or a timeout: back off hard. limit = max(floor, limit * 0.7)
// Re-evaluate on a fixed interval, not per request, so one blip does not oscillate.

Multiplicative decrease matters because the cost of being too high is much worse than the cost of being too low: too high means 429s, which means retries, which means more load on something already unhappy. Add a floor so the system can always make some progress, a ceiling so a quiet night does not leave you configured for a throughput the provider will not honour tomorrow, and a minimum sample count before reacting so a single timeout does not halve your capacity.

A latency signal can be used alongside the error signal — if the p95 duration has doubled while your concurrency stayed flat, the dependency is saturating and reducing concurrency is the polite and self-interested response. Keep the controller’s current limit as a metric; when throughput drops, the first question is always whether the limiter or the provider caused it.

Where the limiter lives

An in-process limiter is correct only if there is one process. With ten replicas each allowing twenty in flight, the provider sees two hundred, and each replica believes it is behaving. The options, in increasing order of effort: divide the allowance statically by the replica count (simple, wastes capacity when replicas are idle, breaks on autoscaling); use a shared counter in Redis with a short lease per slot (accurate, adds a round trip and a dependency); or put a single proxy in front and let it hold the limit.

Whichever you pick, scope the limit the way the provider does. Limits are usually per key, sometimes per model, occasionally per organisation. A single global limiter across models you are limited on separately will throttle one to protect another for no reason, and a per-model limiter under a shared account limit will not protect you at all.