Handling 429s Without Melting Down

A 429 is not an error in the sense that a 500 is. It is a correct, successful response meaning “you are asking for more than you may have”. Systems melt down when they treat it as a fault and retry it like one, because the response to a capacity signal cannot be more load.

What a 429 is telling you

The server has measured your offered load against a limit and found it higher. Nothing was generated, nothing was billed, and — this is the part that changes the design — the condition will persist until either your rate drops or time passes. Neither of those is affected by retrying.

So the question is not “how do I retry this” but “where does this work wait, and what happens when the waiting room is full”. Answering it requires an explicit decision about which requests you are prepared to drop, and making that decision in advance is the entire difference between a system that degrades and one that collapses.

Three responses, and the one that is wrong

Retrying is not banned — it is the tail of the queue strategy, governed by Retry-After and a budget, as in retries and backoff. What is banned is retrying instead of deciding.

Sizing the queue with Little’s Law

A queue length is not a taste question. Little’s Law gives it to you directly from the throughput you actually have and the wait you are willing to impose:

L = lambda * W          queue length = throughput x acceptable wait

  throughput            = 20 requests/second   (your actual admitted rate)
  acceptable wait       = 5 seconds            (a product decision)
  ---------------------------------------------------------------
  max queue length      = 100 requests

Anything beyond 100 is a request that WILL wait longer than 5 s. Queueing it
does not help it; it only delays the moment you tell the caller the truth.

This is the argument against the large queue that feels safer. A deeper queue does not increase throughput by one request per second — it is bounded by the limit, not by your buffer — it only increases latency for everything in it. The extra depth converts fast, honest rejections into slow, expensive ones, and in the worst case into timeouts on both sides of the connection while work is still being done for a caller who left.

A bounded, deadline-aware queue

type Job<T> = {
  run: () => Promise<T>;
  deadline: number;                 // absolute ms; from the caller's budget
  priority: 0 | 1;                  // 0 = interactive, 1 = background
  resolve: (v: T) => void;
  reject: (e: Error) => void;
};

export class Shed extends Error {
  constructor(msg: string, readonly retryAfterMs: number) { super(msg); }
}

export class AdmissionQueue {
  private q: Job<any>[] = [];
  private inFlight = 0;
  private paused = 0;               // absolute ms; set by a 429's Retry-After

  constructor(
    private readonly maxQueue: number,      // from Little's Law, above
    private readonly maxConcurrent: number, // the provider's concurrency limit
  ) {}

  submit<T>(run: () => Promise<T>, deadline: number, priority: 0 | 1): Promise<T> {
    // 1. Reject work that cannot possibly meet its deadline. Cheapest check,
    //    so it goes first: never queue something you already know will fail.
    if (Date.now() >= deadline) {
      return Promise.reject(new Shed("deadline already passed", 0));
    }

    // 2. Reject when full -- but let an interactive job evict a background one
    //    rather than shedding the request a user is waiting on.
    if (this.q.length >= this.maxQueue) {
      const victim = priority === 0 ? this.q.findIndex((j) => j.priority === 1) : -1;
      if (victim === -1) {
        return Promise.reject(new Shed("queue full", this.estimateWaitMs()));
      }
      this.q.splice(victim, 1)[0].reject(new Shed("evicted by priority", 1000));
    }

    return new Promise<T>((resolve, reject) => {
      const job: Job<T> = { run, deadline, priority, resolve, reject };
      // Priority insert; FIFO within a class so nothing starves.
      const at = this.q.findIndex((j) => j.priority > priority);
      if (at === -1) this.q.push(job); else this.q.splice(at, 0, job);
      this.pump();
    });
  }

  /** Called on a 429: stop pulling until the server says we may resume. */
  penalise(retryAfterMs: number) {
    this.paused = Math.max(this.paused, Date.now() + retryAfterMs);
    setTimeout(() => this.pump(), retryAfterMs + 10);
  }

  private estimateWaitMs(): number {
    return Math.max(0, this.paused - Date.now()) || 1000;
  }

  private pump() {
    if (Date.now() < this.paused) return;

    while (this.inFlight < this.maxConcurrent && this.q.length > 0) {
      const job = this.q.shift()!;

      // 3. Re-check the deadline at DEQUEUE time, not only at submit time.
      //    Time spent queued is exactly what invalidates it, and skipping
      //    this is why queues keep working on abandoned requests.
      if (Date.now() >= job.deadline) {
        job.reject(new Shed("deadline expired while queued", 0));
        continue;
      }

      this.inFlight++;
      job.run().then(job.resolve, job.reject).finally(() => {
        this.inFlight--;
        this.pump();
      });
    }
  }
}

The three numbered comments are the whole design. Checking the deadline on the way in stops you queueing doomed work; checking it again on the way out stops you executing work that became doomed while it waited — under overload that is most of the queue, and skipping it is the single most common reason an overloaded service never recovers. Priority eviction ensures the thing you drop is the thing nobody is watching.

Where the queue should live

The class above is an in-process queue, and in-process is the right answer more often than people expect — but it has three properties you must accept knowingly.

• It dies with the process. A deploy, a crash or an autoscaler scale-in drops everything waiting. That is fine when the caller is a synchronous HTTP request that would have been abandoned anyway, and unacceptable when the work is a job somebody expects to complete.
• It does not coordinate. Ten replicas each holding a queue of 100 and a concurrency of 8 offer the provider 80 concurrent requests, not 8. Every limit you set locally must be divided by the replica count, or enforced centrally — and dividing by a replica count that autoscales is a bug waiting for a traffic spike.
• It cannot prioritise across replicas. An interactive request on a busy replica waits behind background work while another replica sits idle.

The decision rule is the caller. If a synchronous caller is holding a connection open, keep the queue in-process and short — a durable queue buys nothing for work whose consumer will have given up, and adds a network hop to the latency you are trying to protect. If the work is asynchronous and must eventually complete, use a real broker with durability, visibility timeouts and a dead-letter destination, and let the in-process queue exist only as the concurrency limiter in front of the provider.

For the cross-replica limit, the cheapest correct option is usually a shared counter in Redis with a short TTL rather than a distributed queue: you keep the local queue for ordering and the shared counter for admission, which is far less machinery than moving the whole queue out of process and fixes the failure that actually bites.

Shedding well

Load shedding has a bad reputation because it is usually implemented as an unexplained 500. Done properly it is the most user-respecting behaviour available under overload:

• Return 429 with your own Retry-After. You know roughly when capacity returns; say so. A client that can schedule around you is a client that stops hammering you.
• Shed the cheapest thing that helps. Background enrichment before interactive chat; a re-ranking pass before the answer itself.
• Degrade rather than fail where you can. A smaller model, a cached answer, or a retrieval-only response is worth more than an error page — and a semantic cache hit costs nothing against the limit at all.
• Make shedding visible. Count shed requests separately from errors. A system quietly shedding 20% of its traffic with a green error dashboard is worse than one that is visibly failing.
• Never shed silently into a retry loop of your own making. If your client retries your own 429 immediately, you have built the amplifier you were trying to avoid, one layer up.