Testing That a Rate Limiter Queues Requests Instead of Dropping Them

A client-side limiter that drops excess requests and one that defers them have the same effect on your outbound request rate, which is the thing you were measuring when you wrote it. They differ entirely in what the caller sees, and the caller is who finds out in production.

Dropping and deferring look identical at first

Both keep you under the provider’s limit. Both make the graph of outbound requests flat. The difference is what happens to request number six in a one-second window when the limit is five: a deferring limiter returns a promise that settles about a second later with a real result, and a dropping limiter returns something now — a rejection, or worse, a resolved null that a caller treats as an empty answer.

The silent version is the one worth a test. Token-bucket implementations that reject when the bucket is empty are a legitimate design; the bug is a limiter documented as queueing that rejects under a load nobody exercised, or a queue with a maximum length whose overflow path returns a default. In an LLM client this surfaces as a batch job that reports success on 940 of 1,000 items and nobody can say where the 60 went.

Fake timers and a settled-time recorder

The test needs virtual time, because the real thing takes as long as the limiter says. Vitest’s timer helpers replace setTimeout, setInterval and Date; vi.useFakeTimers() installs them and vi.advanceTimersByTimeAsync(ms) advances the clock while also running timers scheduled asynchronously, which is the variant you need because a queue drains through promise callbacks.

Record the virtual time at which each request actually reaches the transport. That single array is what every assertion reads.

import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { createLimiter } from "../src/limiter";

beforeEach(() => vi.useFakeTimers());
afterEach(() => vi.useRealTimers());

it("defers excess requests rather than dropping them", async () => {
  const sentAt: number[] = [];
  const limiter = createLimiter({ perSecond: 5 });
  const send = (i: number) =>
    limiter.schedule(async () => {
      sentAt.push(Date.now());
      return i;
    });

  const all = Promise.all(Array.from({ length: 20 }, (_, i) => send(i)));
  await vi.advanceTimersByTimeAsync(5_000);
  const results = await all;

  expect(results).toEqual(Array.from({ length: 20 }, (_, i) => i));
  expect(sentAt).toHaveLength(20);
});

Note that Promise.all is created before the clock moves and awaited after. Awaiting each call in a loop under fake timers deadlocks: the first await never settles because nothing is advancing the clock, and the loop never reaches the line that would.

The four assertions

• Completeness. Twenty in, twenty out. This is the assertion the page is named for and it is one line: expect(sentAt).toHaveLength(20). A dropping limiter fails here, or fails on the Promise.all rejecting.
• Rate. Bucket the recorded timestamps by second and assert no bucket exceeds the limit. Do not assert exact timestamps; an implementation that spaces requests evenly at 200 ms and one that fires five at the top of each second are both correct against a five-per-second limit, and pinning the timestamps makes the test reject a legal implementation.
• Order. If your limiter promises FIFO, assert the results come back in submission order, as above. If it does not promise FIFO, assert the set rather than the sequence — and write down which one you chose, because a caller that assumes ordering will be broken by a fair-queue rewrite.
• No busy-wait. Assert the total virtual time is close to the theoretical minimum. Twenty requests at five per second should complete in about four seconds of virtual time; a limiter that polls with a fixed 1 s sleep before each check will take twenty, and that difference is invisible in production except as latency nobody can explain.

const perSecond = new Map<number, number>();
for (const t of sentAt) {
  const bucket = Math.floor(t / 1000);
  perSecond.set(bucket, (perSecond.get(bucket) ?? 0) + 1);
}
for (const [, count] of perSecond) expect(count).toBeLessThanOrEqual(5);
expect(Math.max(...sentAt) - Math.min(...sentAt)).toBeLessThan(4_500);

An unbounded queue is also a bug

Having proved nothing is dropped, prove that the queue has a limit. A limiter that accepts arbitrarily many deferred requests converts a rate problem into a memory problem, and then into a timeout problem: the request at position 40,000 resolves long after the HTTP request that created it gave up, so the work is done and paid for and thrown away.

Configure a maximum depth and write the mirror-image test. Submit more than the maximum, and assert that the excess rejects immediately with a specific error — not at the back of the queue, and not silently. Immediate, named rejection is a backpressure signal a caller can act on; a promise that resolves in four minutes is not.

it("rejects immediately once the queue is full", async () => {
  const limiter = createLimiter({ perSecond: 5, maxQueueDepth: 10 });
  const tasks = Array.from({ length: 30 }, () => limiter.schedule(async () => "ok"));
  const settled = await Promise.allSettled(tasks);
  const rejected = settled.filter((s) => s.status === "rejected");

  expect(rejected).toHaveLength(19);
  expect((rejected[0] as PromiseRejectedResult).reason.name).toBe("QueueFullError");
});

Nineteen rather than twenty because the first task is in flight rather than queued. That kind of off-by-one is worth writing out explicitly in the test rather than computing, since computing it in the test means reimplementing the limiter inside its own test.

Traps that make this test lie

Three things commonly make this suite pass against a broken limiter. The first is real timers left installed by a previous test file — the suite then takes four real seconds, usually still passes, and stops catching the busy-wait case. Assert on virtual duration, and it becomes visible.

The second is a limiter that shares state across tests. A module-level singleton keeps its bucket between cases, so the second test in a file starts with an empty bucket and a full queue. Construct the limiter inside the test, or reset it in beforeEach.

The third is testing the limiter with a task that never yields. Recording a timestamp and returning synchronously never gives the queue a chance to misbehave. Make the fake task await something — even a zero-delay timer — so the interleaving in the test resembles a network call. Once that passes, the same harness will also show you how the limiter interacts with retries, which is the next test to write: a retry scheduled inside a limiter must re-enter the queue rather than bypass it, or your retry policy quietly becomes a way to exceed your own rate limit. The mechanics of that are set out in testing retry backoff, and the interaction compounds, because the retries provoked by a limit breach are themselves subject to the limit.

One last check worth adding while the harness is in front of you. Assert that a task which throws does not wedge the queue. A limiter that decrements its in-flight counter only on the success path will leak a slot per failure, and after five failures a five-per-second limiter permits nothing at all — a total outage whose only symptom is that every request times out. Submit ten tasks, make three of them reject, and assert that the remaining seven still complete and that the recorded rate is unchanged. It is two extra lines and it covers the worst failure this component has.