Testing Retry and Backoff Without Waiting for the Backoff

A retry ladder of five attempts with exponential backoff and full jitter can take four minutes to exhaust. A test suite cannot spend four minutes on it, and a test that shortens the base delay to make it fast is no longer testing the ladder you ship. The way out is to stop measuring time and start recording it.

Why elapsed time is the wrong assertion

The instinct is to wrap the call, measure wall-clock duration, and assert it was roughly what the ladder predicts. That test is slow and flaky in the same breath: slow because it genuinely waits, flaky because a loaded CI runner adds hundreds of milliseconds of scheduling noise on top of a value you are trying to bound. Widen the bound enough to be stable and it no longer distinguishes a correct ladder from a broken one.

The behaviour you actually care about is a sequence of intended delays: 1s, 2s, 4s, 8s, each with jitter, five attempts, give up on the sixth. That sequence is a value your code computes. If you can get hold of it, you can assert on it exactly, in a test that finishes in single-digit milliseconds and never flakes.

Inject the sleep and the randomness

The cleanest way to get hold of it is to make the two non-deterministic dependencies parameters. A sleep function and a random function, both defaulting to the real thing, both replaceable in a test.

// retry.ts
export type RetryDeps = {
  sleep: (ms: number) => Promise<void>;
  random: () => number;
};

const realDeps: RetryDeps = {
  sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
  random: Math.random,
};

export class RetryableError extends Error {
  constructor(readonly status: number, readonly retryAfterMs?: number) {
    super("retryable " + status);
  }
}

export async function withRetry<T>(
  fn: (attempt: number) => Promise<T>,
  opts: { attempts?: number; baseMs?: number; maxMs?: number } = {},
  deps: RetryDeps = realDeps,
): Promise<T> {
  const attempts = opts.attempts ?? 5;
  const baseMs = opts.baseMs ?? 1000;
  const maxMs = opts.maxMs ?? 30000;

  for (let i = 0; i < attempts; i++) {
    try {
      return await fn(i);
    } catch (err) {
      const retryable = err instanceof RetryableError;
      if (!retryable || i === attempts - 1) throw err;

      const ceiling = Math.min(maxMs, baseMs * 2 ** i);
      // Full jitter: uniform in [0, ceiling). Server-supplied Retry-After wins.
      const delay = err.retryAfterMs ?? Math.floor(deps.random() * ceiling);
      await deps.sleep(delay);
    }
  }
  throw new Error("unreachable");
}

Two details in there are the ones that break in real code. The ceiling is clamped by maxMs before jitter is applied, not after, so a long ladder cannot produce a twelve-minute wait. And a server-supplied Retry-After overrides the computed delay entirely rather than being added to it — a provider that tells you when to come back knows something you do not.

The four assertions worth writing

With sleep injected, the test records the delay sequence instead of experiencing it. Everything else follows.

// retry.test.ts
import { describe, expect, it } from "vitest";
import { RetryableError, withRetry } from "./retry";

function recorder(random = 0.5) {
  const delays: number[] = [];
  return {
    delays,
    deps: {
      sleep: async (ms: number) => { delays.push(ms); },
      random: () => random,
    },
  };
}

describe("withRetry", () => {
  it("gives up after the configured number of attempts", async () => {
    const { delays, deps } = recorder();
    let calls = 0;
    await expect(
      withRetry(async () => { calls++; throw new RetryableError(429); },
        { attempts: 5, baseMs: 1000 }, deps),
    ).rejects.toBeInstanceOf(RetryableError);

    expect(calls).toBe(5);      // five attempts, not five retries
    expect(delays).toHaveLength(4); // four waits between them
  });

  it("keeps every delay inside its jitter window", async () => {
    for (const r of [0, 0.999]) {
      const { delays, deps } = recorder(r);
      await withRetry(async () => { throw new RetryableError(503); },
        { attempts: 5, baseMs: 1000, maxMs: 30000 }, deps).catch(() => {});
      const ceilings = [1000, 2000, 4000, 8000];
      delays.forEach((d, i) => {
        expect(d).toBeGreaterThanOrEqual(0);
        expect(d).toBeLessThan(ceilings[i]);
      });
    }
  });

  it("does not retry a 400", async () => {
    const { delays, deps } = recorder();
    let calls = 0;
    await expect(
      withRetry(async () => { calls++; throw new Error("bad request"); },
        {}, deps),
    ).rejects.toThrow("bad request");
    expect(calls).toBe(1);
    expect(delays).toEqual([]);
  });

  it("succeeds on a later attempt without extra waits", async () => {
    const { delays, deps } = recorder();
    const out = await withRetry(
      async (attempt) => {
        if (attempt < 2) throw new RetryableError(429);
        return "ok";
      }, {}, deps);
    expect(out).toBe("ok");
    expect(delays).toHaveLength(2);
  });
});

The attempt-versus-retry off-by-one in the first test is the bug this catches most often. Five attempts means four waits; five retries means six requests. Teams disagree about which their config field means, and the disagreement shows up as a provider bill or a rate-limit incident rather than as a failing test — until you assert on both numbers.

The jitter test deserves a word too, because it is the one people skip as untestable. Randomness is only untestable while it is buried inside the function; once random is a parameter, the interesting cases are the endpoints of its range, and two runs — one at 0 and one at just under 1 — pin both edges of every window in the ladder. That catches the two mistakes full jitter is prone to: a ceiling computed before the clamp, so the top of a late window exceeds maxMs, and an exponent taken from the retry count rather than the attempt index, which shifts the whole ladder by one rung and doubles the first wait.

When you cannot inject the clock

Sometimes the retry lives inside a vendor SDK and there is no seam. Then fake timers are the tool. Vitest documents vi.useFakeTimers() for replacing setTimeout, setInterval and Date, and vi.advanceTimersByTimeAsync(ms) for advancing them in a way that also drains timers scheduled from promise callbacks — which is the one you need, because a retry loop schedules its next wait from inside an async continuation, and the synchronous vi.advanceTimersByTime will not see it.

import { afterEach, expect, it, vi } from "vitest";

afterEach(() => { vi.useRealTimers(); });

it("retries a rate limit without real waiting", async () => {
  vi.useFakeTimers();
  const promise = callTheSdk();          // starts, hits 429, schedules a wait
  await vi.advanceTimersByTimeAsync(60_000);
  await expect(promise).resolves.toBeDefined();
});

Restore real timers in afterEach without exception. A leaked fake clock is one of the more baffling failures in a suite, because the test that breaks is a later, unrelated one whose own timeout never fires.

Retry-After and the give-up condition

Two cases remain and both are worth a test of their own. The first is that Retry-After can arrive as either a delay in seconds or an HTTP date, and a parser that assumes one will produce NaN for the other. Assert the parsed delay for both forms.

The second is the give-up condition, which is where retrying stops being free. Every retried request is a request you pay for, so a ladder that retries a request that will never succeed is a cost bug as much as a latency bug. And a retry of a request that had a side effect is worse than a slow one: if the call executed a tool, charged a card or posted a message, the second attempt does it twice. That is the argument for an idempotency key on anything retried, and it is a genuine invariant rather than a vendor detail: assert that the side effect happened exactly once, in its own test, separately from testing that the delays are right.

The last thing to test is the interaction between the ladder and whatever sits above it. A retry loop nested inside a client that also retries multiplies rather than adds — three attempts wrapping three attempts is nine requests at a provider that is already telling you it is overloaded, and it is one of the more common ways a rate-limit incident becomes a self-inflicted outage. Both the vendor SDK and your HTTP client probably have a retry setting; find them, set one of them to zero, and write a test that counts requests at the transport to prove which one is active. The same test is what tells you a circuit breaker is doing its job, because a breaker that opens should reduce the request count to zero rather than merely spacing the failures out.