Testing a Timeout Around a Slow LLM Call

The reason timeout code is so often wrong is that testing it honestly means waiting for it, so nobody does, and the branch ships unexercised. A fake clock and a resolver that never answers remove the wait entirely.

Three assertions, not one

“It threw a timeout error” is the assertion everybody writes and it is the least valuable of the three.

• It rejects at the deadline, not near it. Assert still-pending at one millisecond before and rejected at the deadline. A timeout that is quietly ten times its configured value because somebody passed seconds where milliseconds were expected passes a vaguer test easily.
• The underlying request is aborted. Abandoning a promise is not cancelling a request. The socket stays open, the provider keeps generating, and you are billed for output you threw away — and under load your connection pool fills with requests nobody is waiting for, which turns one slow call into a general outage.
• The error is classified as transient. A timeout is retryable, but only if the operation is safe to repeat. If it is not, that is an idempotency question and the answer is not “retry anyway”.

Simulating a hang that costs nothing

You need a response that never arrives. At the HTTP boundary, a resolver that returns a promise which is never resolved does it, and msw also exports a delay helper that accepts the string "infinite" for exactly this purpose — documented by the msw project at mswjs.io/docs/api/delay.

import { delay, http } from "msw";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { server } from "./setup";
import { classify } from "../src/classify";

beforeEach(() => vi.useFakeTimers({ shouldAdvanceTime: false }));
afterEach(() => vi.useRealTimers());

it("rejects at 10s and not before", async () => {
  server.use(
    http.post("https://api.openai.com/v1/chat/completions", async () => {
      await delay("infinite");
      return new Response();   // never reached
    }),
  );

  const promise = classify("…", { timeoutMs: 10_000 });
  let settled = false;
  promise.catch(() => { settled = true; });

  await vi.advanceTimersByTimeAsync(9_999);
  expect(settled).toBe(false);

  await vi.advanceTimersByTimeAsync(1);
  await expect(promise).rejects.toThrow(/timed out after 10000ms/);
});

Attaching the catch immediately matters for a reason unrelated to the assertion: a promise that rejects while nothing is attached triggers an unhandled-rejection warning, and in some runner configurations that alone fails the run with an error that points nowhere near the cause.

Proving the request was actually cancelled

This is the assertion that distinguishes a real timeout from a give-up, and it is the reason the page exists. Pass an AbortSignal down to the client — the OpenAI Node SDK accepts signal in its per-request options alongside timeout and maxRetries — and assert in the resolver that it fired.

it("aborts the in-flight request rather than abandoning it", async () => {
  let aborted = false;

  server.use(
    http.post("https://api.openai.com/v1/chat/completions", async ({ request }) => {
      request.signal.addEventListener("abort", () => { aborted = true; });
      await delay("infinite");
      return new Response();
    }),
  );

  const promise = classify("…", { timeoutMs: 5_000 });
  promise.catch(() => {});

  await vi.advanceTimersByTimeAsync(5_000);
  await expect(promise).rejects.toThrow();

  expect(aborted).toBe(true);
});

If aborted is false, your timeout is implemented as a race between the request promise and a sleep. That pattern is extremely common, it looks correct in every code review, and it leaks a socket and an unbounded generation on every timeout. The fix is to have the deadline trigger an AbortController whose signal is handed to the client, and the assertion above is what stops it regressing back to a race the next time someone simplifies the function.

A stream needs an idle timeout, not a total one

A total timeout is wrong for streaming and applying it there is a bug the tests above will not catch. A long answer legitimately takes longer than a short one, so a fixed total either kills healthy long generations or is set so high it never fires. What has actually failed is a stream that has stopped producing, and the right deadline is on the gap between events.

So the timer resets on each chunk, and the test needs a handler that emits some events and then stops:

it("survives a slow but progressing stream and fails on a stalled one", async () => {
  const chunks = [
    'data: {"choices":[{"delta":{"content":"Hel"}}]}\n\n',
    'data: {"choices":[{"delta":{"content":"lo"}}]}\n\n',
  ];

  server.use(
    http.post("https://api.openai.com/v1/chat/completions", () => {
      const stream = new ReadableStream({
        async start(controller) {
          const enc = new TextEncoder();
          for (const c of chunks) {
            controller.enqueue(enc.encode(c));
            await delay(4_000);         // under a 5s idle budget
          }
          await delay("infinite");      // then stall, and never send [DONE]
        },
      });
      return new Response(stream, {
        headers: { "content-type": "text/event-stream" },
      });
    }),
  );

  const promise = streamClassify("…", { idleTimeoutMs: 5_000 });
  promise.catch(() => {});

  await vi.advanceTimersByTimeAsync(8_000);   // two chunks, gaps under budget
  await vi.advanceTimersByTimeAsync(5_000);   // then the stall exceeds it
  await expect(promise).rejects.toThrow(/no data for 5000ms/);
});

The two advances encode the whole property: 8,000ms of total elapsed time did not trip a 5,000ms budget because the gaps were smaller, and the stall did. That is the difference between an idle timeout and a total one stated as an executable assertion. The transport mechanics underneath are covered in streaming transport.

Add one more case while you are here: a stream that ends without a terminating sentinel. A truncated text/event-stream that simply closes should be an error rather than a successful short answer, and if your parser treats end-of-stream as completion it will silently hand a half-sentence to your users.

The idle budget also has to survive the first chunk. Time to first token on a long prompt is legitimately much longer than the gaps that follow it, so a single idle value tight enough to catch a stall mid-stream will kill healthy calls before they start. Two numbers — a generous budget until the first event, a tighter one between events after that — is the configuration that works, and it needs its own test case: a stream whose first chunk arrives just inside the opening budget and whose later gaps are well inside the idle one.

Which layer owns the deadline

Finally, write down the arithmetic in a test, because in production there are usually three deadlines and they are frequently in the wrong order: the HTTP client’s per-request timeout, your wrapper’s budget across all attempts, and whatever the caller upstream is willing to wait.

The invariant is that the innermost is strictly the shortest. A per-attempt timeout of 30s inside a total budget of 20s means the budget always fires first and the per-attempt value is dead configuration; three attempts of 30s inside a caller with a 60s deadline means the third attempt is always wasted. Assert the ordering directly on your config object — it is a two-line test that catches an entire class of misconfiguration that is otherwise only visible under load.

Pin the starting point too, in one test rather than in all of them. A deadline that begins when your function is entered includes any time spent queueing behind a concurrency limiter, so a service under load times out on calls that never reached the provider at all. A deadline that begins when the request is dispatched measures the provider and nothing else. Both are defensible choices and they produce very different incident reports; a test that names which one you made stops the next person changing it by accident while refactoring the limiter.