Timeouts: Choosing One That Isn’t a Guess

Thirty seconds is the most popular timeout in the industry and it is a number somebody typed. A timeout is a claim about your latency distribution — specifically, the point past which waiting longer is worse than giving up — and you can compute that point from data you already have.

What a timeout is for

Not “to stop waiting forever”. TCP already does that, eventually. A timeout exists to bound resource occupancy — a connection, a worker, a user’s attention — when the probability that waiting longer will pay off has dropped below the cost of continuing to wait.

That framing has an immediate consequence. Set the timeout too low and you abandon requests that would have succeeded, and if you retry them you pay for both. Set it too high and a stuck request holds a connection, a thread and a user for the duration. The correct value is therefore a property of two distributions: how long successful requests take, and how much a wasted wait costs you.

Four timeouts, not one

A single overall deadline conflates events with wildly different meanings. Split it:

The stall timeout is the important addition. A stream that is producing tokens is healthy no matter how long it has been running; a stream that has produced nothing for fifteen seconds is not, even if it started two seconds ago. An overall deadline gets both of those exactly backwards: it kills a long healthy generation and tolerates a short dead one.

Deriving each from your distribution

The procedure needs one input: a few thousand recorded durations of successful requests, segmented by whatever changes the shape — model, prompt-length bucket, streaming or not. Then:

timeout = quantile(successful_durations, q) * safety

  q = 0.99   for a user-facing path        -> abandons ~1% of good requests
  q = 0.999  for a background job          -> abandons ~0.1%
  safety = 1.5 to 2                        -> covers drift between reviews

WORKED, using placeholder numbers you replace with your own:
  suppose p99 first-token, short prompts   = 2.4 s
          p99 first-token, 30k prompts     = 9.1 s
          p99 inter-token gap              = 1.8 s

  first-token timeout(short)  = 2.4 * 2   =  5 s
  first-token timeout(30k)    = 9.1 * 2   = 18 s
  stall timeout               = 1.8 * 5   =  9 s   (higher multiple: gaps are
                                                    scheduler noise, and their
                                                    tail is much heavier)

Two notes on doing this honestly. Compute the quantile over successes only — including timed-out requests censors the distribution at your current timeout and makes the next value you derive too small, a feedback loop that ratchets the timeout downward over successive reviews. And bucket by prompt size, because time to first token grows with prefill; a single value applied to both a 200-token and a 30,000-token prompt is either far too loose for one or far too tight for the other.

A stall timeout that actually works

The mechanism is a timer reset by every chunk, racing the read. Written as a wrapper so it composes with any async iterable of chunks:

export class StreamStalled extends Error {
  constructor(readonly afterMs: number, readonly tokensSoFar: number) {
    super("stream stalled for " + afterMs + "ms after " + tokensSoFar + " tokens");
  }
}

/**
 * Enforce a gap limit between chunks, and a separate (larger) limit before
 * the first one. Aborts the underlying request rather than merely rejecting,
 * so the socket is released instead of leaking.
 */
export async function* withStallTimeout<T>(
  src: AsyncIterable<T>,
  ac: AbortController,
  opts: { firstChunkMs: number; stallMs: number },
): AsyncGenerator<T> {
  const it = src[Symbol.asyncIterator]();
  let n = 0;

  try {
    for (;;) {
      const limit = n === 0 ? opts.firstChunkMs : opts.stallMs;

      let timer: ReturnType<typeof setTimeout>;
      const expiry = new Promise<never>((_, reject) => {
        timer = setTimeout(() => {
          ac.abort();                       // free the socket, do not just reject
          reject(new StreamStalled(limit, n));
        }, limit);
      });

      let step: IteratorResult<T>;
      try {
        step = await Promise.race([it.next(), expiry]);
      } finally {
        clearTimeout(timer!);               // or the timer keeps the loop alive
      }

      if (step.done) return;
      n++;
      yield step.value;
    }
  } finally {
    await it.return?.(undefined);           // propagate cancellation upstream
  }
}

Three details are load-bearing. The first-chunk limit is separate, because that gap legitimately contains queueing and prefill while later gaps do not. The abort is inside the timer, not left to the caller, because a rejected promise on its own leaves the HTTP request running. And clearTimeout is in a finally, because in Node an un-cleared timer keeps the event loop alive and turns a fast script into one that hangs for the length of your longest timeout.

A timeout is also a concurrency limit

This is the consequence people discover during an incident rather than during design. Little’s Law again: the number of requests in flight is the arrival rate multiplied by how long each one stays. Your timeout is the upper bound on how long each one stays, so it is also the upper bound on your own concurrency.

in_flight_worst_case = arrival_rate * timeout

  50 req/s x  2 s  =    100 concurrent   -- fine
  50 req/s x 30 s  =  1,500 concurrent   -- 1,500 sockets, 1,500 response
                                            buffers, 1,500 stalled handlers
  50 req/s x 120 s =  6,000 concurrent   -- your process dies long before this

When the provider slows down, the arrival rate does not fall to match it — callers keep arriving. Occupancy climbs to whatever the timeout permits, and if that exceeds your connection pool, your memory, or your file-descriptor limit, your service fails for reasons that have nothing to do with the model. A generous timeout, chosen to be kind to slow requests, is the mechanism by which one dependency’s slowdown becomes your outage.

So the timeout has to be chosen against two constraints, not one: long enough that you are not abandoning requests that would have succeeded, and short enough that rate × timeout stays inside the resources you actually have. When those conflict, the resolution is not a compromise value — it is an explicit concurrency limit that sheds rather than queues, which is the subject of handling overload. Bounding occupancy directly is strictly better than bounding it as a side effect of a duration.

Deadlines beat timeouts in a call chain

One more failure mode, common in agent systems. If a handler has a 10-second budget and makes three model calls each with a 10-second timeout, the timeouts are decorative: the caller gives up first, and the work continues downstream, billing you, unobserved.

The fix is to pass a deadline rather than a duration. Compute an absolute time once at the edge, and let each step take the smaller of its own timeout and the remaining budget:

const deadline = Date.now() + 10_000;

function budget(preferred: number): number {
  const left = deadline - Date.now();
  if (left <= 0) throw new Error("deadline exceeded before call");
  return Math.min(preferred, left);
}

await callModel({ timeoutMs: budget(8_000) });   // step 1
await callModel({ timeoutMs: budget(8_000) });   // gets whatever survived

The same absolute deadline should also gate whether a retry is worth attempting at all — retrying with 200 ms of budget left spends money on a request whose result nobody will wait for. That interaction is covered in retries and backoff.