Timeouts and Deadlines Across a Multi-Step Pipeline

A timeout says how long one call may take. A deadline says when the whole operation stops being worth doing. They look similar in a two-step pipeline and diverge badly in a five-step one, and the divergence is expensive when every step is billed.

Timeouts do not compose

Consider a pipeline: classify the request, retrieve context, generate an answer, check it against a policy model. Four calls, each given a sensible ten-second timeout. The worst case is forty seconds plus your own overhead, and the caller in front of you gives up at thirty.

Add retries — two attempts per step, which is modest — and the worst case is eighty seconds. Every one of those seconds after the caller gave up is work you are paying for and nobody will read. This is the defining property of per-step timeouts: the bound they give you is the sum, it grows with the length of the pipeline, and it is unrelated to the only number that matters, which is how long the person at the front is willing to wait.

Worse, the sum is invisible in any individual file. Each step looks reasonable. The problem only exists in the composition, which is exactly the kind of bug that survives code review.

A deadline is an absolute time

The fix is to compute one absolute instant at the edge — the moment the operation stops being worth continuing — and pass it down. Every step derives its own timeout from the deadline rather than from a constant, and every step can ask the question that a per-step timeout cannot answer: is there enough time left for me to be worth starting?

Absolute rather than relative matters. A relative budget passed down as “you have 12 seconds” loses the time spent in the handoff, in queueing, and in the part of the parent that ran before the call. Those add up across a chain. An instant does not drift, because it is not being recomputed.

Carrying it through the chain

export class Deadline {
  constructor(readonly at: number) {}

  static in(ms: number) { return new Deadline(Date.now() + ms); }

  /** The earlier of two deadlines always wins. */
  static min(a: Deadline, b: Deadline) { return new Deadline(Math.min(a.at, b.at)); }

  remaining() { return this.at - Date.now(); }
  expired()   { return this.remaining() <= 0; }

  /** An AbortSignal that fires when the deadline does. */
  signal(): AbortSignal {
    const ac = new AbortController();
    const ms = this.remaining();
    if (ms <= 0) ac.abort(new DeadlineExceeded());
    else setTimeout(() => ac.abort(new DeadlineExceeded()), ms).unref?.();
    return ac.signal;
  }

  /** Reserve time for work that must happen after this step. */
  reserve(ms: number) { return new Deadline(this.at - ms); }
}

async function step<T>(d: Deadline, minMs: number, run: (s: AbortSignal) => Promise<T>) {
  if (d.remaining() < minMs) throw new InsufficientBudget();
  return run(d.signal());
}

The minMs parameter is the whole point. Each step declares the smallest amount of time in which it could plausibly succeed — for a model call, roughly its own p95, not its median — and refuses to start below that. Starting a call you know will be aborted is the purest form of paying for nothing: the provider begins generating, you cancel, and depending on the provider you may still be billed for what was produced before the cancellation landed.

If your runtime has an ambient context mechanism — Go’s context.Context, Node’s AsyncLocalStorage, a request-scoped container — put the deadline in it. A deadline that has to be threaded through every signature by hand will be dropped somewhere, and the place it is dropped is where the eighty-second worst case comes back.

Splitting the budget

A total budget has to be divided, and the naive division — equal shares — is wrong, because the steps are not equally valuable. Two rules do better.

• Reserve backwards from the mandatory tail. If a policy check must run after generation, subtract its budget from the deadline handed to generation. Use reserve() at the point where the tail becomes known, so generation cannot consume the time the check needs. A pipeline that runs out of budget before its mandatory safety step has failed in the worst available direction.
• Give the optional steps what is left over. Retrieval, reranking, enrichment — anything whose absence degrades the answer without invalidating it — should take the remainder and skip themselves when it is too small. This is the same continue as in a degradation ladder, applied along the pipeline rather than down it.

Retries live inside the step’s budget, not outside it. A step with four seconds left does not get two two-second attempts by default — the second attempt is only worth starting if it can finish, so the retry loop must consult the same deadline. This is the single most common place the pattern is implemented halfway.

Deadlines and streams

Streaming breaks the simple model, because a streamed call has no single duration. It has a time to first token and then a rate, and the useful failure detector is different for each. There are really three numbers:

The asymmetry is worth internalising. Before the first token, aborting is cheap and recoverable. After it, aborting is visible to the user and the tokens produced so far are already spent. So put your aggression in the first bound and your patience in the third.

Where the number comes from

The edge deadline is a product decision, not an engineering one, and it should be written down as such: how long is this user willing to wait before an empty result is better than a late one? Then subtract the timeouts you do not control. Every hop in front of you — browser, CDN, load balancer, platform function limit — imposes its own cap, and your deadline must be the minimum of all of them minus enough time to serialise a fallback response. Discovering that your platform caps a response at some fixed duration while your pipeline was budgeted for longer is a familiar and entirely avoidable outage.

One habit makes all of this debuggable: put the remaining budget in the log line of every step. When something takes too long, the question is always “which step ate the budget”, and a countdown across the log answers it instantly.