Deploying an AI App Without Timing Out

“It works locally and times out in production” is the most common serverless AI complaint, and it is hard to fix because four different clocks can end the request and they produce nearly identical symptoms. Identify which one fired before changing anything.

Four clocks, not one

They fail differently, and the difference is diagnostic. The platform limit ends the function, so nothing after it runs — no log line, no database write, no error handler. Your own fetch timeout throws inside the function, so your catch runs and you get a log. An idle proxy timeout closes the connection while the function keeps running, which is the confusing one: the user sees a failure and your logs show a success.

Local development has none of these. next dev is a long-running Node process with no duration ceiling and no proxy in front, which is precisely why the bug does not reproduce.

Raising the function ceiling

// app/api/chat/route.ts
export const runtime = "nodejs";
export const maxDuration = 60;   // seconds; a request, not a guarantee
export const dynamic = "force-dynamic";

// app/api/how-long/route.ts — measures the real ceiling for your project.
export const runtime = "nodejs";
export const maxDuration = 300;      // ask for a lot, and see what you get
export const dynamic = "force-dynamic";

export async function GET() {
  const started = Date.now();
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      // Emit every second until something kills us. The last line the client
      // receives is the effective ceiling, in seconds.
      for (let i = 1; i <= 600; i++) {
        await new Promise((r) => setTimeout(r, 1000));
        controller.enqueue(
          encoder.encode("elapsed " + Math.round((Date.now() - started) / 1000) + "s\n"),
        );
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

Run it with curl -N against the deployment. The last number printed is your real limit under your real configuration, which is the only figure worth designing against. Delete the route afterwards, or protect it — it is a five-minute function invocation anybody can trigger.

What streaming actually buys

Streaming is often described as the fix for timeouts. It fixes two of the four clocks and neither of the others, and being precise about which is what stops people being surprised twice.

• It fixes the browser clock. The response starts at time-to-first-token, so from the browser’s point of view the request succeeded almost immediately and the rest is body.
• It fixes the idle-proxy clock. Every chunk is activity, and an idle timer that keeps resetting never fires.
• It does not fix the function duration limit. The function is still running while it streams. A 90-second generation under a 60-second ceiling is killed mid-stream, and the user sees a truncated answer with no error — the response already carried a 200 status, so there is no way to signal failure in the status code.
• It does not fix a slow model. Total time is unchanged. It moves when the user first sees something, which is a large perceived win and no change at all to the clock.

The truncation case deserves a defence, because it fails silently. Emit a terminal event, and have the client treat its absence as a failure:

// Server: always end with an explicit terminal event.
controller.enqueue(encoder.encode('data: {"type":"done","reason":"stop"}\n\n'));
controller.close();

// Client: a stream that ended without one was cut off.
let sawDone = false;
for await (const evt of readSse(res.body!)) {
  if (evt.type === "done") { sawDone = true; break; }
  append(evt.delta ?? "");
}
if (!sawDone) {
  setError("The answer was cut short. Retry, or shorten the request.");
}

Keepalives, and their limits

When the model takes a long time before its first token — reasoning models, long prompts, a cold provider — the connection is open with nothing on it, which is exactly what an idle proxy timeout looks for. Send a comment line periodically: the SSE format defines a line beginning with : as a comment, and every conformant client ignores it.

function withKeepalive(upstream: ReadableStream<Uint8Array>, everyMs = 15_000) {
  const encoder = new TextEncoder();

  return new ReadableStream<Uint8Array>({
    async start(controller) {
      const timer = setInterval(() => {
        controller.enqueue(encoder.encode(": keepalive\n\n"));
      }, everyMs);

      const reader = upstream.getReader();
      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          controller.enqueue(value);
        }
        controller.close();
      } catch (err) {
        controller.error(err);
      } finally {
        clearInterval(timer);            // must run on every exit path
        reader.releaseLock();
      }
    },
  });
}

The finally is the whole correctness of it: an interval that outlives its stream keeps enqueuing into a closed controller, which throws on a timer you can no longer see. Keepalives also do nothing about the function duration limit — they keep the connection alive, and the connection was never the constraint.

The point where you move to a queue

There is a clean test. If the work can exceed the function ceiling under any realistic input, it does not belong in a request. Not “usually finishes in time” — the tail is what pages you.

Concretely: move to a queue when the job is a batch over many documents, when it is an agent loop whose length depends on the model’s own choices, when it needs a retry against a different model on failure, or when the user does not need to watch it happen. The shape is always the same.

1. The request enqueues a job and returns an id immediately. Response time is milliseconds and no ceiling is in play.
2. A worker — a background function, a container, a queue consumer — runs the job with a duration budget appropriate to the work rather than to an HTTP request.
3. The worker writes progress and the result to the database, keyed by the job id.
4. The client polls that id, or subscribes over SSE to a channel keyed by it.
5. Failures are retried by the queue with backoff, and a poison message lands in a dead-letter queue instead of retrying forever. This is the part a request-scoped implementation never gets right.

Resist the intermediate design where the request kicks off work and hopes the platform keeps the process alive after responding. On serverless it may not, and the failure is intermittent and load-dependent, which is the worst combination to debug. More on the pattern in running AI work as a background job.

Diagnosing which clock fired