Streaming Responses From Cloudflare Workers AI

Streaming on Workers AI is one extra key in the request. Keeping it streamed all the way to the browser is the part that goes wrong, and it goes wrong in three specific, recognisable ways.

Enabling it changes the return type

Add stream: true to the model inputs and env.AI.run() stops resolving to an object with a response string. It resolves to a ReadableStream of server-sent events instead. That is a different type, not a different field, so any code path that reads result.response silently yields undefined the moment somebody flips the flag.

const stream = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
  messages: [{ role: "user", content: question }],
  max_tokens: 512,
  stream: true,
});
// stream is a ReadableStream, not { response: string }

The events are SSE frames: each one is a line beginning data: followed by a JSON object carrying the next fragment of text, and the sequence terminates with the literal frame data: [DONE]. The [DONE] sentinel is not JSON, so a parser that calls JSON.parse on every frame will throw on the last one. Check for it before parsing.

The pass-through that does not buffer

If you are not modifying the events, do not touch them. Hand the stream to the Response constructor as the body and set the content type. Nothing accumulates in the Worker, so nothing counts against the 128 MB isolate memory Cloudflare documents, and the first token reaches the client at time-to-first-token rather than at completion.

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const question = new URL(request.url).searchParams.get("q") ?? "hello";

    const stream = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
      messages: [{ role: "user", content: question }],
      stream: true,
    });

    return new Response(stream, {
      headers: {
        "content-type": "text/event-stream",
        "cache-control": "no-cache",
        "connection": "keep-alive",
      },
    });
  },
} satisfies ExportedHandler<Env>;

cache-control: no-cache is not decoration. An intermediary that decides an event stream is cacheable will hold it, and a held stream is a buffered stream by another name.

Three ways to accidentally buffer

• Awaiting the body. Any call to await new Response(stream).text(), or a for await loop that pushes chunks into an array before returning, converts the stream back into a string. The model call is no faster and no slower; the reader now waits for the last token before seeing the first.
• Returning JSON. Response.json(...) has to serialise a complete value, so it cannot be streamed by construction. If a helper in your codebase wraps every handler in a JSON envelope, streaming endpoints must bypass it.
• A middleware that reads the body. Logging middleware that records response size, or a framework layer that rewrites HTML, will consume the stream to do it. This one is the hardest to spot because your handler looks correct.

The symptom of all three is identical and distinctive: the response is correct, the total time is unchanged, and time-to-first-byte equals total time. If you are measuring only total latency you will not see it at all.

Transforming without collecting

When you do need to touch the events — to strip the SSE framing and emit plain text, or to count tokens as they pass — use a TransformStream and pipeThrough. Chunks flow through one at a time and nothing is held.

const decoder = new TextDecoder();
const encoder = new TextEncoder();
let buffered = "";

const toPlainText = new TransformStream({
  transform(chunk, controller) {
    buffered += decoder.decode(chunk, { stream: true });
    const lines = buffered.split("\n");
    buffered = lines.pop() ?? "";

    for (const line of lines) {
      if (!line.startsWith("data: ")) continue;
      const payload = line.slice(6).trim();
      if (payload === "[DONE]") return;
      try {
        const event = JSON.parse(payload);
        if (event.response) controller.enqueue(encoder.encode(event.response));
      } catch {
        // a partial frame; the remainder arrives in the next chunk
      }
    }
  },
});

return new Response(stream.pipeThrough(toPlainText), {
  headers: { "content-type": "text/plain; charset=utf-8" },
});

The buffered variable is the detail people skip. Network chunks do not respect line boundaries, so a frame can be split across two chunks and a naive line split will corrupt it. Holding the trailing partial line and prepending it to the next chunk is the whole fix, and it holds at most one line in memory rather than the whole response.

Reading it on the other end

In a browser, an endpoint that responds to GET can be consumed with EventSource, which handles framing and reconnection for you. Anything that needs POST — which is most chat UIs, because the conversation does not fit in a query string — has to use fetch and read response.body through a reader, applying the same partial-line discipline as the transform above.

Note that EventSource will reconnect on its own when the stream ends, which for a completion endpoint means it will ask the model the same question again. If you use it, close the connection explicitly when you see the terminal frame. Streaming across a persistent socket instead — where the server pushes without the client re-requesting — is what a Durable Object with a hibernating WebSocket is for.

What streaming does not fix

Streaming changes when the reader sees the answer. It does not change how long the answer takes to produce, and it does not change what it costs — the same tokens are generated and the same neurons are charged whether you deliver them in one piece or five hundred. If your problem is total latency rather than perceived latency, streaming is not the lever; a smaller model or a shorter answer is.

The thing streaming genuinely makes harder is error handling, and the reason is structural. Once you have returned a Response, the status line and the headers have gone. A provider failure at token 300 cannot become a 500, because you already sent a 200. From the client’s side, a stream that dies mid-flight and a stream that finished cleanly are the same thing: the body ended.

So a streaming endpoint needs an in-band completion signal, not only an in-band error one. Emit a terminal event on success and a distinct error event on failure, and have the client treat a stream that ends with neither as a failure. That is the only way to tell a truncated answer from a short one, and truncation is precisely the failure that produces a plausible-looking wrong result rather than an obvious one.

try {
  // ... pipe model events through ...
  controller.enqueue(encoder.encode("event: done\ndata: {}\n\n"));
} catch (err) {
  controller.enqueue(
    encoder.encode('event: error\ndata: {"stage":"generation"}\n\n')
  );
} finally {
  controller.close();
}

The other asymmetry worth knowing is what happens when the reader leaves. Cloudflare documents that when the client disconnects or the response completes, tasks associated with that request may be cancelled. That is the behaviour you want, since an abandoned stream stops generating rather than running to completion. But it means the tokens already produced have already been charged, and any bookkeeping you do after the loop will simply never run. If you record token counts per request, record them incrementally or inside ctx.waitUntil(), not as the last statement of a loop that may never reach its end.