Streaming Tokens Into a React UI

Streaming does not make the model faster. It moves the moment the user sees something from the end of generation to the first token, which for a 600-token answer is the difference between eight seconds of nothing and 400 milliseconds of something. The work is on your side: read the body as it arrives, parse it, and get it into React without paying a full render per token.

What the stream actually contains

With stream: true, the response body is a sequence of server-sent events. Each event is a line beginning data: followed by JSON, and events are separated by a blank line. The terminal event is the literal string data: [DONE], which is not JSON and will throw if you hand it to JSON.parse.

data: {"choices":[{"delta":{"role":"assistant","content":""}}]}

data: {"choices":[{"delta":{"content":"Because"}}]}

data: {"choices":[{"delta":{"content":" each"}}]}

data: {"choices":[{"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Three facts about that stream decide whether your parser works. The chunks that arrive from the network are not the events: a single read() may hand you two and a half events, or half of one. A multi-byte character can be split across two chunks. And the first delta usually carries role with empty content, so a component that renders on first delta renders nothing and looks broken.

Parsing SSE correctly

TextDecoder with { stream: true } handles the split multi-byte character — it holds the incomplete sequence and emits it when the rest arrives. The split-event problem is yours: keep a buffer, split on the blank line, and leave the trailing fragment in the buffer.

// stream.ts — framework-free, works in the browser and in Node 18+
export async function* readSse(body: ReadableStream<Uint8Array>) {
  const reader = body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });

      // Events are separated by a blank line. The last element is a
      // fragment and must stay in the buffer until more arrives.
      const parts = buffer.split("\n\n");
      buffer = parts.pop() ?? "";

      for (const part of parts) {
        const line = part.split("\n").find((l) => l.startsWith("data:"));
        if (!line) continue;
        const payload = line.slice(5).trim();
        if (payload === "[DONE]") return;
        yield JSON.parse(payload);
      }
    }
  } finally {
    reader.releaseLock();
  }
}

An async generator is the right shape here because the consumer controls the pace: a for await that awaits inside its body applies backpressure all the way to the socket, which a callback-based reader does not. Note the finally — releasing the lock matters when the loop exits by break or by a thrown error, and that is the path an aborted request takes.

The naive React version, and what it costs

// Works. Do not ship it.
for await (const chunk of readSse(res.body!)) {
  const delta = chunk.choices?.[0]?.delta?.content;
  if (delta) setText((t) => t + delta);   // one render per token
}

Count the work. A 600-token answer at 60 tokens per second is 600 state updates over ten seconds — about one every 17ms, which is coincidentally one per frame, so on a trivial component this is fine. It stops being fine for two reasons that compound.

First, string concatenation in state means every update allocates a new string of the full length so far. Six hundred appends to a growing string is quadratic in total bytes copied: roughly n²/2 characters, or about 800KB of copying for a 1,800-character answer. That alone is survivable. Second, and much worse: if that state lives high in the tree, every token re-renders every descendant. Markdown rendering, syntax highlighting and virtualised lists are all expensive per render, and 600 of them inside ten seconds is where the typing animation starts stuttering.

Fast models make this worse, not better. At 200 tokens per second you are asking for a render every 5ms, and the browser only has a frame every 16.7ms.

Flushing on animation frames

The fix is to decouple arrival from rendering. Accumulate into a ref, which does not trigger a render, and flush to state once per animation frame. The user cannot see more than one frame anyway, so nothing is lost.

"use client";
import { useCallback, useEffect, useRef, useState } from "react";
import { readSse } from "./stream";

export function useStreamedCompletion() {
  const [text, setText] = useState("");
  const [status, setStatus] = useState<"idle" | "streaming" | "done" | "error">("idle");

  const pending = useRef("");        // arrives here
  const frame = useRef<number | null>(null);
  const abort = useRef<AbortController | null>(null);

  const flush = useCallback(() => {
    frame.current = null;
    if (!pending.current) return;
    const next = pending.current;
    pending.current = "";
    setText((t) => t + next);
  }, []);

  const schedule = useCallback(() => {
    if (frame.current !== null) return;   // already queued for this frame
    frame.current = requestAnimationFrame(flush);
  }, [flush]);

  const send = useCallback(
    async (prompt: string) => {
      abort.current?.abort();
      const controller = new AbortController();
      abort.current = controller;

      setText("");
      pending.current = "";
      setStatus("streaming");

      try {
        const res = await fetch("/api/chat", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ prompt }),
          signal: controller.signal,
        });
        if (!res.ok || !res.body) throw new Error("HTTP " + res.status);

        for await (const chunk of readSse(res.body)) {
          const delta = chunk.choices?.[0]?.delta?.content;
          if (delta) {
            pending.current += delta;
            schedule();
          }
        }
        flush();
        setStatus("done");
      } catch (err) {
        if ((err as Error).name === "AbortError") return;  // deliberate, not a failure
        setStatus("error");
      }
    },
    [flush, schedule],
  );

  useEffect(() => {
    return () => {
      abort.current?.abort();
      if (frame.current !== null) cancelAnimationFrame(frame.current);
    };
  }, []);

  return { text, status, send, stop: () => abort.current?.abort() };
}

The render count is now bounded by the frame rate rather than by the model’s speed: at most 60 renders per second regardless of whether tokens arrive at 30/s or 300/s. The final flush() after the loop is not optional — without it the last few tokens sit in the ref until a frame that never comes, and the answer is silently truncated by a word or two. That bug is intermittent and it is a miserable one to find.

Unmounting mid-stream

A user who navigates away mid-answer leaves a reader holding an open socket and a setState queued against a component that no longer exists. React 18 and later stopped warning about the second, which means the leak is now silent. The cleanup function above handles both: it aborts the fetch, which rejects the pending read() with an AbortError and closes the connection, and it cancels the queued frame.

One subtlety worth internalising: aborting stops the bytes reaching you, but the tokens already generated upstream are generated and, depending on the provider, billed. That is covered properly in cancelling an in-flight request, and it is the reason a “stop” button is a UX feature rather than a cost control.

If the text you are streaming is markdown, do not pipe it straight into dangerouslySetInnerHTML as it grows — a half-arrived link is a genuinely exploitable state, which is the whole subject of rendering model markdown without an XSS hole.