A Chat UI That Survives 500 Messages

A chat UI is easy at twenty messages and unpleasant at five hundred. Three things break, in this order: the DOM gets too big to lay out smoothly, the scroll position starts fighting the user, and a dropped connection mid-answer leaves a half-message with no way back. All three have concrete fixes.

The DOM budget, derived

“Virtualise your list” is advice with no threshold attached, which makes it useless for deciding whether you need to. Count nodes instead. The arithmetic is simple and it is yours to re-run with your own component.

Nodes in one rendered assistant message, counted from devtools
on a typical markdown answer:

  wrapper + avatar + header             ≈   6
  4 paragraphs (p + text node each)     ≈   8
  1 list of 5 items (ul + 5×(li+text))  ≈  11
  1 fenced code block, highlighted
     (pre + code + ~40 <span> tokens)   ≈  43
  action row (copy / retry / feedback)  ≈   8
                                          ----
                                          ≈  76 nodes

500 messages, half of them assistant answers of that shape,
half of them short user messages at ~8 nodes:

  250 × 76  = 19,000
  250 ×  8  =  2,000
              -------
              21,000 nodes

Browsers handle tens of thousands of nodes; the problem is not existence, it is that every streamed token can trigger style recalculation and layout over that tree. Layout cost scales with the number of nodes affected, and a naive implementation makes the affected set the whole conversation because the growing message changes the height of its container.

The practical thresholds that fall out of that, which you should confirm with the Performance panel on your own component rather than take on faith:

The cheapest fix comes first because it removes the cause rather than managing the symptom:

// The history never re-renders while a new answer streams.
const HistoryMessage = memo(
  function HistoryMessage({ message }: { message: Message }) {
    return <MessageBody message={message} />;
  },
  (a, b) => a.message.id === b.message.id && a.message.text === b.message.text,
);

function Conversation({ history, streaming }: Props) {
  return (
    <>
      {history.map((m) => <HistoryMessage key={m.id} message={m} />)}
      {streaming && <StreamingMessage text={streaming.text} />}   {/* only this re-renders */}
    </>
  );
}

Windowing without a library

Chat messages have variable, unknown heights, which is the hard case for virtualisation: you cannot compute the scroll height without measuring, and you cannot measure without rendering. A measure-and-cache windowed list is genuinely fiddly, and a maintained library is the right answer for production.

But there is a simpler technique that covers most chat apps and has almost no failure modes: render a bounded window of recent messages and expand it when the user scrolls up. Chat is read from the bottom, so the far history is usually not wanted at all.

"use client";
import { useEffect, useRef, useState } from "react";

const PAGE = 50;

export function MessageList({ messages }: { messages: Message[] }) {
  const [visible, setVisible] = useState(PAGE);
  const sentinel = useRef<HTMLDivElement>(null);

  // Show more when the top sentinel scrolls into view.
  useEffect(() => {
    const el = sentinel.current;
    if (!el) return;
    const io = new IntersectionObserver(
      ([entry]) => {
        if (entry.isIntersecting) {
          setVisible((v) => Math.min(v + PAGE, messages.length));
        }
      },
      { rootMargin: "400px 0px 0px 0px" },   // start before it is visible
    );
    io.observe(el);
    return () => io.disconnect();
  }, [messages.length]);

  const shown = messages.slice(Math.max(0, messages.length - visible));

  return (
    <>
      {visible < messages.length && <div ref={sentinel} aria-hidden />}
      {shown.map((m) => <HistoryMessage key={m.id} message={m} />)}
    </>
  );
}

IntersectionObserver rather than a scroll handler because it does not run on the main thread per scroll event, and the rootMargin makes the expansion happen before the user reaches the top so there is no visible gap. The node count is now bounded by PAGE times the per-message figure — about 3,800 for fifty messages of the shape derived above, which is comfortable.

Sticky scroll that the user can escape

This is the bug every chat UI ships and then fixes: the container scrolls to the bottom on every token, so a user who scrolls up to read something earlier gets yanked back down 60 times a second. The fix is to track whether the user is at the bottom and only auto-scroll if they are.

"use client";
import { useEffect, useLayoutEffect, useRef, useState } from "react";

const AT_BOTTOM_PX = 80;   // treat "close enough" as at the bottom

export function useStickyBottom(dep: unknown) {
  const ref = useRef<HTMLDivElement>(null);
  const [stuck, setStuck] = useState(true);

  // Recompute on user scroll. Passive: we never preventDefault.
  useEffect(() => {
    const el = ref.current;
    if (!el) return;
    const onScroll = () => {
      const distance = el.scrollHeight - el.scrollTop - el.clientHeight;
      setStuck(distance < AT_BOTTOM_PX);
    };
    el.addEventListener("scroll", onScroll, { passive: true });
    return () => el.removeEventListener("scroll", onScroll);
  }, []);

  // Before paint, so the jump is never visible.
  useLayoutEffect(() => {
    if (!stuck) return;
    const el = ref.current;
    if (el) el.scrollTop = el.scrollHeight;
  }, [dep, stuck]);

  return { ref, stuck, jump: () => setStuck(true) };
}

Three details that are not decoration. useLayoutEffect rather than useEffect, because the latter runs after paint and the user sees a frame at the old position — a visible flicker on every token. The 80-pixel tolerance, because exact equality fails on fractional device pixel ratios and the sticky state flickers. And the passive: true listener, which lets the browser scroll without waiting to see whether you will cancel the event.

When stuck is false, show a “jump to latest” button. That is not a nicety: without it a user who has scrolled up has no affordance to get back to a stream they can no longer see.

Keeping position when older messages load

The second scroll bug: prepending older messages changes scrollHeight, so the content the user was reading jumps upward by the height of what was inserted. Correct it by recording the distance from the bottom, which is invariant under prepending, and restoring it before paint.

const el = containerRef.current!;
const before = el.scrollHeight - el.scrollTop;   // distance from the bottom

setMessages((prev) => [...older, ...prev]);

// After React commits the new nodes, restore the same distance.
useLayoutEffect(() => {
  el.scrollTop = el.scrollHeight - before;
}, [messages.length]);

The CSS property overflow-anchor does something similar automatically in browsers that implement scroll anchoring, but it is disabled by many scroll containers and interacts badly with programmatic scrolling. Measure and restore explicitly; it is four lines and it behaves the same everywhere.

Resuming a stream that dropped

Mobile networks drop connections. If the answer only exists in the browser’s memory, a drop at token 300 of 800 loses everything and the only recovery is regenerating — which costs a second full request and produces a different answer.

The fix is that the server, not the client, owns the answer. This costs one extra write and it converts an unrecoverable failure into a reload.

1. Client POSTs the message. The server creates a row with a generation_id and status streaming, and returns the id before the first token.
2. The server streams tokens to the client and appends them to its own record of that generation — a Redis key, a row, or an append-only buffer. Batch the writes; once every 500ms is plenty and once per token is a needless write amplification.
3. On disconnect, the client reconnects to GET /api/generations/:id/stream?from=<chars-received>.
4. The server replays from that offset, then continues live if the generation is still running, or closes if it finished while the client was away.
5. On completion the server writes the final text and flips the status. A page reload now shows the whole answer with no model call. See the chat history schema for where these rows live.

// Client side: reconnect with backoff, resuming from what we have.
async function readWithResume(id: string, onText: (s: string) => void) {
  let received = 0;
  let attempt = 0;

  while (attempt < 5) {
    try {
      const res = await fetch("/api/generations/" + id + "/stream?from=" + received);
      if (!res.ok || !res.body) throw new Error("HTTP " + res.status);

      for await (const chunk of readSse(res.body)) {
        if (chunk.done) return;
        const delta: string = chunk.text ?? "";
        received += delta.length;
        onText(delta);
      }
      return;                                    // clean end
    } catch {
      attempt += 1;
      const wait = Math.min(1000 * 2 ** attempt, 10_000);
      await new Promise((r) => setTimeout(r, wait));
    }
  }
  throw new Error("could not resume after 5 attempts");
}

Resuming by character offset works because the text is append-only: the server can always answer “what came after character N” without knowing anything about tokens. Offsetting by token index does not work as well, because the token boundaries the provider used are not something the client can reconstruct.