End-to-End Testing a Chat UI That Renders Tokens as They Arrive

The bug class this test exists for is the one where every layer is correct and the screen is still wrong: the network delivers forty chunks, the store receives forty updates, and the user sees the answer appear once, at the end, or sees it flicker between a partial and a complete version.

Why route interception is not enough

The reflex is page.route() plus route.fulfill() with the SSE body as a string. It does not test streaming. Playwright’s fulfill takes a body as a string or Buffer, a json object, or a file path, and delivers the response as one unit; there is no option for a chunked or streamed body, and fulfilling from a readable stream is an open feature request against the project rather than a shipped API.

A test built that way passes, which is the problem. The page receives the whole body at once, the reader loop runs to completion in a single task, and the final DOM is correct — so it cannot distinguish a UI that renders incrementally from one that waits for the end. The bugs you wrote the test for are exactly the ones it cannot see.

Two approaches do work. Run a real HTTP server inside the test process and let the test decide when each frame goes out; or override window.fetch for the one endpoint with page.addInitScript, returning a ReadableStream whose controller the test can enqueue into over page.exposeFunction. The first is closer to production and exercises your real network code, so it is what follows; the second is useful when the app talks to a URL you cannot redirect.

A stream the test drives

// tests/e2e/stream-fixture.ts
import { test as base, expect } from "@playwright/test";
import { createServer } from "node:http";
import { once } from "node:events";
import type { ServerResponse } from "node:http";

type StreamHandle = {
  url: string;
  waitForRequest: () => Promise<void>;
  push: (text: string) => void;
  finish: () => void;
  destroy: () => void;
};

export const test = base.extend<{ upstream: StreamHandle }>({
  upstream: async ({}, use) => {
    let open: ServerResponse | null = null;
    let announce: (() => void) | null = null;
    const arrived = new Promise<void>((resolve) => { announce = resolve; });

    const server = createServer((_req, res) => {
      res.writeHead(200, {
        "content-type": "text/event-stream",
        "cache-control": "no-cache",
        connection: "keep-alive",
        "x-accel-buffering": "no",
      });
      open = res;
      announce?.();
    });
    server.listen(0);
    await once(server, "listening");
    const { port } = server.address() as { port: number };

    const frame = (delta: object, finish: string | null = null) =>
      'data: ' + JSON.stringify({
        object: "chat.completion.chunk",
        choices: [{ index: 0, delta, finish_reason: finish }],
      }) + '\n\n';

    await use({
      url: "http://127.0.0.1:" + port,
      waitForRequest: () => arrived,
      push: (text) => { open?.write(frame({ content: text })); },
      finish: () => {
        open?.write(frame({}, "stop"));
        open?.write("data: [DONE]\n\n");
        open?.end();
      },
      destroy: () => { open?.socket?.destroy(); },
    });

    open?.end();
    server.close();
  },
});

export { expect };

The handle gives the test four verbs — wait, push, finish, destroy — and those four cover every case on this page and the drop and cancel cases too. Point the app at upstream.url through whatever configuration it already has: an environment variable read at server start, or a query parameter your test build honours. Do not add production code paths that exist only for tests; a base-URL setting you already have is the seam.

Assertions that are actually about streaming

The point of controlling the frames by hand is that you can assert between them. Playwright’s web-first assertions retry until they pass or time out, so an assertion made while the stream is still open is well defined rather than a race.

• Partial text is on screen before the stream ends. Push two frames, then await expect(bubble).toHaveText("The quick "). If this fails and the final assertion passes, the UI is buffering.
• The streaming affordances reflect state. A stop button visible while open, hidden after the terminal frame; the send button disabled during; a typing indicator that clears. These are the controls users complain about and they are one line each.
• The terminal frame ends the pending state. Assert the spinner is gone only after finish(), not after the last content frame. A UI that clears on the last content delta looks fine until a stream sends usage after it.
• A destroyed connection surfaces. Call destroy() instead of finish() and assert an error state appears and the partial text is still visible and marked incomplete.
• Markdown does not flicker. If the UI renders markdown, push a frame ending mid-syntax — a lone ** or a half-written code fence — and assert no stray asterisks are visible and that the fence closes correctly once complete.

The prefix-monotonicity check

The strongest assertion available is that the rendered text only ever grew, and that each rendering was a prefix of the next. That single property rules out flicker, out-of-order application of deltas, duplicated chunks, and a re-render that briefly replaces the partial answer with an empty string.

import { test, expect } from "./stream-fixture";

test("the bubble grows by prefix and never flashes", async ({ page, upstream }) => {
  const seen: string[] = [];
  await page.exposeFunction("__record", (t: string) => { seen.push(t); });
  await page.addInitScript(() => {
    const start = () => {
      new MutationObserver(() => {
        const el = document.querySelector('[data-testid="assistant-message"]');
        if (el) (window as any).__record(el.textContent ?? "");
      }).observe(document.body, { childList: true, characterData: true, subtree: true });
    };
    if (document.body) start();
    else document.addEventListener("DOMContentLoaded", start);
  });

  await page.goto("/chat");
  await page.getByRole("textbox", { name: "Message" }).fill("hello");
  await page.getByRole("button", { name: "Send" }).click();
  await upstream.waitForRequest();

  upstream.push("The ");
  await expect(page.getByTestId("assistant-message")).toHaveText("The ");
  await expect(page.getByRole("button", { name: "Stop" })).toBeVisible();

  upstream.push("quick ");
  upstream.push("brown fox.");
  upstream.finish();

  await expect(page.getByTestId("assistant-message")).toHaveText("The quick brown fox.");
  await expect(page.getByRole("button", { name: "Stop" })).toBeHidden();

  expect(seen.length).toBeGreaterThan(1);
  for (let i = 1; i < seen.length; i++) {
    expect(seen[i].startsWith(seen[i - 1]), seen[i - 1] + " -> " + seen[i]).toBe(true);
  }
});

page.exposeFunction must be called before the script that uses it runs, and page.addInitScript before navigation — both orderings are in the code above and getting either wrong produces an undefined-function error inside the page that is easy to misread as a selector problem.

The observer watches document.body with characterData and subtree set, which is deliberately broad: a React re-render that replaces the text node rather than mutating it produces a childList record and no characterData one, and a narrower observer would miss exactly the re-render pattern you are trying to catch. Recording the element’s whole textContent on every record, rather than the mutation payload, keeps the assertion simple — you are comparing rendered states, not diffing them.

Putting it together

1. Add the fixture file above and make the upstream URL configurable at app start. Verify by running the app manually against the fixture server once.
2. Write the happy path first: wait for the request, push, assert partial, push, finish, assert final. Confirm it fails if you make the UI buffer — comment out the incremental update and check the partial assertion is the one that fails.
3. Add the monotonicity check. Expect to find at least one real re-render issue the first time you run it on an existing app.
4. Add the drop variant with destroy() and the cancel variant where the test clicks Stop mid-stream.
5. Keep the fixture deterministic: no timers, no waitForTimeout. Every wait in this test is either a web-first assertion or waitForRequest, which is why it does not flake.
6. Add a long-answer case. Push two hundred short frames and assert the view is scrolled to the bottom, and that scrolling up part-way through stops the auto-scroll rather than fighting the user. Both are streaming-specific behaviours no other test covers, and both are trivial once the fixture exists.