End-to-End Testing an AI Chat Widget With Playwright

An end-to-end test of a chat widget that calls a real model is a test of the model. Cut the network and you get back a test of your widget — but only if you assert on something your fixture did not already contain.

Where to cut the seam

Playwright intercepts at the browser’s network layer with page.route(urlPattern, handler). The handler receives a Route, and route.fulfill() completes the request with a response you construct. That is one seam. The other is the application itself: a build-time flag that swaps the transport for a fake. Cutting at the network layer is nearly always right, because everything from the fetch call down through the SSE parser, the reducer that appends tokens, and the renderer stays in the test. Swap the transport and you have deleted the parser from coverage, which is where the bugs are.

There is a third seam people reach for and should not: pointing the test at the real provider with a cheap model. It makes the suite depend on a vendor’s availability, its rate limits and its sampler, so a failure means one of four things and you cannot tell which. Keep exactly one test that talks to the real endpoint, run it outside the end-to-end suite, and let it assert only that authentication works and a response arrives — testing without the model covers where that line sits.

The one thing to check before writing anything: your widget must send its request to a URL Playwright can match. A chat widget that talks to the same origin under /api/chat is easy. One that opens a WebSocket is a different problem with a different API, covered in mocking a WebSocket-based chat.

The SSE body as a fixture

A server-sent-events body is text with a specific frame shape: lines beginning data:, frames separated by a blank line, and a terminating sentinel that depends on the provider you emulate. Build it as a string so the test reads as data rather than as plumbing.

import { test, expect } from "@playwright/test";

const frames = [
  { choices: [{ delta: { role: "assistant" } }] },
  { choices: [{ delta: { content: "The invoice " } }] },
  { choices: [{ delta: { content: "is overdue by " } }] },
  { choices: [{ delta: { content: "12 days." } }] },
  { choices: [{ delta: {}, finish_reason: "stop" }] },
];

const sse =
  frames.map((f) => "data: " + JSON.stringify(f) + "\n\n").join("") +
  "data: [DONE]\n\n";

test("renders a streamed answer", async ({ page }) => {
  await page.route("**/api/chat", async (route) => {
    await route.fulfill({
      status: 200,
      headers: {
        "content-type": "text/event-stream",
        "cache-control": "no-cache",
      },
      body: sse,
    });
  });

  await page.goto("/support");
  await page.getByTestId("composer").fill("where is my invoice");
  await page.getByTestId("send").click();

  await expect(page.getByTestId("message-assistant")).toHaveText(
    "The invoice is overdue by 12 days.",
  );
});

Note that route.fulfill() takes one complete body. The browser still reads it through a streaming reader, so your SSE parser runs for real and a malformed frame still breaks the test — but every byte is available at once. This matters, and it is the next section.

What is actually worth asserting

The assertion above is nearly worthless on its own. You wrote the text “The invoice is overdue by 12 days” into the fixture and then asserted the page shows it; the only thing that could fail is the concatenation. The value of a widget test is in the transformations — everything the widget does to the stream that the stream did not already say.

• Frame kind to UI element. A tool_calls delta should render a tool chip carrying the tool name, not the raw JSON. Assert the chip exists and reads lookup_invoice.
• Markdown to DOM. Emit a fenced code block across three deltas so the fence opener and closer land in different frames. A parser that re-parses the whole buffer each tick handles it; one that parses per chunk does not. Assert a single pre exists.
• Terminal state. After the last frame the composer re-enables, the stop button disappears, and the busy attribute clears. Assert await expect(page.getByTestId("send")).toBeEnabled(). This is the assertion that catches a stream whose finish handler never ran.
• Message count. Exactly one assistant bubble, not two. Double-append on the first delta is a common reducer bug and is invisible if you only assert text with toContainText.
• The outbound request. Capture route.request().postDataJSON() inside the handler and assert the conversation history you expect was sent — not truncated, not duplicated, with the system prompt present exactly once.

A useful discipline when deciding whether an assertion earns its place: ask what fixture change would make it fail. If the only answer is “editing the string I already wrote into the fixture”, the assertion is a tautology. If the answer is “emitting the fence opener and closer in separate frames” or “omitting the finish frame”, it is testing your widget. Keep one text assertion as a smoke check and spend the rest of the file on the second kind.

When fulfil is not enough

Because fulfill() delivers the whole body, it cannot express “the third token arrives 400 ms after the second”. Any assertion about intermediate state — the typing indicator visible while tokens are still arriving, the stop button working mid-stream, the auto-scroll behaviour as the bubble grows — needs a source that actually pauses.

Start a small HTTP server inside the test process and point the route at it, or serve it through Playwright’s webServer config. The server writes frames with a delay between them and exposes a handle the test can use to release the next one, which turns a timing test into a deterministic one:

import { createServer } from "node:http";

// A stub that holds the connection open until the test releases each frame.
function streamServer(frames) {
  let release;
  const gate = () => new Promise((r) => (release = r));
  const server = createServer(async (req, res) => {
    res.writeHead(200, { "content-type": "text/event-stream" });
    for (const f of frames) {
      await gate();
      res.write("data: " + JSON.stringify(f) + "\n\n");
    }
    res.end("data: [DONE]\n\n");
  });
  return { server, next: () => release && release() };
}

Now the test drives the clock: send the message, assert the typing indicator is visible, release one frame, assert the partial text, click stop, and assert no further text appears. Nothing depends on a waitForTimeout, which is the usual reason a streaming test is flaky in CI and fine locally.

The paths nobody stubs

The happy path gets tested and the four failure paths do not, which is backwards — the happy path is the one a human would notice was broken. Each of these is one more page.route handler:

1. A 429 before any frame. Fulfil with status: 429 and a retry-after header. Assert the widget shows a retry affordance rather than an empty bubble.
2. An error frame mid-stream. Providers emit these as a normal SSE frame carrying an error object after a 200 has already been sent. Assert the partial text stays on screen and is marked incomplete, rather than being replaced by an error toast that loses it.
3. A stream that ends without its terminal frame. Drop the [DONE] line. A widget that treats end-of-body as success renders a truncated answer as finished — see testing a timeout on a stream that never finishes.
4. An aborted request. Call route.abort() to simulate a dropped connection and assert the widget offers to resend rather than silently discarding the user’s message.