Writing an Integration Test for a Streaming Chat Endpoint

Most streaming tests call the provider SDK and check that tokens arrive. That tests the provider. The bugs are almost always in the fifty lines of your own code between the provider socket and your client, and those lines only run when you make a request to your own endpoint.

The seam worth testing

A streaming chat endpoint is a proxy. It opens an upstream request, reads server-sent events, possibly transforms them, and writes its own events downstream. Every step in that chain has a failure mode that a test of the provider call alone cannot see:

• Compression middleware buffers the whole response. A default compression() or gzip layer collects output until it has enough to compress. The stream is byte-for-byte correct and arrives all at once, several seconds late.
• A reverse proxy buffers it instead. nginx does this by default; the fix is the X-Accel-Buffering: no response header, which your endpoint has to emit.
• Re-framing loses a frame. A TCP read does not align with an SSE frame boundary. Code that does chunk.toString().split("\n\n") per read and forgets the remainder silently drops content whenever a frame straddles two reads — which happens on long answers and almost never on short test ones.
• The terminal frame is forwarded but the socket stays open. The client renders a complete answer and keeps a connection until it times out.

None of those are provider bugs and none of them are visible from a unit test of your token-handling function. They need a request that goes in the front door.

A fake upstream that really streams

The upstream has to be faked, because a real provider call makes the test slow, non-deterministic and billable. It also has to actually stream: a mock that returns the whole body at once will hide the buffering bugs above, because there is nothing to buffer. Mock Service Worker version 2 handles this, because its HttpResponse takes the same body types as the Fetch Response constructor, including a ReadableStream.

// tests/streaming-endpoint.test.ts
import { afterAll, afterEach, beforeAll, expect, test } from "vitest";
import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";

function sseBody(frames: string[], gapMs = 0) {
  const encoder = new TextEncoder();
  return new ReadableStream({
    async start(controller) {
      for (const frame of frames) {
        controller.enqueue(encoder.encode(frame));
        if (gapMs) await new Promise((r) => setTimeout(r, gapMs));
      }
      controller.close();
    },
  });
}

const chunk = (delta: object, finish: string | null = null) =>
  'data: ' +
  JSON.stringify({
    id: 'chatcmpl-test',
    object: "chat.completion.chunk",
    model: "gpt-4o-mini",
    choices: [{ index: 0, delta, finish_reason: finish }],
  }) +
  '\n\n';

const upstream = setupServer(
  http.post("https://api.openai.com/v1/chat/completions", () =>
    new HttpResponse(
      sseBody(
        [
          chunk({ role: "assistant", content: "" }),
          chunk({ content: "Hel" }),
          chunk({ content: "lo" }),
          chunk({}, "stop"),
          "data: [DONE]\n\n",
        ],
        5,
      ),
      { headers: { "content-type": "text/event-stream" } },
    ),
  ),
);

beforeAll(() => upstream.listen({ onUnhandledRequest: "error" }));
afterEach(() => upstream.resetHandlers());
afterAll(() => upstream.close());

onUnhandledRequest: "error" is doing real work. Without it, a code path that reaches a second, unmocked provider fails silently or hits the network; with it, the test fails and names the URL. The five-millisecond gap between frames is what makes buffering observable — a buffering proxy turns four separate arrivals into one.

Reading SSE inside a test

Do not use EventSource in the test. It only issues GET requests, it reconnects on its own, and it hides the raw bytes you want to assert on. Read the response body directly and do your own framing — and write that framing once, correctly, as a helper you reuse, because the reader you write for the test is also the reader whose bugs you are hunting.

async function readFrames(res: Response) {
  const frames: string[] = [];
  const arrivals: number[] = [];
  const started = performance.now();
  let buffer = "";

  const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    let i: number;
    while ((i = buffer.indexOf("\n\n")) !== -1) {
      frames.push(buffer.slice(0, i));
      arrivals.push(performance.now() - started);
      buffer = buffer.slice(i + 2);
    }
  }
  // Anything still here is a frame that was never terminated.
  return { frames, arrivals, trailing: buffer };
}

TextDecoderStream is the important detail. It carries decoder state across reads, so a multi-byte character split across two network reads decodes correctly instead of becoming a replacement character. Doing String(value) on each raw chunk instead is a bug that only appears when somebody sends an emoji or an accented word. That failure mode has its own page: testing reassembly of streamed tokens.

What to assert

Not the text. The words come from a model in production and from your fixture here, so asserting them tests the fixture. Assert the properties of the transport that have to hold whatever the model says:

• Headers. content-type starts with text/event-stream, cache-control contains no-cache, and content-encoding is absent — that last one is the compression check, and it is a one-line assertion for a bug that is miserable to diagnose in production.
• Incrementality. More than one frame arrived, and the gap between the first and the last arrival is non-zero. If your endpoint buffers, every recorded arrival time is within a millisecond of every other.
• Framing. trailing is empty. A non-empty remainder means the last frame was written without its blank-line terminator, which most clients will hold forever rather than delivering.
• Termination. The stream ended with your terminal frame, exactly once, and the reader reached done. See testing that a stream closes cleanly.
• Isolation from upstream shape. If your endpoint emits its own event format rather than passing the provider’s through, assert that no upstream field leaked. A test that passes equally well against a proxy and a pass-through is not testing your translation layer.

Building the test

1. Start your app on an ephemeral port in beforeAll, or import the route handler and call it with a Request if your framework allows that. An ephemeral port is worth the extra lines: it exercises the real HTTP server, which is where header and buffering behaviour actually lives.
2. Install the MSW server above so the provider call is intercepted. Point your app at the same base URL the handler matches.
3. POST to your endpoint with fetch and a realistic body. Do not pass a signal yet — cancellation gets its own test.
4. Assert the four headers before reading a byte. A failure here is cheaper to read than a failure sixty frames later.
5. Read with readFrames, then assert on frames.length, trailing, the terminal frame and the arrival spread.
6. Add a second case where the fixture emits a frame split across two enqueues — half a data: line, then the rest. Your endpoint must still produce the same downstream frames. This is the one case that catches the re-framing bug, and it is two lines.