Recording Streaming Responses as Test Fixtures

Cassette tools were built for request and response. A streamed completion is one response whose body arrives in pieces over several seconds, and the piece boundaries are the only thing your streaming code actually has to cope with. Most recording tools throw them away.

Why the recorded version passes trivially

A recording layer stores a response body as a single value: a string in a YAML file, a JSON field in a fixture. When it replays, your client reads that whole value in one go. Every data: line is complete, every JSON object is whole, every event boundary falls exactly where the parser expects.

Your parser therefore never encounters the situation it exists for. Over a real network a read returns whatever bytes have arrived: half a JSON object, two events at once, a data: prefix with its payload still in flight. A parser that assumes one read is one event works perfectly against the fixture and drops tokens in production. The test is green and it is testing the fixture, not the code.

So the useful streaming fixture is not just a capture of the bytes. It is the bytes plus a decision about where they are cut.

Capturing the stream

Capture the raw bytes rather than the SDK’s parsed objects, so the fixture is a wire recording and not a recording of a library version. curl with buffering off is enough:

curl -N --no-buffer https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,
       "stream_options":{"include_usage":true},
       "messages":[{"role":"user","content":"Say hello in five words."}]}' \
  > tests/fixtures/hello.sse

What you get is the OpenAI-style event stream: repeated data: lines each holding one chunk object, separated by blank lines, ending with a literal [DONE] sentinel. Anthropic’s format is the same transport with named events — each block carries an event: line such as content_block_delta alongside its data: — which is exactly why a fixture per provider is worth keeping rather than one canonical one.

data: {"id":"chatcmpl-abc","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-abc","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-abc","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":5}}

data: [DONE]

Two capture details worth getting right the first time. Request the usage chunk if your code reads token counts, because it only appears when asked for and a fixture without it will not exercise that path. And strip the key out of the fixture and out of your shell history before committing anything — redacting keys from fixtures applies just as much to a hand-captured file as to a cassette.

What to assert on

The assembled text is the least interesting assertion, and the only one most people write. The fixture is fixed, so asserting that the deltas concatenate to a known sentence tests string concatenation.

• Boundary handling. The same fixture, delivered in different cuts, must produce the same assembled result. This is a metamorphic assertion, and it is the strongest thing you can say about a stream parser.
• Event ordering. The role delta arrives before content; finish_reason arrives on a chunk with an empty delta; usage arrives after the final content chunk. Code that assumes the last chunk carries the content is wrong on real traffic.
• Sentinel handling. [DONE] is not JSON. A parser that decodes every data: payload unconditionally throws on the final line, and this is one of the two most common streaming bugs.
• Token accounting. That the usage chunk was read and recorded, rather than dropped because it arrived after the code considered the stream finished.
• Cancellation. That abandoning the stream half way closes the underlying response rather than leaking it, and that partial output is still surfaced.

Replaying it in pieces

Read the fixture, cut it where you choose, and feed the pieces through whichever mocking library you already use. In Node, nock accepts a function returning a readable stream:

import { readFileSync } from "node:fs";
import { Readable } from "node:stream";
import nock from "nock";

function chunked(path, size) {
  const body = readFileSync(path, "utf8");
  const parts = [];
  for (let i = 0; i < body.length; i += size) parts.push(body.slice(i, i + size));
  return Readable.from(parts);
}

nock("https://api.openai.com")
  .post("/v1/chat/completions")
  .reply(200, () => chunked("tests/fixtures/hello.sse", 17), {
    "content-type": "text/event-stream",
  });

Seventeen is chosen for being unhelpful. A fixed byte size unrelated to the line lengths guarantees that events are split at arbitrary points, which is precisely the property a real socket has and a whole-body replay does not.

In a browser test, MSW does the same with a ReadableStream whose start enqueues encoded slices, with an await between them so the UI actually re-renders — see mocking the OpenAI API with MSW. On the JVM, WireMock’s chunked dribble delay does it at the server, which is closer still to real behaviour because the chunking happens below your client rather than inside it.

Splitting in the wrong places on purpose

Rather than one fixture, write one parametrised test that replays the same bytes under several cutting strategies and asserts the same result each time:

1. Whole body in one chunk. The baseline, and the only case a plain cassette gives you.
2. One chunk per event. The case people imagine when they say “streaming”.
3. Fixed small byte size. Splits mid-JSON, mid-key, mid-UTF-8 sequence. If your parser decodes bytes to text per chunk rather than through a streaming decoder, a multi-byte character split across a boundary produces a replacement character here.
4. One byte at a time. Slow, but it is the strongest statement of the invariant, and it is fine on a fixture of a few kilobytes.
5. Two events in one chunk. A parser that handles one event per read and discards the remainder of the buffer drops output here, and this is the other most common streaming bug.

Same bytes, same expected assembled output, five deliveries. If all five agree, your parser is buffering correctly; if one disagrees, the cut tells you where the assumption is. The general parsing technique is in streaming JSON parsing.

Endings that are not [DONE]

Capture the unhappy endings too, because they are the ones production produces and no happy-path fixture contains.

Truncate a copy of the fixture mid-event and assert that your code raises rather than silently returning half an answer — a stream cut by a dropped connection ends exactly like that, with no [DONE]. Build another where an error object arrives as a data: payload after content has already streamed, which is how a mid-stream failure is reported and which breaks code that assumes an error can only arrive in place of a 200. And build one where a chunk carries an empty choices array, which is normal for the usage chunk and crashes anything that indexes into element zero without checking.