Testing Reassembly of Streamed Tokens Into a Full Message

Reassembly code is written against a fixture where every chunk is a whole frame containing whole characters, and it works. Then a user writes in Japanese, or the answer contains an emoji, or the response is long enough that a frame straddles a TCP read, and single characters turn into question marks or whole sentences disappear.

Three boundaries, one cause

The bug always has the same shape: code assumes that the chunks it receives are aligned with some unit of meaning, and the network makes no such promise. A read from a socket gives you whatever bytes happened to arrive. There are three units this breaks:

• The character boundary. UTF-8 encodes most non-ASCII characters in two to four bytes. If a read ends between them, decoding that read in isolation produces U+FFFD, the replacement character, and the remaining bytes decode as a second one. The character is gone and cannot be recovered downstream.
• The frame boundary. An SSE frame ends at a blank line. If a read ends halfway through data: {"choices", code that splits that read and discards the remainder loses the whole frame, and code that JSON-parses it throws.
• The token boundary. Tokens are not words and are certainly not characters — a single model token can be a fragment such as Franc, and one character can span two tokens. Any logic that inspects deltas individually (trimming, capitalising, matching a stop phrase) is wrong for this reason. Match on the accumulated text, never on a delta.

All three are testable exhaustively, because the input space is small: for a fixture of n bytes there are n − 1 places to cut it, and a loop over all of them runs in under a second.

Testing the character boundary

The correct primitive is a stateful decoder. TextDecoder with { stream: true } holds an incomplete sequence back until the next call supplies the rest; the final call with no argument flushes. The test asserts that this holds at every cut point.

import { expect, test } from "vitest";

const MESSAGE = "caf\u00e9 \ud83d\ude42 \u65e5\u672c\u8a9e na\u00efve";

test("a stateful decoder survives a split at every byte offset", () => {
  const bytes = new TextEncoder().encode(MESSAGE);
  for (let cut = 1; cut < bytes.length; cut++) {
    const decoder = new TextDecoder("utf-8");
    const out =
      decoder.decode(bytes.subarray(0, cut), { stream: true }) +
      decoder.decode(bytes.subarray(cut), { stream: true }) +
      decoder.decode();
    expect(out, "split at byte " + cut).toBe(MESSAGE);
  }
});

test("a stateless decode does not, which is why the first test matters", () => {
  const bytes = new TextEncoder().encode(MESSAGE);
  const broken: number[] = [];
  for (let cut = 1; cut < bytes.length; cut++) {
    const out =
      new TextDecoder().decode(bytes.subarray(0, cut)) +
      new TextDecoder().decode(bytes.subarray(cut));
    if (out !== MESSAGE) broken.push(cut);
  }
  expect(broken.length).toBeGreaterThan(0);
});

The second test is not decoration. It documents that the fixture actually contains multi-byte characters, so that if somebody later “tidies” MESSAGE down to ASCII, the suite fails and says why rather than passing vacuously. A fixture with an emoji, a two-byte accented letter and a three-byte CJK character covers the two-, three- and four-byte cases in one string.

Testing the frame boundary

Same technique, one level up. Build a complete SSE response as a byte array, cut it at every offset, feed the two halves to your framing function, and assert the frames that come out are identical every time.

function frameSplitter() {
  let buffer = "";
  return {
    push(text: string): string[] {
      buffer += text;
      const out: string[] = [];
      let i: number;
      while ((i = buffer.indexOf("\n\n")) !== -1) {
        out.push(buffer.slice(0, i));
        buffer = buffer.slice(i + 2);
      }
      return out;
    },
    get remainder() { return buffer; },
  };
}

test("framing is independent of where the reads land", () => {
  const wire =
    'data: {"choices":[{"delta":{"content":"caf\u00e9 "}}]}\n\n' +
    ': keepalive\n\n' +
    'data: {"choices":[{"delta":{"content":"\ud83d\ude42"}}]}\n\n' +
    'data: [DONE]\n\n';
  const bytes = new TextEncoder().encode(wire);

  const expected = ["data: " , ": keepalive", "data: ", "data: [DONE]"].length;
  for (let cut = 1; cut < bytes.length; cut++) {
    const decoder = new TextDecoder();
    const splitter = frameSplitter();
    const frames = [
      ...splitter.push(decoder.decode(bytes.subarray(0, cut), { stream: true })),
      ...splitter.push(decoder.decode(bytes.subarray(cut), { stream: true })),
      ...splitter.push(decoder.decode()),
    ];
    expect(frames, "split at byte " + cut).toHaveLength(expected);
    expect(splitter.remainder, "split at byte " + cut).toBe("");
  }
});

Two details make this test find real bugs rather than confirm the obvious. The splitter keeps its buffer across calls, so the test fails loudly if somebody replaces it with a stateless text.split(). And the fixture includes a comment frame and the non-JSON [DONE] line, so a splitter that parses while framing fails here rather than in production.

Worth handling explicitly: some servers and intermediaries use \r\n line endings, so a splitter that only looks for \n\n never finds a boundary and buffers the entire response before emitting nothing. Add a fixture variant with CRLF and assert the same frame count. You do not need to worry about a blank line appearing inside a JSON payload — a literal newline is escaped as \\n inside a JSON string, so it cannot look like a frame terminator.

The whitespace bug nobody expects

The SSE grammar says that a single leading space after the colon in a data: line is part of the syntax and is stripped; anything beyond that is data. Reassembly code very often does line.slice(5).trim() instead, and that quietly deletes the leading and trailing spaces of the payload.

For a JSON payload this is harmless, because the whitespace is outside the object. But plenty of endpoints stream raw text rather than JSON, and there the difference is visible in the output: every token that began with a space loses it, and the quick brown fox is rendered as thequickbrownfox. Assert it directly — build a fixture whose deltas are "The", " quick", " brown" and assert the joined result has its spaces. Use trimStart of at most one character, or a regex that removes /^data: ?/ exactly.

Assembling the suite

1. Pick one fixture string containing a two-byte, a three-byte and a four-byte character, and reuse it in every test in the file.
2. Test the decoder alone at every byte offset. This isolates character-boundary bugs from framing bugs.
3. Test the framer alone at every byte offset, with a comment frame, a non-JSON terminal line and a CRLF variant.
4. Test the whole reader end to end at every offset, asserting that the accumulated text equals the fixture exactly — including leading and trailing spaces.
5. Assert that no assembled output contains U+FFFD. It is a single check, it is cheap, and it is the one symptom that identifies this entire family of bugs from a production log line as readily as from a test failure — which makes it worth asserting in the reader itself, not only in the suite.
6. Add a randomised case: split into k random pieces rather than two, with a seed printed on failure, and run a few hundred iterations. Two-way splits cover the boundary cases; many-way splits catch state that is only wrong on the third call.

Once framing is proven correct, the remaining reassembly problem is the JSON payload arriving across frames, which is a different mechanism and has its own page: testing that partial JSON mid-stream does not crash the parser.