Unit Testing an LLM Output Parser

The mistake this page exists to correct is a test that asserts the model said a particular thing. What you can test, completely and deterministically, is whether your parser copes with everything a model plausibly returns — and that is a different suite with a different input.

Two questions, and only one is a unit test

Put the two questions side by side and the confusion dissolves. “Does the model return valid JSON for this prompt?” is a question about a third party’s behaviour on a distribution of inputs. Answering it needs many samples, a scoring function and a statistical claim; it belongs in an evaluation run, not in a unit suite, and the library already covers that ground in evaluation frameworks and golden datasets.

“Given this exact string, does my parser produce the right value or the right error?” is a pure function over a fixed input. It runs in a millisecond, it cannot be flaky, and it is where every parser bug you have ever shipped actually lives. That is the suite this page builds.

The practical consequence is that the parser must be extractable. If the JSON handling is inline in the same function that makes the HTTP call, you cannot run it against a fixture without a transport, and you will end up testing both at once and blaming the wrong one when it fails.

The fixture corpus

The corpus is the deliverable. Everything else on this page is scaffolding around it. It should contain, as literal strings, every shape a model has plausibly produced for your prompt:

• The clean case: exactly the object you asked for and nothing else.
• Fenced: the same object wrapped in a triple-backtick block, with and without a json language tag. Extremely common even when the prompt forbids it.
• Preamble and postscript: “Sure! Here is the JSON you requested:” before it, or a helpful explanation after it.
• Truncated: valid JSON up to the point where finish_reason came back as length. Your parser must distinguish this from malformed output, because the remedy differs — raise the token limit rather than fix the prompt.
• Typographically mangled: smart quotes instead of straight ones, usually arriving with non-English output.
• Schema-shaped but wrong: a number as a string, a null where you expected a value, an extra field, an enum value that is not in your enum, a nested object flattened.
• A refusal, and a content_filter stop, both of which are structurally valid responses containing no data.
• The empty string, and a response whose content is null because the model returned tool calls instead.

Keep them as files under a fixtures directory rather than as inline string literals. Inline strings tempt people to edit them until the test passes, which inverts the whole point; a file recorded from a real response is evidence.

A parser that returns instead of throwing

The corpus above only becomes testable if the parser has a total signature — every input maps to a value, including the bad ones. A parser that throws for six different reasons forces every test to match on message text.

// src/parse-extraction.ts
import { z } from "zod";

const Schema = z.object({
  invoiceNumber: z.string().min(1),
  totalCents: z.number().int().nonnegative(),
  currency: z.string().length(3),
});

export type ParseResult =
  | { ok: true; value: z.infer<typeof Schema> }
  | { ok: false; reason: "empty" | "no_json" | "truncated" | "invalid_json" | "schema"; detail: string };

export function parseExtraction(
  content: string | null,
  finishReason: string,
): ParseResult {
  if (finishReason === "length") return { ok: false, reason: "truncated", detail: "stopped at max_tokens" };
  if (!content || !content.trim()) return { ok: false, reason: "empty", detail: "no content" };

  const fenced = content.match(/```(?:json)?\s*([\s\S]*?)```/i);
  const candidate = fenced ? fenced[1] : content.slice(content.indexOf("{"), content.lastIndexOf("}") + 1);
  if (!candidate.trim().startsWith("{")) return { ok: false, reason: "no_json", detail: content.slice(0, 80) };

  let raw: unknown;
  try { raw = JSON.parse(candidate); }
  catch (e) { return { ok: false, reason: "invalid_json", detail: String(e) }; }

  const parsed = Schema.safeParse(raw);
  if (!parsed.success) {
    return { ok: false, reason: "schema", detail: parsed.error.issues.map((i) => i.path.join(".")).join(", ") };
  }
  return { ok: true, value: parsed.data };
}

The reason union is the real interface. Each value maps to a different operational response: truncated means raise the budget, no_json means the prompt or the response format needs work, schema means the model is answering a slightly different question than you asked, and empty usually means a refusal or a tool call you did not handle. Emitting one string reason as a metric label makes a dashboard that tells you which fix to make.

Table-driven tests over the corpus

import { describe, expect, it } from "vitest";
import { parseExtraction } from "../src/parse-extraction";
import { readFileSync } from "node:fs";

const fixture = (name: string) =>
  readFileSync(new URL(`./fixtures/${name}.txt`, import.meta.url), "utf8");

describe("parseExtraction", () => {
  it.each([
    ["clean",            "stop",   true],
    ["fenced-json",      "stop",   true],
    ["fenced-untagged",  "stop",   true],
    ["with-preamble",    "stop",   true],
    ["with-postscript",  "stop",   true],
  ])("accepts %s", (name, finish, ok) => {
    const result = parseExtraction(fixture(name), finish as string);
    expect(result.ok).toBe(ok);
    if (result.ok) expect(result.value.invoiceNumber).toBe("INV-7781");
  });

  it.each([
    ["truncated",      "length", "truncated"],
    ["prose-only",     "stop",   "no_json"],
    ["smart-quotes",   "stop",   "invalid_json"],
    ["total-as-string","stop",   "schema"],
    ["refusal",        "stop",   "no_json"],
  ])("rejects %s with reason %s", (name, finish, reason) => {
    const result = parseExtraction(fixture(name), finish as string);
    expect(result.ok).toBe(false);
    if (!result.ok) expect(result.reason).toBe(reason);
  });

  it("never throws on any fixture", () => {
    for (const name of ALL_FIXTURES) {
      expect(() => parseExtraction(fixture(name), "stop")).not.toThrow();
    }
  });
});

That last test is worth more than it looks. A parser that throws on an input you did not anticipate takes down the request path rather than degrading it, and the set of inputs you did not anticipate is by definition unknown. Asserting totality over the whole corpus is the cheapest available approximation, and it gets stronger every time somebody adds a fixture.

One assertion you will be tempted to write and should not: comparing the parsed object against an expected object for the prose fields. If your schema has a summary string, assert its constraints — non-empty, under a length cap, no leading “Sure,” — never its content. The moment a fixture is re-recorded from a newer model, an exact-content assertion fails for a reason that is not a bug.

Growing the corpus from production

The corpus is only as good as its coverage of what really happens, and you will not imagine the weird cases. Wire the pipeline so that everyok: false in production writes the raw content to storage with its reason and its model id. Then adding a fixture is a copy, and each production surprise permanently becomes a regression test.

1. Log the raw content on every parse failure, redacted for anything sensitive that was in the prompt. Cap the length so an enormous response cannot fill the log.
2. Once a week, take the distinct failure reasons and pull one example of each into test/fixtures/, named for the reason rather than for the incident.
3. Add a row to the table. If the parser should have handled it, fix the parser; if the shape is genuinely unhandleable, the fixture documents which error the caller sees, which is still a decision worth pinning.
4. Record the model id and date alongside each fixture. When you switch models, the output shape can move under you, and knowing which fixtures predate the switch tells you which ones to re-record rather than trust.

The same discipline applies to streaming, where the fixture is a sequence of events rather than a string and the interesting cases are a JSON value split across two chunks and a stream that ends mid-object. Capturing those is its own recording problem, and the parser tests on top of them look exactly like the table above.