Unit Testing a Custom Output Validator Before You Wire It to a Model

A validator is a pure function from a candidate output to a verdict. Everything hard about testing language models — sampling, latency, cost, drift — is absent here, which is exactly why this is the part of the pipeline that should have the most tests and usually has the fewest.

Why this is a pure-function test

The validator sits between the model and everything downstream. It receives a string or a parsed object, applies rules, and returns a verdict. If it calls the model — to judge a field, to repair the output, to score a rubric — then it is not a validator, it is a second inference step, and it needs a completely different testing approach. Keep the deterministic checks in a function with no I/O and push anything model-shaped outside it. That separation is the single decision that makes this testable in milliseconds.

The practical consequence is that this suite needs no credential, no network, no fixture recording, and no mock. It runs on every commit, it runs on a fork’s pull request where secrets are unavailable, and it runs before the expensive suite so that a broken validator does not waste a model-calling job. That is worth engineering for even if it costs a small amount of indirection.

The table

Drive it from a list of named cases rather than writing a test per rule. A table makes the coverage visible: you can read down the names and see which rules have a negative case and which only have a positive one.

import { describe, expect, it } from "vitest";
import { validateInvoice } from "../src/validate";

const CASES = [
  { name: "complete invoice", input: { vendor: "Acme", total_cents: 1250, currency: "GBP" }, valid: true, codes: [] },
  { name: "negative total", input: { vendor: "Acme", total_cents: -1, currency: "GBP" }, valid: false, codes: ["total_cents.negative"] },
  { name: "total as string", input: { vendor: "Acme", total_cents: "1250", currency: "GBP" }, valid: false, codes: ["total_cents.type"] },
  { name: "unknown currency", input: { vendor: "Acme", total_cents: 1250, currency: "XYZ" }, valid: false, codes: ["currency.enum"] },
  { name: "empty vendor", input: { vendor: "", total_cents: 1250, currency: "GBP" }, valid: false, codes: ["vendor.empty"] },
  { name: "whitespace vendor", input: { vendor: "   ", total_cents: 1250, currency: "GBP" }, valid: false, codes: ["vendor.empty"] },
  { name: "two problems at once", input: { vendor: "", total_cents: -1, currency: "GBP" }, valid: false, codes: ["vendor.empty", "total_cents.negative"] },
];

describe("validateInvoice", () => {
  it.each(CASES)("$name", ({ input, valid, codes }) => {
    const result = validateInvoice(input);
    expect(result.ok).toBe(valid);
    expect(result.errors.map((e) => e.code).sort()).toEqual([...codes].sort());
  });
});

Assert on the errors, not on the boolean

This is the assertion that earns the page. In almost every real pipeline, a failed validation is not the end — the errors are formatted into a message and sent back to the model as a repair attempt. That makes the error list a load-bearing output, and a validator tested only on ok being false has an untested output that the model reads.

Two properties follow. First, the validator must collect all errors rather than returning the first one. A fail-fast validator drives a repair loop that fixes one problem per round trip, which turns a two-problem output into three model calls, and each of those calls can introduce a new problem. Assert the multi-error case explicitly — the last row in the table above exists only for that.

Second, assert on a stable error code and separately on the human-readable message. The code is what your retry logic branches on and must not change casually; the message is what the model sees and should be specific enough to act on. “Invalid input” tells a model nothing; “total_cents must be a non-negative integer in minor units; received the string 1250” tells it exactly what to change. Write one assertion per rule that the message names the field and the constraint, so that nobody can degrade the message without a test noticing.

The rows that matter

• Absent versus null versus empty. Three different inputs that non-trivially different code paths conflate. Under schema-constrained decoding you will see explicit nulls far more often than missing keys, so the null row is not hypothetical.
• Whitespace-only strings. A model asked for a vendor name it cannot find will sometimes return a space, a hyphen, or the literal text “N/A”. Decide which of those are valid and write the row either way.
• Numbers that arrive as strings. Extremely common, and the row that catches a validator built on truthiness rather than on types.
• Unicode and length. A name with combining characters, a right-to-left string, and one at your maximum length plus one character. If the validator enforces a length, it must be explicit about whether it counts code points or UTF-16 units, because the model’s idea of length is neither.
• Input mutation. Assert the validator does not modify what it was given. A validator that trims, coerces or fills defaults while reporting is two functions wearing one name, and the bug it causes — the caller keeps the object it passed in and finds it changed — is hard to find later.

One more row belongs in every table and is almost always missing: the output that is valid but empty of information. An extraction where every optional field is null and every required one is a placeholder satisfies the schema, satisfies the type checks, and is worthless. If your validator is meant to reject that, it needs a rule and a row; if it is not, write the row anyway and assert that it passes, so the next person can see the decision was made rather than overlooked.

Property-testing the accept side

Table tests cover the rejections you thought of. The complementary check is generative: if you already have a schema, generate arbitrary instances that conform to it and assert the validator accepts every one. A false rejection is the more expensive failure, because it sends a correct output back for repair and costs a full round trip each time it fires.

Both fast-check in JavaScript and Hypothesis in Python can build generators from a schema definition, and either way the property is one line: anything the schema admits, the validator admits. Where the validator is deliberately stricter than the schema — a business rule the schema cannot express — encode that as a filter on the generator, and the act of writing the filter documents the gap between the two, which is usually knowledge that lives nowhere else. For the broader case of testing pipeline logic with the model removed entirely, see testing without the model.

A last note on where the validator sits. Because it is cheap and pure, there is a temptation to run it in more places than one — at the model boundary, again before persistence, again at the API edge. That is fine, and it is much better than the alternative of validating nowhere, but it makes the error codes a public contract between layers rather than an internal detail. Version them the way you would version any other contract, keep the code strings out of user-facing messages, and write one test asserting the full set of codes the validator can emit. That test fails when somebody adds a rule without telling the repair prompt about it, which is the drift that makes a retry loop quietly stop working.