Property-Based Testing LLM Output in JavaScript With fast-check

fast-check is the property-based testing library for JavaScript and TypeScript, and a model call is asynchronous, which puts you on the async path immediately. That path has one failure mode that silently turns a red test green, so it is worth reading before the arbitraries.

The async property, and the trap in it

Two functions build a property: fc.property for a synchronous predicate and fc.asyncProperty for one that returns a promise. You hand the result to fc.assert. The signatures matter here: when given an async property, fc.assert returns a Promise<void> rather than void.

If you do not await it, the test function returns before a single model call has resolved, the runner records a pass, and the assertion failures surface later as unhandled rejections that most CI configurations will not fail on. The test is green and tests nothing. Await every fc.assert on an async property, and give the surrounding test a timeout large enough for the whole run — not for one call.

import fc from "fast-check";
import { expect, test } from "vitest";

test(
  "extraction preserves the amount",
  async () => {
    await fc.assert(
      fc.asyncProperty(refundRequest, async (req) => {
        const out = await classifyRefund(req);

        expect(Object.keys(out).sort()).toEqual(
          ["amountCents", "currency", "decision", "reason"],
        );
        expect(["approve", "review", "decline"]).toContain(out.decision);
        expect(out.currency).toBe(req.currency);
        expect(out.amountCents).toBe(req.amountCents);
      }),
      { numRuns: 20, timeout: 30_000 },
    );
  },
  600_000,
);

The synchronous fc.property is not an escape route here. If you hand it a predicate that returns a promise, fast-check sees a truthy object rather than a boolean and treats every run as a pass. The rule is simple and absolute: a predicate that awaits anything needs fc.asyncProperty, and the fc.assert around it needs an await. If your suite has ever gone green suspiciously fast after you added a model call, this is the first thing to check.

Arbitraries that produce plausible requests

fc.record builds an object from a map of arbitraries, and takes a requiredKeys constraint if you want some of them dropped sometimes — which is a good way to find out what your prompt does with a missing field. fc.constantFrom picks one of its arguments. fc.string takes { minLength, maxLength, unit }, where unit chooses what a “character” means — the default is grapheme-based ASCII, and 'grapheme' widens it to full Unicode graphemes, which is the setting that will find your emoji and combining-mark bugs.

const orderId = fc
  .tuple(fc.constantFrom("NL", "DE", "GB"), fc.integer({ min: 0, max: 999_999 }))
  .map(([cc, n]) => cc + "-" + String(n).padStart(6, "0"));

const refundRequest = fc.record({
  orderId,
  currency: fc.constantFrom("EUR", "USD", "GBP", "JPY"),
  amountCents: fc.integer({ min: 1, max: 1_000_000 }),
  locale: fc.constantFrom("en-GB", "nl-NL", "de-DE", "ja-JP"),
  note: fc.string({ maxLength: 120, unit: "grapheme" }),
});

The order of arguments to fc.constantFrom is not cosmetic. fast-check shrinks a constantFrom toward its first argument, so putting your most ordinary value first means a failing case involving currency will be reported with EUR unless the currency is actually implicated. The same instinct applies throughout: integers shrink toward their minimum, strings toward empty, and fc.oneof shrinks within the branch it chose but not across branches unless you pass withCrossShrink. You are choosing what a failure will look like when you declare the arbitrary.

Parameters that decide what the run costs

The second argument to fc.assert is a Parameters object. The fields that change for a model-backed property:

• numRuns — documented default 100. That is 100 requests per property before any shrinking, and shrinking adds more. Set it deliberately.
• timeout — a per-predicate limit in milliseconds, disabled by default. Without it a hung provider connection stalls the whole suite rather than failing one example.
• seed — defaults to Date.now(), so every run generates different data. Pin it in CI if you want two runs of one commit to make the same calls; leave it floating if you want the suite to explore. Do not do both by accident.
• endOnFailure — stops at the first failure without shrinking. Worth setting temporarily when you only need to know that it fails and shrinking would cost another few dozen calls.
• examples — an array of values prepended to the generated ones. This is where a previously found counterexample lives permanently, and it is the cheapest regression test in the file.
• ignoreEqualValues — discards runs whose generated value was already tried. With a small constantFrom space and a paid call behind it, duplicate draws are money.
• interruptAfterTimeLimit — stops the whole run after a wall-clock budget, reporting what it managed rather than failing. This is the closest thing fast-check offers to a spend cap: a property that has run for four minutes against a slow provider has already cost what it was going to cost, and the remaining runs are unlikely to tell you something the first sixty did not.

These interact in a way worth stating once. numRuns bounds the generation phase only; a failure then enters shrinking, which has no run cap of its own and can comfortably issue more calls than the search that found the failure did. If you are budgeting a suite, budget it as generation plus an unbounded tail, or set endOnFailure and accept an unminimised counterexample.

Replaying the exact failure: seed and path

When a property fails, fast-check reports the counterexample together with the seed and a path, and the path identifies the exact position in the shrink tree that the reported value came from. Passing both back reproduces that single case without regenerating the run:

await fc.assert(fc.asyncProperty(refundRequest, predicate), {
  seed: 1_732_998_401,
  path: "12:3:1",
  endOnFailure: true,
});

This replay is exact for the input and not for the outcome, because the model is not a pure function of that input. Reproducing a model failure needs the seed and path to rebuild the request, plus a pinned model version, plus a temperature of zero, and even then the honest expectation is “usually” rather than “always”. That is a property of the system under test, not of fast-check, and it is the same reason shrinking behaves differently here than in ordinary property testing.

Running it

1. npm i -D fast-check vitest. fast-check has no peer dependency on a runner; it works the same under Jest, Vitest or node:test.
2. Check the arbitrary alone before wiring the model in: console.log(fc.sample(refundRequest, 10)) prints ten generated requests. Look at them. If they do not resemble your traffic, the property will be testing a distribution nobody sends.
3. Add fc.statistics(refundRequest, (r) => r.currency, 1000) once, to see how the generation is actually distributed across the dimension you care about. Uniform is rarely what you want and is always what you get by default.
4. Write the predicate with the assertions in cheapest-first order: shape, then enum membership, then conservation. The first failing assertion is the one reported, so ordering them decides how legible the failure is.
5. Run with a low numRuns, confirm it passes, then raise it and leave it. When it fails, copy the counterexample into examples and keep it there.