Testing an AI Feature Without Calling the Model

“You cannot unit test it, the output is non-deterministic” is true of one function and false of the twenty around it. The non-determinism is confined to a single call; the reason it feels pervasive is that the call is usually not behind a seam.

Most of your AI feature is not AI

Write out what a typical feature does between receiving a request and returning an answer: authorise the caller, load and validate the input, select a model, assemble a prompt from a template and some retrieved context, count tokens and truncate to fit, call the model, parse the output, validate it against a schema, apply business rules to the parsed result, compute the cost, record usage, and cache.

Exactly one of those steps is non-deterministic. Everything else is ordinary code with ordinary inputs and outputs, and it is where most of your bugs live — truncation that cuts mid-token, a prompt template that silently drops a variable, a parser that accepts a fenced code block but not a bare one, a cost calculation that ignores cached input tokens.

The reason those bugs escape is not that they are hard to test. It is that they are entangled with the call, so the only way to exercise them is an integration test that costs money and fails intermittently, so nobody writes many. Break the entanglement and they become boring.

The seam

One interface, injected rather than imported, with a fake that does not touch the network. The task-shaped interface described elsewhere in this cluster gives you this for free; if you have a thinner wrapper, put the seam at its narrowest point.

export interface ModelClient {
  complete(req: Request, ctx: Ctx): Promise<{ text: string; meta: RunMeta }>;
}

/** A fake with a script. Not a mocking framework: a class with a queue. */
export class ScriptedClient implements ModelClient {
  constructor(private script: (Response | Error)[]) {}
  readonly seen: Request[] = [];

  async complete(req: Request): Promise<{ text: string; meta: RunMeta }> {
    this.seen.push(req);                    // assertions about the prompt live here
    const next = this.script.shift();
    if (!next) throw new Error("scripted client exhausted: unexpected extra call");
    if (next instanceof Error) throw next;
    return next;
  }
}

Two features of that fake matter more than they look. It records the requests, so a test can assert that the retrieved context reached the prompt, that the system prompt was not duplicated, and that the truncation kept the most recent turns. And it throws on an unexpected extra call, which is how you catch a retry loop that runs one more time than it should — a bug that is invisible in a test that only checks the final value, and expensive in production.

The script is a list, so error sequences are trivial to express: [new Timeout(), new Timeout(), ok] tests that two failures are survived; [new RateLimited(2000), ok] tests that Retry-After is honoured; [malformedJson, ok] tests that a parse failure is treated as a retryable rung.

Parsers deserve adversarial tests

The function that turns model output into a typed value is the single highest-value thing to test, because it faces an adversary that is not malicious but is inexhaustibly creative. Build a corpus of real malformed outputs as you encounter them and keep it in the repository. The recurring shapes:

• JSON wrapped in a fenced code block, sometimes with a language tag, sometimes with prose before and after it.
• Valid JSON with an extra key, a missing optional key, or a number where a string was requested.
• Truncated output, because the answer hit max_tokens mid- object. This is why the finish reason must be checked before the parse: length means the output is incomplete no matter how plausible the prefix looks.
• Trailing commas, single quotes, unescaped newlines inside strings, and the word “json” on its own first line.
• An empty string, or a polite refusal where a schema was expected.

Two testing habits pay off here. Property-based tests over generated objects — serialise, wrap in random prose and fences, parse, assert round-trip — find whole classes of bug that examples miss. And every parse failure seen in production becomes a test case, which turns your incident log into a growing suite.

Test the strict path too: when the input is genuinely unparseable, the function must fail loudly rather than return a plausible half-filled object. A lenient parser that guesses is worse than a strict one that triggers a retry.

Testing the policies: retry, deadline, breaker

Retry policy, deadline propagation and circuit breaking are pure control flow over a clock and an error stream, and both are injectable. Pass in a clock rather than calling Date.now()directly, and these become fast deterministic tests:

• A 429 with Retry-After: 2 waits about two seconds, not the backoff schedule’s value.
• A 400 is not retried at all — assert on the fake’s call count, because this is the bug that quietly triples the bill.
• With 500ms left on the deadline, a step whose minimum is 2s is skipped rather than started.
• Eleven failures in a window of twenty opens the breaker; the twenty-first call fails without reaching the fake.
• After the cool-down, exactly one probe is admitted and the others are still rejected.

Every one of those runs in milliseconds and none of them costs anything. They are also the tests most likely to catch something genuinely expensive, because a bug in the retry policy is a bug that multiplies spend.

What genuinely needs a model

A small residue does require the real thing, and being clear about it keeps the expensive suite small:

• Contract checks. Does this model still accept this schema, still emit tool calls in this shape, still exist? These are about the provider, not your logic. A handful of tiny calls, scheduled rather than on every commit.
• Quality evaluation. Whether the output is good is not a unit test and should not pretend to be one. It belongs in a separate suite with its own dataset, its own scoring and its own cadence, and it reports a score rather than passing or failing.
• Prompt regressions. When a prompt changes, the question is whether behaviour moved on your evaluation set. Same suite as above, triggered by a change to the prompt version.

Keep those out of the commit path. A test suite that costs money and fails randomly gets disabled within a month, and then you have neither the fast tests nor the slow ones.