Skip to content
Multigrid

Testing code that calls an LLM

Unit tests around a non-deterministic dependency — stubbing, recording and asserting on the parts that are yours rather than the model's.

A test suite that calls a real model is not a test suite. It is a monitoring job with a slow feedback loop and a bill, and it fails for reasons that have nothing to do with the change under review. The instinct that follows is usually to assert on the model’s words anyway and mark the test flaky — which throws away the one thing a test is for, the ability to say that a red run means somebody broke something.

The way out is a boundary. Almost none of the code you are worried about is the model: it is the prompt you assembled, the retry you wrote, the timeout you set, the parser that turns a completion into a domain object, and the tool the model asked you to run. All of that is deterministic, and all of it can be tested at full speed with no network if the seam is in the right place. These pages are about putting it there and then about the harder half — deciding what is worth asserting once the output on the other side is not fixed.

Testing Node.js Code That Calls an LLM

Where to cut a Node service so the model is out of the test, using msw at the HTTP boundary and assertions on the request you sent rather than the answer you got.

10 min read

Dependency Injection for Testing an LLM Client

Why a module-level client turns a five-line fake into a rewrite, and the smallest seam that makes an LLM call testable without a mocking framework.

9 min read

Testing a Repository Pattern Wrapped Around an LLM Call

How the repository pattern changes when the store behind it is a model — partial results, a price per read, and a return value you cannot compare for equality.

10 min read

Unit Testing a Prompt-Building Function

Prompt assembly is string code with no network in it, and the five assertions worth making on it — substitution, delimiting, prefix stability, budget and loud failure.

9 min read

Stubbing an LLM Client Without a Mocking Library

A hand-written fake that implements the same interface, records its calls and scripts its answers — and an honest account of the four things it does worse than a mocking framework.

9 min read

Testing Retry Logic Around an LLM API Call

Forcing a transient failure on the first attempt so the test proves the retry executed, and turning off the SDK's own retries so you are testing your code and not theirs.

10 min read

Testing Exponential Backoff on a 429 From an LLM Provider

Asserting the delay sequence itself with a fake clock, so a backoff test finishes in milliseconds and still proves the waits were the right length.

10 min read

Testing a Timeout Around a Slow LLM Call

Simulating a hung response without waiting for one, and asserting the part everyone forgets — that the abandoned request was actually aborted.

10 min read

Testing Idempotency of a Retried LLM Request

The retry whose first attempt actually succeeded — how to reproduce it in a test, and why the dangerous duplicate is usually the tool call rather than the completion.

10 min read

Unit Testing an LLM Output Parser

Building a fixture corpus of everything a model plausibly returns, and testing the parser against it — never the model's behaviour, only your handling of it.

10 min read

VCR Cassettes for LLM Tests in Python

A working VCR.py setup for a Python LLM client, and the match_on mistake that quietly sends every test run back to the live API.

10 min read

Recording and Replaying OpenAI API Calls With nock

How to intercept the OpenAI Node SDK with nock, why older nock versions silently miss it entirely, and how to match a body the SDK rewrote.

10 min read

Mocking the OpenAI API With MSW in a Browser Test

Using Mock Service Worker for a model call made from the browser, including which URL you should actually be intercepting and the worker startup race.

10 min read

Stubbing an LLM API With WebMock in Ruby

Stubbing a chat completions call with WebMock's stub_request, and why hash_including does not do what you expect against a nested messages array.

9 min read

Mocking an LLM API With gock in Go

Using gock against an OpenAI-compatible Go client, starting with the transport interception that provider SDKs quietly break.

9 min read

Stubbing an LLM API With WireMock in Java

Running WireMock as a local stand-in for a chat completions API, for the common case where the Java client gives you no transport to patch.

10 min read

Mocking an LLM API With responses in Python

Hand-writing mock LLM responses with the responses library, and the client-library check that decides whether it can work for you at all.

9 min read

When a Cassette Goes Stale: Refreshing Recorded LLM Fixtures

Why a recorded fixture keeps a suite green against an API shape the provider retired, and the two mechanisms that make the drift visible.

9 min read

Redacting API Keys From Recorded LLM Cassettes

What to do the moment you find a live provider key in a committed test fixture, in the order that limits the damage.

9 min read

Recording Streaming Responses as Test Fixtures

Capturing a server-sent event stream as a fixture, and replaying it in chunks that split where a real network splits.

10 min read

Other topics