Using Pact for Consumer-Driven Contract Tests Against an LLM API

Pact is built on an assumption that does not hold for a model vendor: that the provider will run your contract in their pipeline. Knowing exactly where that assumption fails tells you the one shape in which Pact is worth adopting here.

What Pact actually does

Pact is consumer-driven contract testing, and it works in two halves that are usually separated by an organisational boundary. In the first half, the consumer’s test suite declares the requests it makes and the responses it needs. Pact spins up a mock provider on localhost that serves exactly those responses, runs your real client code against it, and — if your code behaved — writes a pact file describing the interactions.

In the second half, that pact file is handed to the provider, whose own pipeline replays every recorded request against the real service and checks the real responses still satisfy the consumer’s expectations. This is the half that makes Pact different from a mock library. The mock proves your code works against your assumptions; the verification proves your assumptions are true of the running service. Without it, a pact file is a fixture with a schema attached.

The current JavaScript API is the PactV3 class from pact-js, constructed with a consumer name, a provider name and an output directory, then driven through a fluent chain. Note that the matchers live under MatchersV3, which is a distinct export from the older Matchers; mixing the two is the usual first error.

The reason this is more than a mocking library is the direction the specification flows. Nobody writes a document describing the API and then checks both sides against it. The consumer’s real code, running its real parser, generates the description as a side effect of passing its own tests — so the recorded expectations are by construction the subset of the response the consumer actually depends on, and no larger. That is the property worth paying for, because a hand-written specification always over-specifies and every over-specified field is a future false alarm.

The half you cannot run against a vendor

You cannot make OpenAI, Anthropic or Google run your pact file. There is no mechanism for it, no broker they publish to, and no reason they would. The verification half — the half that is the entire argument for Pact over a hand-written mock — is unavailable against a third-party model provider.

What remains is the consumer half: a local mock server, driven from a declarative interaction description, that your client code runs against. That is genuinely useful, but it is what a mocking library already gives you, with a file format and a broker added. If somebody proposes Pact for “contract testing our OpenAI integration” and means only this, the honest comparison is against Mock Service Worker, and Pact loses on setup cost.

There is a partial recovery, and it is worth knowing: you can run the Pact provider verifier yourself, in your own pipeline, pointed at the vendor’s live endpoint instead of at a service you deploy. Pact replays the recorded requests and checks the real responses against the recorded expectations. That gets you the scheduled live check described in catching a provider’s breaking change, with the pact file as the specification. It costs real tokens on every run, and provider states — Pact’s mechanism for putting a provider into a known condition before a request — have no meaning against a vendor you cannot instruct, so every interaction must be written stateless.

The topology where it fits

Pact earns its cost when the provider is a service you operate. If your applications talk to an internal LLM gateway, a routing layer or an inference service your team deploys, then both halves run: the application teams publish pacts describing what they need, and the gateway’s pipeline verifies it still satisfies all of them before it deploys.

That is a real and common shape, and it is where the consumer-driven part becomes the point rather than a formality. A gateway serving six internal consumers has no other way to know which of its response fields anyone depends on. The pacts are that list, generated from the consumers’ actual code rather than from a wiki page, and the gateway team gets a red pipeline when it is about to remove a field somebody parses. The rest of what the gateway owes its callers is in the gateway’s own contract suite.

Writing the pact without pinning prose

The trap specific to this domain is that Pact’s default is value matching. Give it an example body and it asserts that body exactly, which for a chat completion means asserting a generated sentence, an id that changes every call and a timestamp. Every field in the response must therefore be wrapped in a type matcher, and the generated text must be matched as “some string” and nothing more.

import path from "path";
import { PactV3, MatchersV3 } from "@pact-foundation/pact";
import { describe, it, expect } from "vitest";
import { summarise } from "../src/summarise";

const { like, integer, string, eachLike } = MatchersV3;

const provider = new PactV3({
  dir: path.resolve(process.cwd(), "pacts"),
  consumer: "support-inbox",
  provider: "llm-gateway",
});

describe("chat completions", () => {
  it("parses a completion into a summary", async () => {
    provider
      .uponReceiving("a chat completion request for a summary")
      .withRequest({
        method: "POST",
        path: "/v1/chat/completions",
        headers: { "Content-Type": "application/json" },
        body: like({
          model: string("alias-fast"),
          messages: eachLike({ role: string("user"), content: string("summarise this") }),
        }),
      })
      .willRespondWith({
        status: 200,
        headers: { "Content-Type": "application/json" },
        body: {
          id: string("chatcmpl-1"),
          object: "chat.completion",
          created: integer(1754870400),
          model: string("provider-model-v2"),
          choices: eachLike({
            index: integer(0),
            message: like({ role: "assistant", content: string("a summary") }),
            finish_reason: string("stop"),
          }),
          usage: like({
            prompt_tokens: integer(11),
            completion_tokens: integer(7),
            total_tokens: integer(18),
          }),
        },
      });

    await provider.executeTest(async (mockserver) => {
      const result = await summarise(mockserver.url, "summarise this");
      expect(typeof result.text).toBe("string");
      expect(result.tokensUsed).toBe(18);
    });
  });
});

Two details in that file are the whole technique. object and message.role are written as bare literals, so Pact matches them exactly — they are documented constants and you want a failure if they change. Everything else is wrapped, so the pact records types and the verification passes against any real response of the right shape. And the assertion inside executeTest is on your own function’s output, not on the mock’s body: the point of the consumer half is to prove your parser handles the shape, which is only demonstrated by running it.

Verifying the provider side

1. Publish the pact file from the consumer’s pipeline. A broker is the normal route; a committed file in the provider’s repo works for two teams and stops working at five.
2. In the provider’s pipeline, run the Pact verifier against a deployed instance. Consult the pact-js verifier documentation for the current option names rather than copying an older example — this is the surface that has changed most between major versions.
3. Handle non-determinism at the provider by pointing verification at a deployment configured with a deterministic stub model, or at temperature zero with a tiny token cap. You are verifying shape; paying for real generation on every verification run is a cost with no assertion attached to it.
4. Write every interaction stateless if the provider is a vendor you cannot instruct. Provider states require the provider to set something up on request, and no vendor will.
5. Gate deployment on verification results, which is the point of the whole apparatus. A pact that is published and never verified is a fixture with extra steps.