Stubbing an LLM API With WebMock in Ruby

WebMock stubs at the adapter level, so it catches Net::HTTP, Faraday, HTTParty and the gems built on them without any change to your code. The stub is easy. Matching the body of a chat completions request is where people lose an hour.

What the stub is for

Two things, and neither is the model’s prose. The first is your request: that the system prompt was assembled in the right order, that the retrieved context was included, that the temperature you set in a config file survived the trip through three objects. The second is your response handling: that a finish_reason of length produces a truncation warning rather than a silently short answer, that a refusal is surfaced, that a 429 triggers your backoff.

Write the assertion on the request as an expectation, not as a condition on the stub. A stub that does not match produces a WebMock::NetConnectNotAllowedError with a large dump, which tells you what was sent but buries which part you cared about. A separate have_been_made expectation names the mismatch.

There is a third thing worth stubbing that people forget: the error bodies. A provider returns a JSON envelope on a 400 or a 429, and the code that reads it is written once, early, from an example in the documentation, and then never exercised again because errors are rare in development. Stubbing them is a two-line change and it is where the bugs are — a client that raises NoMethodError on a nil because the error shape differed from the one it expected fails in production at exactly the moment you least want a second bug.

The stub

# spec/spec_helper.rb
require "webmock/rspec"
WebMock.disable_net_connect!(allow_localhost: true)

# spec/summariser_spec.rb
RSpec.describe Summariser do
  let(:url) { "https://api.openai.com/v1/chat/completions" }

  let(:body) do
    {
      object: "chat.completion",
      id: "chatcmpl-abc",
      model: "gpt-4o-mini",
      choices: [
        {
          index: 0,
          finish_reason: "stop",
          message: { role: "assistant", content: "Three bullet points." }
        }
      ],
      usage: { prompt_tokens: 214, completion_tokens: 9, total_tokens: 223 }
    }.to_json
  end

  it "sends the system prompt first and parses the choice" do
    stub_request(:post, url)
      .to_return(status: 200, body: body,
                 headers: { "Content-Type" => "application/json" })

    expect(described_class.new.call("some input")).to eq("Three bullet points.")

    expect(
      a_request(:post, url).with { |req|
        JSON.parse(req.body)["messages"].first["role"] == "system"
      }
    ).to have_been_made.once
  end
end

have_been_made.once rather than a bare have_been_made matters more here than in most testing. A retry bug that fires the same completion three times is invisible to an at-least-once assertion, and it is a bug that costs money in production rather than merely being wrong.

The response headers in the stub are not decoration either. Faraday and most wrappers decide whether to parse the body as JSON from the content type, so a stub that returns a JSON string without Content-Type: application/json hands your code a String where production hands it a Hash. The test then fails for a reason unrelated to the change you made, or — worse — passes because your code calls JSON.parse defensively, hiding the fact that it parses twice in production.

The nested body trap

WebMock lets you match a body as a hash, and when the request carries Content-Type: application/json it will parse the JSON and compare structurally. hash_including relaxes that so you only have to name the keys you care about. That works cleanly at the top level:

stub_request(:post, url)
  .with(body: hash_including(model: "gpt-4o-mini"))
  .to_return(status: 200, body: body)

Where it stops behaving the way people expect is one level down. The interesting part of a chat request is messages, which is an array of hashes. hash_including loosens the matching of keys in the hash you hand it; it does not turn every nested collection into a partial match. Write this and you are asserting on the full contents of that array, in order, with every key present:

# Matches only if messages is EXACTLY this array — one element,
# with exactly these two keys. Add a system prompt later and it breaks.
stub_request(:post, url)
  .with(body: hash_including(
    messages: [{ role: "user", content: "some input" }]
  ))

That is a brittle assertion dressed as a lenient one, and it fails on the day someone prepends a system message — a change that did not break anything. The failure is also confusing, because the stub simply does not match and you get a connection error rather than a diff.

The block matcher is the answer. It receives the request, you parse the body yourself, and you assert exactly what you mean:

stub_request(:post, url)
  .with { |req|
    payload = JSON.parse(req.body)
    payload["model"] == "gpt-4o-mini" &&
      payload["messages"].any? { |m| m["role"] == "system" } &&
      payload["messages"].last["content"].include?("some input")
  }
  .to_return(status: 200, body: body)

Same idea as the custom matcher on the VCR.py page and the function matcher on the nock page: every one of these libraries defaults to something close to deep equality, and the LLM request body is the wrong shape for deep equality because it grows.

A block matcher has one cost worth knowing about. Because the block decides matching, a stub with a block that returns false behaves as though no stub existed at all, and WebMock reports a connection error rather than telling you the block rejected the request. When a previously passing spec starts failing this way, add a puts req.body inside the block before you start rewriting the production code — nine times out of ten the request changed in a way that is obvious the moment you look at it, and the block was asserting on a key that got renamed.

The same reasoning applies to the URL. Stubbing an exact string is fine when the base URL is fixed, but if your application supports a configurable endpoint — Azure, a self-hosted server, a gateway — the specs should stub the configured value rather than a hard-coded api.openai.com, or they will pass in a configuration nobody runs. WebMock accepts a regular expression or a URI template for the URL argument, which is usually the least brittle option.

Sequences, for retries and rate limits

WebMock can return a different response on each call, which is how you test the code that most needs testing: what happens when the provider says no. Chain responses with then:

stub_request(:post, url)
  .to_return(status: 429,
             headers: { "Retry-After" => "2" },
             body: { error: { type: "rate_limit_error" } }.to_json)
  .then
  .to_return(status: 200, body: body,
             headers: { "Content-Type" => "application/json" })

expect(described_class.new.call("some input")).to eq("Three bullet points.")
expect(a_request(:post, url)).to have_been_made.twice

Assert on the count, and if your client honours Retry-After, assert that it slept rather than hammering — usually by injecting the sleeper so the test can record the delay instead of waiting for it. Testing backoff on a 429 goes into that properly.

Turn the network off

WebMock.disable_net_connect!(allow_localhost: true) in your spec helper is not a nicety. It is the line that converts “we think the suite does not call the provider” into a fact: any unstubbed request raises immediately and the error names the URL and shows the body it was going to send.

allow_localhost: true keeps Capybara and a local OpenAI-compatible server working. If you run a self-hosted model on another host, allow that host explicitly by name rather than reopening the network for everything. And remember the error dump prints request headers — your Authorization header goes into the CI log with it, which is the same class of leak as a key in a recorded cassette.