Using a Local Model in Tests Instead of Paying for API Calls

A small local model is not a cheaper version of the provider. It is a different tool that answers a different question, and it is genuinely better than a mock at exactly one of the three things a test suite needs.

Three questions, three fakes

Sort your tests by what they are actually asking and the choice makes itself.

• “Does my code do the right thing with this response?” Parsing, validation, retries, error handling, state updates. The response is an input to the test, so you must control it exactly. A mock or a stub. A model cannot help and can only introduce variance — this is most of your suite.
• “Does my code talk to an inference server correctly?” Request serialisation, streaming, tool call round-trips, timeouts, cancellation. Here a hand-written stub is weak evidence, because you wrote the thing it is being checked against. A real local server is the right fake.
• “Does the provider still behave as I assume?” Whether a parameter is honoured, whether a schema is enforced, what a rate limit looks like. Only the provider can answer, and a local model answering confidently is worse than no answer.

What a local model buys you

Against a mock, it buys realism in the parts of the exchange you did not write. A real server produces a real SSE stream with real chunk boundaries, which is where streaming clients break — a JSON object split across two chunks, a keepalive comment line, a final [DONE] sentinel. It produces a real tool call with arguments as a JSON string rather than an object, which is the detail that catches every first implementation. It applies a real stop sequence and reports a real finish_reason.

Against the provider, it buys three things that matter more than the money: it works offline, it does not rate limit you when a suite runs in parallel, and it does not change underneath you. That last one is the strongest argument. A test that talks to a hosted model is a test whose expected behaviour can change without any commit, and a suite with that property teaches people to ignore red.

Where it will lie to you

The differences are systematic rather than random, which makes them predictable and therefore avoidable if you know the list.

• Parameter coverage. A compatible endpoint is not a complete one. Ollama, for instance, documents that its OpenAI endpoint does not implement tool_choice, logit_bias, n or logprobs, so a test asserting your forced-tool path works will pass locally for the wrong reason and fail against a provider that honours it, or the reverse.
• Tokenisation. Different tokeniser, different counts. Anything asserting on token numbers, cost estimates or context-budget arithmetic gets a locally correct answer that is wrong for the model you ship.
• Error surface. Local servers rarely rate limit, never return the provider’s error taxonomy, and have different timeout behaviour. Your retry ladder cannot be validated here; it needs a stub that produces the exact statuses and headers.
• Instruction following. A small model ignores instructions a frontier model obeys. Any test whose assertion depends on the model complying will be flaky locally, and the temptation is to weaken the assertion until it passes — at which point it no longer tests anything.

The costs nobody budgets for

The weights are the obvious one: every developer machine and every CI runner needs them cached, and a cold cache is minutes. Less obvious is the machine variance. The same model on a GPU runner and on a CPU laptop will not produce identical output, because different kernels accumulate floating-point arithmetic in a different order, so any assertion on exact text is a test that passes on your machine and fails on somebody else’s. Assert on structure, not on strings — and if you find yourself wanting an exact-text assertion, that is the signal that this test wanted a stub all along.

There is also a maintenance cost that arrives later. A local model in the suite is a second inference stack to keep working: an image tag to bump, a runtime whose API surface changes, a model name that gets retired from the registry. It is worth paying where the tests are genuinely about wiring, and not worth paying to avoid writing a stub.

Choosing per test, not per suite

The decision is not “do we use a local model”. It is made test by test, and the rule is short: if the test's assertion mentions the content of the model's output, use a stub; if it mentions the transport, use the local model; if it mentions the provider, use the provider and accept that this test runs on a schedule rather than on a commit. Most suites end up roughly eighty per cent stub, fifteen per cent local, five per cent live, and the five per cent is the part that needs a budget and an owner.

Make the choice legible in the test file rather than in a wiki. A tag or a directory per tier — unit, wired, live — means a reader can tell at a glance what a failing test is claiming, and it gives you somewhere to hang the policy: unit tests run on every save, wired tests on every push, live tests nightly and before a release. The alternative, one suite where the backend is chosen by an environment variable, produces the worst outcome available: a test whose meaning depends on how it was invoked, and a green run that nobody can interpret.