Local Development Against Expensive APIs

Every team has the story: a retry loop with no ceiling, left running over a weekend, against the expensive model, on the shared key. It is not a discipline problem. It is a missing mechanism, and the mechanism takes an afternoon.

Three modes, one switch

Local development against a metered, slow, non-deterministic API wants three distinct modes, selected by one environment variable so that switching is trivial and the current mode is obvious.

Make fixture mode the default in every environment that is not production, and make the mode visible: print it at startup, put it in the log line of every call, and show it in the developer UI if there is one. The expensive mistakes happen when someone believes they are in one mode and are in another.

The switch belongs behind the same interface everything else uses. If the mode is chosen inside the model client, no call site knows or cares, and there is no path by which one forgotten code path escapes to the network.

Seeding fixtures from one real run

Hand-written fixtures are wrong in ways that matter — they are too clean, too short, correctly formatted, and they never contain the preamble the model actually emits. Generate them instead, once, from a real run.

The workflow: run the feature once in live mode with recording on, against a small corpus of inputs chosen to cover the shapes you care about. Commit the recordings. Everyone else develops against them indefinitely. When the prompt changes materially, re-run the seeding job and review the diff, exactly as with test cassettes — the two are the same mechanism used at different times, and there is no reason to build both.

For inputs with no recording, a fixture mode needs a fallback generator rather than an error, or every new test input becomes a blocked developer. A deterministic generator that produces schema-valid nonsense derived from a hash of the request is enough: it never matches reality, but it keeps the application running, and the obviously fake output is a feature because nobody mistakes it for a real answer.

The cheap-model mode and what it hides

Routing development traffic to a small model cuts the cost of an afternoon’s work substantially and keeps the whole transport path real: authentication, streaming, error shapes, usage accounting, cancellation. For debugging plumbing it is the right mode.

Be clear about what it hides, though, or it will mislead you. A small model is faster, so every timing-related bug is masked — your streaming timeout, your deadline arithmetic and your idle detector are all being exercised against a duration you will not see in production. It follows instructions differently, so a prompt that works in development may have been tuned against the wrong target. And its failure modes differ: it truncates more often, it produces malformed structured output more often, and it may not support tools at all.

The practical rule: use cheap mode for anything about the mechanics, and never for a judgement about whether a prompt is good. Those judgements need the model you will ship, on the inputs you will see, in the evaluation suite rather than in a terminal.

Guardrails that are mechanisms

“Be careful with the API key” is not a control. These are:

• A per-process spend ceiling. Accumulate estimated cost in the client; when the total for this process exceeds a small ceiling, throw rather than call. A developer hitting it sees a clear error and raises the ceiling deliberately; a runaway loop hits it in seconds. This single mechanism prevents the story in the lede.
• A call-count ceiling per process, too. Cheap calls do not trip a spend limit but a million of them will trip a rate limit and annoy everyone sharing the account.
• Loop detection. If the identical request key is issued more than a handful of times in a minute, something is looping. Fail loudly with the request in the message rather than continuing politely.
• A hard cap on the key itself. Client-side limits protect against your own bugs; a limit enforced where the money is protects against everything, including the script somebody ran outside your application entirely.
• An idle shutdown for long-running dev processes. A worker started on Friday should not still be polling on Monday.

Keys and environments

One key per environment, minimum, and never the production key on a laptop. The reasons compound: a leaked development key can be revoked without an outage, per-environment spend is separable so you can see what development actually costs, and rate limits do not interfere between a load test and real traffic.

Per-developer keys are better still where the provider makes them cheap to issue, because attribution becomes automatic — the runaway loop has a name on it, and the person who owns it finds out first rather than last. The cost is key management, which is a real cost and the subject of its own page.