Cost Regression: Catching a 10× Bill Before It Lands

Inference bills rarely drift. They step, on a deploy, and then sit at the new level until someone reconciles an invoice weeks later. The useful thing about a step change is that something caused it, and the cause was in a diff.

What a cost regression looks like

It is not an incident. Latency is normal, error rates are normal, the feature works, every dashboard is green. The only signal is the cost per request, and if that is not on a dashboard there is no signal at all until finance asks a question.

It is also usually not a bug in the ordinary sense. Someone added a paragraph to a system prompt, raised a retrieval parameter from 3 to 10, or turned on a model setting that made the answers better. Each of those is a correct, reviewed, intentional change whose cost consequence was simply not part of the review, because the diff doesn’t show it.

The change classes and their multipliers

Each of these is a one-line diff with a computable effect. Knowing the multiplier is what turns a code review comment from “hmm” into a number.

Note how many of these are multiplicative rather than additive. Two of them in the same release — say, a longer prompt and an enabled reasoning setting — compose, which is how a 3× and a 4× become the twelve-fold surprise that gets noticed.

A cost test that runs in CI

The good news is that cost is deterministic given a prompt and a token count, so most of this can be caught without calling a model. Build the prompt, count the tokens with the tokenizer, apply pinned prices, and compare against a committed baseline. No network, no flakiness, no spend — the check itself is free.

// tests/prompt-cost.test.ts
import { buildPrompt } from "../src/prompt";
import { countTokens } from "../src/tokenizer";
import fixtures from "./fixtures/requests.json";
import baseline from "./prompt-cost.baseline.json";

// Prices pinned in the repo, not fetched. This test measures
// OUR change, not the provider's. Bump them deliberately.
const P_IN = 1.00, P_OUT = 5.00;   // $ per million tokens
const TOLERANCE = 0.05;            // 5%

for (const fx of fixtures) {
  it(`input cost is stable for ${fx.name}`, () => {
    const prompt = buildPrompt(fx.input);
    const inTokens = countTokens(prompt);
    const cost = (inTokens * P_IN + fx.expectedOut * P_OUT) / 1e6;
    const was = baseline[fx.name];
    const delta = (cost - was) / was;

    expect(Math.abs(delta)).toBeLessThan(TOLERANCE);
  });
}

Four properties make this worth the hour it takes to write.

• It fails the pull request. The author sees “input cost for the support fixture rose 240%” while they still remember why, rather than an analyst seeing it six weeks later.
• The baseline is a committed file. Accepting an increase is an explicit diff with a reviewer on it. That is the entire mechanism: it does not prevent expensive changes, it prevents accidental ones.
• Prices are pinned deliberately. Fetching live prices would make the test measure the provider instead of your diff, and would make it fail on days you did nothing.
• It catches the cache break. Add a second assertion that the first N tokens of the built prompt are byte-identical to the baseline prefix, and the timestamp-at-the-top mistake becomes a test failure rather than a silently forfeited discount.

What CI cannot catch is output length, since that depends on the model. For that, keep a small live check — twenty fixtures, run nightly rather than per-commit, asserting that mean completion tokens per fixture is within tolerance of the baseline. Twenty calls a night is a rounding error against what it protects.

The runtime alert, and what to alert on

CI catches what you changed. Production catches what changed around you: a shift in the traffic mix, a customer who started uploading hundred-page documents, a silently updated model.

• Alert on cost per request, per route, not on total spend. Total spend rising with traffic is the product working. Cost per request rising is always worth a look.
• Compare against the same hour last week. Inference traffic has strong daily and weekly seasonality, so a fixed-threshold alert either pages on Monday mornings or misses everything. A ratio against the seasonal comparison point does neither.
• Alert on p95 as well as the median. A new failure mode where a small share of requests generate enormous answers moves p95 long before it moves the median, and it is exactly the shape a runaway loop produces.
• Track input and output tokens separately. They have different causes and different fixes, and a combined cost number hides which one moved.
• Watch the cached-token ratio. A sudden drop is the single highest-value cost alert available, because it is unambiguous — nothing legitimate makes it fall.

When it fires

The diagnosis is short because the causes are enumerable. In order:

• Did we deploy? Correlate the step with the deploy timeline first; this resolves most of them in a minute.
• Which token line moved — input, output, cached, reasoning? Each points at a different half of the list above.
• Is it all traffic or one route, one tenant, one key? A single-tenant spike is an abuse or integration question, not a code question, and it is what per-key caps exist for.
• Did request count move too? If cost per request is flat and volume tripled, nothing is wrong with the code and the conversation is a capacity one.
• If none of the above: check whether the model changed underneath you. An alias repointed at a newer version can change both output length and price without any diff on your side.