A Test That Fails When Estimated Cost Diverges From Actual Cost by More Than 5%

Cost estimation drifts for two independent reasons: the token counts are wrong, or the prices are wrong. A single divergence assertion catches both, but only if it is built on the right aggregate — a per-request percentage check will fire constantly on small requests and tell you nothing.

Assert on the ratio of sums

The naive form is to compute a percentage error per request and assert each is under 5%. It fails immediately in practice, because a request with 40 actual input tokens and a four-token serialisation overhead your estimator missed is 10% off, while contributing a rounding error to your bill. You end up either raising the threshold until it catches nothing or excluding small requests by an arbitrary cutoff.

Sum first, divide second. Over a window of requests, compute total estimated cost and total actual cost, and assert the ratio is within tolerance of one. This weights each request by the money it represents, which is exactly the weighting you want, since a 10% error on a request costing a hundredth of a cent is not a thing worth failing a build over.

divergence = (sum(estimated) - sum(actual)) / sum(actual)

Keep the sign. A signed divergence tells you the direction, and the direction usually names the cause: consistently under means a term is missing from the estimate — tool schemas, cached tokens, reasoning tokens — while consistently over usually means a stale price that has since come down. An absolute value throws away the most useful bit of the signal.

A worked divergence

Every figure below is a placeholder, chosen to make the arithmetic legible. Substitute your provider’s current published rates — the point is the shape of the calculation, not these numbers.

Assume a window of 10,000 requests. Assume a labelled placeholder price of $3.00 per million input tokens and $15.00 per million output tokens. Your estimator records an average of 1,800 input and 400 output tokens per request; the logged usage records 2,050 input and 400 output.

estimated input   10,000 × 1,800 = 18,000,000 tok × $3.00/1M  = $54.00
estimated output  10,000 ×   400 =  4,000,000 tok × $15.00/1M = $60.00
estimated total                                                 $114.00

actual input      10,000 × 2,050 = 20,500,000 tok × $3.00/1M  = $61.50
actual output     10,000 ×   400 =  4,000,000 tok × $15.00/1M = $60.00
actual total                                                    $121.50

divergence = (114.00 - 121.50) / 121.50 = -6.2%   → fails a 5% gate

The 250-token per-request input gap is 14% of input tokens but only 6.2% of cost, because output is priced higher here and output was estimated correctly. That relationship is worth internalising: a divergence gate on cost is less sensitive to input-token errors than a gate on token counts, which is an argument for having both. The token-level check lives in testing that your token counting matches the bill.

The price table is the other half

Half of all cost-estimation drift is a price constant that no longer matches the provider’s published rate. Nothing about your code changes when a provider adjusts pricing or you switch to a model tier you did not add to the table, and a missing entry usually falls back to a default that is quietly wrong.

Two structural fixes, both cheap. Put prices in a data file with an explicit updated date and a source URL per entry, never inline in code. Then write a second test, unrelated to divergence, that fails when the file is older than your review period — ninety days is a reasonable default — and that fails when any model id appearing in production traffic has no entry. The second half of that matters more than the first: a model with no price entry contributes zero to your estimate and its full cost to your invoice, which produces a divergence that no amount of tokenizer work will explain.

it("prices every model that appeared in traffic this week", async () => {
  const seen = await distinctModelIds({ since: "7d" });
  const priced = new Set(Object.keys(prices.models));
  expect([...seen].filter((m) => !priced.has(m))).toEqual([]);
});

it("has a price table reviewed in the last 90 days", () => {
  const ageDays = (Date.now() - Date.parse(prices.updated)) / 86_400_000;
  expect(ageDays).toBeLessThan(90);
});

The assertion

The divergence test itself reads production data rather than fixtures, because it is checking a property of your live estimate against your live bill. Run it on a schedule, not on commit.

import { expect, it } from "vitest";
import { costWindow } from "../src/analytics";

it("estimated cost tracks actual cost within 5% over the last day", async () => {
  const { estimated, actual, requests } = await costWindow({ since: "24h" });

  // Below a few hundred requests the ratio is noise; skip rather than assert.
  expect(requests).toBeGreaterThan(500);

  const divergence = (estimated - actual) / actual;
  expect(divergence, `estimated $${estimated.toFixed(2)} vs actual $${actual.toFixed(2)}`)
    .toBeGreaterThan(-0.05);
  expect(divergence).toBeLessThan(0.05);
});

The minimum-request guard is not optional. A window with eleven requests can diverge by 40% for reasons that mean nothing, and a scheduled test that fails on quiet Sundays gets muted within a month. Make the guard explicit so that its purpose survives the next person reading the file.

Where 5% comes from, and when it is wrong

5% is not a law. Derive it from the decision the estimate feeds. If the estimate drives a customer-facing usage meter or a prepaid balance, the tolerance is whatever gap you are willing to eat or to over-bill by, and it will be tighter than 5% — over-billing on a wrong estimate is a refund and a support conversation. If it drives internal capacity planning, 10% is fine and a tighter gate is just noise.

If it drives a hard spend cap, the asymmetry matters more than the magnitude: underestimating means you blow through the cap, while overestimating means you stop early. Set an asymmetric pair of bounds in that case rather than a symmetric window, and say in the test why.

One thing the gate cannot do is tell you which of the two causes fired. Print both components when it fails — the aggregate token divergence and the price-table age — and the person paged at least starts in the right half of the problem.

There is a third cause the gate will surface and that neither component explains, so it is worth naming here: a change in traffic mix. If a new feature sends long documents through a model that was previously only answering short questions, both your token counting and your prices can be individually correct while the aggregate divergence moves, because the model with the estimation error is now a larger share of spend. The diagnostic is to compute the divergence per model rather than in total. A single model responsible for nearly all of the gap points at the price table or that model’s serialisation; a divergence spread evenly across every model points at something systemic in the estimator, such as a missing per-message overhead.

Finally, decide deliberately what happens when the gate fails, because a scheduled test with no consequence is a scheduled test that gets muted. This one should not block a deploy — it is measuring a property of yesterday’s traffic and nothing in the pending change caused it. It should open a ticket with the two components attached and a link to the window it measured. Blocking deploys on a signal nobody can fix in the next ten minutes is how a good check earns a permanent exemption.