Testing That a Cost Alert Actually Fires at the Threshold You Set

A cost alert is discovered to be broken at the same moment it was supposed to be useful. It is also ordinary code with an ordinary input, so there is no reason to find out that way.

The alert is code, so test it like code

Separate the three parts, because only the middle one is interesting. Collection turns requests into usage records. Evaluation turns a sequence of usage records and a threshold into a decision. Delivery turns a decision into a page or a message. Evaluation is a pure function of a list and a number, and if it is not — if it queries a metrics backend inside the comparison — the first job is to make it one, because otherwise nothing below is possible without standing up a monitoring stack in CI.

Once evaluation is pure, the test is a synthetic stream of usage records with known costs, fed in one at a time, asserting on when the decision flips. No model is called, nothing is billed, and the run takes milliseconds.

Deriving the expected crossing point

The expected event number should be arithmetic, not a number somebody tuned until the test went green. Every input below is a stated assumption you substitute with your own; none of them is an observed figure from any particular deployment.

• Assumed input price: $3.00 per million input tokens.
• Assumed output price: $15.00 per million output tokens.
• Assumed request shape: 12,000 input tokens and 800 output tokens per request — a retrieval-augmented call with several chunks and a short structured answer.
• Assumed threshold: $500 in a rolling day.

input  cost = 12,000 / 1,000,000 x $3.00  = $0.036 per request
output cost =    800 / 1,000,000 x $15.00 = $0.012 per request
                                            ---------------------
total                                       $0.048 per request

events to cross $500 = ceil(500 / 0.048) = ceil(10,416.67) = 10,417

cost after 10,416 events = 10,416 x 0.048 = $499.968   -> must NOT fire
cost after 10,417 events = 10,417 x 0.048 = $500.016   -> must fire, once

Those two rows are the test. Note that the boundary is not round: the per-request cost does not divide the threshold evenly, which is exactly the situation where an implementation using a strict rather than non-strict comparison, or accumulating in floating point, gets the crossing wrong by one event. A test written against a threshold of $500 and a per-request cost of $0.05 would have divided evenly and would have caught neither.

import { describe, expect, it, vi } from "vitest";
import { makeCostAlert } from "../src/alerts";

const PRICE_IN_PER_M = 3.0;    // assumption, substitute your own
const PRICE_OUT_PER_M = 15.0;  // assumption, substitute your own

const usage = () => ({ inputTokens: 12_000, outputTokens: 800 });
const costOf = (u: ReturnType<typeof usage>) =>
  (u.inputTokens / 1e6) * PRICE_IN_PER_M + (u.outputTokens / 1e6) * PRICE_OUT_PER_M;

describe("daily spend alert at $500", () => {
  it("fires exactly once, on event 10417", () => {
    const notify = vi.fn();
    const alert = makeCostAlert({ thresholdUsd: 500, window: "1d", notify });

    for (let i = 1; i <= 10_500; i += 1) {
      alert.record({ at: new Date("2026-08-11T00:00:00Z"), cost: costOf(usage()) });
      if (i === 10_416) expect(notify).not.toHaveBeenCalled();
      if (i === 10_417) expect(notify).toHaveBeenCalledTimes(1);
    }

    expect(notify).toHaveBeenCalledTimes(1);
  });
});

Five assertions

• It does not fire below the threshold. The negative case first. An alert that fires on every evaluation once any spend exists is a common bug and is indistinguishable from a working alert on the day of a real spike.
• It fires on the crossing, exactly once. The remaining 83 events in the loop above exist only to assert that the count stays at one. An alert that re-fires per event pages somebody ten thousand times and is muted within the hour, which is worse than no alert at all.
• It re-arms after the window resets. The complement of the previous assertion: an alert that fires once and then never again is equally broken. Advance to the next window with spend still above the threshold and assert a second notification.
• A late or out-of-order record does not un-fire it. Usage records arrive with lag. A record timestamped inside the window but received after the alert fired must not cause a second alert, and a correction that reduces the total must not cause the alert to resolve and immediately re-fire.
• Projection alerts have their own case. If you also alert on a projected end-of-period total rather than a realised one, the projection is a second piece of arithmetic and needs its own derivation: at the same $0.048 per request and 5,000 requests a day, the projected daily total is $240, and an alert set at $500 on projection must not fire — whereas at 12,000 requests a day the projection is $576 and it must. Assert both.

Units are where this goes wrong

The single most common defect in this code is a factor of a hundred or a million. Prices are quoted per million tokens, money is often stored in minor units, and somewhere a value crosses a boundary as a float in one convention and is read in another. The symptom is an alert that either never fires or fires immediately, and both are usually attributed to the threshold being wrong.

Two defences. Carry money as an integer count of minor units through the whole pipeline and convert only for display, which removes the floating-point accumulation problem as well — ten thousand additions of 0.048 do not sum to exactly 500.016 in binary floating point, and a test asserting an exact total will fail for that reason alone. And write one test per boundary that asserts the unit explicitly: a usage record priced at a known rate produces a known integer, in cents, with the assertion naming the unit in its message.

The clock

Windows, resets and rate limits are all clock-dependent, and a test that reads the real clock is a test that behaves differently at 23:59 on the last day of a month. Use the runner’s fake timers — in Vitest, install them and advance time explicitly — or better, inject the clock as a function so the alerting code never reads the system time at all. Then write the awkward cases deliberately: the window boundary, a daylight-saving transition if your window is a calendar day in a local timezone, and the month rollover. The month-boundary case matters most because a monthly budget alert that resets on the wrong day is silently wrong for a month at a time.

Two things this test does not tell you. It does not tell you that delivery works — that the notification actually reaches a human — which needs its own periodic synthetic notification. And it does not tell you that the cost numbers are right in the first place; if your estimate diverges from the invoice, a perfectly tested alert fires at the wrong real spend. That is a separate reconciliation problem, and it is worth doing before trusting any of this. See setting service level objectives for AI systems for how a cost threshold fits with the rest of what you alert on.