Testing That a Tool-Calling Loop Stops Before It Exceeds a Cost Budget

An agent that can call a tool can call it again. A search that returns results that prompt another search is an ordinary loop with an extraordinary bill, and the ceiling that stops it is worth more than the agent.

Check before the call, not after

Most budget code is written as a check at the top of the loop against spend so far. That is one call too late: it stops the loop after the expensive call, having already paid for it. If a single turn can carry a hundred thousand tokens of accumulated context, the difference between checking before and after is the most expensive request of the run.

The correct check is against a projection. Before dispatching turn n, you know the input token count exactly — it is the context you are about to send — and you know the maximum output, because you set it. So the worst-case cost of the next turn is computable, and the guard compares spent-plus-projected against the ceiling.

// budget.ts
export function wouldExceed(budget, inTokens, maxOut, price) {
  const projected =
    (inTokens / 1e6) * price.inputPerMTok + (maxOut / 1e6) * price.outputPerMTok;
  return budget.spentUsd + projected > budget.limitUsd;
}

// in the loop, before dispatch:
if (wouldExceed(budget, countTokens(messages), MAX_OUT, price)) {
  return { status: "budget_exhausted", turns, partial: lastGoodResult, budget };
}

Two properties of that code are what the tests below assert. The guard runs before the request, and the loop returns a typed result rather than throwing — because the partial work is usually worth something, and an exception discards it.

A model that never stops asking

The fixture is a fake completion function that always returns a tool call, with a usage block you control. It is four lines and it is a perfect runaway: without a ceiling, the loop is infinite.

const alwaysCallsTool = vi.fn(async () => ({
  message: {
    content: null,
    tool_calls: [{ "id": "c1", "function": { "name": "search", "arguments": "{}" } }],
  },
  finish_reason: "tool_calls",
  usage: { prompt_tokens: 12000, completion_tokens: 400 },
}));

// Assumed rates for the test, not quoted prices: read yours from config.
const price = { inputPerMTok: 3, outputPerMTok: 15 };

Grow the prompt tokens per turn if you want the test to reflect reality, because an agent loop’s context accumulates: each turn appends the tool result, so turn ten is much more expensive than turn one. A fake that returns a constant usage understates the growth and will make a linear-looking budget test pass on code that is quadratic in practice.

What to assert

• The loop terminates. With a timeout on the test itself, so a broken ceiling fails in seconds rather than hanging the suite. This is the assertion that would have caught the incident.
• It stopped before breaching, not after. Assert final spend is at or below the limit, and separately that the call count is the largest number of turns that fits — not one more. Only the second catches an off-by-one that costs a full turn every time.
• The result is typed and carries the reason. The status names the budget, the turn count is reported, and the partial result is present. A caller that cannot distinguish “finished” from “ran out of money” will present a half-done answer as a complete one.
• Output tokens are counted. Feed a fixture whose completion tokens dominate and assert the ceiling still holds. A budget that counts only input tokens is the most common wrong implementation, and it is wrong in the direction of the more expensive number.
• Nothing is dispatched after the stop. Assert the call count exactly, and that no tool executor ran after the final model call. A loop that stops calling the model but still executes a queued tool has side effects after its own halt.

it("stops before the turn that would breach the budget", async () => {
  const res = await runAgent({
    budget: { limitUsd: 1.0, spentUsd: 0 }, price, complete: alwaysCallsTool,
  });
  expect(res.status).toBe("budget_exhausted");
  expect(res.budget.spentUsd).toBeLessThanOrEqual(1.0);
  expect(alwaysCallsTool).toHaveBeenCalledTimes(res.turns);
  expect(res.partial).toBeDefined();
}, 5000);

Two ceilings, not one

A cost ceiling alone is insufficient, because the failure modes it misses are the cheap ones. A loop calling a free local tool in a tight cycle spends almost nothing and never terminates; a loop whose model call fails and is retried burns wall time without accruing token cost. So a second, independent ceiling on step count belongs alongside it, and each needs its own test with the other set high enough not to interfere.

A third guard is worth the same treatment: a repetition detector. If the model requests the identical tool with identical arguments three turns running, it is stuck, and continuing to the budget ceiling is money spent proving it. Assert that the loop halts with a distinct status for that case, because the operational response is different — a budget stop is a capacity question, a repetition stop is a prompt or tool-description question, and the general treatment of that is tool description design. Wall-clock is the fourth guard, and with fake timers it is as testable as the rest.

Choosing the number

The ceiling should be derived rather than picked, and the derivation belongs in a comment next to the constant. Work it from the shape of your own loop and your own price sheet — the figures below are assumptions for a worked example, not quoted prices.

Assume, for one agent run:
  starting context            8,000 tokens
  tool result per turn        2,000 tokens
  max output per turn           500 tokens
  assumed input price         $3.00 per million tokens
  assumed output price       $15.00 per million tokens

Input tokens on turn n = 8,000 + 2,000 x (n - 1), so over N turns:
  total input  = N x 8,000 + 2,000 x N(N - 1)/2
  total output = N x 500

N = 10:  input  = 80,000 + 90,000 = 170,000 tokens
         output = 5,000 tokens
         cost   = 0.170 x $3.00 + 0.005 x $15.00 = $0.51 + $0.075 = $0.59

N = 20:  input  = 160,000 + 380,000 = 540,000 tokens
         output = 10,000 tokens
         cost   = 0.540 x $3.00 + 0.010 x $15.00 = $1.62 + $0.15 = $1.77

Doubling the step limit triples the cost, because the context grows with the loop. That quadratic is the reason a step ceiling of fifty “just to be safe” is not safe, and it is the number to put in front of whoever sets the limit. Multiply the per-run figure by your expected runs per day for the exposure; the same population arithmetic is worked in the rollout reach calculation.