Setting a Per-Request Cost Ceiling Before Calling a Model API

Every budget control that runs after the fact shares one weakness: it finds out about the expensive request once you have already paid for it. A per-request ceiling is the only control that can refuse, and it works because the worst-case cost of a request is knowable before you send it.

Counting the input without guessing

The input side is exactly determined: it is a token count against a specific model’s tokenizer, multiplied by that model’s input rate. The trap is the word “specific”. Tokenizers differ between vendors and between model generations within a vendor, so a count produced by one tokenizer is not a count for another model.

In particular, reaching for tiktoken because it is the library you already have is a mistake for any non-OpenAI model. It is OpenAI’s tokenizer; against Claude it undercounts typical prose by roughly 15–20%, and by considerably more on code or non-English text. A ceiling built on an undercount lets through exactly the requests it exists to stop.

Anthropic exposes a counting endpoint that takes the same request shape as the inference call, so the count is against the real tokenizer, the real system prompt, and the real tool definitions:

from anthropic import Anthropic

client = Anthropic()

def count_input_tokens(model: str, system: str, messages: list, tools=None) -> int:
    resp = client.messages.count_tokens(
        model=model,
        system=system,
        messages=messages,
        tools=tools or [],
    )
    return resp.input_tokens

Passing tools matters more than it looks. Tool definitions are serialised into the prompt, so a request with a dozen tools carries thousands of tokens that a count over messages alone never sees — and it is a fixed overhead on every request, which makes it the most repeatedly-underestimated part of the bill.

The count is a network round trip. On a hot path where the prefix is stable, count the stable part once, cache it against a hash of that prefix, and count only the variable tail per request.

The output side is the unbounded half

Input is measurable; output is not, because it has not been generated yet. This is where naive estimators give up and guess an average, and an average is precisely the wrong statistic — the requests you are trying to catch are the ones in the tail.

You do not have to guess, because you already control the bound. max_tokens is a hard cap the API enforces: generation stops there. So the worst case is not unknown, it is arithmetic.

worst_case_cost = (input_tokens  / 1_000_000) * input_rate
                + (max_tokens    / 1_000_000) * output_rate

That reframes max_tokens from a truncation setting into the cost-control parameter it actually is. A request whose worst case is over the ceiling has three ways to come under it, and lowering max_tokens is one of them.

Two adjustments make the worst case more honest. Output rates are typically several times input rates, so the output term dominates for anything with a generous max_tokens and a short prompt. And on a reasoning model, thinking tokens are billed as output and count against max_tokens, so a request configured for extended reasoning can spend most of its budget before producing a visible word. Both push in the same direction: the output term is the one to scrutinise.

The ceiling function

1. Put the per-model rates in configuration, not in code. They change, they differ by provider, and a hard-coded rate is a silently wrong ceiling the day after a price change.
2. Count the input tokens against the model you are about to call.
3. Compute the worst case with max_tokens as the output term.
4. Compare to the ceiling for this caller, this tenant, or this route — a single global ceiling is rarely the right shape.
5. Emit the estimate on the outcome either way. A ceiling you cannot observe is a ceiling you cannot tune.

RATES = {  # USD per million tokens; from config, checked against the
           # vendor pricing page — see the note below
    "claude-opus-5":  {"in": 5.00, "out": 25.00},
    "claude-sonnet-5": {"in": 3.00, "out": 15.00},
    "claude-haiku-4-5": {"in": 1.00, "out": 5.00},
}

class CostCeilingExceeded(Exception):
    def __init__(self, estimate, ceiling):
        self.estimate, self.ceiling = estimate, ceiling
        super().__init__(f"estimated ${estimate:.4f} exceeds ceiling ${ceiling:.4f}")

def check_ceiling(model, system, messages, max_tokens, ceiling_usd, tools=None):
    rate = RATES[model]
    n_in = count_input_tokens(model, system, messages, tools)
    estimate = (n_in / 1e6) * rate["in"] + (max_tokens / 1e6) * rate["out"]
    if estimate > ceiling_usd:
        raise CostCeilingExceeded(estimate, ceiling_usd)
    return estimate

One deliberate conservatism: this ignores prompt caching. A cached prefix is read at a fraction of the input rate, so a request against a warm cache costs materially less than the estimate says. That is the right direction for a safety control to be wrong in — but it does mean a ceiling set tightly against real observed spend will reject requests that would in practice have been cheap, so tune against the estimate rather than against the invoice.

Where to enforce it

Put the check in one place that every model call passes through. A ceiling implemented at each call site is a ceiling that the next call site forgets, and the request that blows the budget is always the one added last.

In practice that means a client wrapper, a middleware, or a proxy. The wrapper is easiest and the weakest — nothing stops someone importing the vendor SDK directly. A proxy is strongest, because the network path is the enforcement point and bypassing it requires intent.

What to do when a request is over

Rejecting outright is the simplest behaviour and rarely the best one. Three alternatives are usually available before you reach a hard refusal.

• Lower max_tokens. If the output term is what breached the ceiling, capping the answer shorter brings the request under it and produces a shorter answer rather than no answer. Handle stop_reason: max_tokens in the response so a truncated answer is labelled as truncated rather than presented as complete.
• Trim the context. If the input term is what breached it, the usual culprit is a retrieval step returning more passages than the task needs. Dropping the lowest-scoring passages is both cheaper and, often, more accurate.
• Downgrade the model. The same request against a smaller model can be several times cheaper. This is a policy decision, not a technical one, and it belongs in configuration next to the ceiling.
• Then refuse, loudly. When none of those brings it under, raise a typed error the caller can catch, and record the estimate, the model, the ceiling and the caller.

That last record is what turns the ceiling into something maintainable. A ceiling that fires constantly is set too low, and a ceiling that has never fired is not protecting anything. Both are only visible if the rejections are counted.