Token Budgets: Designing a Prompt Around a Cost Ceiling

Most teams design the prompt, ship it, and then discover the unit economics. Reversing that order takes twenty minutes and changes what the prompt looks like, because a token budget is a design constraint in exactly the way a latency budget is.

Working backwards

Start from the business constraint and derive the tokens. A worked example, with every assumption stated so you can swap yours in: a product charging $0.05 per assisted action, wanting inference to stay under 20% of that, at illustrative rates of $3.00 per million input and $15.00 per million output tokens.

ceiling:            $0.05 x 20%                    = $0.0100 per request

output first (it is the expensive side):
  target answer     300 tokens x $15.00/M          = $0.0045

remaining for input: $0.0100 - $0.0045             = $0.0055
input budget:        $0.0055 / $3.00 per M         = 1,833 tokens

Eighteen hundred tokens is the entire prompt: system, tools, retrieved context, history and the question. Most teams discover at this point that their system prompt alone is over budget, which is the useful shock. The number also makes the output side legible — those 300 tokens of answer consume 45% of the budget, so “answer in three sentences” is not a style preference but a line item.

The allocation table

Split the input budget explicitly, and treat each line as a cap that something enforces rather than a hope:

Budget against the p95, not the mean, on the two lines that vary. Retrieved context and output length both have long tails, and a feature that averages inside its ceiling while a fifth of requests double it is a feature with a cost profile nobody planned. The mean is what you report; the p95 is what you design the caps around, and the gap between them is a decent proxy for how much variance your prompt is allowing.

Once written down, the trade-offs become negotiable in a way they never are in prose. Retrieval wants more room; the honest question is what it takes it from. Adding a sixth tool costs about 60 tokens, which is a quarter of the history allowance.

When it does not fit

It will not fit the first time. The levers, in the order that usually gives the most relief per unit of effort:

• Cache the fixed prefix. With a read multiplier around 0.1×, the 650 tokens of system prompt and schemas cost about a tenth as much, which returns roughly 585 tokens of budget for nothing. This is almost always the first move — see cached tokens.
• Shorten the output. Worth 5× input per token. Structured output instead of prose typically halves it, and a schema enforces the brevity that an instruction only requests.
• Split the model. Classification and extraction on a small cheap model, generation on the expensive one. The budget is per model, so two calls can be cheaper than one.
• Retrieve harder, not more. A better reranker returning three chunks beats a weaker one returning eight, and costs fewer tokens for a better answer.
• Revisit the ceiling. Sometimes the honest conclusion is that the feature is worth more than $0.05 and the price is wrong, not the prompt.

Budget the session, not the request

The per-request budget is the wrong unit for anything agentic. An agent that makes eight tool-calling round trips pays the system prompt and the tool schemas eight times, and carries a history that grows on every one of them. A budget of $0.01 per request is $0.08 per task, and if the history grows the later turns cost more than the earlier ones.

per-task ceiling            $0.02
expected turns              8
naive per-turn budget       $0.0025

but history grows, so the real profile is roughly:
  turn 1   700 in                      turn 8  ~4,000 in
  cost is back-loaded; a cap that only checks per-turn
  will pass every turn and blow the task budget.

So the ceiling has to be a running total with a hard stop, and the stop needs a defined behaviour: return the best answer so far, escalate to a human, or fail loudly. Choosing that behaviour in advance is the difference between a cost incident and a degraded response.

Making it structural

A budget in a document is a wish. A budget in the call path is a constraint:

class Budget:
    def __init__(self, usd, p_in, p_out, p_cached=None):
        self.left = usd
        self.p_in, self.p_out = p_in / 1e6, p_out / 1e6
        self.p_cached = (p_cached or p_in * 0.1) / 1e6

    def affordable_output(self, prompt_tokens, cached_tokens=0):
        fresh = prompt_tokens - cached_tokens
        spent = fresh * self.p_in + cached_tokens * self.p_cached
        if spent >= self.left:
            raise OverBudget(spent, self.left)
        return int((self.left - spent) / self.p_out)

    def record(self, usage):
        fresh = usage.input_tokens - usage.cached_tokens
        self.left -= (fresh * self.p_in
                      + usage.cached_tokens * self.p_cached
                      + usage.output_tokens * self.p_out)

Three things this buys you. max_tokens becomes a computed consequence of the budget rather than a constant somebody guessed. Running out of budget is an exception with a stack trace rather than an invoice. And because record takes the returned usage rather than your estimate, the budget tracks reality — including the reasoning tokens and cache writes that an estimate would have missed.

Two extensions are worth adding once the basic object is in place. Attribute spend to a feature and a tenant, not just to a model: the question you will actually be asked is which feature became expensive this month, and a total per model cannot answer it. And log the budget remaining at the end of every task, not only when it runs out — the distribution of that number is a better early warning than any alert threshold, because it degrades gradually while an alert fires only after the failure.

The last thing to say is about what the exercise is really for. Very few teams end up with the budget they first wrote down; what they get instead is an explicit account of what the prompt spends money on, and that account survives every subsequent change of model and price. The arithmetic gets rerun in ten minutes. The allocation table is the durable artefact.