Output Length Control: The Most Ignored Cost Lever

Output tokens cost several times what input tokens cost and are generated one at a time, so they dominate both the bill and the latency of a chatty endpoint. They are also the only part of the request you did not write, which is why nobody audits them.

How much of your bill is the answer

Before optimising anything, compute the share. With k = P_out / P_in as the price ratio:

output_share = (k * T_out) / (T_in + k * T_out)

Three worked cases, with k = 5 throughout as an assumption:

RAG answer      T_in 8,000  T_out   300   share = 1500/9500  = 16%
chat turn       T_in 2,400  T_out   300   share = 1500/3900  = 38%
drafting tool   T_in   600  T_out 1,200   share = 6000/6600  = 91%
classification  T_in 1,200  T_out     5   share =   25/1225  =  2%

The instruction is direct. On the drafting tool, halving the output cuts 45% off the bill and nothing else you do will come close. On the classifier, output is a rounding error and any effort spent here is wasted — go and look at the input instead. Most engineers guess wrong about which of these describes their own traffic, and it is one division to find out.

Note also what the formula does when k rises. A model with a higher output premium shifts every one of those shares upward, so switching to a model with a different price shape can make a length problem you did not previously have.

max_tokens is a fuse, not a control

Setting max_tokens lower does not make the model write more concisely. It makes the model write exactly what it was going to write and then stop mid-sentence when the budget runs out. You are billed for every token it produced up to the cut, you have an unusable answer, and the usual next step is a retry at full price.

effective_cost = C * (1 + q)

  q  the truncation rate, i.e. share of responses with
     finish_reason = "length" that you retry

At q = 0.08 a too-tight cap has added 8% to the bill while appearing to reduce it. Treat max_tokens as a safety fuse set well above the length you expect — its job is to bound the damage from a repetition loop or a runaway generation, not to shape normal output — and get the actual shortening from the prompt.

The fuse is still worth setting, because it is the only hard bound on a single request’s cost. The computable ceiling for one call is (T_in × P_in + max_tokens × P_out) / 1e6, which is the number a pre-request spend check has to work with.

Five things that actually shorten output

• 1. Ask for a shape, not a length. “Be concise” is weak; models comply with it inconsistently and it gives them nothing to aim at. “Answer in at most three bullets, no more than fifteen words each” is a structure, and structures are followed far more reliably than adjectives. Word and token counts in the instruction are approximate at best — models do not count well — but a structural constraint bounds the length anyway.
• 2. Use structured output. A JSON schema with three string fields cannot produce a five-paragraph essay. This is the strongest available control because it is enforced by constrained decoding rather than by persuasion, and it removes the prose scaffolding around the answer entirely. Where the consumer of the answer is code, there was never a reason for the prose.
• 3. Ban the ritual. A large share of a typical answer is preamble and postamble: restating the question, “Certainly! Here is...”, a summary of what was just said, an offer of further help. An explicit instruction to start with the answer and stop after it removes tokens that carried no information. This is the highest-value single sentence you can add to a system prompt, and it is free.
• 4. Stop sequences. If the output has a natural terminator — a closing fence, a delimiter, the start of a section you do not want — a stop sequence halts generation there and you are not billed for what follows. Particularly effective against models that helpfully continue with a second example.
• 5. Split the call. When the endpoint needs a short answer and a long explanation, generating both every time bills the explanation to every request. Return the short answer, and generate the explanation on demand for the small fraction of users who ask. If 10% expand, you have removed 90% of the explanation tokens.

The reasoning multiplier

On models that think before answering, thinking tokens are billed at the output rate and are typically several times the length of the visible answer. That makes the reasoning setting the single largest output-length control available, and it is usually a configuration value rather than a prompt change.

billed_output = T_visible + T_reasoning

If T_reasoning is m times T_visible, then output cost is
(1 + m) times what the visible answer suggests.

At m = 4, an answer that looks like 200 tokens bills as 1,000.

The practical rule: reasoning effort should be a per-route setting, not a global default. Routes that classify, extract, format or route rarely benefit from it; routes that do multi-step analysis often do. Leaving a high effort level on across every call is the most expensive default available in current inference, and it is invisible unless you are reading the reasoning token count in the usage object.

Verifying it worked

Output length is the easiest cost metric to regression-test because it is a single integer in every response. Log completion_tokens per route, alert on the p95 rather than the mean — a mean hides a tail of enormous answers — and treat a step change in it as a deploy artefact to investigate. A prompt edit that adds a paragraph of instructions can double the answers it produces, which is one of the classic cost regressions and is trivially catchable if the number is on a dashboard.

One caution: shorter is not automatically better. If the shortening removes information the user needed, the follow-up question costs a whole additional request, input included, and you have made the bill larger while making the metric smaller. Check the follow-up rate alongside the token count.