Token Accounting Across a Long Session

A fifty-turn conversation contains maybe 20,000 tokens of text. It will bill several hundred thousand input tokens. The gap is not overhead or a rounding error — it is the direct consequence of one rule, and the growth it produces is quadratic.

The rule that causes everything

Models are stateless. Nothing is retained between calls, so every turn re-sends the entire conversation as input. Turn 30 does not send the user’s new message; it sends the system prompt, the tool schemas, and all twenty-nine previous exchanges, plus the new message.

Which means a message written once at turn 3 is paid for at turns 3, 4, 5 … 50. Forty-eight times. That is the whole mechanism, and every surprising number below falls out of it.

The derivation

Let B be the fixed prefix in tokens — system prompt plus tool schemas, present on every request — and g the average tokens added per turn, counting both the user message and the assistant reply. Input billed on turn n:

in(n) = B + g*(n-1)

Total input across N turns:

  TOTAL_in = sum over n=1..N of [ B + g*(n-1) ]
           = B*N + g * N*(N-1)/2

Output is linear (each turn generates once):

  TOTAL_out = a*N            where a = average answer length

Cost = P_in * TOTAL_in + P_out * TOTAL_out
     = P_in * ( B*N + g*N*(N-1)/2 ) + P_out * a * N

The term that matters is g·N(N−1)/2. It is quadratic in N, so doubling the length of a conversation roughly quadruples the history component of its cost. Everything else in the expression is linear and therefore eventually irrelevant.

The second observation is the one that surprises people who have reasoned about this in terms of price per token. Output tokens are several times dearer than input, so for a short conversation the output term dominates and it feels like generation is what you pay for. The input term grows with N² and the output term with N, so there is a crossover, and past it the expensive part of a chat is re-reading what has already been said. Set the two equal:

P_in * g * N*(N-1)/2   =   P_out * a * N
                   N   ≈   1 + 2 * a * (P_out / P_in) / g

With a = 300, an output-to-input price ratio of 5, and g = 400 — all assumptions — the crossover is at about turn 8. From turn eight onwards, re-sent history costs more than generation, and it never stops pulling further ahead.

A worked session

Take B = 2,500, g = 400, a = 300, and assume P_in = $1.00 per million and P_out = $5.00 per million. Every one of those is an input to the formula, not a quoted price.

Read the last row carefully. Roughly 80,000 tokens of actual conversation produced 8.5 million billed input tokens: each token was paid for about a hundred times. Nothing pathological happened. That is simply what the re-send rule does over two hundred turns.

Which term each fix attacks

With the formula in front of you, every mitigation can be classified by which term it touches — and that immediately explains why some scale and some do not.

• Compaction changes the growth law. Capping history replaces g·N(N−1)/2 with a constant per turn, making total input linear. This is the only intervention that changes the order, which is why bounding history is the answer to long sessions rather than one of several options.
• Caching changes the constant. A cache hit on the prefix multiplies part of the input by r. Enormously valuable — potentially most of the bill — but N² × a small number is still N². Caching buys time, not a different curve, and it only applies to the stable prefix.
• Shorter system prompts change the smallest term. Trimming B affects B·N, which is linear and usually the least of the three. This is the optimisation people reach for first and it is the one that matters least at length.
• Shorter answers hit two terms at once. Reducing a reduces output cost directly and reduces g, which enters the quadratic. A verbose assistant is expensive twice, and the second time compounds.
• Ending the session resets N to zero. Blunt, unfashionable, and the largest available saving. A handoff document plus a fresh session costs one summarisation call and restarts the quadratic from the bottom of the curve.

Budgeting a session, not a request

The practical failure this arithmetic explains is a budget set per request. A cap of “two cents per call” is satisfied by every single turn of the 200-turn session above and still permits an $8.50 conversation, because the cap never sees the aggregate.

Three habits follow directly. Track cumulative spend per session, not per request, and set the limit there. Alert on session length as a leading indicator, since N is knowable before the cost arrives and is the variable in the quadratic. And put a hard turn cap on any interface that allows unbounded conversation, with a handoff at the cap rather than a refusal — the cap is not a restriction on the user, it is the point at which continuing linearly stops being possible.

One last note for anyone modelling this at the fleet level. The quadratic means your cost is dominated by your longest sessions, not your median one. A distribution where 5% of sessions run past a hundred turns will have those sessions accounting for the majority of spend, and an average-session cost model will understate the bill by a wide margin. Model the tail.

It is worth noticing what this arithmetic does not say. It does not say long conversations are wasteful, because the re-sent history is what makes the assistant coherent — you are paying for a real thing. What it says is that the price of coherence rises with the square of the conversation while its value rises much more slowly, which is precisely the shape that justifies bounding history rather than eliminating it. The engineering question is where on that curve the marginal turn stops being worth its cost, and the formula is what lets you answer it for your own prices instead of by feel.