How OpenAI Counts Tokens in a Multi-Turn Conversation With Tool Results

A conversation has no server-side state. Every request re-sends the whole transcript, so the cost of turn five includes turns one to four again — and a tool result, which is usually the largest message in the conversation, is re-sent every time after it lands.

What is re-sent on every request

The Chat Completions endpoint is stateless. Each call carries the entire messages array, and usage.prompt_tokens in the response is the token count of that whole array. There is no incremental billing: the tenth request in a conversation pays for the first message for the tenth time.

Written out, if turn i adds m_i tokens to the transcript, the prompt billed on request t is the sum of m_1 through m_(t-1). Total prompt tokens across a conversation of T requests is a sum of prefixes, which grows with the square of the number of turns rather than linearly. That is the single most important fact about the economics of a long chat, and it is why a conversation that feels cheap at turn three is not cheap at turn thirty.

The published per-message overhead

The count is not simply the sum of the content strings. Each message is wrapped in role delimiters that the tokeniser also counts, and OpenAI publishes the arithmetic in the OpenAI Cookbook recipe “How to count tokens with tiktoken”. Its num_tokens_from_messages function, for the gpt-4 and gpt-4o families, uses:

tokens_per_message = 3     # role delimiters wrapping each message
tokens_per_name    = 1     # extra, only if the message has a "name" field
priming            = 3     # every reply is primed with <|start|>assistant<|message|>

total = 3 + Σ over messages of ( 3 + tokens(content) [+ 1 if name] )

Three tokens per message sounds negligible and is not, in one specific case: a long conversation of short messages. Twenty exchanges of ten-token messages is 400 tokens of content and 120 tokens of overhead, a 30% surcharge on the visible text. On a conversation of long messages it rounds to nothing.

Five requests, counted

Here is a support conversation that makes a tool call on the first turn. The content-token counts below are assumptions, stated so you can substitute your own — they are plausible sizes for each message, not measurements of a particular string. The overhead arithmetic is the published rule applied to them.

assumed content tokens

  system   "You are a support assistant for..."      40
  user 1   "Where is my order 10482?"                12
  asst 1   tool call, get_order(id: "10482")         25
  tool 1   order JSON returned by your API          120
  asst 2   "Order 10482 shipped on the 4th..."       45
  user 2   "Can I change the delivery address?"      15
  asst 3   "Not once it has shipped, but..."         60
  user 3   "What about a redirect?"                  10
  asst 4   "Carrier redirects are possible..."       55
  user 4   "Ok, do that for me please."              14

Applying total = 3 + Σ(3 + content) to the messages present at each request:

request  messages in the array                              prompt_tokens
   1     system, user1                                          61
   2     + asst1(tool_call), tool1                              212
   3     + asst2, user2                                         278
   4     + asst3, user3                                         354
   5     + asst4, user4                                         429
                                                          ─────────
         total prompt tokens billed across the conversation   1,334

Request 1 is 3 + (3+40) + (3+12) = 61. Request 2 adds the assistant’s tool call and the tool result: 61 + (3+25) + (3+120) = 212. Each later request adds one assistant message and one user message with their three-token wrappers.

Where the count jumps. Between request 1 and request 2 the prompt more than triples, from 61 to 212. Of the 151 tokens added, 123 are the tool result and its wrapper. Nothing the user typed caused that jump; an API response did.

It is worth seeing the general form, because the five-turn case understates it. If every exchange adds a constant m tokens, the prompt on request t is about (t − 1) × m and the total across T requests is about m × T(T − 1) / 2. Doubling the length of a conversation roughly quadruples what you have spent on prompts by the end of it. A twenty-turn conversation of the same shape as the one above bills something like ten times the prompt tokens of a ten-turn one, not twice.

And it is paid for again on every subsequent request. That single 123-token tool message appears in requests 2, 3, 4 and 5 — four times, 492 tokens, which is 37% of the 1,334 prompt tokens billed across the whole conversation. One JSON blob from your own database is the largest line item in a five-turn support conversation.

The same arithmetic says something about trimming strategy that is not obvious. Dropping the oldest messages is the usual approach and it is the wrong end of the array to look at first: the oldest messages here are the system prompt and a twelve-token question. The expensive message is in the middle. A history policy that keeps the last N turns will faithfully carry a 120-token tool result forward forever while discarding the user question that explains it, which is both more expensive and worse. Trimming by size, or replacing an old tool result with a one-line summary of what it said, beats trimming by age on exactly this shape of conversation.

The lever this hands you is specific and unglamorous: shrink tool results before appending them. A 400-token API response trimmed to the 40 tokens the model actually needs saves 360 tokens multiplied by every remaining turn, not once. Filtering the fields your tool returns is usually the highest-value token optimisation available in an agent loop, and it is a change to your own code rather than to your prompt.

The part that is not documented

Everything above counts messages. A request that uses tools also carries a tools array of function definitions, and those definitions are serialised into the prompt too — you are billed for your schemas on every single request, whether or not any tool is called.

The exact serialisation is not published. The cookbook’s own treatment of function counting is presented as an approximation derived by observation rather than as a specification, and it carries the caveat that it may be wrong. So the honest statement is: the token cost of a tool definition is real, it is roughly the size of the schema’s JSON plus a per-function overhead, and it is not exactly predictable from the client. The same applies to the assistant message that contains a tool call — the 25 in the table above is an assumption about the serialised call, not a rule.

The consequence for planning is that client-side counting is a budget estimate, not an invoice. Use it to decide whether a request will fit in the context window, with headroom. Do not use it to reconcile a bill.

Getting the real number

There are two numbers and only one of them is authoritative.

1. Before the request, estimate with tiktoken. Get the encoding with tiktoken.encoding_for_model(model) rather than naming one directly, since the 4o family moved to o200k_base and a hard-coded cl100k_base will quietly count the wrong vocabulary. Add the per-message overhead from the recipe above. Use this for the fit-in-context decision and for trimming history.
2. After the request, read usage.prompt_tokens from the response. This is the number you are billed on. It includes the tool definitions, the tool-call serialisation and anything else the server added, so it is the only figure to use for cost accounting or for calibrating your estimator.
3. Log the difference. Estimate minus actual, per request, is a cheap regression test on your own accounting. When the gap changes shape, something about the serialisation changed and your context-fit headroom is no longer what you thought.

One qualifier on all of the above: prompt caching changes what you pay, not what is counted. When a long prefix is served from cache the tokens still appear in prompt_tokens, with the discounted portion broken out in usage.prompt_tokens_details.cached_tokens. Because caching keys on an exact prefix, the quadratic growth described here is exactly the access pattern it is designed for — the transcript so far is a prefix of the transcript next turn — provided you only ever append to the array and never edit an earlier message. Rewriting the system prompt at turn ten invalidates the cache for the entire conversation.