Characters to Tokens Converter
Converts characters into tokens at a ratio measured on your own text, and shows how wrong the familiar four-characters-per-token rule would be for it.
usage object on any completion response — prompt_tokens and completion_tokens are what you are billed on, they cost nothing to read, and they settle the question for the model you are actually calling.At 4.37 characters per token (estimated from the sample).
- Characters in the sample
- 354
- Tokens in the sample (estimated)
- 81
- Characters per token, from the sample
- 4.37
- Ratio used for the conversion
- 4.37
- The same characters at 4 chars/token
- 2,500
- How wrong 4 chars/token would be here
- +9.3%
- Characters that fit in 100,000 tokens
- 437,037
Where four characters per token comes from, and where it breaks
The constant is an average over English web text, and on English web text it is a decent one: prose typically lands between 4 and 5 characters per token. Its usefulness comes from being conservative in the right direction for a spending check — divide by four and you get a number slightly above the truth, which is what you want when the question is “can this account afford the request” rather than “what will this cost”.
It breaks in four predictable ways. Structured text — JSON, XML, HTML, minified code — packs punctuation densely and lands nearer 2 to 3 characters per token, so dividing by four underestimates by a third or more. Numbers are split into groups of about three digits, so a table of figures is far more expensive per character than a paragraph about them. Non-Latin scripts often reach one token per character, and sometimes worse. And emoji, unusual symbols and rare characters fall back to something close to per-byte encoding, at two to four tokens each.
So use the constant for a rough ceiling and your own measured ratio for anything you plan against. The measurement costs one API call: send a representative sample, read the token count out of the response, and divide your character count by it. Do that once per model family and once per content type — prose, code, and whatever your retrieved context looks like — and you have a small table of ratios that describes your system instead of the internet’s average.