Words to Tokens Converter
Converts a word count into tokens using a ratio derived from your own text, rather than the single constant every other converter uses.
usage object on any completion response — prompt_tokens and completion_tokens are what you are billed on, they cost nothing to read, and they settle the question for the model you are actually calling.At 1.286 tokens per word, from this page's estimate for the text above.
- Words in the sample
- 63
- Characters in the sample
- 354
- Tokens in the sample (estimated)
- 81
- Tokens per word, from the sample
- 1.286
- Words per token, from the sample
- 0.778
- Ratio used for the conversion
- 1.286
- Tokens per 1,000 words at that ratio
- 1,286
- Words that fit in 100,000 tokens
- 77,778
Why the ratio moves, and what moves it
Tokenizers learn their vocabulary from a corpus, and the merges they learn are the character sequences that corpus contained most often. Common English words end up as a single token; longer or rarer ones split into two or three pieces. That is why English prose sits near 1.2 to 1.4 tokens per word and why the figure is so stable: most of the words in any English paragraph are drawn from the few thousand most common words in the language.
Everything that departs from that raises the ratio. Technical vocabulary, product names and identifiers split more. Inflected languages — where one word carries endings that English spreads across three — pack more meaning into a word and get charged more tokens for it. Languages written in a script the vocabulary has few merges for fragment much further still, sometimes to one token per character or worse. And any text with numbers in it breaks the model entirely, because digits are usually split into groups of three regardless of how they read.
The practical consequence: derive the ratio once, from a real sample of your own traffic, and re-derive it whenever you change model family or add a language. A ratio taken from someone else’s blog post is a ratio for someone else’s text. If you want a number you can actually plan a budget against, send one representative request and divide the token count in the response by your own word count — that division is the honest version of this whole tool.