Token Cost per Language
Put the same sentence in two languages side by side and see the token difference, the mechanism behind it, and what it costs you per month.
Estimated. Paste the exact counts from your own responses into both boxes to turn this into a measurement of your own workload.
- 1 · English — characters
- 81
- 1 · English — tokens (estimated)
- 17
- 1 · English — characters per token
- 4.76
- 2 · Japanese — characters
- 35
- 2 · Japanese — tokens (estimated)
- 37
- 2 · Japanese — characters per token
- 0.95
- Token difference per call
- 20
- Monthly cost, 1 · English
- $5.10
- Monthly cost, 2 · Japanese
- $11.10
- Monthly difference
- $6.00
Why the same sentence costs more in some languages
A tokenizer’s vocabulary is learned by repeatedly merging the most frequent character sequences in a training corpus, and those corpora are dominated by Latin-script text, mostly English. The result is a vocabulary that has a single entry for common English words and nothing much for sequences it rarely saw. Text in a script with few entries falls back to short fragments, and in the worst case to individual UTF-8 bytes — where one character can be three or four bytes before any merging happens at all.
Two consequences follow, and they compound. The direct one is price: if the same request costs twice as many tokens in one language as another, the speaker of that language pays twice as much for the same answer, and their requests take longer because generation is per-token. The indirect one is capacity: the same context window holds proportionally less of their document, so retrieval, summarisation and long-document workflows all get harder at the same time as they get dearer.
Character count is not a shortcut here, and the default example on this page shows why. Japanese says the same thing in far fewer characters than English, so a character-based comparison would suggest it is cheaper, while the token count frequently goes the other way. The only number that settles it is a real token count for your real text — which is why both exact-count fields exist, and why the honest version of “the non-English tax, quantified” is a measurement you take rather than a table you read.