Token Segment Visualiser
Colours your text by the character classes that drive its token count, so you can see which part — indentation, digits, emoji, non-Latin script — is doing the damage.
usage object on any completion response — prompt_tokens and completion_tokens are what you are billed on, they cost nothing to read, and they settle the question for the model you are actually calling.Latin letters account for about 43.9% of it.
Indentation·costs·money: ····const·total·=·1_284_993; So·do·emoji·🎉🎉·and·scripts·with·few·merges·—·Пример,·これは例です,·उदाहरण·—·while·ordinary·English·prose·is·the·cheapest·thing·you·can·send.
- Latin letters
- 116 chars → ~27 tokens (43.9%)
- Punctuation and ASCII symbols
- 10 chars → ~10 tokens (16.4%)
- Chinese, Japanese, Korean
- 6 chars → ~6 tokens (9.8%)
- Indic and South-East Asian scripts
- 6 chars → ~5 tokens (8.9%)
- Emoji and less common scripts
- 2 chars → ~4 tokens (6.6%)
- Cyrillic, Greek, Arabic, Hebrew
- 6 chars → ~3 tokens (4.9%)
- Digits
- 7 chars → ~3 tokens (4.9%)
- Whitespace and line breaks
- 36 chars → ~3 tokens (4.9%)
Reading the picture
The useful thing about seeing text coloured by class is that the expensive parts stop being invisible. Four spaces of indentation on every line of a pretty-printed JSON payload is a real, recurring line on your bill, and it looks like nothing at all in an editor. So does a blank line between every paragraph of a long system prompt. So does an emoji, which is frequently two to four tokens on its own and more once you add a skin-tone modifier or a zero-width joiner.
The other thing it shows is why the “four characters to a token” rule of thumb only works on the kind of text it was derived from. English prose is one long run of Latin letters separated by single spaces, which is the case every merge in the vocabulary was optimised for. Change any of those conditions — swap the script, replace the words with numbers, add structure — and the ratio moves, sometimes by a factor of three.
What this cannot show you is where a real tokenizer would actually cut. Byte-pair encoding builds its vocabulary from whatever corpus it was trained on, so the boundaries are an artefact of that corpus rather than of any rule you could reproduce here. If you need the true segmentation, run the tokenizer for your model family locally; if you only need the total, the response you were going to send anyway will tell you.