Unicode and Emoji Token Cost
Counts graphemes, code points and UTF-8 bytes exactly for any text, and shows the token range those bytes imply for a byte-level tokenizer.
197 UTF-8 bytes, of which 87 come from 31 non-ASCII characters. A byte-level tokenizer can never charge more than one token per byte, so 197 is the ceiling for this text.
- Characters you would count by eye (graphemes)
- 134
- Unicode code points
- 141
- UTF-16 units (JavaScript's .length)
- 148
- UTF-8 bytes
- 197
- ASCII characters (1 byte each)
- 110
- Bytes spent on non-ASCII characters
- 87 (44.2% of the text)
- Estimated tokens
- 65
- Hard ceiling, one token per byte
- 197
Estimated, not tokenized. Token figures here come from an approximation running in your browser: no tokenizer vocabulary is downloaded, because a real one is megabytes and nothing on this page fetches anything. It imitates how a byte-level BPE splits text — words, digit groups, punctuation runs, whitespace runs — but it has no merge table, so treat it as a planning number. The authoritative count is the usage object on a real API response.
Every non-ASCII character in your text
| Character | Code points | UTF-8 bytes | Est. tokens | Worst case | Times |
|---|---|---|---|---|---|
| 👨👩👧👦 | U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F466 | 25 | 13 | 25 | 1 |
| 🙏🏽 | U+1F64F U+1F3FD | 8 | 4 | 8 | 1 |
| — | U+2014 | 3 | 2 | 3 | 2 |
| é | U+00E9 | 2 | 1 | 2 | 3 |
| 🚀 | U+1F680 | 4 | 2 | 4 | 1 |
| р | U+0440 | 2 | 1 | 2 | 2 |
| и | U+0438 | 2 | 1 | 2 | 2 |
| ✅ | U+2705 | 3 | 2 | 3 | 1 |
| 你 | U+4F60 | 3 | 1 | 3 | 1 |
| 好 | U+597D | 3 | 1 | 3 | 1 |
| ， | U+FF0C | 3 | 2 | 3 | 1 |
| 世 | U+4E16 | 3 | 1 | 3 | 1 |
| 界 | U+754C | 3 | 1 | 3 | 1 |
| ï | U+00EF | 2 | 1 | 2 | 1 |
| П | U+041F | 2 | 1 | 2 | 1 |
| в | U+0432 | 2 | 1 | 2 | 1 |
| е | U+0435 | 2 | 1 | 2 | 1 |
| т | U+0442 | 2 | 1 | 2 | 1 |
| м | U+043C | 2 | 1 | 2 | 1 |
"Worst case" is the byte count: if a tokenizer's merge table has no entry for that byte sequence, it falls back to one token per byte.
Intl.Segmenter where it exists and falls back to code points where it does not, which will over-count flags and family emoji on very old browsers. The token column is the estimate described above and assumes a byte-level BPE, which is what the GPT-2-descended family uses; a SentencePiece model with a byte-fallback behaves similarly, and a model with a character-level vocabulary for a given script does not.The reason a single emoji can cost four tokens is arithmetic about bytes, not a quirk. Modern tokenizers are byte-level: before any merging happens, text is UTF-8 encoded and the tokenizer works on bytes. ASCII is one byte per character. Accented Latin and Cyrillic are two. Most CJK is three. Emoji live outside the Basic Multilingual Plane and are four. If the merge table learned that particular four-byte sequence during training — and it will have, for common emoji — it costs one token. If it did not, the tokenizer falls back to the bytes themselves, and you pay four.
That is the ceiling, and it is why the table above prints the byte count as "worst case": no byte-level tokenizer can ever charge more than one token per UTF-8 byte, because the bytes are the base vocabulary. Everything above that base is a merge, and merges only ever make things cheaper.
Compound emoji are where the bill gets strange. 👨👩👧👦 is not one character; it is four emoji joined by three zero-width joiners, seven code points and 25 bytes. 🙏🏽 is a base emoji plus a skin-tone modifier, eight bytes. A flag is two regional indicator symbols, also eight. So a message that looks like three characters to a human can easily be thirty bytes, and the friendly ✅ at the end of a status line is not the cheap punctuation it looks like.
The practical version: emoji in a system prompt sent on every request are a fixed tax you pay forever, and stripping decorative ones from templates is one of the few prompt edits with no behavioural risk at all. Emoji in user content are not worth fighting — you cannot control what people type, and the honest response is to budget for them. The same logic explains why the non-English "token tax" exists: it is fundamentally a question of how many bytes a script needs and how densely the merge table covers them.