Tokenizer Playground: Train a BPE and Watch Every Merge
Trains a real byte-pair encoding on text you supply, then lets you drag the merge count and watch your text regroup from characters into subwords with the token count moving as you type.
3.58 characters per token, against 44 tokens with no merges at all — a 3.67× compression bought with 40 merge rules.
▁ marks the start of a word — it is how the tokenizer remembers that a space was there, and why the same word costs a different number of tokens at the start of a line than in the middle of one.
Tokens in your text against merges learned. The steep part at the left is where almost all of the compression is; the long flat tail is why vocabularies stop growing.
| Rank | Merged pair | New token | Occurrences when merged |
|---|---|---|---|
| 40 | ▁an + d | ▁and | 3 |
| 39 | u + nt | unt | 3 |
| 38 | o + del | odel | 3 |
| 37 | iz + er | izer | 3 |
| 36 | i + z | iz | 3 |
| 35 | d + el | del | 3 |
| 34 | ▁w + id | ▁wid | 4 |
| 33 | ▁ne + wer | ▁newer | 4 |
- Characters in your text
- 43
- Tokens at 0 merges (one per character)
- 44
- Tokens now
- 12
- Characters per token
- 3.58
- Base symbols learned from your corpus
- 22
- Merge rules applied
- 40
- Vocabulary size (base + merges)
- 62
- Distinct tokens used in your text
- 10
- Merges this corpus supports before pairs stop repeating
- 74
Why your text costs what it costs
A tokenizer is not a dictionary of words. It is an ordered list of merge rules, learned by repeatedly gluing together whichever adjacent pair of symbols is most common in some corpus, and it has no notion of meaning, morphology or word boundaries beyond what frequency happened to produce. Drag the slider from zero and watch the first merges arrive: they are not words, they are the fragments your text repeats — th, er, ▁the. Whole words appear only once they have out-competed every fragment inside them.
That is the whole explanation for the effect people notice on their invoice. A word that was frequent in the training corpus is one token. A word that was not gets assembled out of pieces, and you pay per piece. Train on the text above and lower becomes a single token while tokenizer stays in fragments — swap the corpus for something technical and the two trade places. The same mechanism is why non-English text costs two to four times more per word through tokenizers trained mostly on English, why code and JSON tokenise badly unless the corpus contained them, and why a rare product name in your system prompt is quietly five tokens on every request you will ever make.
The curve explains the other half: why vocabularies are the size they are. Almost all the compression arrives in the first handful of merges, and the tail flattens out. Doubling a 50,000-token vocabulary buys a few percent fewer tokens and costs a proportionally larger embedding table and output head — you can watch that trade in /tools/transformer-explainer, where vocabulary size drives the two largest matrices in the model. What this page cannot tell you is your real bill, and it will not pretend to: for that you need the actual vocabulary your provider uses, which is theirs to publish and theirs to change.