Why Korean Text Costs More Tokens Than English

Korean gets grouped with Chinese and Japanese under “CJK” and given the same multiplier. The byte widths are identical and the outcome is not, because Hangul encodes sound where Han encodes meaning, and the tokenizer is paying for sound.

The syllable block is the unit

Hangul is an alphabet arranged into syllable blocks. 한 is not one letter; it is three jamo — ㅎ, ㅏ, ㄴ — composed into one square. Unicode stores the composed form as a single code point in the Hangul Syllables block, U+AC00 to U+D7A3, which is 11,172 precomposed syllables covering every combination that modern Korean permits. Every one of those code points is in the three-byte UTF-8 range.

That number, 11,172, is the crux. A byte-level BPE tokenizer would need two merges for each syllable it wants to represent as a single token, and there are eleven thousand candidates competing for slots against every other language in the corpus. Only the frequent few thousand realistically win. The rest fall back to two tokens or three, and because Korean draws on the full inventory far more evenly than Japanese draws on its hundred kana, the tail is long and it is hit often.

The arithmetic on a labelled sentence

이 기능은 컴퓨터 메모리를 많이 사용할 수 있습니다.
"This feature can use a lot of computer memory."

Count it: 21 Hangul syllable blocks, seven spaces, one full stop. The blocks are three bytes each (63 bytes), the spaces and stop are one byte each (8 bytes), so the sentence is exactly 71 UTF-8 bytes. The English is 46 bytes and, at four characters per token, near 11 or 12 tokens.

Now derive the Korean, with the assumption stated. Assume the common blocks in this sentence — 이, 기, 능, 은, 사, 용, 할, 수 and the endings 습, 니, 다 — are single tokens, and that the less common ones in 컴퓨터 and 메모리 cost two. That puts the sentence in the region of 25 to 35 tokens against the English 11 or 12, so a derived multiplier of roughly 2× to 3×. The unconditional bound is again the byte count: it cannot exceed 71.

Note what is doing the work there. It is not that Korean characters are wide — Chinese characters are exactly as wide. It is that this sentence needs 21 of them where the Chinese equivalent would need around 12.

Why this lands worse than Chinese

Chinese pays three bytes per character and gets a whole morpheme in return; 機 is a morpheme, and two of them make a word. Korean pays three bytes per syllable, and a Korean word is typically two to four syllables. 컴퓨터, the Korean for “computer”, is three blocks and nine bytes for a word that Chinese writes with two characters and English with eight ASCII bytes.

Then agglutination adds to it. Korean marks case, topic, politeness and tense with suffixed particles and endings that attach to the stem: 기능은 is 기능 plus the topic marker 은, and 있습니다 is four blocks of which three are the polite ending. Those endings are frequent enough to tokenise well, but they are still characters that English does not write at all. The result is a language that is three bytes per syllable like Chinese, but needs roughly twice as many of them per sentence — which is why the density discount that softens the Chinese multiplier does not apply here.

This also means the fix that helps Chinese retrieval — treating characters as near-morphemes — is wrong for Korean. Korean chunking works on morpheme boundaries, not syllables; see morpheme-aware chunking for Korean.

One thing does work in Korean’s favour and is worth naming because it is the difference from Chinese and Japanese that reduces the bill rather than raising it: Korean is written with spaces. The tokenizer’s pre-tokenization step can therefore cut Korean into word-sized pieces and attach a leading space to each, which is the exact mechanism that makes English words single tokens. Korean does not get single-token words out of it, because the inventory is too large, but it does get stable, learnable word-initial sequences — something Chinese and Japanese cannot offer at all.

The offsetting problem is that Korean spacing rules are widely and legitimately varied in practice, and user-generated text ignores them freely. Compound nouns can be written joined or separated, both acceptably, and social and messaging text often drops spaces entirely. Two spellings of the same phrase therefore carry different token counts and do not match each other in retrieval. Korean chat text adds another layer: the compatibility jamo block U+3130–U+318F, which is where the standalone letters in ㅋㅋㅋ and ㅠㅠ come from. Those are three-byte code points, distinct from both the precomposed syllables and the conjoining jamo, and they are extremely frequent in social corpora.

The decomposition trap that triples the count

Everything above assumes composed Hangul, Unicode normalisation form NFC, which is what a keyboard and almost every web form produce. Hangul can also be represented in decomposed form, NFD, where 한 is stored as three separate conjoining jamo code points instead of one precomposed syllable.

The two forms look identical on screen. They are not identical to a tokenizer. Decomposed Hangul is three code points per syllable, each in the three-byte range, so a syllable that cost three bytes now costs nine — and the conjoining jamo are far rarer in training corpora than the precomposed blocks, so merge coverage is worse on top of that. A file that has passed through a macOS filesystem, which stores filenames in a decomposition close to NFD, can arrive with a token cost roughly three times what the same visible text costs in NFC.

Measuring it, and what to do about it

import unicodedata
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

ko = "이 기능은 컴퓨터 메모리를 많이 사용할 수 있습니다."
en = "This feature can use a lot of computer memory."

for label, s in (("ko NFC", unicodedata.normalize("NFC", ko)),
                 ("ko NFD", unicodedata.normalize("NFD", ko)),
                 ("en", en)):
    b = len(s.encode("utf-8"))
    n = len(enc.encode(s))
    print(label, "codepoints", len(s), "bytes", b, "tokens", n)

Run that and the NFC and NFD rows should differ sharply while looking the same when printed. If your production corpus shows an NFD-shaped token count, you have found a real bug rather than a property of Korean.

• Normalise to NFC before counting, indexing or sending. Do it once at ingestion, not at query time, or your index and your queries will disagree.
• Budget chunks in tokens with a Korean-specific ratio. A 4,000-token chunk holds roughly 16,000 English characters and, on the derivation above, something closer to 1,500–2,500 Hangul syllables. That is a short document section, not a chapter.
• Hanja is a different cost again. Korean text that mixes in Chinese characters for disambiguation shifts toward the Chinese profile for those runs, which is worth knowing if you process legal or academic Korean.