Why Vietnamese Text Costs More Tokens Than English

Vietnamese uses the same alphabet as English and pays a token penalty comparable to Cyrillic. The reason is that six tones and a set of vowel modifiers are written as diacritics, and the resulting characters are not in ASCII — several of them are not even in the two-byte range.

Latin letters that are not ASCII

Quốc ngữ, the Vietnamese writing system, is a Latin alphabet extended in two directions at once. First there are the modified base letters: ă, â, ê, ô, ơ, ư and đ. These are the vowel-quality and consonant modifiers, and they sit in Latin Extended-A and Latin Extended-B, below U+0800, so UTF-8 encodes them in two bytes.

Then there are the tones. Vietnamese has six, five of which are marked: acute, grave, hook above, tilde and dot below. A tone mark can land on a plain vowel or on one that already carries a modifier, so the system needs precomposed code points for every combination — ế, ệ, ộ, ứ, ữ, ị, ọ, ằ, ẵ and dozens more. Unicode places these in Latin Extended Additional, U+1E00–U+1EFF, which is above U+0800 and therefore three bytes in UTF-8.

So a Vietnamese word is a mixture of one-, two- and three-byte characters that looks entirely Latin on screen. That mismatch is the whole story: the merge table’s enormous investment in English byte sequences applies only to the ASCII fragments, and every tone-bearing vowel is a wall the merges cannot cross.

The arithmetic on a labelled sentence

Chức năng này sử dụng nhiều bộ nhớ.
"This feature uses a lot of memory."

Both lines are exactly 35 code points. That is the useful coincidence here, because it removes every confound about verbosity and density: the two sentences are the same length as a reader counts, and the only difference is which code points they use.

The English is 35 bytes, all ASCII, near 9 tokens. The Vietnamese is 49 bytes: 20 ASCII bytes for the plain consonants, spaces and full stop, plus two two-byte characters (ă, à) and seven three-byte characters (ứ, ử, ụ, ề, ộ, ớ and one more) — 41 bytes of letters and 8 of ASCII punctuation and spacing. So the byte multiplier alone is 1.4× at identical character counts.

Now derive the tokens with the assumption stated. Assume the ASCII runs inside each word merge normally — nh, ng, ch are extremely frequent English digraphs and are certainly single tokens — but that every tone-bearing vowel terminates a merge and costs one to three tokens by itself. Each of these eight words then breaks into roughly two to four pieces, giving something in the region of 18 to 30 tokens against the English 9: a derived multiplier of about 2× to 3.3×, with an unconditional ceiling of 49 from the byte floor.

Note how the multiplier exceeds the byte multiplier. That gap is the fragmentation effect, and it is the part that makes Vietnamese more expensive than its byte count predicts. Compare Indonesian, which is also a Latin-script Southeast Asian language and pays almost nothing, because it is written in plain ASCII.

Spaces separate syllables, not words

The second Vietnamese-specific effect is orthographic convention rather than encoding. Vietnamese puts a space between every syllable, and multi-syllable words are written as separate space-delimited units: máy tính is “computer”, bộ nhớ is “memory”, ngôn ngữ is “language”.

For the tokenizer’s pre-tokenization step, this means a Vietnamese word generates two or three pre-tokens where an English word generates one. Since merges cannot cross a pre-token boundary, the tokenizer has no way to represent máy tính as a single unit no matter how frequent it is. There is a floor of one token per syllable that no amount of vocabulary would remove.

It also breaks word-count-based assumptions elsewhere: a Vietnamese document has far more whitespace-delimited units than an English document of the same content, so any heuristic phrased in words — chunk sizes, truncation limits, snippet lengths — produces the wrong result.

The same word, two or three encodings

Everything above assumes NFC, the precomposed form. Vietnamese has a particularly bad case of the normalisation problem because there are multiple legitimate ways to encode the same visible word.

• NFC: ế is one code point, U+1EBF, three bytes.
• NFD: the same ế is e, then the circumflex U+0302, then the acute U+0301 — three code points, five bytes, and each combining mark is a merge-breaker.
• Partially composed: ê as a single code point followed by a combining acute. This form is common in real data because it is what some input methods emit.

All three render identically. They have different lengths, different token counts and different byte sequences, so they do not compare equal, do not match in search, and do not deduplicate. On top of that, Vietnamese has a live orthographic disagreement about tone mark placement in certain diphthongs — hòa and hoà are both current spellings of the same word, with the mark on a different vowel, and they are genuinely different strings even after normalisation.

Measuring it, and what to normalise

import unicodedata
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

vi = "Chức năng này sử dụng nhiều bộ nhớ."
en = "This feature uses a lot of memory."

rows = (
    ("vi NFC", unicodedata.normalize("NFC", vi)),
    ("vi NFD", unicodedata.normalize("NFD", vi)),
    ("en", en),
)

for label, s in rows:
    print(label,
          "codepoints", len(s),
          "bytes", len(s.encode("utf-8")),
          "tokens", len(enc.encode(s)),
          "syllables", len(s.split()))

The NFC and NFD rows should differ substantially in both bytes and tokens while printing identically. If your production corpus resembles the NFD row, you have a normalisation bug that is inflating your bill as well as breaking your search. The syllables column is the number to size chunks against, not a word count.

• Stripping diacritics is not a compression strategy. It makes Vietnamese ambiguous — ma, má, mà, mả, mã and mạ are six different words — and degrades output quality accordingly.
• Budget roughly a third of the English character count. On the derivation above, a 4,000-token chunk that would hold about 16,000 English characters holds something closer to 5,000–7,000 Vietnamese ones.
• Check what your database stores. Collations and column types that silently strip or reorder combining marks will change the token count between what you wrote and what you read back.