Why Sinhala Text Costs More Tokens Than English

Most pages about token cost treat the multiplier as a property of a language. For Sinhala it is not. The same sentence, semantically and visually identical, has two different byte lengths depending on which Unicode normalisation form your pipeline produced, and therefore two different bills.

The floor, and how far above it Sinhala sits

The Sinhala block runs from U+0D80 to U+0DFF, three UTF-8 bytes per codepoint. The word සිංහල is five codepoints — sa, the i vowel sign, the anusvara, ha, la — and therefore fifteen bytes. English “Sinhala” is seven ASCII bytes.

The three-byte floor is shared with every Indic script in this cluster. What is not shared is how far above the floor Sinhala sits. Sinhala is written by roughly the population of Sri Lanka and by essentially nobody outside it; unlike Devanagari, it is not shared with a larger language that could fund its merges, the way Nepali free-rides on Hindi. A script used by one language of moderate size, with no larger sibling in the same encoding, is close to the worst position in the merge auction.

The Unicode Sinhala code chart is the reference for every codepoint named on this page, including the decompositions in the next section.

The same word, two byte lengths

Sinhala’s two-part vowel signs are the unusual part. Several of them are encoded as single codepoints that Unicode also defines a canonical decomposition for. The code chart lists, among others:

• U+0DDA, diga kombuva, which decomposes to U+0DD9 followed by U+0DCA.
• U+0DDC, kombuva haa aela-pilla, which decomposes to U+0DD9 followed by U+0DCF.
• U+0DDD, which decomposes to a three-codepoint sequence: U+0DD9, U+0DCF, U+0DCA.
• U+0DDE, which decomposes to U+0DD9 followed by U+0DDF.

Under NFC those are one codepoint each, three bytes. Under NFD the first three become two, two and three codepoints — six, six and nine bytes. A Sinhala word containing one U+0DDD costs six bytes more in NFD than in NFC, and a passage with a normal density of these vowels can differ by ten to twenty per cent in total byte length between the two forms.

Because a byte-level tokenizer sees only bytes, that byte difference is a token difference, and it is usually worse than proportional: whatever merges exist for Sinhala were learned from whichever form dominated the training corpus, which is NFC, so the decomposed form fires fewer of them and falls closer to the floor. The penalty is the extra bytes plus the lost merges.

Decomposed text arrives from more places than people expect: macOS filesystem APIs, some PDF extractors, some database drivers, and any pipeline that concatenates text from mixed sources without normalising. If you have never explicitly normalised, you probably have both forms in your corpus, which also means your retrieval index has two entries for some words that are the same word — the failure described in the difference between NFC and NFKC.

Invisible characters you are billed for

Sinhala also uses the al-lakuna, U+0DCA, to form conjunct consonants, and touching or ligated conjunct forms are requested with a zero-width joiner, U+200D, between the al-lakuna and the following consonant.

The ZWJ is three UTF-8 bytes and completely invisible. A conjunct written with it costs three bytes more than the same conjunct written without it, and in most fonts the two render similarly enough that no one reviewing the text will spot the difference. It also breaks string equality, so the two spellings will not match each other in a lookup.

On a corpus with heavy conjunct usage this is not a rounding error. Every joined conjunct is one extra codepoint’s worth of bytes and, because the joined sequence is rarer in training data than the unjoined one, quite possibly one extra token on top of that.

Deriving both multipliers

Assumptions: English at four characters per token, per OpenAI’s published rule of thumb, so 0.25 tokens per character. Three bytes per Sinhala codepoint. NFC merge efficiency of 1.3 to 1.8 bytes per token, on the argument above that Sinhala has no larger sibling to borrow from — near the floor, but not at it, because function words and frequent endings do earn some merges. NFD efficiency lower still, call it 1.1 to 1.3. And a length assumption of 0.7 Sinhala codepoints per English character.

NFC. A 200-character English passage at 50 tokens is 140 Sinhala codepoints, so 420 bytes, so at m of 1.55 about 271 tokens. Multiplier about 5.4x.

NFD. Assume the two-part vowel signs add 12 per cent to the byte count, giving 470 bytes, and m drops to 1.2. That is about 392 tokens, a multiplier near 7.8x.

Same text, same model, same prompt. A 45 per cent difference in cost, decided by a normalisation call somewhere in a pipeline that nobody documented. That is the point of this page: for Sinhala, before you argue about which model to use, find out which normalisation form you are sending.

Fix the pipeline before the prompt

import unicodedata, tiktoken
enc = tiktoken.get_encoding("o200k_base")

raw = open("corpus.si.txt", encoding="utf-8").read()

for label, text in [
    ("as-received", raw),
    ("NFC", unicodedata.normalize("NFC", raw)),
    ("NFD", unicodedata.normalize("NFD", raw)),
]:
    b = len(text.encode("utf-8"))
    t = len(enc.encode(text))
    print(f"{label:12s} {len(text):8d} cp  {b:8d} B  {t:8d} tok  m={b/t:4.2f}")

# how much invisible joiner are you paying for?
print("ZWJ occurrences:", raw.count("\u200d"))

If “as-received” matches NFD, normalising to NFC is a one-line change that reduces your Sinhala bill immediately and improves retrieval at the same time. If it matches neither, your corpus is mixed, which is the worst case and the one most worth fixing.

After that, the remaining cost is the merge auction, and no preprocessing removes it. At a realistic 5.4x, a 4,000-token budget holds roughly 740 English tokens of Sinhala content, and a 512-token retrieval chunk holds about 95 — well under a paragraph. Size chunks on sentence boundaries in the source text rather than on a token count borrowed from an English pipeline. The general form of this constraint, and what it means for languages with no larger sibling in their script, is covered in the tokenizer vocabulary bottleneck.