Why Odia Text Costs More Tokens Than English

Odia has more than thirty-five million speakers and one of the thinnest presences of any major Indic script in public tokenizer training data. Two specific mechanisms make it worse than the three-bytes-per-character arithmetic alone predicts, and both are checkable against published Unicode data rather than being general statements about low-resource languages.

Deriving the multiplier

Odia    : ଲାଇବ୍ରେରୀରେ ଓଡ଼ିଆ ବହି ଅଛି।
English : There are Odia books in the library.

Odia    : 22 Odia code points     = 22 × 3 bytes = 66
          1 danda (।, U+0964)     =  1 × 3 bytes =  3
          3 spaces                =  3 × 1 byte  =  3
                                    total        = 72 bytes
          4 orthographic words

English : 36 ASCII characters     = 36 bytes  ≈ 9 tokens

Byte-fallback ceiling: 72 tokens against roughly nine, giving a derived worst case of 8.0x. Odia occupies U+0B00–U+0B7F, which is above U+0800 and so three bytes per code point throughout. As elsewhere in this cluster the ceiling is arithmetic and not a measured count; the useful work is in establishing where a real tokenizer lands within the range, and for Odia there are two identifiable reasons it lands high.

Note the sentence terminator. Odia does not have its own danda; it borrows U+0964 from the Devanagari block, as most Indic scripts do. It is three bytes, and it is the one code point in an Odia document that a Devanagari-trained merge could plausibly cover.

The consonants that cost six bytes

Odia writes two everyday consonants with a nukta — a subscript dot — on a base letter: ଡ଼ and ଢ଼, the retroflex flaps. Unicode does define precomposed single code points for them, U+0B5C and U+0B5D. The practical problem is that those precomposed forms appear in the Unicode Consortium’s composition exclusions list, which means normalisation form C will not produce them. Run NFC over Odia text, as almost every well-behaved pipeline does, and the flaps come out as base letter plus U+0B3C: two code points, six bytes, rather than one and three.

This is not a corner case. ଓଡ଼ି ଆ — the name of the language — contains one. The flaps are common enough in ordinary vocabulary that a document carries many of them, and each one is an extra three-byte code point that a reader does not perceive as a separate character and that a character-based size estimate does not count. The normalisation rules themselves are specified in UAX #15.

It also creates a comparison hazard. Text that has been through NFC and text that has not will contain different byte sequences for the same word, will tokenize differently, will hash differently and will fail an equality check. If you deduplicate an Odia corpus without normalising first, you keep both variants; if you normalise inconsistently between indexing and querying, retrieval misses.

Odia, Oriya, and where corpora lose it

The language was officially renamed from Oriya to Odia in India in 2011. Unicode block names are stable by policy and cannot be changed once published, so the block is still named U+0B00 Oriya, and every character in it still carries a name beginning ORIYA. The ISO 639 codes are or and ori, which are neutral, but a great deal of tooling and a great deal of metadata in the wild says Oriya.

That split is a real mechanism for underrepresentation rather than a curiosity. Corpus construction filters and language allowlists are written by people, and a list written with modern language names may say Odia while the metadata on the documents says Oriya, or the reverse. Documents that match neither string are dropped. The result is a corpus share below what the speaker population would suggest, and a tokenizer vocabulary with correspondingly few Odia merges — the bottleneck described in general terms on the tokenizer vocabulary page.

Odia’s script mechanics are otherwise shared with its neighbours: the abugida structure, the vowel signs and the virama at U+0B4D behave exactly as described for Kannada, and the nearest large relative in both geography and script family is Bengali, which has roughly ten times the speakers and a correspondingly better-served block.

The naming problem has a smaller cousin that is worth checking in your own stack. Locale tags for Odia are written both as or and, in older data, alongside the informal ory, and some libraries key font selection, hyphenation and language detection off those strings. A mismatch there does not usually produce an error; it produces silent fallback to a default that treats the text as unknown, which in a retrieval pipeline means the document is indexed without whatever language-specific handling the rest of your corpus receives. The symptom is Odia documents that are present in the index and never returned.

What this means for a real budget

• Character-based estimates understate Odia twice. Once for three bytes per code point, and again for the code points a reader does not count — viramas and nuktas. An estimate built from len(text) in a language like Python, which counts code points, is closer than one built from perceived characters but still wrong about which of those code points merge.
• Retrieval chunks must be sized in tokens. On the derivation above, a chunk that is 400 tokens of English is somewhere between 1,200 and 3,000 tokens of Odia. Embedding models with a 512 token limit will silently truncate most of it.
• Normalise once, early, and record that you did. Because NFC changes Odia byte counts, a token estimate made before normalisation does not match the one made after. Normalise at ingestion, before counting, chunking, hashing or embedding.

Measuring it on your own text

This checks both claims at once: the normalisation effect on the flaps, and the overall bytes-per-token on your corpus.

import tiktoken, unicodedata

enc = tiktoken.get_encoding("o200k_base")

# 1. the nukta consonants under each normalisation form
for form in ("NFC", "NFD"):
    w = unicodedata.normalize(form, "ଓଡ଼ିଆ")
    print(f"{form}: {len(w)} code points, "
          f"{len(w.encode('utf-8'))} bytes, {len(enc.encode(w))} tokens")

# 2. the corpus multiplier, after normalising
text = unicodedata.normalize("NFC",
        open("odia.txt", encoding="utf-8").read())
b, t = len(text.encode("utf-8")), len(enc.encode(text))
print("bytes/token:", round(b / t, 2))    # near 1.0 means byte fallback

The first block should show NFC and NFD agreeing, because NFC cannot compose the excluded forms — which is the point. If you have Odia text from a source that used the precomposed U+0B5C directly, run it through the same script and watch the code-point count change; that is the deduplication hazard, made visible.