Why Indonesian Tokenizes Almost as Efficiently as English

Indonesian is the control case for this whole territory. It is a national language with more than 200 million speakers that costs roughly what English costs, and understanding why is the fastest route to understanding why every other language in this cluster does not.

Token cost has exactly two axes

A byte-level BPE tokenizer turns text into tokens in two steps, and a language can be penalised at either. Every multiplier on every page in this cluster decomposes into these two and nothing else.

• Bytes per character. Set by UTF-8 and the script’s Unicode block. ASCII is one byte. Latin letters with diacritics, Greek, Cyrillic, Hebrew and Arabic are two. Indic, South-East Asian and CJK are three. This is fixed physics: no vendor can change it and no amount of training data helps.
• Bytes per token. Set by how many merges the vocabulary learned for this language, which is set by how much of the language was in the training corpus. English sits around four bytes per token. A script the tokenizer never saw sits at one.

Multiply them and you get tokens per character. Divide by English’s and you get a multiplier. A language pays a large penalty only if it loses on both axes, which is why Finnish — one byte per character, poor merges — and Hindi — three bytes per character, moderate merges — end up in surprisingly similar territory from completely different directions.

Axis one: Indonesian is pure ASCII

Indonesian orthography, standardised in the 1972 spelling reform shared with Malaysia, uses the twenty-six unaccented Latin letters and nothing else. No diacritics, no digraph characters outside a–z, no eszett, no dotless i. Ordinary Indonesian prose is one byte per character all the way through.

That is a rarer property than it sounds. Vietnamese uses the same alphabet with stacked tone and vowel diacritics and pays two or three bytes for many of its vowels; the Vietnamese page works through what that costs. German pays two bytes for every umlaut. Turkish pays two for every dotless i. Indonesian pays nothing, and it gets the further benefit that its bytes are the same bytes English uses, so English merges partially apply to it — a shared prefix like ing or an was learned from English text and fires on Indonesian words for free.

Axis two: almost nothing inflects

Indonesian has no grammatical case, no gender, no number agreement, and no verb conjugation for person or tense. A noun does not change shape. A verb does not change shape for who did it or when. Tense is carried by separate adverbs — sudah, akan, sedang — which are frequent independent words and therefore precisely the kind of thing BPE gives a single token.

Compare what that saves. A Finnish noun has on the order of a thousand written surface forms once case, number and possessive suffixes combine. A vocabulary of any size can hold a merge for the handful most frequent and must reconstruct the rest from fragments. An Indonesian noun has one form. Every occurrence is the same bytes, so the frequency of that one form is the sum of what would have been spread across a thousand forms, and it wins a merge slot easily.

Indonesian is not entirely free, and the two places it is not are worth naming because they are where the ratio comes from:

• Affixation. Derivational affixes are productive and written closed — meN-, peN-, ber-, ter-, -kan, -i, -an, and circumfixes like ke-...-an. A derived word such as ketidakadilan is long and infrequent, so it splits. Note this is derivation, not inflection: it creates a new dictionary word rather than a grammatical variant, so the number of such forms is bounded by the lexicon rather than multiplied by a paradigm.
• Reduplication. Plurality and intensity are marked by repeating the word with a hyphen: buku-buku, jalan-jalan. That doubles the character count of the word and usually more than doubles its token count, because the hyphenated form is rarer than the bare one.

Deriving the small penalty that remains

Assumptions: English at about four characters per token, per OpenAI’s published rule of thumb. Indonesian at one byte per character, from the alphabet. Indonesian merge efficiency of 3.2 to 3.6 bytes per token — below English’s four because Indonesian has a smaller share of the training corpus and because affixed and reduplicated forms split, but well above any non-Latin script.

At m of 3.4, Indonesian is 1 divided by 3.4, or 0.294 tokens per character, against English’s 0.25. That is a per-character ratio of about 1.18x. Indonesian also tends to write the same content in slightly more characters than English, largely because of affixation and the separate tense adverbs; assume 1.1 characters per English character and the per-meaning ratio comes out near 1.3x.

Why English is the denominator at all

It is worth being explicit that English is not the baseline because it is special. It is the baseline because it is the plurality language in every large web corpus, which means BPE spends the largest share of its merge budget on English byte sequences, which means English gets the highest bytes-per-token figure, which means it looks free. The baseline is an artefact of corpus composition, and a tokenizer trained on a differently balanced corpus would move it.

Two research groups have documented the spread this produces across languages. Petrov and colleagues at “Language Model Tokenizers Introduce Unfairness Between Languages” (2023) measure tokenization ratios across a large multilingual parallel corpus and find differences exceeding an order of magnitude between the best- and worst-served languages. Ahia and colleagues, in “Do All Languages Cost the Same?” (2023), connect the same measurements directly to per-token API pricing. Both are worth reading before quoting anyone’s multiplier, including this page’s.

The practical upshot for Indonesian specifically: at around 1.3x, you can size context windows, chunk lengths and rate limits with English intuitions and a small safety margin. That is not true anywhere else in this cluster, and it is why Swahili — another Latin, low-diacritic language with a very different corpus share — is the useful next page to read.