Why Zulu Text Costs More Tokens Than English

Every other page in this cluster spends most of its argument on bytes. This one cannot: Zulu is written in unadorned ASCII Latin letters, so a Zulu sentence and its English translation take almost exactly the same number of bytes. The multiplier is still real, and everything that produces it is in the shape of the words.

The script costs nothing

Standard Zulu orthography uses the twenty-six Latin letters and no diacritics. There is no tone marking in the standard orthography even though Zulu is tonal; the clicks are written with the plain letters c, q and x and their digraphs; long vowels are not marked. Every character is one byte.

That removes the mechanism the rest of this cluster relies on. A Zulu document has no three-byte code points to pay for, no combining marks, no virama, no block-disjointness from a better-resourced neighbour. Whatever multiplier remains has to come from somewhere else, which makes Zulu the clean case for showing that a script penalty and a language penalty are genuinely independent — the same point Azerbaijani makes against Kazakh, here in its strongest form.

Deriving the multiplier

Zulu    : Izingane zifunda izincwadi esikoleni.
English : The children read books at school.

Zulu    : 37 ASCII characters = 37 bytes,  4 orthographic words
English : 34 ASCII characters = 34 bytes,  6 orthographic words

The byte-fallback ceiling for Zulu here is 37 tokens; for English, roughly nine at four characters per token. That would be a 4.1x worst case, but for once the ceiling is not the useful figure, because a vocabulary trained on any amount of English text has thousands of merges that fire on ASCII letter sequences regardless of what language produced them. Zulu will never come near byte fallback.

The realistic derivation goes the other way, from word structure. Each Zulu word above is a stem with one or more affixes, and a vocabulary with little Zulu in it will split each into pieces that correspond to English fragments rather than to Zulu morphemes — something in the region of three to five pieces per word for four words, against six English words that are mostly one token each. That puts the derived estimate around two to three times English, with the assumption being pieces-per-word rather than bytes-per-token. It is a derivation from structure and it is stated as an estimate, not a measurement.

Two to three times is modest next to the eight and nine times derived for the Indic pages in this cluster, and it is worth saying so plainly: Zulu is not the expensive case. It is the instructive one, because the entire multiplier survives the removal of the script.

Noun classes and the concord chain

Zulu nouns belong to one of about fifteen classes, and the class is marked by a prefix on the noun itself. umuntu is a person, abantu is people; isikole is a school, izikole is schools; incwadi is a book, izincwadi is books. The stem is the short part and the prefix carries number and class.

The prefix does not stay on the noun. Zulu grammar requires concord: every word that agrees with a noun echoes its class marker. An adjective, a possessive, a relative clause and the verb all carry a form of the same prefix, so a single sentence repeats the class marker three or four times across different words.

Izingane ezinkulu zifunda izincwadi.
  izi-ngane      class 8/10 noun prefix + stem   "children"
  ezi-n-kulu     concord + stem                  "big"
  zi-funda       concord + verb stem             "read"
  izi-n-cwadi    class prefix + stem             "books"

For a tokenizer this is a specific kind of bad luck. The morpheme doing the grammatical work is a two- or three-letter prefix, appearing in a different word each time, glued to a different stem each time. Byte-pair merges are learned on frequent contiguous sequences; a prefix that is frequent but always followed by something different produces a large number of moderately frequent bigrams rather than one very frequent one. That is precisely the distribution BPE handles least efficiently, and it is discussed in general terms on the agglutinative languages page.

Why one Zulu word is a whole English clause

Zulu uses conjunctive orthography: the subject marker, tense marker, object marker and verb stem are written as a single orthographic word. The most-quoted example is the ordinary way to say “I love you”:

ngiyakuthanda
  ngi-    subject marker, 1st person singular   "I"
  ya-     present tense marker
  ku-     object marker, 2nd person singular    "you"
  thanda  verb stem                             "love"

Thirteen characters, one word, four English words. Sotho and Tswana, which are closely related, write the same construction disjunctively as separate words — so this is a spelling convention rather than a fact about the grammar, and it has a direct cost consequence. The conjunctive convention produces a very large inventory of distinct surface forms, because every combination of subject, tense and object is a different string.

The practical effect is a mismatch between two intuitions that usually agree. Zulu sentences have fewer words than their English translations, which suggests they should be cheaper. Zulu words are split into far more pieces than English words, which more than cancels it. Anyone estimating cost from word count — and word count is the estimate people reach for when the script looks familiar — will underestimate Zulu badly.

Swahili is the useful contrast within Bantu: the same noun-class system and the same conjunctive verb, but far more text in public corpora, which buys it merges Zulu does not have. The mechanism is shared and the outcome is not, which is the training-share argument again with the grammar held constant.

Measuring it on your own text

The claim to test is that the splits ignore morpheme boundaries. Print the pieces and compare them with the morphology.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

words = ["ngiyakuthanda", "izingane", "izincwadi", "esikoleni", "abantwana"]

for w in words:
    pieces = [enc.decode([t]) for t in enc.encode(w)]
    print(f"{w:16s} {len(pieces)} tokens  {pieces}")

zu = "Izingane zifunda izincwadi esikoleni."
en = "The children read books at school."
print("zulu   ", len(enc.encode(zu)), "tokens for", len(zu.split()), "words")
print("english", len(enc.encode(en)), "tokens for", len(en.split()), "words")

If ngiyakuthanda comes back as something like four pieces that happen to align with ngi, ya, ku and thanda, the vocabulary has learned real Zulu structure. It is much more likely to come back split at English-looking boundaries — fragments that are common in English words and mean nothing in Zulu. That is the cost, and it is also why short Zulu queries embed poorly: the pieces the model sees are not the units the language is built from.