Why Swahili Text Costs Fewer Tokens Than Most African Languages

Swahili is the useful control case for this entire subject. It is a major African language with a modest presence in training corpora, so you would expect a large multiplier — and it does not have one, because standard Swahili orthography uses nothing but the twenty-six ASCII letters. Removing the script variable leaves morphology, and morphology alone is measurable.

One byte per letter, no exceptions

Standard Swahili as written in Tanzania and Kenya uses the basic Latin alphabet with no diacritics at all. There is no ą, no ł, no circumflex, no tone mark. The digraphs ch, dh, gh, kh, ng’, ny, sh and th are written as two ASCII letters. The one near-exception is the apostrophe in ng’, which represents the velar nasal and which should be a straight ASCII apostrophe or a U+02BC modifier letter depending on your house style — a distinction that matters for indexing, not for byte count.

So every character in a Swahili document is one UTF-8 byte, exactly like English. Whatever multiplier Swahili has, none of it comes from the encoding. That makes it the one language in this cluster where the derivation can attribute the entire penalty to a single cause.

Swahili also carries a large stock of Arabic and English loanwords written in that same ASCII alphabet, and those tokenize unusually well: kitabu, safari, hoteli and kompyuta are all byte sequences a vocabulary has seen before, either directly or as familiar-looking Latin fragments. That is a real and slightly unusual advantage. It also has a limit worth stating, because it is the source of a common misreading: cheap tokens on loanwords do not mean the model understands the sentence around them. The one historical exception to the Latin-script rule is Ajami, the Arabic-script tradition used for Swahili before colonisation and still found in manuscripts; anything from that corpus is Arabic-script text and behaves like Arabic, not like this page.

A verb that is a whole English clause

Swahili is a Bantu language with an agglutinative verb and a noun-class system with roughly fifteen classes, each triggering its own agreement prefixes on verbs, adjectives and possessives. The Swahili verb is built as an ordered chain of slots: subject prefix, negation, tense, object prefix, root, extension, final vowel.

Hawatakusaidia is one word and it means “they will not help you”. It decomposes as ha- (negative) + wa- (third person plural subject) + ta- (future) + ku- (second person singular object) + saidia (help). Five morphemes, one orthographic word, fourteen characters. English spends four words on the same content, and three of those four are among the most frequent tokens in any English vocabulary.

This inverts the usual arithmetic in an interesting way. Because the prefixes are short and highly regular, Swahili sentences are oftenshorter in characters than their English translations. The cost is not length. The cost is that each combination of prefixes produces a distinct surface form, so the number of distinct Swahili word forms is very large and the frequency of each is low, and BPE buys merges by frequency. Zulu takes the same mechanism further with longer prefix chains and conjunctive orthography; see Zulu.

Deriving the multiplier

The sentence is Hawatakusaidia kesho kwa sababu wamechoka. — “They will not help you tomorrow because they are tired.”

Swahili
  characters ............. 42
  UTF-8 bytes ............ 42   (pure ASCII, 1 byte each)
  orthographic words ..... 5

English "They will not help you tomorrow because they are tired."
  characters ............. 55
  UTF-8 bytes ............ 55
  tokens (assumption: ~4 chars/token for English) ..... ~14

  Note the direction: the Swahili is 24% SHORTER in bytes.

Ceiling (1 token per byte -- unreachable for ASCII, quoted to bound it)
  42 / 14 = 3.0x English

Band    (assume Swahili resolves at 2.5 bytes/token, well below
         English's ~4, because agglutinated forms are individually rare)
  42 / 2.5 = 17 tokens  ->  17 / 14 = 1.2x English

Band    (assume 2 bytes/token, a vocabulary with almost no Swahili merges)
  42 / 2 = 21 tokens  ->  21 / 14 = 1.5x English

Derived, not measured. The four-characters-per-token figure used for the English baseline is OpenAI’s own published rule of thumb, set out in its help article on counting tokens, and it is a heuristic with perhaps twenty percent of slack in it. Roughly 1.2 to 1.5 times English is a small number by the standards of this cluster, and it is small for a precise reason: shorter text at a worse tokens-per-character rate nearly cancels. The measurement that isolates morphology is to tokenize the verb chain against its own morphemes:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

# Same five morphemes, packed as Swahili writes them and then
# separated. Byte count is identical apart from the added spaces,
# so any token difference is the merge table's opinion of the
# combined surface form.
packed = "hawatakusaidia"
split  = "ha wa ta ku saidia"

for label, s in (("packed", packed), ("split", split)):
    ids = enc.encode(s)
    print(label, len(s), "bytes", len(ids), "tokens",
          [enc.decode_single_token_bytes(i).decode("utf-8", "replace")
           for i in ids])

Printing the fragments is the point. You will typically see the packed form broken at boundaries that are not morpheme boundaries, which is the specific reason short Swahili queries embed poorly: the fragments a retrieval model sees do not correspond to any unit of meaning.

Why this is cheaper than Amharic or Tigrinya

Grouping “African languages” into one token-cost bucket is the mistake this page exists to prevent. The variable that dominates is script, not continent, and it is worth stating the comparison numerically rather than qualitatively.

• Swahili, Yoruba, Hausa, Zulu: Latin script, one to two bytes per character. Yoruba pays a little more than Swahili because its tone marks and subdot vowels are combining or precomposed non-ASCII characters; Swahili pays nothing.
• Amharic and Tigrinya: Ge’ez syllabary, three bytes per character, and a 275-symbol grid to cover before any merges are bought. See Amharic.
• Egyptian and Maghrebi Arabic: Arabic script, two bytes, with dialect forms that the vocabulary’s Modern Standard Arabic merges do not cover.

The token multiplier for Swahili is close to English. The quality gap is not, and confusing the two leads to bad planning; see embedding quality for African languages and what Swahili support actually means.

Where the remaining cost shows up

Because the multiplier is low, the practical problems with Swahili are not budget problems. They are segmentation and retrieval problems.

• Keyword search fails on the verb. A user searching for “saidia” will not match hawatakusaidia under a whole-word index, and stemming rules written for English strip suffixes when Swahili carries its grammar as prefixes.
• Short queries are the worst case. A two-word Swahili query is two rare surface forms with no context to disambiguate the fragments, which is where the retrieval quality gap is widest.
• Do not apply an African-languages contingency to the budget. A 1.2x planning factor is right for Swahili. A 3x factor would over-provision by more than double and would push you toward a smaller context than you need to buy.
• Normalise the ng’ apostrophe on ingest. U+2019, U+02BC and ASCII apostrophe are three different byte sequences, so the same word indexes three ways and none of the three matches the others.