Why Yoruba Text Costs More Tokens Because of Tone Marks

Yoruba is written in the Latin alphabet, which ought to make it cheap. It is not, and the reason is unusually specific: the marks Yoruba needs are a combination Unicode never gave a single code point to, so each marked vowel arrives as two characters and five bytes.

What a Yoruba vowel is made of

Yoruba orthography carries two independent layers of diacritic, and they mean different things. A subscript dot distinguishes vowel quality and one consonant: ẹ and ọ are separate letters from e and o, and ṣ is a separate letter from s. On top of that, an acute or grave accent marks high or low tone, with mid tone left unmarked. Both layers can land on the same vowel, and frequently do.

Unicode handles the first layer with precomposed characters in the Latin Extended Additional block: ẹ is U+1EB9, ọ is U+1ECD, ṣ is U+1E63, all charted at U+1E00 Latin Extended Additional. Those are above U+07FF, so each is three bytes in UTF-8 — already three times an ASCII letter before any tone is written.

For the second layer there is no precomposed form at all. Unicode has no character meaning “e with dot below and acute”. Normalisation form C composes what it can and then stops, so ẹ́ normalises to U+1EB9 followed by the combining acute U+0301, which is itself two bytes. One vowel, two code points, five bytes.

Deriving the multiplier

Yoruba  : Mo fẹ́ràn oúnjẹ Yorùbá.
English : I like Yoruba food.

Yoruba, byte by byte:
  M o ␣                       3 × 1 = 3
  f                                 = 1
  ẹ  U+1EB9                         = 3
  ́  U+0301  (combining acute)      = 2
  r                                 = 1
  à  U+00E0                         = 2
  n ␣                         2 × 1 = 2
  o                                 = 1
  ú  U+00FA                         = 2
  n j                         2 × 1 = 2
  ẹ  U+1EB9                         = 3
  ␣ Y o r                     4 × 1 = 4
  ù  U+00F9                         = 2
  b                                 = 1
  á  U+00E1                         = 2
  .                                 = 1
                              total = 32 bytes  (23 code points)

English : 19 ASCII characters = 19 bytes ≈ 5 tokens

Twenty-two visible letters, twenty-three code points, thirty-two bytes. The byte-fallback ceiling of 32 tokens against roughly five for the English gloss gives a derived worst case of 6.4x — from a Latin-script language, which is the surprising part. The floor, if the four Yoruba words merged perfectly, would be under one. As always in this cluster these are computed from byte counts and not from a tokenizer run.

The word fẹ́ràn is the one to look at. Five visible letters, six code points, ten bytes. An English word of five letters is five bytes and usually one token. Yoruba doubles the byte count of its vocabulary at the encoding layer, and then a vocabulary with little Yoruba in it cannot merge across the diacritics, so the ASCII runs either side of each mark are cut into separate pieces.

That last point is the specific damage. A tokenizer holds excellent merges for English letter sequences, and Yoruba words are made of those same letters — but every marked vowel is a wall the merge cannot cross. fẹ́ràn cannot be reached by any merge that would have handled feran, because the bytes in the middle are not the bytes those merges were learned on.

Why Vietnamese gets a better deal

Vietnamese needs the same thing Yoruba needs: a vowel quality mark and a tone mark on the same letter. Unicode gave Vietnamese precomposed characters for the full set. ệ is U+1EC7, a single code point meaning “e with circumflex and dot below”, and the Vietnamese tone-plus-quality combinations occupy a large contiguous run of U+1EA0–U+1EF9 for exactly this purpose.

Vietnamese  ệ   U+1EC7               1 code point,  3 bytes
Yoruba      ẹ́   U+1EB9 + U+0301      2 code points, 5 bytes

Same amount of linguistic information, two-thirds more bytes, and one extra code point per marked vowel for every occurrence in every document. The asymmetry is a historical accident of which national character sets were submitted for encoding in the early years of Unicode, not a judgement about the languages, but it is permanent and it is paid on every request. The Vietnamese page covers the other half of the comparison, where Vietnamese still ends up expensive for reasons of its own.

There is a compounding factor. Vietnamese has a far larger web presence than Yoruba, so a vocabulary has seen the U+1EA0 block often enough to learn merges over it. Yoruba’s combination of three bytes for the base and two for the combining mark has no such support, and the situation is discussed more broadly on the Yoruba language support page.

The tempting fix that loses meaning

Because most of the cost is in the marks, stripping them roughly halves the byte count and improves tokenization dramatically. A great deal of Yoruba on the web is already written this way, because keyboards are inconvenient. It is still the wrong thing to do to text you are processing, and the reason is that Yoruba tone is lexical rather than prosodic: the marks are not emphasis, they are part of the word.

• owó is money.
• ọwọ́ is hand.
• ọwọ̀ is respect.

Strip the marks and all three become the same string. A model asked to work with the stripped form has to recover the distinction from context, and often cannot; a retrieval index built on stripped text conflates the three; a diacritic-insensitive search over Yoruba is not a convenience feature, it is a semantic merge of unrelated words.

The defensible version of the optimisation is to strip for matching and keep the marked form as the stored and displayed text — the same discipline that normalisation form choice requires, where NFKC will happily fold distinctions you needed. Never strip on the way into a prompt, and never strip on the way into storage.

Measuring it on your own text

Two things worth checking: that your text is actually in NFC, and what the marks cost you.

import tiktoken, unicodedata

enc = tiktoken.get_encoding("o200k_base")

def strip_marks(s):
    d = unicodedata.normalize("NFD", s)
    return "".join(c for c in d if not unicodedata.combining(c))

for word in ["fẹ́ràn", "ọwọ́", "owó", "ilé-ìwé", "Yorùbá"]:
    w = unicodedata.normalize("NFC", word)
    bare = strip_marks(w)
    print(f"{w:10s} {len(w):2d} cp  {len(w.encode('utf-8')):3d} B  "
          f"{len(enc.encode(w)):2d} tok   |   "
          f"stripped {bare:10s} {len(enc.encode(bare)):2d} tok")

Note that strip_marks above removes the combining acute and also decomposes and removes the dot below, so it destroys the vowel quality distinction as well as the tone — which is exactly why it is shown here as a measurement tool and not as a preprocessing step. The gap between the two token counts is what the orthography costs you, and it is the number to weigh against the three meanings of ọwọ.