Why Hebrew Text Costs More Tokens Than English

Hebrew belongs in this cluster, but it is the language here with the strongest claim to being cheap. It omits vowels, attaches prepositions to words, and fits into two bytes a letter. The cost arrives from three specific places, and two of them are avoidable.

An abjad is compact before it is expensive

Hebrew is an abjad: the letters are consonants, and the vowels are either omitted entirely or implied by a small set of letters doing double duty. Ordinary written Hebrew — a newspaper, a website, an email — has no vowel marks at all. A reader supplies them from context.

This makes Hebrew orthography dense in a way that works against the usual argument in this cluster. Where English writes seven characters for “memory”, Hebrew writes זיכרון in six. Where English needs three words for “and in the house”, Hebrew writes ובבית — one orthographic word, five letters, because the conjunction ו, the preposition ב and the definite article all attach to the front of the stem. Per unit of meaning, unvocalised Hebrew uses fewer characters than English.

Against that, the Hebrew block U+0590–U+05FF sits in the two-byte UTF-8 range, so each of those characters costs twice what an ASCII character does, and Hebrew’s share of any training corpus is small. The two effects partly cancel, which is why the derived multiplier below is the mildest in this cluster.

The arithmetic on a labelled sentence

פונקציה זו משתמשת בהרבה זיכרון.
"This function uses a lot of memory."

Twenty-six Hebrew letters, four spaces, one full stop. Two bytes per letter gives 52, plus 5 bytes of ASCII: exactly 57 bytes. The English is 35 bytes, near 9 tokens. The byte floor caps the Hebrew at 57 tokens.

Derive the middle, assumption named. Assume merges exist for the frequent Hebrew letter pairs and for the very common prefix clitics, so a five- or six-letter word costs two or three tokens rather than five or six. Five words of that shape gives roughly 11 to 18 tokens against the English 9 — a derived multiplier of about 1.2× to 2×. That is a modest penalty, and it is the correct baseline for clean modern Hebrew.

Everything that makes Hebrew expensive in practice is a deviation from that baseline, and the two below are the deviations worth knowing.

Five letters with two code points each

Five Hebrew consonants change shape when they end a word: כ becomes ך, מ becomes ם, נ becomes ן, פ becomes ף, צ becomes ץ. Unlike Arabic’s contextual forms, which are a rendering matter with one code point per letter, the Hebrew final forms are separate code points in their own right — U+05DA, U+05DD, U+05DF, U+05E3 and U+05E5 — and they are the correct, standard way to write those letters at the end of a word.

For a merge table this is a quiet tax. The word-final position is exactly where a tokenizer would like to learn strong merges, because word endings are grammatically regular and highly repetitive. Hebrew splits five of its twenty-two consonants into two distinct code points depending on position, so any merge that ends in one of them has to be learned separately from the merge that ends in its non-final twin. The merges are not shared, and the same morpheme — the plural ending ים, for instance — is a byte sequence unrelated to anything containing מ.

It also breaks naive string operations: a substring search for a stem written with מ will not find the same stem written with ם at a word boundary. This is a legitimate orthographic distinction, not a normalisation error, so the fix is a Hebrew-aware matcher rather than a Unicode transformation.

What niqqud adds

Niqqud are the vowel points: a system of dots and dashes below, inside and above the consonants, encoded as combining marks in U+05B0–U+05C7. Each is a separate two-byte code point placed after the consonant it modifies.

Fully vocalised Hebrew carries roughly one point per consonant and sometimes two, so the arithmetic is direct: adding about 25 marks to the 26-letter sentence above roughly doubles it in bytes, from 57 to around 107. The token count rises further than the byte count does, for the same reason it does in vocalised Arabic: an unfamiliar combining code point inserted between two consonants destroys the merge that would have joined them, so a two-letter pair that was one token becomes three or four. That mechanism is identical across the two languages and is derived in full on the Arabic page rather than repeated here.

Where Hebrew differs from Arabic is in which corpora are vocalised. Vocalised Hebrew is not a stylistic choice; it is essentially confined to liturgical and biblical text, poetry, children’s books, dictionaries and language teaching. If your corpus is any of those, you are on the doubled figure, and if it is not, you are on the mild one. There is very little in between, which makes it worth checking rather than assuming.

Measuring it, and what to normalise

import unicodedata
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def strip_niqqud(s):
    return "".join(ch for ch in s
                   if not (0x0591 <= ord(ch) <= 0x05C7))

he = "פונקציה זו משתמשת בהרבה זיכרון."

for label, s in (("as given", he), ("niqqud stripped", strip_niqqud(he))):
    print(label,
          "codepoints", len(s),
          "bytes", len(s.encode("utf-8")),
          "tokens", len(enc.encode(s)))

marks = sum(1 for ch in he if unicodedata.category(ch).startswith("M"))
print("combining marks", marks)

If the two rows are identical your text is unvocalised and the mild multiplier applies. If they differ by roughly a factor of two, your corpus carries niqqud and you should decide deliberately whether to index it. The range U+0591–U+05AF also covers the cantillation marks used in biblical text, which are denser still than niqqud and even less represented in training corpora.

• Strip niqqud for retrieval, keep it for display. Queries are typed without vowel points, so an index that keeps them will not match.
• Do not normalise final forms away. They are correct Hebrew, and replacing them changes the text. Handle them in the matcher instead.
• Geresh and gershayim are not quotes. Hebrew acronyms use U+05F3 and U+05F4, which look like apostrophes and are different code points; a quote-normalisation pass will corrupt them.
• Budget accordingly. On the derived range, a 4,000-token chunk holds something in the order of 5,000–9,000 Hebrew letters unvocalised, and roughly half that vocalised.