Why Compound Words Make German Text Cost More Tokens

German writes with the letters English writes with, sits high in every training corpus, and still costs meaningfully more per sentence. That combination makes it the cleanest available demonstration that token cost is a morphology problem as much as a script problem.

The script is almost free

German orthography is ASCII plus five characters: ä, ö, ü, their capitals, and ß. Those live in the Latin-1 Supplement block, which UTF-8 encodes in two bytes rather than one. Umlauts are common but not dense — a typical German paragraph is well over 95 per cent one-byte characters — so the script axis contributes a few per cent, not a multiplier.

There is one script-adjacent trap worth knowing before you measure anything: ä can be a single codepoint (U+00E4, two bytes) or a base a followed by a combining diaeresis (U+0061 U+0308, three bytes). Those are different byte sequences and therefore different token sequences for the same word. Normalise to NFC before you count, or your German corpus will appear to cost more than it does — see umlaut and eszett normalisation for the full treatment.

With the script accounted for, everything that remains is morphology.

Closed compounds have no boundary to split on

German writes compound nouns closed. English writes “speed limit”, two whitespace-delimited words, each frequent, each almost certainly a single token. German writes Geschwindigkeitsbegrenzung: one whitespace-delimited word of twenty-five characters.

This matters because of how BPE merges are actually learned. The merges that survive into a vocabulary are dominated by frequent whitespace-bounded units, typically including the leading space, which is why the space-prefixed form of a word is a different token from the bare form. Compounding is productive: German speakers create compounds freely, and the great majority of the compounds in any real corpus appear a handful of times. A word that appears a handful of times earns no merge.

So the compound gets assembled from whatever fragments the vocabulary does contain, and each fragment is a token. The parts are frequent — Geschwindigkeit and Begrenzung are both common German words — but the compound is not, and BPE has no mechanism that says “this long word is two known words stuck together”. It only has byte-pair merges, applied greedily to whatever the frequency table happens to contain.

Add the linking morphemes. German inserts a Fugen-s, or an -en, or an -er between compound elements according to rules with many exceptions: Geschwindigkeit plus Begrenzung becomes Geschwindigkeits-begrenzung. That inserted s means the first element’s bytes are not the bytes of the standalone word, so even the merge for the standalone word does not fire. The linking morpheme costs a token by itself and destroys the merge on either side of it.

BPE cuts in the wrong places

The splits BPE produces on a long compound are not the morpheme boundaries a German speaker would draw. Merges are chosen by frequency over the whole corpus, so a compound tends to be cut into whatever high-frequency byte sequences happen to align, which frequently spans a morpheme boundary in the middle of a fragment.

This has two consequences beyond cost, and they are the ones that bite in production. Embeddings for the compound are built from fragments that do not correspond to units of meaning, so nearest-neighbour retrieval on German compounds behaves worse than the token count alone predicts. And constrained generation — anything that forces the model to produce an exact string, such as a schema field name or an enumerated value — has to navigate a fragmentation the model did not see often during training, which is where German enum values tend to come back subtly misspelled.

Every noun is capitalised, and that costs

German capitalises all nouns, not only proper ones. A tokenizer has no notion of case-insensitivity — haus, Haus, HAUS and the space-prefixed variants of each are all distinct byte sequences competing separately for merge slots.

For English this costs little, because a common noun appears capitalised only at the start of a sentence and the lowercase form takes nearly all the frequency. For German the noun’s frequency is genuinely split: the capitalised form dominates in running text, but the lowercase form appears as the tail of every compound it participates in. Both need merge slots. The vocabulary budget spent on German is effectively serving twice as many distinct forms as the language has words.

Deriving the multiplier

Assumptions: English at four characters per token, which is the rule of thumb OpenAI publishes in its own documentation on counting tokens. German at 1.03 bytes per character, allowing for umlaut density. German merge efficiency of 2.6 to 3.2 bytes per token — well below English’s four, because of compounds and the capitalisation split, but far above any non-Latin script, because German is one of the best-represented languages in every web corpus. And a length assumption: German writes the same content in roughly 1.15 characters per English character.

At m of 2.9, German is 1.03 divided by 2.9, or 0.355 tokens per character, against English’s 0.25 — a per-character ratio of about 1.42x. Multiply by the 1.15 length factor and the per-meaning ratio is about 1.63x. A 200-character English sentence at 50 tokens becomes roughly 82 tokens in German.

That average understates the tail badly, and the tail is what breaks things. Legal, administrative and insurance German is compound-dense in a way that ordinary prose is not: Arbeitsunfähigkeitsbescheinigung is thirty-two characters that no vocabulary contains and that will fragment heavily. If your German corpus is contracts or claims rather than marketing copy, derive your own figure rather than using this one.

import tiktoken, unicodedata

enc = tiktoken.get_encoding("o200k_base")
text = unicodedata.normalize("NFC", open("corpus.de.txt", encoding="utf-8").read())

# where does the cost concentrate? rank words by tokens-per-character
worst = {}
for w in set(text.split()):
    if len(w) >= 12:
        worst[w] = len(enc.encode(" " + w)) / len(w)

for w, r in sorted(worst.items(), key=lambda kv: -kv[1])[:25]:
    print(f"{r:5.2f}  {w}")

That listing is the useful artefact. It names the specific compounds your corpus pays for, and in most German document sets a few dozen recurring administrative compounds account for a disproportionate share of the overhead. The same technique applied to Finnish finds a structurally similar tail for a completely different morphological reason.