Why Agglutinative Languages Break Word-Based NLP Assumptions

In English, a word is roughly a concept and a sentence is roughly a proposition. Almost every piece of text tooling assumes that ratio somewhere. In Turkish or Finnish a single orthographic word can carry a subject, a possessor, a case, a negation, a tense and a question, and every assumption built on the ratio fails at once.

One word, morpheme by morpheme

Take the Turkish word evlerinizden. It is one word by every mechanical definition — no spaces, one token to a whitespace splitter, one entry in a word count:

ev      house          root
-ler    plural         "houses"
-iniz   your (plural)  "your houses"
-den    ablative case  "from your houses"

Four English words, one Turkish word. Push further and it keeps going: evlerinizdeymişsiniz adds a locative, an evidential past and a second-person plural agreement, landing at roughly “apparently you were at your houses”. Finnish does the same thing with a case system of fifteen cases plus possessive and clitic suffixes: taloissanikin decomposes as talo (house) + -i- (plural) + -ssa (inessive, “in”) + -ni (my) + -kin (also) — “in my houses too”.

The suffixes are also not free-standing strings you can search for. Turkish vowel harmony changes their vowels to agree with the stem, so the ablative is -den, -dan, -ten or -tan depending on the preceding vowels and consonants. There is no substring that reliably means “ablative”.

It is worth naming what agglutination is not, because the distinction decides whether standard tooling helps. German compounds — the long nouns everyone quotes — are concatenations of independent words, and splitting them is a dictionary lookup over word-sized pieces that mostly works. Fusional inflection, as in Latin or Russian, packs several grammatical features into a single unsegmentable ending, so the number of forms per word is large but bounded and can be tabulated. Agglutination is the third case: each suffix carries one feature, suffixes stack freely, and the number of forms is unbounded. Only the third defeats enumeration, which is why Turkish and German break tooling in different ways despite both being famous for long words.

The assumption that breaks

The broken assumption has a precise name: one orthographic word corresponds to roughly one lexical unit. It is rarely stated, because in English it is close enough to true that nobody notices they depend on it. Once it fails, so does everything derived from it.

• Word count as a proxy for content. A 100-word Turkish document and a 100-word English document do not contain comparable amounts of information. Anything that budgets, truncates or sizes by words is now measuring a different quantity per language.
• Stopword lists. English function words are free-standing and can be removed by lookup. Turkish equivalents are suffixes welded to content words, so there is nothing to remove — and removing whole words instead deletes the content with the function.
• Exact-match retrieval. A user searching for ev will not lexically match evlerinizden, even though the document is exactly about houses. Keyword search degrades from “imperfect” to “mostly returns nothing”.
• Stemming and lemmatisation. A suffix-stripping stemmer of the Porter kind assumes a small closed set of suffixes in a fixed order. Agglutination gives an open set in a productive order, so stemming becomes full morphological analysis — a different and much larger piece of software.

Why the vocabulary cannot be enumerated

This is the mechanism underneath all of it. If a language has a few thousand roots and each root can take a sequence of suffixes chosen from several slots, the number of possible word forms is the product of the options, not the sum. Turkish and Finnish word forms per root run into the thousands; Hungarian and Korean behave similarly; the agglutinating Bantu languages do the same thing with prefixes as well as suffixes.

A fixed vocabulary therefore cannot cover the language. It does not matter how large you make it — the tail of legitimate, ordinary, fully grammatical word forms extends past any list. This is exactly why subword tokenization exists, and it is also why subword tokenization costs these languages more tokens per unit of meaning: the model is reassembling a word from pieces every time, and the pieces are chosen by frequency in a corpus dominated by other languages. The token-cost consequence is worked through on the Turkish token cost and Finnish token cost pages.

What breaks downstream

Two failures are worth calling out because they are hard to attribute when you meet them in production.

Short queries embed badly. An embedding model sees a two-word Turkish query as a handful of subword fragments with little context. Because the morphology is where the meaning lives and the morphology has been shredded into frequency-chosen pieces, semantically distant queries can land close together in vector space. The effect is much weaker for long documents, where surrounding text supplies context, which is why retrieval quality can look fine in offline evaluation over documents and poor in production over queries.

Chunk boundaries land inside words. A character-based splitter with a fixed window will cut evlerinizden in half, and the two halves are not two things — they are one thing broken. The cure is the same as for CJK: size chunks in tokens and split on linguistic boundaries, as described in sizing CJK chunks in tokens rather than characters, with sentence boundaries as the split points rather than a raw character offset.

Designing around it

Three decisions carry most of the benefit, and none of them requires a per-language pipeline.

• Measure in tokens, everywhere. Budgets, truncation, chunk sizes, cost estimates and length limits should all be expressed in the tokenizer’s units. Word and character counts silently mean different things per language; tokens at least mean the same thing to the model.
• Prefer dense retrieval, and if you keep lexical search, use a morphological analyser. Elasticsearch and its relatives ship Turkish and Finnish analysers that do real stemming. Using the default analyser on agglutinative text is the single most common cause of “search works in English and not in Turkish”.
• Do not translate prompts word for word. Instructions like “answer in three words” or “extract each noun phrase” carry assumptions about word granularity that do not transfer. Write the constraint in terms the target language can satisfy — a sentence, a clause, a named entity — as writing prompts natively argues at length.