Stemming and Lemmatisation: Do They Still Matter?

Both reduce a word to a canonical form. One does it by chopping suffixes with rules and does not care whether the result is a word; the other looks the word up and returns a real one. The interesting question is not how they differ but whether either is worth running, and the honest answer depends entirely on the language.

The difference, in one example

Stemming is a rule-based suffix chopper. Porter’s algorithm (1980) applies about sixty ordered rewrite rules and produces a stem, which is frequently not a word: studies becomes studi, argument becomes argument, and operational, operating and operates all become oper. It runs in microseconds, needs no dictionary, and knows nothing about the sentence.

Lemmatisation looks the word up in a lexicon and returns the dictionary form, the lemma. That requires knowing the part of speech, which is why it is slower and why it is wrong when you do not supply one. The classic demonstration is that WordNet’s lemmatiser, asked for the lemma of meeting with no part of speech supplied, defaults to noun and returns meeting; told it is a verb, it returns meet. Both answers are correct for different sentences, and a pipeline that never passes the tag gets the noun reading every time.

A second worked case: better. A stemmer returns better, because there is no suffix to remove. A lemmatiser told it is an adjective returns good. If your goal is that a query for good retrieves a document containing better, only one of these tools can help, and it costs a tagger to use it.

How Porter’s algorithm fails

Stemmers fail in two directions, and both have names. Overstemming is conflating words that should stay apart: the much-cited example is that Porter maps both university and universe to univers, so a search for one retrieves the other. Understemming is failing to conflate words that belong together, which is why irregular morphology — ran and run, mice and mouse — passes straight through untouched.

Neither failure is a bug to be fixed. They are the price of a dictionary-free algorithm, and Porter himself later produced Snowball (also called Porter2) with corrections and a framework for other languages rather than claiming the problem away. The practical consequence is that stemming should never be applied to a field where a false conflation is expensive — product codes, surnames, chemical names, ticker symbols. Index those exactly, in their own field.

What the retrieval literature found

This question was studied properly long before anyone was arguing about it on the web. Harman’s How effective is suffixing? (Journal of the American Society for Information Science, 1991) tested three stemmers on standard English test collections and found no reliable improvement in average retrieval performance — individual queries moved in both directions and the gains cancelled. Hull’s Stemming algorithms: a case study for detailed evaluation (1996) went finer-grained and found small but real average gains, concentrated in short queries, where there are few terms and losing a match to a suffix is costly.

Both are thirty years old and both still describe the situation accurately, because English morphology has not changed. The summary a practitioner should carry: for English full-text retrieval, stemming is a small effect that helps short queries, hurts precision on rare proper nouns, and is not the lever anyone should be spending a week on.

Where the answer flips

English is nearly the least inflected language you could have picked to test this on. A regular English verb has four or five surface forms. A Finnish noun has roughly fifteen cases and, with clitics and possessive suffixes, thousands of distinct surface forms; Turkish is agglutinative, Arabic has templatic morphology plus attached prepositions and pronouns, and Russian inflects nouns, adjectives and verbs together.

In those languages, an unstemmed lexical index simply cannot match a query against a document that uses a different case of the same noun, and the multilingual retrieval evaluations run under CLEF from 1999 onwards found morphological normalisation to be a substantial improvement rather than a marginal one. Snowball ships stemmers for around twenty languages precisely because of this. If you are building lexical search over Finnish, Turkish, Hungarian, Arabic or Russian text and you are debating whether stemming is worth it, the debate is settled and the answer is not the English one.

The one place both still belong

Inside a lexical index, and nowhere else in a modern pipeline. A BM25 or Elasticsearch analysis chain is exactly the setting these tools were designed for: the representation is a bag of terms, matching is exact string equality on terms, and morphological variants are therefore invisible to each other unless you collapse them.

Two rules make it safe. First, apply the identical analyser at index time and query time — a mismatch here produces zero results for obviously correct queries and is the single most common lexical search bug. Second, index a raw and an analysed field, and search both, so exact matches on identifiers still work. The cost of doing this is index size, which is the cheapest resource in the system, and the benefit is that a stemmer can never make an exact query fail.

Between the two tools in that setting, prefer the stemmer. It needs no tagger, no lexicon and no language model, it runs in microseconds, and the extra precision a lemmatiser buys is largely wasted on a bag of terms that has already discarded the syntax the lemmatiser needed. Reach for lemmatisation when the canonical form is the output rather than an index key — building a controlled vocabulary, normalising extracted terms for a report, or preparing candidates for keyphrase extraction, where oper would be an embarrassing thing to show a user.

Everywhere else, skip it. Do not stem text going into a language model or an embedding model — both were trained on natural text, both have subword vocabularies that already relate run to running, and a stem is an out-of-distribution input that usually costs extra tokens. And do not stem before a classifier over character n-grams, which picks up the same morphological relationships on its own.