Skip to content
Multigrid

NLP fundamentals & classical tasks

The pre-2020 toolkit — tokenizing, BM25, taggers, classifiers, edit distance — and an honest account of which parts a language model has replaced and which parts still win.

Natural language processing did not begin in 2022 and it did not end there either. Fifty years of work produced a toolkit that is small, fast, deterministic and extremely well documented, and a good deal of it is still the correct answer — not out of nostalgia, but because a classifier that returns in three milliseconds on a CPU, costs nothing per document, and gives the same answer next year is a different kind of component from a network call to a model that may be silently updated.

The trouble is that almost every explanation of these techniques on the open web was written before language models and never revised, so it teaches the mechanism and never says whether you should still use it. These pages carry that comparison on every one of them, in both directions: where the classical option wins on cost, latency, determinism or auditability, and where it plainly loses and you should stop defending it. Where a number appears it is either from a named published result or derived from assumptions labelled as assumptions, because nobody here ran a benchmark and pretending otherwise would make the whole cluster worth less than nothing.

Natural Language Processing: The Field Before and After LLMs

What NLP means as a field, the three eras that built it, and a clear account of which parts of the classical toolkit survived the arrival of language models.

4 min read

Text Preprocessing: What You Still Need to Do

The classical preprocessing pipeline, which steps subword tokenizers made pointless, and the four steps that matter more now than they did before.

5 min read

Stemming and Lemmatisation: Do They Still Matter?

What each one does, where Porter's algorithm goes wrong, what the retrieval literature found about whether it helps, and the one place both still belong.

5 min read

Stop Words: A Practice Worth Questioning

Why stop word lists existed, why term weighting already does the job, and the three query types where removing them makes retrieval strictly worse.

4 min read

TF-IDF Explained and Implemented

The two intuitions behind term weighting, the formula and its variant traps, thirty lines that implement it, and what the representation cannot express.

5 min read

BM25: The Algorithm That Refuses to Die

What BM25 fixes about TF-IDF, what the BEIR benchmark found when it was tested against dense retrievers, and how to set the two parameters that matter.

4 min read

Named Entity Recognition Then and Now

What NER is as a task, what the published shared-task results actually show, the cost arithmetic against LLM extraction, and the hybrid that beats both.

5 min read

Part-of-Speech Tagging and Dependency Parsing

What a tagger and a parser give you, why English tagging accuracy stalled around 97%, and four places explicit syntax still earns its keep in a modern pipeline.

5 min read

Sentiment Analysis: Harder Than It Looks

Three different tasks hide under one name, negation defeats bag-of-words, sarcasm is a labelling problem before it is a modelling one, and domain shift breaks everything.

4 min read

Topic Modelling: LDA vs Embedding Clustering

What LDA assumes, what embedding clustering assumes instead, the evaluation trap that published work identified, and a script that runs both on your corpus.

5 min read

Text Similarity: Every Method and What It Actually Detects

Nine ways to measure how similar two texts are, sorted by what each can and cannot detect, with complexity classes and the arithmetic that rules out all-pairs comparison.

5 min read

Word2Vec and GloVe: The Ancestors of Embeddings

The distributional idea, how skip-gram with negative sampling actually trains, what is wrong with the famous analogy demo, and when a 2013 method is still the right tool.

4 min read

Language Detection and Script Handling

How language identifiers work, the named libraries worth using, and the seven edge cases that make multilingual pipelines fail quietly rather than loudly.

5 min read

Text Normalisation for Search and Retrieval

Unicode normalisation forms, why case folding is not lowercasing, and the named bugs that follow from getting either wrong in a search pipeline.

5 min read

Regular Expressions for Text Extraction: Still Undefeated

The class of extraction problems where a regular expression is exact by construction, the cost gap derived, the outage catastrophic backtracking caused, and where regex is the wrong tool.

5 min read

Coreference Resolution and Why Chatbots Get Confused

What coreference is, the datasets that define its hard cases, and why unresolved pronouns are a concrete cause of retrieval failure in multi-turn assistants.

5 min read

Text Classification Without an LLM

Four approaches to putting text in buckets, with the per-million-document cost derived from labelled assumptions and the label budget and latency floor treated as first-class inputs.

6 min read

Keyword Extraction and Summarising Without Generation

Statistical and graph-based methods for pulling keyphrases and representative sentences out of text, with no per-document API fee and no possibility of inventing content.

4 min read

Spell Correction and Fuzzy Matching

Edit distance and its faster relatives, why candidate generation rather than scoring is the real problem, and the special case of matching personal names.

5 min read

Evaluating Classical NLP vs LLM Approaches

A decision table by task, the break-even volume derived from labelled assumptions, and the three considerations that are not about cost at all.

6 min read

Other topics

NLP fundamentals & classical tasks · Multigrid