Natural Language Processing: The Field Before and After LLMs

Natural language processing is not a technique, it is a set of jobs. Language models took over some of those jobs completely, left others untouched, and made a third group cheaper to solve badly. Knowing which group you are in is worth more than knowing any single algorithm.

What NLP actually means

NLP is the business of getting a computer to do something useful with human language. That is deliberately broad, because the field is broad: deciding whether a support ticket is about billing, pulling company names out of a filing, ranking ten thousand documents against a query, detecting that a string is Turkish, correcting a misspelt surname, summarising a transcript. These have almost nothing in common as engineering problems. They are grouped together only because the input happens to be text.

It is worth separating them along one axis before anything else: whether the task has a closed output space or an open one. Classifying a ticket into one of eight queues, detecting a language, matching a name against a customer list, ranking a fixed corpus — the set of possible correct answers is enumerable and often small. Writing a summary, answering a question, rewriting a paragraph in another register — the output space is every sentence in the language. Generative models are the only good answer to the second kind. They are frequently the worst answer to the first.

Three eras, and what each left behind

Symbolic, roughly 1950 to 1990. Hand-written rules and grammars. Weizenbaum’s ELIZA (1966) is the famous artefact, and the era’s honest legacy is not the chatbots but the formal machinery: finite-state transducers, regular expressions, morphological analysers, lexicons. Rules do not generalise to open-domain language, which is why the era ended. They are still unbeatable when the language you are parsing was produced by a machine to a specification.

Statistical, roughly 1990 to 2013. Annotate a corpus, count things, fit a model. The Penn Treebank (Marcus et al., 1993) and the IBM alignment models for machine translation (Brown et al., 1993) define the shift. This era produced the parts that are still in production everywhere: TF-IDF weighting after Spärck Jones (1972), Okapi BM25 (Robertson and Walker, TREC-3, 1994), hidden Markov and maximum-entropy taggers, and support vector machines over sparse features. It also produced the working assumption that labelled data is the expensive input, which is still true.

Neural, 2013 onwards. word2vec (Mikolov et al., 2013) made dense word vectors practical, sequence-to-sequence (Sutskever et al., 2014) and attention (Bahdanau et al., 2015) made translation end-to-end, the Transformer (Vaswani et al., 2017) made it scale, BERT (Devlin et al., 2019) made pre-training plus fine-tuning the default, and GPT-3 (Brown et al., 2020) made the prompt the interface. The practical consequence for the tasks above is that a general model can now do many of them with no task-specific training at all.

What survived the transition

• Everything that feeds a retrieval index. BM25 is not a legacy component in a RAG system; it is half of hybrid search, and the BEIR evaluation is the reason people stopped assuming vectors dominate it. Tokenization, Unicode normalization and analysis chains are what make that index correct.
• High-volume, low-stakes classification. Routing, spam, language ID, deduplication, safety pre-filters. These run on every document, so the per-document cost is the whole cost, and a linear model over sparse features is thousands of times cheaper.
• Anything with a formal specification. Dates in a known format, identifiers, machine-generated logs. A regular expression is exact by construction here; a model is a probability.
• String distance. Deduplication, record linkage, fuzzy lookup against a known list. Edit distance answers a question about characters, and a model does not.

What genuinely died

Being honest in the other direction matters just as much, or the cluster is nostalgia. Hand-built feature pipelines for semantic tasks are gone: nobody should start a new sentiment, entailment, question-answering or intent-detection system by engineering lexical features, because a general model with a prompt will beat a week of that work before lunch. Hand-written grammars for open-domain parsing are gone. Task-specific sequence models trained from scratch on a few thousand labels are mostly gone, replaced by fine-tuning a small pre-trained encoder, which needs an order of magnitude less data. And any task where the output is free text — summarising, rewriting, explaining — was never really solved before and is solved now.

Telling which half your problem is in

Four questions decide it, and none of them is about accuracy:

• Volume. Cost scales linearly with documents for an API and effectively not at all for a local model. At a hundred documents a day this never matters. At ten million a month it is the only thing that matters.
• Latency floor. A small local classifier answers in single-digit milliseconds. A network round trip to a hosted model is hundreds. If the answer sits in front of a user typing, that gap is the product.
• Determinism. A fitted logistic regression returns the identical answer for the identical input for as long as you keep the weights. A hosted model can be updated underneath you — see why that changes results without warning.
• Auditability. You can print the ten features that drove a linear model’s decision and defend it to a regulator. An explanation generated after the fact by a model is not the same object.

Where all four point one way the decision is easy. Where they conflict, the decision table at the end of this cluster works through the arithmetic.