Evaluating Classical NLP vs LLM Approaches
6 min read · updated August 3, 2026
The reflexive answer is “use a model” and the contrarian answer is “classical methods are underrated”. Both are slogans. The decision is a calculation with four inputs, and once you write it down it usually answers itself.
The four variables
- Volume. Documents per month. This decides whether per-document cost is a rounding error or the entire budget, and it is the only variable that changes the answer by orders of magnitude.
- Output space. Closed (one of eight buckets, a span, a ranked list) or open (a sentence nobody wrote yet). Closed spaces are what classical methods were built for; open spaces are what models are for, and no amount of cleverness moves a task across this line.
- Label availability. Do you have, or can you cheaply get, a few thousand labelled examples? This is the fixed cost of the classical route and the reason the model route wins on day one.
- Constraint profile. Latency budget, determinism requirement, audit requirement, data-residency requirement. Any one of these can decide the question regardless of cost.
The decision table by task
| Task | Description |
|---|---|
| format-defined extraction | Regex, always. Dates, identifiers, log fields, anything with a written specification. Exact by construction, microseconds, no fee. A model here is slower, costlier and less correct. See the regex page. |
| fixed-taxonomy classification | Local model at scale, hosted model below the break-even. A linear model is roughly three orders of magnitude cheaper per document and sub-millisecond; the crossover is set by your label budget rather than by quality. |
| open-taxonomy classification | Hosted model. If the categories change monthly or were invented this morning, there is nothing to train on and a prompt is the only option that ships. |
| standard entity types | Local NER pipeline. Person, organisation, location, dates and money in a common domain are well covered, and you get character offsets a generated list does not have. |
| bespoke entity types | Hosted model, unless volume is very high — in which case use it to label a few thousand documents, review them, and train a local model on the result. |
| lexical retrieval | BM25. It is the baseline a dense system must beat, and BEIR (2021) found that out of domain it frequently is not beaten. Hybrid with embeddings; do not replace. |
| semantic retrieval | Embeddings. Nothing lexical relates 'cancel' to 'terminate', and no weighting scheme will. |
| deduplication | MinHash or SimHash. Detects exactly what deduplication means, indexable, and about a hundredth the cost of embedding the corpus. |
| summarising, rewriting, explaining | Hosted model. The output space is open. Extractive methods are a different product, not a cheaper version of this one. |
| anything needing multi-step reasoning | Hosted model. Classical pipelines do not do this at all, and pretending otherwise wastes a quarter. |
The break-even volume, derived
For the rows where both options are genuinely available, the crossover is arithmetic. Define:
V— documents per month.c— hosted cost per document, which is tokens per document × price per token, input and output.F— one-off cost of building the local system: engineering time plus labelling.m— monthly cost of running it: infrastructure plus maintenance.T— the horizon you are amortising over, in months.
Local is cheaper over T months when F + mT < cVT, which rearranges to V > (F/T + m) / c. Everything about the decision is in that expression: the fixed cost divided by your horizon, plus the running cost, over the per-document price you avoid.
A worked example with every input labelled as an assumption. Assume classification at 300 input tokens per document and an assumed $0.20 per million tokens, so c = $0.00006 per document. Assume the local route costs two engineer-weeks at an assumed $6,000 per week plus three analyst-days of labelling at an assumed $400 per day, so F = $13,200. Assume $50 per month to run and maintain it, and a 24-month horizon.
Then V > (13,200/24 + 50) / 0.00006 = (550 + 50) / 0.00006 = 10 million documents per month. Below that, the hosted model is cheaper and you should not be building anything. Above it, the gap widens without limit, because one side of the comparison scales with volume and the other does not.
Three sensitivities worth noticing before trusting any such number. Halve the horizon and the break-even nearly doubles — building for a product that might be cancelled is a bad trade. Raise the assumed token price fivefold, as a reasoning model or a much longer prompt would, and the break-even falls to about 2 million. And if F is near zero because a maintained off-the-shelf model already covers your task — language ID, standard NER, deduplication — the break-even collapses to almost nothing and the local option wins at any volume. That last case is more common than it looks, and it is the one people skip.
Three things that are not about cost
Latency. A local model answers in single-digit milliseconds; a hosted call is hundreds. If the result is needed inside a request a user is waiting on, that is not a cost difference, it is an architecture difference — one is a function call, the other needs a queue, timeouts, retries and a story for what the user sees meanwhile. At 10 ms you can afford to classify every document twice; at 400 ms you cannot afford it once, synchronously.
Determinism. A pinned local model returns the same answer forever. A hosted endpoint can change under you, and an update you did not initiate will move your outputs without moving your code. If a decision must be reproducible — a compliance classification, a scoring rule someone can appeal — the local option has a property the hosted one cannot offer at any price.
Auditability. A linear model’s decision decomposes into features and weights you can print. A generated explanation is text produced after the fact by the same process that produced the answer, and it is not evidence about how the answer was computed — the two can disagree without anything erroring. Where a person is affected and can appeal, this difference tends to be decisive on its own.
The hybrid that usually wins
Almost every mature system in this space is a cascade rather than a choice, and the shape is the same across every task in this cluster: something cheap and deterministic runs on everything, and only the residue escalates.
Regex finds candidates and a model adjudicates the ambiguous ones. A local classifier handles documents above a confidence threshold and escalates the rest. BM25 and embeddings each retrieve fifty documents, and a cross-encoder re-ranks the union. Take the arithmetic above with a 5% escalation rate and the effective per-document cost falls twentyfold while the hard cases still get the better treatment — which is a strictly better outcome than either pure option, at the price of one threshold you have to tune and monitor.
The two mistakes to avoid are symmetrical. Do not send ten million documents to a model because it was faster to write, when the task has a specification and the specification is a regex. And do not spend a quarter building a classical pipeline for a task with an open output space, or for a volume the arithmetic above says will never repay it. Write down the four variables first; the answer is usually already in them.