Named Entity Recognition Then and Now

Named entity recognition is span labelling: find the character offsets of every person, organisation and place in a text and tag each one. That framing — offsets, not a list — is where most comparisons between classical NER and LLM extraction quietly go wrong.

The task, precisely

A classical NER system consumes a tokenised sentence and emits one tag per token in a scheme such as BIO: B-ORG for the first token of an organisation, I-ORG for subsequent ones, O for everything else. Two properties follow and both matter. Output is aligned to the input, so every entity comes with exact offsets you can highlight, store or redact. And output is structurally constrained: the system cannot return an entity that is not literally in the text, because it is labelling tokens rather than writing them.

Ask a language model to “extract all organisations” and you get a list of strings. That is a different object. It has no offsets, it may normalise the surface form (IBM for International Business Machines Corp.), it may merge two mentions into one, and it can return something that never appeared — which is the ordinary failure mode of generation applied to extraction. For redaction, highlighting or any use where you must point at the source, you need the offsets, and getting them back means string-matching the model’s output against the document, which fails whenever it normalised anything.

What the published record shows

NER has an unusually clean public record, so there is no need to invent numbers. The reference benchmark is the CoNLL-2003 shared task (Tjong Kim Sang and De Meulder), which defined four entity types over Reuters newswire and has been the comparison point ever since. The best systems at the 2003 workshop — Florian et al. among them — reported F1 in the high eighties on English, using classifier ensembles over hand-engineered features plus gazetteers.

Twenty years of neural work moved that into the mid nineties on the same dataset, with entity-aware pre-trained encoders such as LUKE (Yamada et al., EMNLP 2020) among the systems reporting the highest figures. The important observation is the shape of the curve rather than any single number: two decades of architecture work bought roughly five to seven F1 points on newswire, which means the classical systems were already close to the ceiling on that domain.

The catch, and it is the one that decides real projects, is that CoNLL-2003 is 1996 Reuters newswire. Performance on clinical notes, legal contracts, chat logs, product listings or non-English text is a different question with a different answer, and a system trained on newswire degrades sharply on all of them. Do not read a benchmark number as a prediction about your documents. For the model you actually install — spaCy, Stanza, Flair — read the accuracy figures published for that specific version and corpus in its own documentation, and then check them on a hundred of your own sentences, because that is the only number that describes your problem.

The cost arithmetic

Nobody here has benchmarked either option, so what follows is arithmetic with every input labelled as an assumption. Substitute your own.

• Assumption: 10 million documents per month, averaging 300 tokens each — 3 billion input tokens.
• Assumption: a cheap hosted model at $0.10 per million input tokens, with extraction output small enough to ignore. That is 3,000 × $0.10 = $300 per month, before retries.
• Assumption: a small CPU NER pipeline processes a 300-token document in 20 ms on one core. 10 million documents is 200,000 core-seconds, about 56 core-hours.
• Assumption: a 4-vCPU instance at $0.05 per hour run continuously is $36 per month and supplies roughly 2,900 core-hours, fifty times the requirement.

So the monthly ratio is roughly 8:1 in this example, and it widens with volume because one side scales and the other does not. But the ratio is not the interesting number — the latency is. Twenty milliseconds locally against a few hundred for a network round trip is a factor of ten to twenty, and it is the difference between running extraction inline in a request handler and building a queue, a worker, a retry policy and a dead-letter path. That engineering is the real cost of the API option and it does not appear on the bill.

Against that, the classical option has a fixed cost the API does not: if no off-the-shelf model covers your entity types you need labelled data, and annotation is measured in analyst-days. The break-even derivation treats that fixed cost properly.

Where an LLM genuinely wins

• Entity types nobody has a model for. “Contract termination clauses”, “mentions of a competitor’s pricing”, “symptoms with their onset”. A prompt describes these in a sentence; a trained model requires a labelled corpus that does not exist.
• Extraction that needs reasoning. Deciding whether an organisation mention is the buyer or the seller is not span labelling. Neither is resolving the company back to a name mentioned two paragraphs earlier — see coreference.
• Low volume, immediate need. At ten thousand documents a month the API bill is a rounding error and the classical route costs a week. Volume is what flips this.
• Long-tail languages. Where no maintained NER model exists for the language at all, a multilingual model is the only option on the table.

The hybrid that keeps both

The arrangement that tends to survive contact with production runs the cheap system on everything and the expensive one on the residue. A local NER model tags all ten million documents; a confidence threshold and a set of rules identify the few per cent that are ambiguous — no entities found in a document that should have some, overlapping spans, types the model handles badly — and only those go to a model. If five per cent escalate, the API cost in the arithmetic above falls to about $15 a month while the hard cases still get the better treatment.

The same pattern also solves the offsets problem. Use the local model for spans, and use the language model only for the judgement it is genuinely better at: classifying a span you already found, or linking it to a canonical record.