Encoder Models Are Not Dead: Where BERT-Style Still Wins

The transformer that lost the assistant race is still the correct answer for classification, retrieval and span extraction. Not because it is cheaper — though it is, by a lot — but because for those jobs it computes a different shape of answer.

The two structural differences

One pass, not a loop. An encoder reads the whole input bidirectionally in a single forward pass and produces a fixed-size output: a class distribution, a vector, a label per token. A generative model must prefill the input and then decode its answer one token at a time, with everything that follows from that for latency and variance.

Offsets, not a string. This one is a correctness difference and gets less attention than it deserves. Ask an encoder to find the company names in a contract and it labels tokens, so you know the character span of every match. Ask a generative model and it returns a string that you must then locate in the source — a fuzzy match that fails on the exact cases you care about, where the model normalised the capitalisation, expanded an abbreviation or quietly corrected a typo. For redaction, highlighting, annotation or anything that must point back at the original document, the encoder gives you something the generative model structurally cannot.

Three more properties follow: the output is deterministic, the class probabilities are directly available and can be calibrated against labelled data, and the model can be fine-tuned on a few thousand examples on a single GPU.

The arithmetic, with assumptions

BERT-base is 110M parameters (Devlin et al., 2018). Compare it with a 7B decoder on one classification of a 200-token input producing a five-token answer, using the same two-FLOPs-per-parameter-per-token approximation used elsewhere in this cluster.

assumption: forward cost ~= 2N FLOPs per token; attention's quadratic
term ignored; batching, memory bandwidth and overheads ignored.

encoder  (110M):  200 tokens x 2 x 1.1e8   = 4.4e10 FLOPs, one pass
decoder  (7B):    prefill 200 x 2 x 7e9    = 2.8e12
                + decode    5 x 2 x 7e9    = 7.0e10
                                           = 2.87e12 FLOPs

ratio: about 65x more arithmetic for the generative call.

Two caveats on that number, stated because a ratio without them is misleading. Real serving cost is often bound by memory bandwidth and batching efficiency rather than by FLOPs, so wall-clock and price ratios will not match this cleanly. And the five decode steps carry a latency penalty out of proportion to their FLOPs, since each is a separate sequential pass. The arithmetic gives the order of magnitude and the direction; your own measurement gives the number.

Where this stops being an abstraction is volume. At a hundred classifications a day, nothing here matters and you should use whatever is fastest to build. At ten million a day, a 65× difference in arithmetic is the difference between a line item and a project.

The deployment consequence is larger than the ratio suggests. A model of this size fits in ordinary memory and runs acceptably on a CPU, so it can sit inside the service that needs it — no GPU pool, no external call, no rate limit, no per-request network hop, and a latency distribution with a thin tail because there is no queue and no cold start to absorb. For a synchronous path where a classification blocks a user action, that changes what is architecturally possible, not just what is affordable.

The jobs where this decides it

• High-volume fixed-label classification. Moderation, intent routing, spam, language ID, ticket triage. The label set is stable and you have or can cheaply get training data.
• Embedding and retrieval. Nearly every production embedding model is an encoder, because a single bidirectional pass producing one vector is exactly the required shape — see embeddings and choosing an embedding model.
• Reranking. A cross-encoder scores a query-document pair by reading both together, which is more accurate than comparing two independently-computed vectors and cheap enough to run over a shortlist — late interaction sits between the two designs.
• Span extraction. Named entities, PII detection, clause identification — anything needing offsets, per the argument above.
• Entailment checks. A natural-language-inference model answering “does this passage support this claim?” is a fast, deterministic guardrail for a grounded pipeline.

The family is also still under active development, which surprises people who filed it under 2019: RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020) improved on the original recipe, and ModernBERT (Warner et al., 2024) brought a modernised encoder with a much longer context and current architectural choices.

Where the generative model wins outright

Anything open-ended. Any label set that does not exist yet or changes weekly — a decoder handles a new class with a sentence in the prompt, an encoder needs labelled examples and a training run. Anything requiring reasoning across the input rather than pattern recognition over it. And anything where you have no labelled data at all and no budget to make some, which is the most common situation and the honest reason so many pipelines are generative end to end.

The pattern that uses both

The arrangement that gets the best of it: use a generative model to label a few thousand examples, review a sample by hand, train an encoder on the result, and serve the encoder on the high-volume path with the generative model kept for the uncertain tail. That is distillation in its most practical form, and it converts a per-request cost into a one-time one.

The routing rule for the tail follows from the encoder’s own class probabilities: confident predictions are served locally, the low-confidence decile goes to the larger model. You keep the accuracy where it matters and pay generative prices on a fraction of the traffic.