Why Almost Every Modern LLM Is Decoder-Only

In 2019 all three transformer shapes were live options. By 2023 nearly every model you could call through an API was decoder-only. The reasons are specific, and none of them is that attention works better in one direction.

Three shapes, one job

• Encoder-only — bidirectional, trained by masking tokens and predicting them. BERT. Produces representations, not text.
• Encoder-decoder — an unmasked encoder for the input, a masked decoder for the output, joined by cross-attention. T5, BART.
• Decoder-only — one masked stack, trained to predict the next token. GPT and everything shaped like it.

The signal-density argument

Start with how much learning each objective extracts from a fixed pile of text. BERT’s masked language modelling, as described by Devlin et al. (2018), corrupts 15% of tokens and predicts those. So a 512-token passage yields roughly 77 prediction targets.

Causal language modelling predicts every position. The same 512-token passage yields 512 targets, each conditioned on a different prefix. That is about 6.7× more supervision per token of data read, from the same forward pass, at the same cost. When the binding constraint is how much text exists and how much compute you can afford to push through it — which is precisely the regime scaling laws describe — a 6.7× difference in signal per token is not a detail.

The masked objective buys something with that cost: bidirectional context. For a classifier that is worth paying for, which is why encoders did not vanish. For a model that must generate, bidirectional context over the output is not available at inference anyway.

Masking has a second, subtler cost. The [MASK] token appears during training and never at inference, so the model is optimised on inputs that differ from the ones it will see — a mismatch Devlin et al. mitigated by replacing only 80% of selected tokens with the mask, leaving 10% unchanged and corrupting 10% at random. That is a patch on a structural problem. Causal language modelling has no such gap: the training task and the inference task are the same operation, which is also why a pretrained decoder is immediately useful for few-shot prompting with no adaptation at all.

The interface argument

Encoder-decoder assumes the world splits cleanly into a source and a target. Translation splits that way. Summarisation splits that way. A conversation does not: turn four’s “source” includes the model’s own turn three, and by turn ten the split is a bookkeeping exercise with no benefit.

Decoder-only removes the distinction. Everything is one sequence, so instructions, documents, examples, tool results and the model’s own output are all the same kind of thing in the same stream. That is what made prompting a general interface rather than a per-task adapter, and it is the structural reason few-shot examples work at all: an example is just more sequence.

The architecture comparisons that were actually run — Raffel et al.’s T5 study (2020), which swept architectures and objectives under matched compute, and Wang et al.’s 2022 comparison of architecture against pretraining objective for zero-shot behaviour — are worth reading in the original rather than summarised as a score. What is fair to say without quoting them is that they found the objective and the architecture interact, and that causal decoding paired with next-token prediction was the combination that generalised to unseen tasks with no adaptation.

The serving argument

This one only became visible once these models were products. A decoder-only model in a conversation keeps one causal stream, so the keys and values computed for the prefix stay valid as long as the prefix does not change. Append a turn and the previous work is still good. That is what makes prompt caching possible and what makes a long system prompt affordable at scale.

An encoder-decoder cannot do that when the source changes, because the encoder is bidirectional: adding a token at the end changes every token’s representation, including the first. Re-encode the whole input on every turn. For translation, where the source is fixed, that cost is paid once and the design is excellent. For chat it is paid again on every message.

The single-stack design also makes batching simpler, which is worth more than it sounds. A serving engine packs many requests of many different lengths into one batch and streams them at different rates; doing that for one homogeneous stack is hard enough, and doing it for two stacks with a join between them is harder. Every technique that makes modern serving efficient — continuous batching, paged attention, prefix sharing — was developed against the decoder-only shape, and the architecture that everything is optimised for accumulates an advantage that has nothing to do with the architecture itself.

What the other shapes kept

Neither of the losing shapes is dead, and treating them as obsolete is a mistake that costs money. Encoder-decoder still dominates where the source is fixed, long and complete before generation begins: speech recognition is the clearest current example. Encoder-only models remain the right tool for classification, retrieval and span extraction, where a 110M-parameter forward pass replaces a generative call entirely.

There is also a middle option that never quite went away. A prefix language model uses one stack but applies bidirectional attention over the prompt and causal attention over the completion — the input gets the encoder’s treatment, the output gets the decoder’s, with no second set of weights. It appears in the architecture surveys cited above and in later unified-objective work, and it is a reminder that “decoder-only” describes the masking pattern rather than a law of nature.

The honest summary is that decoder-only won the general assistant job, decisively and for reasons that are about training economics and serving state rather than about representational power. It did not win every job.