Encoder-Decoder Models and the Tasks That Still Fit Them

An encoder-decoder model reads the whole input with bidirectional attention, freezes that into a set of vectors, and then generates output while attending to it. The difference from a decoder-only model is one extra attention block per layer and one structural guarantee: every source token was encoded with the full source visible.

Two stacks, three attentions

ENCODER  (runs once)
  input tokens (L_src)
  -> N layers of: bidirectional self-attention + MLP
  -> memory: (L_src, d)          keys and values computed once

DECODER  (runs per generated token)
  each layer:
    1. causal self-attention over tokens generated so far
    2. cross-attention:  Q from the decoder
                         K, V from the encoder memory
    3. MLP

Three attention operations in total, and the middle one is the whole difference. In self-attention the queries, keys and values all come from the same sequence. In cross-attention the queries come from what is being written and the keys and values come from what was read. The decoder asks a question of the source at every layer and every step.

The encoder memory is computed once and reused for every generated token, which matters for the cost model below.

What bidirectional source encoding buys

This is the substantive advantage and it is easy to state. In the encoder, source token 3 is encoded with source tokens 1 through L_src all visible. In a decoder-only model reading the same source as a prefix, token 3 is encoded with tokens 1 and 2 visible and nothing else, because the causal mask does not lift for the prompt.

For tasks where the meaning of an early word depends on a later one, that is a genuine representational difference rather than a stylistic one. German verb-final clauses, morphological agreement, a pronoun resolved by something said afterwards, an acoustic frame disambiguated by the next syllable — all of these are cases where the right encoding of position i needs position i + k.

A decoder-only model compensates with depth and with scale, and at sufficient scale it compensates well. The point is that it is spending capacity to work around a mask, and per parameter the encoder shape is ahead on exactly these tasks. It is the same argument that keeps encoder-only models in production for classification and reranking.

The training objective is different too

A decoder-only model is pretrained by predicting the next token. An encoder-decoder needs a source and a target, and unlabelled text does not come with that split, so one has to be manufactured. The T5 formulation is span corruption:

Original:  Thank you for inviting me to your party last week.

Source:    Thank you <X> me to your party <Y> week.
Target:    <X> for inviting <Y> last <Z>

Contiguous spans are removed and replaced by numbered sentinels;
the decoder emits the sentinels and what belonged to each.

It is a neat construction: the encoder gets a denoising task over the whole visible sentence, the decoder gets a short target, and both stacks train from raw text with no annotation. T5 then casts every downstream task into the same text-to-text shape with a task prefix, which is a direct ancestor of the instruction formatting used everywhere now.

There is a cost in it that is rarely stated. Next-token prediction supervises every position of every example — a 1,000-token document yields 1,000 prediction targets. Span corruption supervises only the corrupted spans, conventionally around 15 per cent. Per token of training data, the encoder-decoder objective extracts substantially less signal.

When data is the binding constraint rather than compute, that difference compounds across a whole pretraining run, and it is one concrete, unglamorous reason the decoder-only objective scaled better on a fixed corpus. The same argument applies to masked language modelling generally, which is why the models built on it are excellent encoders and were never the ones that scaled to frontier size.

The cost model is different

Source of 2,000 tokens, generating 100 tokens.

Decoder-only:
  the source lives in the same KV cache as the output
  every generated token attends over 2,000 + k entries
  prefill cost 2,000 tokens, then cache grows to 2,100

Encoder-decoder:
  encoder pass over 2,000 tokens, ONCE, bidirectional
  decoder self-attention cache grows only to 100
  cross-attention K/V for the 2,000 source tokens: computed once, fixed

Two consequences. First, the decoder’s own KV cache is sized by the output length, not by input plus output, which for long-input short-output tasks — translation, summarisation, transcription — is a large saving. Second, the encoder memory is a fixed cost per request that does not grow as generation proceeds.

Against that, the encoder is a second stack of parameters to train, store and serve, and the cross-attention adds an attention block per decoder layer. For a task that is mostly generation with a short prompt, all of that is pure overhead.

The tasks that still fit

The T5 family and its descendants remain the reference open encoder-decoder text models, and the translation and speech stacks are where this architecture is not merely surviving but standard.

One worked example: speech to text

Whisper is the clearest case, because the shapes make the argument by themselves.

ENCODER
  audio resampled to 16 kHz
  fixed 30-second window (shorter clips are PADDED to 30 s)
  log-Mel spectrogram, 80 channels, 10 ms hop  -> 3,000 frames
  two convolutions, the second with stride 2   -> 1,500 positions
  transformer encoder over those 1,500

DECODER
  a text transformer, causal, with cross-attention into the 1,500
  prompted with control tokens before any text:
    <|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>
  then emits the transcript, typically well under 100 tokens.

Three things in that are worth pulling out.

• The asymmetry is extreme and fixed. Fifteen hundred source positions, perhaps fifty output tokens. Putting the source in the decoder’s own cache would mean every one of those fifty steps attends over 1,550 entries; here the decoder’s cache reaches fifty and the 1,500 are cross-attended from a memory computed once.
• The window is fixed, and that costs. A two-second clip is padded to thirty and pays the full encoder pass. The architecture trades flexibility for a completely static encoder shape, which is excellent for batching and wasteful for short audio — a real, unglamorous consequence of the design that anyone transcribing short utterances hits immediately.
• The control tokens are the task interface. Language and task are selected by tokens in the decoder’s prompt, not by separate models. That is the T5 text-to-text idea again: one set of weights, the job specified in the prefix.

Every multimodal model is one of these

Worth naming, because it is usually described as something else. A vision-language model consists of a vision encoder that turns an image into a set of embeddings, a projection that maps them into the language model’s space, and a decoder that generates text while attending to them.

That is an encoder-decoder model whose encoder happens to take pixels. The only real variation is whether the image embeddings are injected through dedicated cross-attention layers or simply concatenated into the decoder’s input sequence — the first keeps the two stacks separate and costs less context, the second is simpler and lets the image tokens participate in ordinary self-attention. Both ship.

Why the decoder-only shape won everything else

• One stack scales more simply. Every parameter serves both understanding and generation, and the training objective is a single next-token loss over undifferentiated text. There is no input-output split to define, which means the entire internet is training data without any pairing.
• Chat has no fixed source. A conversation is a growing sequence in which yesterday’s output is today’s input. Freezing an encoder memory makes no sense when the source changes on every turn.
• In-context learning lives in the prefix. Few-shot examples, tool schemas, retrieved documents and system prompts are all just tokens in one sequence. The encoder-decoder split forces a decision about which side each of those belongs on, and there is often no good answer.
• Prefix caching pays. Reusing the KV entries of a shared prompt prefix is a large real saving, and it is natural in one sequence.

The trade: the encoder-decoder shape buys bidirectional understanding of a fixed input and a decoder cache sized by the output alone. It pays with two stacks to train and serve, a rigid distinction between source and target that conversation does not respect, and an ecosystem that moved elsewhere. Where the task really does have a fixed source and a separate target, it is still the right architecture and it is still what ships.