Self-Attention vs Cross-Attention: What Each Is For

These are not two mechanisms. They are one mechanism asked to look at two different places, and every difference in behaviour, cost and caching follows from which place.

One operation, one substitution

Attention takes queries, keys and values, scores each query against each key, and returns a weighted blend of the values — the picture in attention as a soft lookup. The only question that separates self from cross is which sequence the keys and values are derived from.

• Self-attention: queries, keys and values all come from the same sequence. A token looks at its own neighbourhood.
• Cross-attention: queries come from one sequence, keys and values from another. A token in the output looks at the input.

Same matrices, same softmax, same square root of the head dimension. Substituting one tensor for another is the whole distinction.

Self-attention

In a decoder, self-attention is causally masked: position 40 may attend to positions 1 through 40 and no further. This is not politeness; it is what makes training efficient. With the mask, a single forward pass over a 2000-token document produces 2000 separate next-token predictions at once, each one having seen only its own prefix. Remove the mask and each prediction would need its own pass.

In an encoder, self-attention is unmasked. Every token sees every other token in both directions, which is exactly what you want when the job is to understand a fixed input rather than continue it, and exactly what you cannot have when the job is to generate.

Cross-attention

Cross-attention is the join between two streams. In the original encoder-decoder transformer (Vaswani et al., 2017) the decoder has three attention sublayers per block: masked self-attention over what it has generated, cross-attention into the encoder’s output, and then the feed-forward network. The keys and values in that middle sublayer are computed once from the finished encoder output and reused for every generated token.

Two properties fall out of that. There is no causal mask on the source — the decoder may look anywhere in the input, because the input is complete before generation starts. And the source representations are bidirectional, because the encoder was unmasked.

Cross-attention is also the older of the two ideas. It began as alignment in neural machine translation — Bahdanau et al. (2014) let a decoder learn which source words to look at while producing each target word, replacing the fixed-length bottleneck vector that earlier sequence-to-sequence models had squeezed the whole sentence through. Self-attention arrived later, as the observation that the same mechanism works pointed at the sequence itself. The weights of a cross-attention layer still have that alignment reading: for a given output token, they say where in the input it looked. Self-attention weights are far more tempting to read that way and far less reliable when you do.

Where each one appears

The row worth pausing on is the third. A chat model has no cross-attention anywhere. Your system prompt, your documents and the conversation are all concatenated into one sequence and handled by self-attention. There is no architectural distinction between “the input” and “what the model has said” — only position and the chat template’s role markers.

The multimodal case

Vision-language models are where the choice is still live, and there are two families. The projector approach runs an image encoder, maps its patch outputs into the text model’s embedding space with a small trained adapter, and splices the resulting vectors into the sequence as if they were tokens. Nothing about the text model changes; self-attention handles the image because the image is now in the stream. That is why image inputs are billed as tokens and why resolution translates directly into cost.

The gated cross-attention approach instead inserts new cross-attention layers into a frozen text model, letting it attend to image features that never enter the token stream. Flamingo (Alayrac et al., 2022) is the reference design. It keeps the language model’s weights untouched and does not consume context length, at the price of new layers to train and a more complicated serving path. The projector family won on simplicity, which is the same reason decoder-only won generally: one stream is easier to build, cache and price than two.

What the difference costs

Cost scales differently. Self-attention over a sequence of length n is n² scores. Cross-attention is (target length × source length), and for translation-shaped work — long fixed source, short target — that is far cheaper than concatenating the two and running self-attention over the sum, which would be (source + target)².

Put numbers on it. A 2000-token source and a 100-token target gives cross-attention 100 × 2000 = 200,000 query-key pairs per head per layer, plus 100² = 10,000 for the decoder’s self-attention. Concatenating the two into one 2100-token causal sequence gives about 2100² / 2 ≈ 2.2 million pairs — roughly eleven times more. That gap is the entire efficiency case for encoder-decoder, and it is real. What the decoder-only design gets in return is that the 2.2 million pairs for an unchanged prefix can be computed once and reused across turns, which the encoder-decoder’s cheaper arithmetic cannot be when the source changes.

Caching differs too, and this is the practical one. Encoder-decoder’s cross-attention keys and values are computed once per source, so re-decoding the same document with a different target is nearly free. Decoder-only has no such split, but it gets something better for chat: because everything shares one causal stream, an unchanged prefix keeps its keys and values valid across turns, which is what prompt caching sells. An encoder-decoder would have to re-encode whenever the source changed — and in a conversation, the source changes every turn.