Skip to content
Multigrid

Attention Visualiser: The Mask, the Scaling and the Softmax

A real causal mask and a real softmax over a score pattern you choose, on your own sentence, with the row entropy that says how many tokens each query is really attending to.

thecatsatonthematbecausethematwaswarmthecatsatonthematbecausethematwaswarm

Rows are queries, columns are keys. Darker means more weight. Flat grey cells are masked: the query is not allowed to look there at all.

Row entropy for “was”
2.73 bits

2^2.73 = 6.7 tokens' worth of attention. A row that spread itself evenly over all 11 visible tokens would score 3.32 bits; a row that picked exactly one would score 0.

the
6.1%
cat
6.1%
sat
6.1%
on
6.1%
the
6.1%
mat
6.1%
because
6.1%
the
6.1%
mat
6.1%
was
45.1%
warm
masked
Tokens (T)
11
Cells in the score matrix (T²)
121
Cells removed by the causal mask
55
Keys this query may attend to
10
Logit range in this row (max − min)
2.000
Largest weight in this row
45.1% on “was”
Row sums to
1.000000
Entropy
2.734 bits
Effective tokens attended to (2^H)
6.65
Where these numbers come from: The score pattern is one you picked from a dropdown and the contrast is one you dragged. They are not any model's attention weights, and this page has no model to get them from. What is exact here is everything downstream of the scores: the causal mask, the division by √d_head, the softmax, the fact that every row sums to one, and the entropy. Those are the parts of attention that are arithmetic rather than learned, and they are the parts a picture can honestly show.
What this assumes: One head. Scores are given in units of √d_head, which is the assumption the 1/√d_k factor exists for: if the query and key vectors have unit-variance entries, their dot product has standard deviation √d_k, so dividing by √d_k returns the logits to a usable scale. Turning the switch off multiplies every logit by √d_head instead, which is why raising the head dimension with scaling off collapses each row onto a single token. Masked positions are set to −∞ before the softmax and are shown as exactly zero afterwards. Tokens are whitespace-separated words, not subwords — a real tokenizer would split several of these.

What the picture is telling you

Attention is usually drawn as arrows between words, which hides the two properties that actually matter. The first is that every row is a probability distribution. It sums to one, so attention is always a competition — a token cannot attend more to one thing without attending less to something else. Drag the contrast down to zero and every row flattens to 1/T; there is no setting at which a head attends strongly to everything.

The second is the scaling factor, which looks like a footnote in the paper and is not. Turn the √d_head switch off and raise the head dimension to 512. Every row collapses onto its single best key and the entropy falls to zero. That is not a quirk of this page: it is what happens to a softmax when its inputs grow, and with a head dimension in the hundreds the unscaled dot products grow exactly that fast. A saturated softmax has near-zero gradients almost everywhere, which is another way of saying the model would stop learning. One division by a constant is what stands between the architecture and that.

What this page cannot show you is which pattern a real head has learned, because that takes a trained model and a forward pass, and there is neither here. It is worth knowing that published attention maps are also less interpretable than they look: a head that appears to track syntax on one sentence often does something else on the next, weights are not explanations of the output, and a large share of the mass in real models sits on the first token doing nothing in particular — the "attention sink" you can select above. Treat any attention picture, including this one, as a diagram of the mechanism rather than evidence about a model.

The reasoning behind this

Related tools

Attention Visualiser: The Mask, the Scaling and the Softmax · Multigrid