Attention Explained Without the Math

Attention is usually introduced with a formula. The formula is the implementation. The idea underneath it is a lookup table that returns a blend instead of a single row, and that idea is enough to reason about almost everything attention does to your latency and your bill.

The problem attention solves

When a model processes a token, it needs to know which of the earlier tokens matter for this one. In “the trophy did not fit in the suitcase because it was too large”, resolving it requires looking back at two candidate nouns and choosing. Nothing about the position of the words tells you which; the model has to weigh them by relevance.

A lookup that returns a blend

Think of a dictionary. You have a key you are looking for, a set of keys that exist, and a value for each. An ordinary lookup finds the one exact match and returns its value. Attention does the same thing except that every key matches a bit: it scores how well your key matches each stored key, turns those scores into weights that sum to one, and returns the weighted average of all the values.

That is the whole mechanism. Everything else is how the keys, values and scores are produced.

Query, key and value

• Query — what this token is looking for. Derived from the token itself.
• Key — what each earlier token offers as a match target.
• Value — what each earlier token actually contributes if it is attended to.

Splitting key from value is the part worth pausing on: what makes a token findable is allowed to differ from what it contributes. The scoring is a dot product, the weights come from a softmax, and “multi-head” means doing all of this several times in parallel with different learned projections, so one head can track syntax while another tracks subject matter.

Why it is the expensive part

Every token attends to every earlier token, so the work grows with the square of the sequence length. Doubling the context does not double the attention cost — it roughly quadruples it. That is why long context was an unsolved problem for years, why providers price it the way they do, and why the cached prefix of a prompt is worth a discount: its keys and values have already been computed and can be kept.