How a Transformer Actually Works, One Layer at a Time

Diagrams of transformers show boxes. Boxes do not tell you what is in memory or how big it is. This page follows one token through a real architecture with the real numbers attached, and finishes by adding the parameters up to the figure the model is sold under.

The shapes we will use

The dimensions below are those of Llama 2 7B, published by Touvron et al. in 2023 and readable in the released configuration file. Any decoder-only model differs in the numbers and almost never in the structure.

From token id to vector

Your text arrives as a string and leaves the tokenizer as a list of integers. Say the prompt is 200 tokens long. Each integer is an index into an embedding matrix of shape 32000 × 4096, so the lookup returns a tensor of shape 200 × 4096. Nothing has been computed yet — this is a table read.

That 200 × 4096 tensor is the residual stream, and it is the only thing that flows from the bottom of the model to the top. Every block reads it, computes something, and adds the result back. Holding that one picture makes the rest of the page mechanical.

What one block does

A block has two halves, each of which is normalise, compute, add back.

The attention half

Normalise the stream (RMSNorm here), then project it three times: queries, keys and values, each with a 4096 × 4096 matrix, each producing 200 × 4096. Reshape each into 32 heads of 128 dimensions. Rotate the queries and keys by position — that is RoPE, and it adds no parameters. Score every query against every key: for one head, a 200 × 128 times a 128 × 200 gives a 200 × 200 matrix of scores, divided by the square root of 128 and masked so no position sees the future. Softmax each row, multiply by the values, concatenate the 32 heads back to 200 × 4096, and pass through a fourth 4096 × 4096 projection. Add that to the stream.

Note the object that has to be kept: those keys and values, 200 × 4096 each, per layer. That is the KV cache, and it is why long contexts cost memory rather than only arithmetic.

Newer models in this family change one shape here: grouped-query attention keeps 32 query heads but only 8 key and value heads, so the key and value projections are 4096 × 1024 rather than 4096 × 4096 and each key head is shared by four query heads. The attention arithmetic is nearly unchanged; the cache shrinks to a quarter. When you see a model card quote a separate num_key_value_heads, that is what it is telling you.

The feed-forward half

Normalise again, then expand each position independently from 4096 to 11008 — twice, in this architecture, because SwiGLU uses a gate projection and an up projection, applies SiLU to the gate and multiplies the two elementwise — then contract 11008 back to 4096 and add that to the stream. This half has no interaction between positions at all. Every token is processed alone.

Thirty-two of those

The same structure runs 32 times with different weights. There is no architectural difference between layer 3 and layer 30; whatever division of labour exists is learned, not designed. It is a consequential fact for anyone reading interpretability work: the statement “early layers do syntax” is an empirical claim about a particular trained model, not a property of the design.

The shapes also tell you where the time goes. Per layer, the six large matrix multiplies — four attention projections and three feed-forward ones, of which the feed-forward pair dominates — account for the great majority of the arithmetic; the attention score matrix is comparatively small at 200 × 200 per head and only becomes the dominant term when the sequence is thousands of tokens long. That is why a short prompt is bounded by weight multiplication and a very long one starts to be bounded by attention, and why the two halves of a request behave so differently — the subject of prefill versus decode.

Back to a distribution

After the last block, normalise once more and multiply by a 4096 × 32000 matrix. Each of the 200 positions now has 32000 scores — the logits. For generation only the last position matters, because that is the one predicting what comes next. Softmax turns those 32000 numbers into probabilities, and the sampler picks one. Then the whole thing runs again with 201 tokens, which is why output is generated one token at a time.

That final matrix is also the subject of a real capacity limit — a rank bound covered in the softmax bottleneck.

The arithmetic adds up

Count the weight matrices per layer:

• Attention: four 4096 × 4096 projections. 4 × 4096² = 67,108,864.
• Feed-forward: gate and up are 4096 × 11008, down is 11008 × 4096. 3 × 4096 × 11008 = 135,266,304.
• Per layer: 202,375,168. Times 32 layers: 6,476,005,376.
• Input embedding and output projection: 32000 × 4096 each = 131,072,000 each.

Total: 6,738,149,376. The model is distributed as 6.74B parameters, and that is where the number comes from. Two useful consequences fall straight out. At two bytes per parameter the weights occupy about 13.5 GB, which is the first thing that decides whether a given card can serve it. And the standard approximation that a forward pass costs about two FLOPs per parameter per token puts one token at roughly 13.5 GFLOP — the number that makes active parameters rather than total parameters the figure that predicts price.