Skip to content
Multigrid

How a Transformer Works: Step a Sequence Through the Stack

Walk a sequence through all seventeen stages of a transformer block, with the exact parameter count, tensor shape and FLOP cost of each computed from a configuration you set.

■ embedding■ attention■ feed-forward■ output head
Token IDs to embeddings
[2,048] → [2,048, 4,096]

A row lookup in the embedding table. No arithmetic happens here at all — this is the largest single weight matrix in most models and it costs nothing to use.

Weights used: 524.3 MFLOPs: 0
Total parameters
8.03 B

Forward pass over 2,048 tokens: 31.83 TFLOP, of which 3.5% is the quadratic attention term. Raise the sequence length and watch that share move — it is the entire argument for every long-context trick there is.

Embedding table (vocab × d)
524.3 M
One block
218.1 M
All 32 blocks
6.98 B
Output head
524.3 M
Total
8.03 B
Forward FLOPs — weight matmuls, linear in T
30.73 TFLOP
Forward FLOPs — attention scores and values, T²
1.10 TFLOP
Forward FLOPs — total
31.83 TFLOP
Rough training cost per token (6 × non-embedding params)
41.88 GFLOP
weight matmuls 96.5% — grows linearly with Tattention T² term 3.5% — grows with the square of T
Where these numbers come from: Everything on this page is arithmetic on the nine numbers you typed. There is no model behind it and no measurement in it: parameter counts, tensor shapes and FLOP counts are determined entirely by the configuration, which is exactly why they can be shown honestly without running anything. Activations, attention weights and outputs are not shown for the opposite reason — those would have to be invented.
What this assumes: No biases on any projection, which matches most current open architectures; add 2d + 2ff per block if yours has them. RMSNorm rather than LayerNorm, so one scale per channel and no shift. Rotary position embeddings, so no learned positional parameters. FLOPs count a multiply-add as 2, count only the matmuls, and halve the two attention matmuls for causal masking — normalisations, activations, softmax and the rotary rotation are all left out because together they are under 2% of the total. The forward pass is the whole sequence at once, as in training or prefill; single-token generation with a KV cache has a completely different profile. Training cost per token uses the standard 6N approximation (forward 2N, backward 4N) over non-embedding parameters.

What the number is telling you

Two things dominate a transformer and they are not the ones the diagrams emphasise. The first is that the feed-forward block, not attention, holds most of the weights: at a typical 3.5× expansion with SwiGLU it is three matrices of d × ff against attention's four of roughly d × d, which is about two-thirds of every block. When someone says a model is mostly a big lookup table, this is the arithmetic they mean.

The second is that the parameter count and the compute cost do not move together. Step to the score matrix and watch the FLOPs line: it is the only stage with no weights at all, and it is the only one that grows with the square of the sequence. At two thousand tokens it is a rounding error. At a hundred thousand it is most of the bill. Every technique with a name — sliding windows, grouped-query attention, sparse patterns, state-space hybrids — is an attack on that one line, and you can watch each of them work by changing a field above.

What this leaves out is everything that is not a matmul. Memory is the obvious omission: the score matrix is heads × T × T numbers, and on a long sequence that tensor, not the weights, is what exhausts a GPU — which is why FlashAttention, which changes no arithmetic at all, was such a large practical win. Bandwidth is the other. During single-token generation almost nothing here is compute-bound; the GPU spends its time reading weights and the KV cache out of memory, and a FLOP count predicts throughput badly. Treat this page as an answer to "where does the model put its parameters and its arithmetic", not as a latency model.

The reasoning behind this

Related tools

How a Transformer Works: Step a Sequence Through the Stack · Multigrid