Induction Heads and In-Context Learning

Give a model the sequence A B ... A and it will predict B. That is not memorisation — it works on token pairs the model has never seen, including random strings. Two attention heads working in series implement it, and this is the most completely understood mechanism in any language model.

The behaviour

The pattern is prefix matching and copying. Present a sequence containing an earlier occurrence of the current token, and the model raises the probability of whatever followed that earlier occurrence. The clean test uses random tokens so that no prior knowledge can help:

prompt:  qz  ht  mv  qz
predicted next token:  ht

A model with no induction mechanism cannot do this above chance. A model with one does it reliably on sequences it has never seen, which is what makes the behaviour a clean target: the capability is separable, testable in isolation, and impossible to explain by lookup.

Elhage and colleagues at Anthropic described the mechanism in “A Mathematical Framework for Transformer Circuits” in 2021, working in two-layer attention-only models, and Olsson and colleagues followed it in “In-context Learning and Induction Heads” in 2022.

The circuit, in two heads

Write the sequence as … A B … A ?, where the final A is the current position. Two heads in different layers cooperate.

1. The previous-token head, in an earlier layer, attends from each position to the position immediately before it and copies information about that earlier token into the current position’s residual stream. After this head runs, the residual stream at the position holding B carries a marker saying “the token before me was A”.
2. The induction head, in a later layer, forms its query from the current token A and matches it against keys built from that marker. The position holding B is exactly the position whose key says “preceded by A”, so the induction head attends there.
3. The copy. The induction head’s output–value path is arranged so that attending to a position increases the logit of the token at that position. Attending to B therefore predicts B.

The whole mechanism is: mark each token with its predecessor, then search for the position marked with the current token, then copy what is there. It is a lookup implemented in attention, and once you have seen it you can implement it by hand in a few lines of tensor code.

Why it takes two layers

This is the part that explains why induction is a circuit and not a head. A single attention head computes its query and key from the residual stream as it stands. At the final A, the residual stream knows the current token is A. At the position holding B, the residual stream knows the current token is B — but nothing about what preceded it, because no head has yet moved that information there.

So a one-layer attention-only model cannot do induction at all, and this is a hard architectural prediction rather than an observation. The second head’s key must be built from something the first head wrote. Anthropic’s framework calls this K-composition: one head’s output feeds another head’s key computation through the residual stream. The companion cases are Q-composition and V-composition, and the taxonomy is useful because it tells you which pairs of heads can, in principle, form a circuit at all.

How it was found

Worth spelling out, because it is the template for circuit work. The heads were first spotted by their attention pattern — a head that consistently attends to the token after the previous occurrence of the current token is visible in a heatmap. But an attention pattern is not evidence of causation, so the pattern was only the lead.

The confirmation came from ablation and from analysing the head’s learned matrices directly. In a two-layer attention-only model the composition of the output–value matrices with the unembedding can be computed in closed form, and for an induction head that composite matrix is close to a positive multiple of the identity in token space — the mathematical statement of “this head copies the token it attends to”. That is a much stronger form of evidence than a heatmap, and it is only available because the model was small enough and simple enough to write the product out.

The phase change, and the in-context learning claim

The 2022 result that made this famous is about training dynamics. Induction heads do not fade in; they appear over a narrow window of training, and that window coincides with a visible bump in the loss curve. Simultaneously, the model’s ability to use context to improve its predictions later in a sequence — measured as the difference in loss between an early token and a much later one in the same context — jumps.

The evidence tying the two together is a stack rather than a single experiment: the co-occurrence in time, the fact that it holds across model sizes, ablation of induction heads degrading in-context performance, and architectural modifications that make induction easier or harder shifting the phase change accordingly. That is a serious body of evidence and it is the strongest link anyone has drawn between a specific mechanism and a macroscopic capability.

How far the claim reaches

Here is where care is needed, because the popular version of this result is stronger than the result.

• Strict induction is not few-shot learning. Copying B after a repeated A is a narrow behaviour. Learning a classification task from three labelled examples in the prompt is not obviously the same computation, and the original work is explicit that the connection is best supported in small attention-only models and becomes more circumstantial as models get larger.
• Real induction heads are messier than the two-head story. In practice many heads participate, heads do more than one thing, and the clean previous-token-plus-induction pair is an idealisation of a distributed implementation.
• The mechanism is necessary, not sufficient. Nobody has shown that induction heads plus some bookkeeping account for in-context learning. They are a component with a demonstrated causal role in part of it.

Stated carefully: induction heads are a well-characterised mechanism that emerges in a phase change, and their emergence is tightly associated with a jump in the model’s ability to exploit context. Whether they explain few-shot prompting in a 400B model is open, and treating it as settled is the most common overreach in writing about this area.

Finding one yourself

The test is cheap and works on any small open model. Build a sequence of random tokens, repeat it, and score every head by how much attention it places from each position in the second copy onto the token that follows the matching position in the first copy.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained(
    "gpt2", attn_implementation="eager"
).eval()

torch.manual_seed(0)
n = 25
seq = torch.randint(1000, 10000, (n,))          # random, unlikely to co-occur
ids = torch.cat([seq, seq]).unsqueeze(0)        # the repeated block

with torch.no_grad():
    out = model(ids, output_attentions=True)

# For a query at position n+i (second copy), the induction target is i+1.
scores = {}
for layer, A in enumerate(out.attentions):      # A: (1, heads, q, k)
    for head in range(A.shape[1]):
        s = sum(A[0, head, n + i, i + 1].item() for i in range(n - 1))
        scores[(layer, head)] = s / (n - 1)

for (layer, head), s in sorted(scores.items(), key=lambda kv: -kv[1])[:5]:
    print(f"layer {layer:>2} head {head:>2}  induction score {s:.3f}")

A score near 0.01 is chance for a 50-token sequence. Heads scoring above about 0.3 are worth investigating. Then do the part that matters: ablate the top head and check that the model’s accuracy on the repeated-sequence task collapses while its loss on ordinary prose barely moves. Without that comparison you have found a correlation, which is where this page started.