Circuits: Reverse-Engineering One Behaviour End to End

A circuit is a subgraph of a model—specific heads and MLPs, wired in a specific way—claimed to implement one behaviour. The claim is only worth something if it was established by intervention, and the procedure for doing that is the same every time. This page is that procedure, followed through one published example.

What a circuit claim is

The strong form of the claim: on this distribution of inputs, these components and these connections between them account for the model’s behaviour, in the sense that you can replace everything else with its mean value and the behaviour survives, and you can break any of the named components and the behaviour fails in the way the account predicts.

Both halves are needed. Sufficiency without necessity means you found a path that can do the job, not the one the model uses. Necessity without sufficiency means you found components that matter, which is a heatmap, not a circuit.

The genre began in vision. The Circuits thread on Distill, led by Olah and colleagues from 2020, traced features in convolutional networks — curve detectors built from earlier edge detectors — and established the method before transformers were the subject.

The task: indirect object identification

Wang and colleagues published the indirect-object identification circuit in GPT-2 small in 2022, and it remains the most complete worked example in a language model. The task is a single sentence template:

"When Mary and John went to the shop, John gave a drink to ___"

correct completion:  " Mary"

Name the roles:
  IO  = the indirect object, appears once     -> Mary
  S   = the subject, appears twice            -> John
  S2  = the second occurrence of the subject

The behaviour to explain: the model outputs the name that appears once,
not the name that appears twice.

It is a good target for three reasons. It is templated, so you can generate thousands of examples and vary the names, which means results are averages rather than anecdotes. It has a natural metric: the logit difference between the IO name and the S name. And it requires real computation — the model must detect the duplicate, work out which name is duplicated, and suppress it.

The method, in order

1. Define the behaviour and a scalar metric. Here, logit difference between IO and S at the final position. Logit difference rather than probability because it does not saturate and it is linear in what the model computes.
2. Build a paired distribution. For every prompt, a counterfactual that differs minimally — here, swapping which name is duplicated, so the correct answer changes while the sentence structure does not. Equal token length is mandatory.
3. Localise with activation patching. Sweep every (layer, position) site and record the effect on the metric. This gives you the regions that matter, in a couple of minutes on a small model.
4. Narrow to components. Patch individual attention head outputs rather than whole residual-stream sites. Now you have a list of heads instead of a list of layers.
5. Find the edges with path patching. Patch the input to one downstream component while every other consumer keeps reading the original. This distinguishes “head A matters and head B matters” from “head A matters through head B”, and it is the step that turns a list into a graph.
6. Characterise each head. What does it attend to? What does it write? A head whose output–value path consistently promotes the token it attends to is a copier; one that demotes it is a suppressor. This is where the names come from.
7. Verify by knockout. Mean-ablate everything outside the proposed circuit and check the behaviour survives. Ablate members and check it fails as predicted. Ablate a matched control set of heads with similar activation statistics and check it does not.

Step seven is the one that separates a circuit from a story, and it is the one most often skipped in secondhand accounts.

The head roles that came out

The IOI circuit resolved into several classes of head with distinct jobs. The naming is from the original work.

What makes this a satisfying result is that the roles compose into a readable algorithm: detect the duplicate, identify which name it is, inhibit attention to it, copy whatever name is left. That sentence is a hypothesis you can break by intervention, which is exactly what a circuit claim should be.

Self-repair, and why it complicates everything

The backup name movers are the most consequential finding on this page and they are usually a footnote. Ablate the primary name mover heads and the behaviour does not fail as much as it should — other heads that were doing almost nothing step up and partially restore it.

This has since been observed more generally and is sometimes called self-repair or the hydra effect: ablating a component causes downstream components to compensate. The mechanism is plausible enough. Later layers read a residual stream that now lacks a contribution, and components whose behaviour depends on that stream respond differently. Nothing is being repaired on purpose; the network is simply not a pipeline of independent parts.

The methodological consequence is severe and it applies to every ablation result anywhere in this field. A small effect from ablating a component is not evidence that the component was unimportant. It may mean the component was important and something else covered for it. Necessity experiments systematically under-report importance, and there is no general fix — the available response is to ablate sets of components together, and to report that you did.

The other circuits worth knowing

• Induction. The two-head prefix-matching mechanism, and the first transformer circuit described in full. Its own page covers the mechanism and the in-context learning claim attached to it.
• The greater-than circuit. Hanna and colleagues (2023) traced how GPT-2 small completes year ranges like “the war lasted from 1732 to 17__” with a token greater than 32, and found MLPs doing the comparison rather than attention alone — a useful corrective to circuit work’s attention bias.
• Vision circuits. Curve detectors and high-low-frequency detectors in convolutional networks, from the Distill thread. Still the clearest demonstration that a learned feature can be understood as built from named earlier features.

Why this has not scaled yet

Three obstacles, all structural rather than a matter of effort.

Combinatorics. GPT-2 small has 144 attention heads. Pairwise interactions are already tens of thousands of candidate edges, and the analysis above was months of work. A model with thousands of heads is not the same problem with more compute.

Superposition. If components are polysemantic, a head does not have a role. The IOI story works partly because templated prompts activate a narrow slice of what each head does. Superposition is why the field has moved towards sparse feature decompositions as the unit of analysis rather than heads and neurons.

Distribution narrowness. The circuit was established on a template. Whether the same components implement the same behaviour in ordinary prose is a further claim requiring further experiments, and the honest position is that circuit results are about the distribution they were verified on until someone shows otherwise.