Mechanistic Interpretability: Reading a Model's Mind

“Reading a model’s mind” is a headline, not a description. Mechanistic interpretability is the attempt to reverse-engineer the computation a trained network performs into parts a person can name and check — and the interesting question is which parts have actually been recovered.

What the programme claims

A trained transformer is a specific arithmetic procedure. Weights are known, activations are observable, and every intermediate value can be recorded or overwritten. Nothing about it is hidden in the way a human brain is hidden. The difficulty is not access; it is that the procedure is written in a representation nobody chose and nobody documented.

Mechanistic interpretability is the claim that this representation can be decompiled — that there are human-legible algorithms in there, and that finding them is an empirical task rather than a philosophical one. It is distinct from two neighbours it gets confused with. Behavioural interpretability studies inputs and outputs and never opens the model. Attribution methods — saliency maps, gradient-based feature importance — say which inputs mattered, but not what computation ran. Mechanistic work aims at the computation itself.

Why it matters for the rest of this cluster: nearly every safety proposal that involves trusting a model eventually needs a way to check a claim about its internals that its outputs cannot settle. A model that says it is not pursuing a goal and a model that is not pursuing a goal produce the same text.

Features and superposition

The first unit of analysis is the feature: a direction in activation space that corresponds to something a person can describe. The obvious place to look was individual neurons, and the obvious finding was that most neurons are polysemantic — one neuron fires for several unrelated things.

The leading explanation is superposition, set out by Elhage and colleagues in “Toy Models of Superposition” (2022). If a network needs to represent far more distinguishable concepts than it has dimensions, it can pack them as nearly-orthogonal directions and tolerate a little interference, which is a better trade than dropping the concepts. Under that account polysemantic neurons are not a mess to be cleaned up; they are the expected consequence of representing many sparse features in few dimensions.

That reframing produced the current main tool. If features are directions rather than neurons, recover the directions: train a sparse autoencoder (SAE) on the activations, with a wide overcomplete hidden layer and a sparsity penalty, and read off the dictionary elements. Bricken et al., “Towards Monosemanticity” (2023), did this on a small model and reported dictionary elements far more interpretable than the raw neurons; Templeton et al., “Scaling Monosemanticity” (2024), applied it to a production-scale model. The features are checkable in both directions: you can look at what activates them, and you can clamp them and see the output change.

Circuits and causal evidence

A feature is a noun. A circuit is the verb: a subgraph of attention heads and MLP layers that computes something. The vocabulary comes from Olah et al.’s “Zoom In” (2020) on vision models and from Elhage et al.’s “A Mathematical Framework for Transformer Circuits” (2021), which showed that small attention-only transformers decompose into readable paths.

The methodological core is causal, not correlational. Correlating an activation with a behaviour proves nothing about mechanism, so the field intervenes: patch an activation from one run into another and see whether the behaviour follows. Activation patching and path patching localise the responsible components; ablation removes them and checks that the behaviour breaks; causal scrubbing, introduced by Chan and colleagues, formalises a hypothesis about a circuit as a set of resamplings that ought to leave the behaviour intact and treats failure to survive them as refutation.

The standard of evidence is worth stating explicitly, because it is the part that separates this from just-so storytelling. A circuit claim is a claim that a specific intervention has a specific effect. It can be wrong, and published circuits have been revised when a later intervention did not behave as the story required.

What has been demonstrated

• Induction heads. Olsson et al. (2022) identified an attention-head pattern that completes repeated sequences by attending to what followed a token last time, and linked its formation to a visible bump in the training curve. This is the closest the field has to a fully-characterised, general mechanism.
• A named task in a real model. Wang et al. (2022) traced indirect object identification in GPT-2 small to a specific set of heads with distinct roles, and supported it with patching and ablation rather than description alone.
• Grokking made legible. Nanda et al. (2023) showed that a small network trained on modular arithmetic learns a recognisable Fourier-based algorithm, and that the delayed jump in test accuracy corresponds to a phase in that algorithm’s formation.
• Locating and editing a fact. Meng et al. (2022) localised specific factual associations to mid-layer MLP weights and edited them directly, which is a causal claim of an unusually crisp kind.
• Steering by feature. Clamping an SAE feature changes model behaviour in ways that match the feature’s label, which is evidence that the dictionary is picking up something the model uses rather than something the autoencoder invented.

What it cannot do yet

Coverage is the first gap. Named circuits explain narrow behaviours in mostly small models; there is no decomposition of a frontier model’s behaviour into circuits that accounts for most of what it does. Nobody in the field claims otherwise.

Evaluation of SAEs is the second, and it is genuinely unsettled. Sparsity and reconstruction error are proxies, and a dictionary can score well on both while splitting one concept across several elements or merging several into one. There is active disagreement about whether current SAEs recover the model’s features or merely a convenient basis for describing its activations, and the honest position is that this is an open research question rather than a solved one.

Third, understanding is not control. Being able to find and clamp a feature does not tell you that you have found all the features that matter for a behaviour, and an absence of evidence in a partial decomposition is weak evidence of absence. That distinction is the whole subject of whether deception can be detected, and it is why the control agenda deliberately assumes interpretability will not arrive in time.

Finally, the framing. Nothing here reads intentions. It reads computations, and interpreting a computation as an intention is an additional inferential step with its own error rate. Related but separate work asks whether a model’s stated reasoning corresponds to the computation that produced its answer — see faithfulness of chain-of-thought — and the two literatures answer different questions.