Skip to content
Multigrid

Interpretability and model internals

What is actually inside a trained model, how researchers find out, and which of those techniques survive contact with a real system.

Interpretability is the attempt to say what a trained network is doing in terms a person can check. Not a story about what it might be doing — a claim precise enough that an experiment could show it false. That distinction is the whole field, and it is the thing most coverage of it leaves out.

These pages are written for someone who has read an abstract and wants the method. Each technique is explained by what you would run: which module you hook, which tensor you capture, what you change, what you compare it against, and what result would mean the hypothesis was wrong. Where a finding is famous, the group and the year are named. Where a number would be the point, the arithmetic or the script is here instead of a figure copied from a paper nobody re-checked.

The honest summary of the field is that it has produced a handful of genuinely reverse-engineered mechanisms in small models, a set of tools that reliably localise where a behaviour lives, and a much longer list of techniques that looked explanatory and did not survive a control. Several pages here are about that second list, because knowing which methods failed which sanity check is worth more than another walkthrough of an attention heatmap.

What Interpretability Research Is Trying to Do

Three distinct research programmes hide under one word. Each asks a different question, and each would count something different as success.

9 min read

Attention Maps: What They Show and What They Do Not

Attention weights show where information could flow, not why an output happened. The distinction, and the experiments that established it.

10 min read

Probing Classifiers: Finding What a Layer Knows

Train a classifier on hidden states to test what a layer encodes, with the control task that separates the representation's content from the probe's own capacity.

13 min read

Activation Patching and Causal Tracing

Swap one activation from a second forward pass and watch the output move. The experiment design, the metric, the code, and what the result licenses.

12 min read

Induction Heads and In-Context Learning

The two-head circuit that completes a repeated sequence, how it was isolated, and how far the in-context learning claim attached to it actually reaches.

11 min read

Superposition: Why One Neuron Means Several Things

How a layer represents more features than it has dimensions, why sparsity is what makes it possible, and what that costs anyone trying to read the model.

11 min read

Steering Vectors and Activation Engineering

Add a direction to the residual stream at inference time and behaviour changes. How the vector is built, where it is injected, and what it breaks.

11 min read

The Logit Lens: Reading Predictions From Middle Layers

Apply the model's own unembedding to an intermediate layer and read what it was leaning towards. Thirty lines of code, and the reason the output is sometimes noise.

10 min read

Circuits: Reverse-Engineering One Behaviour End to End

How a circuit is actually isolated: the task, the metric, the localisation sweep, the edges, the head roles, and the knockout that verifies it.

13 min read

Grokking and Delayed Generalisation

A network memorises its training set, sits at zero test accuracy for a long time, and then generalises. The setting, the mechanism, and how far it reaches.

10 min read

Memorisation and Generalisation, Measured

Extraction studies measure what can be pulled out under a specific attack budget. What that licenses you to conclude, and what it does not.

12 min read

Model Editing: Changing One Fact

Direct weight edits that change a single association without retraining, the four axes they are scored on, and the ripple effects that follow.

12 min read

Unlearning: Can a Model Forget on Request?

The approaches to removing knowledge from a trained model, and the evaluation problem that undermines most claims made for them.

12 min read

Comparing Representations Across Models

CKA, CCA and their relatives measure whether two networks represent things the same way. What each is invariant to decides what its answer means.

11 min read

Neuron-Level Explanations and Their Limits

How automated pipelines generate and score explanations of individual neurons, and the structural reasons a neuron may not have an explanation at all.

11 min read

Saliency Maps in Vision Models

Grad-CAM and its relatives, and the randomisation sanity checks that showed several widely used methods were largely independent of the model.

11 min read

Interpretability for Debugging, Not Just Research

The short list of internals-based techniques that pay for themselves on a real bug today, and the much longer list that does not.

11 min read

What “Explanation” Means to an Auditor

The regulator's definition of an explanation and the researcher's, side by side, and why an attribution map satisfies neither.

11 min read

Instrumenting an Open Model to Watch It Think

One script that hooks every layer of a local model and prints what each one added, what it was predicting, and how confident it was.

14 min read

What We Still Cannot Explain

Six open questions about model internals, each with the strongest current lead and the result that would settle it.

13 min read

Other topics