What We Still Cannot Explain

Interpretability has produced a handful of genuinely reverse-engineered mechanisms and a much longer list of methods that did not survive their controls. This is an inventory of what remains open, argued from what has been published rather than from anyone here having run it — and for each question, the experiment that would settle it.

What counts as an open question here

Not “we do not fully understand neural networks”, which is true and useless. The questions below share three properties: the field agrees they are unresolved, there is at least one serious research programme aimed at each, and each has a describable result that would count as an answer. A question with no such result is not an open problem, it is a mood.

They are ordered by how much else depends on them. The first is foundational: if features are not the right unit, a large fraction of current work is measuring its own instruments.

1. Are features real?

The question. Sparse autoencoders decompose activations into units that look interpretable. Is that decomposition something the model uses, or an artefact of an objective that always produces a sparse decomposition of anything you point it at?

This is not scepticism for its own sake. A sparse autoencoder trained on random Gaussian activations will also converge and also produce sparse units, and some of them will admit plausible descriptions because a description can be found for almost any set of examples — the same weakness the neuron explanation literature had to build automated scoring to escape. The sparsity penalty is a hyperparameter, and moving it changes how many features you get, with no principled setting that says which count is correct.

The best current lead: causal validation. A feature that is real should support an intervention whose effect its description predicts — suppress it and the behaviour it names should degrade specifically, amplify it and the behaviour should intensify. Published work in this direction is the strongest evidence the framework has, because it tests the decomposition against the model rather than against a human reader.

What would settle it. A trained model with known ground-truth structure — features planted by construction — where a sparse autoencoder recovers them without being told what to look for, at a sparsity setting chosen without reference to the answer. That is a hard experiment to design well, because the planting procedure biases what is recoverable, and it is exactly why the toy models in the superposition work matter more than their size suggests.

2. Does any of this scale?

The question. Every complete circuit result is in a small model on a narrow, usually templated task. Whether the approach extends to a frontier model on ordinary text is unresolved, and it is not obviously a matter of more compute.

Three specific obstacles, all structural. The combinatorics of component interactions grow far faster than model size. Superposition means components do not have single roles, so the unit of analysis is uncertain. And a circuit verified on a template has not been shown to be what the model does on prose, which means the narrowness that makes the analysis tractable is also what limits the conclusion.

The best current lead: automation plus a better unit. Automated circuit discovery replaces the manual search over components with a procedure, and sparse feature decompositions replace heads and neurons with units that may be more nearly monosemantic. Both are bets that the bottleneck is human labour and basis choice rather than irreducible complexity.

What would settle it. A circuit-level account of one non-trivial behaviour in a model of production scale, verified by intervention, on a natural rather than templated distribution. Nothing of that description exists. Its absence is the single most honest summary of where the field is.

3. Is stated reasoning connected to computation?

The question. A model that writes out its reasoning before answering usually does better. Is the written reasoning the computation that produced the answer, or a plausible narrative generated alongside it?

The published evidence is uncomfortable. Studies that perturb a chain of thought — introducing an error partway and seeing whether the conclusion follows the corrupted reasoning or the original answer — find cases where the stated steps are not what the answer depends on. Work on biasing features that a model is demonstrably influenced by, but does not mention, points the same way. The faithfulness question is not a curiosity: a great deal of AI oversight assumes that reading a model’s reasoning tells you something about its process.

The best current lead: intervention on the reasoning itself, treated as an activation-patching problem rather than a reading-comprehension one. Corrupt a step, measure the effect on the conclusion, and compare against corrupting a step the model should not be relying on. That converts a vague worry into a measurable quantity per model and per task, which is what a research programme needs.

What would settle it. Nothing settles it globally — faithfulness is a property of a model and a task, not of reasoning as such. The achievable result is a reliable per-model faithfulness measure that predicts when a stated rationale can be trusted. That would be enough to act on, and it does not exist as a standard instrument today.

4. How do you verify absence?

The question. Showing a model can do something takes one demonstration. Showing it cannot — that a capability was removed, that a fact is gone, that a behaviour will not appear under any prompt — requires ruling out an unbounded space of probes. There is no known experiment of that shape.

This is the deepest problem on the list because it is close to logical rather than technical, and it recurs everywhere: unlearning claims, extraction and memorisation claims, and every safety evaluation that concludes a model is incapable of something. A negative result in all three cases is a statement about the battery of tests, not about the model.

The best current lead: adversarial batteries with a relearning test. Fine-tuning an allegedly-scrubbed model on a small amount of related data and seeing whether the capability returns has proved unusually good at separating suppression from removal, because it probes the weights rather than the output distribution. Internal probes are the complement: decodability of a supposedly removed property is direct evidence it survived.

What would settle it. Realistically, nothing settles it — but the useful target is a lower bound: a procedure that certifies a capability cannot be recovered under a defined computational budget. That is a weaker and achievable claim, it is the form security guarantees already take, and stating results that way would be an improvement on the current practice of reporting an unqualified absence.

5. Why do capabilities appear when they do?

The question. Some abilities seem to arrive sharply rather than improving smoothly. Is that a real property of the underlying computation, or an artefact of how it is measured?

The evidence pulls in two directions, and both directions are instructive. Grokking is a real, sharp transition in a small model, and its mechanism was worked out: a generalising circuit forms gradually beneath a memorising solution and becomes visible only when regularisation erodes the memoriser. That is an existence proof that a sharp behavioural transition can sit on top of continuous internal change. Against that, Schaeffer and colleagues argued in 2023 that many reported emergent jumps at scale are artefacts of discontinuous metrics, and dissolve under smoother measures of the same tasks.

The best current lead: progress measures derived from a mechanistic hypothesis. The restricted and excluded losses used in the grokking analysis moved smoothly across a plateau where every standard metric was flat. That is the template: given a hypothesis about the mechanism, construct a quantity that tracks it, and check whether it moves before the capability appears.

What would settle it. A progress measure computed during training that predicts a capability before it is behaviourally visible, in a model somebody actually cares about. Every existing example was constructed after the mechanism was known, which makes them demonstrations rather than instruments.

6. Can internals catch what behaviour cannot?

The question. The strongest argument for interpretability as a safety technology is that a model behaving well because it is being evaluated is indistinguishable, from the outside, from a model behaving well. If internal state could distinguish them, interpretability would be doing something no amount of testing can.

The evidence that this is even possible is real but limited: probes can recover properties from hidden states that the output does not express, and intervening on directions found that way changes behaviour, which is a genuine causal link rather than a correlation.

The problem with the programme is circularity, and it is worth stating clearly. A probe is trained on labelled examples of the state you want to detect. Producing those labels requires already being able to tell, which is the thing you lacked. Work in this area leans on constructed settings where ground truth is known by design, and whether a detector trained on constructed cases transfers to a case nobody constructed is exactly the open question.

The best current lead: generalisation testing under deliberate distribution shift — train the detector on one family of constructed cases and test it on a structurally different family, rather than on held-out examples of the same family. A detector that survives that has some claim to be tracking the property rather than the construction.

What would settle it. A detector that flags a behaviour nobody had labelled in advance, verified afterwards by an independent route. That is a high bar and it is the right one, because anything less is a detector validated on the distribution it was built for.

What progress would look like

The field’s best results share a shape, and it is worth naming because it is the thing to look for in the next paper. Induction heads: a mechanism, verified by ablation, tied to a macroscopic capability by multiple independent lines of evidence. Grokking: a mystery, a mechanism, and a measure derived from the mechanism that showed something the original observation could not. In both cases the story made a prediction that could have failed.

The results that did not hold up share a shape too. Attention maps and several saliency methods were adopted because their output looked like an explanation, and the controls came later — the “attention is not explanation” exchange and the randomisation sanity checks. Both would have been caught earlier by asking, at the start, what result would show this method is not working.

So the concrete standard for reading anything in this area is unchanged from the top of this cluster: find the intervention, find the control, find the distribution, and check the abstract’s verb against what the method supports. On current evidence, interpretability is a real research programme with a small number of solid results, a serious foundational uncertainty about its own units, and no demonstrated ability yet to explain a production model’s behaviour end to end. Anyone telling you otherwise is selling something, and the list of techniques that pay for themselves today is the honest short version.