What “Explanation” Means to an Auditor

A researcher explaining a model wants a mechanism. A regulator wants something a specific person can act on: why this decision, about me, and what would have to change. Those are different artefacts, and the most common compliance mistake is producing the first when the obligation asks for the second.

Two definitions that do not overlap

What GDPR actually requires

The popular formulation is a “right to explanation”. The careful version is narrower. Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects, and where such processing is permitted it requires safeguards including the right to obtain human intervention, to express a point of view, and to contest the decision. Articles 13 to 15 require, where such processing occurs, meaningful information about the logic involved and the significance and envisaged consequences.

Recital 71 refers to obtaining an explanation of a decision reached after automated assessment. Recitals are not binding operative text, and Wachter, Mittelstadt and Floridi argued in 2017 that a legally enforceable right to an explanation of a specific decision does not straightforwardly exist in the GDPR as drafted — a contested reading, but the debate is why careful practitioners talk about “information about the logic involved” rather than about a right to explanation.

What that phrase demands is the interesting part. It is not the weights and not the algorithm. Regulatory guidance has pointed consistently at the categories of data used, their relative importance, and the consequences — information that lets a person understand and challenge the decision. The same authors proposed counterfactual explanations as a construction that satisfies this without disclosing the model: you were declined; had your income been X rather than Y, with everything else unchanged, you would have been approved.

The EU AI Act’s version

The AI Act, adopted in 2024, splits the requirement in two, and the split is the useful thing to understand.

Transparency towards the deployer. High-risk systems must be accompanied by instructions for use and be designed so that their operation is sufficiently transparent for deployers to interpret the output and use it appropriately. This is a documentation-and-design obligation on the provider, discharged with technical documentation, characteristics and limitations, expected performance, and human oversight provisions — not with an attribution method.

Explanation to the affected person. The Act also provides a right for a person subject to a decision taken by a deployer on the basis of output from a high-risk system, where that decision produces legal or similarly significant effects, to obtain clear and meaningful explanations of the role of the system in the decision procedure and the main elements of the decision taken.

Read that last phrase carefully: the role of the system in the procedure and the main elements of the decision. It is a question about your process and the factors, not about your network. The AI Act overview covers the classification rules that decide whether any of this applies to you.

The oldest requirement: adverse action notices

Long before any of this, United States credit law required lenders to tell applicants why they were declined. The Equal Credit Opportunity Act and its implementing Regulation B require a statement of specific principal reasons for adverse action — not a general statement, not a reference to a scoring system, but the actual principal reasons.

The Consumer Financial Protection Bureau addressed complex models directly in a 2022 circular, stating in substance that the use of a model whose reasons a creditor cannot articulate does not excuse the requirement. That is the clearest existing statement of the principle the whole field runs into: a model you cannot explain is a model you may not be permitted to use for that purpose. Explainability is a constraint on model choice, not a reporting layer added afterwards.

Note also how the industry has satisfied this for decades: reason codes. A fixed vocabulary of factors, ranked, attached to each decision. Not an explanation of the model — an explanation of the decision, in a form a person can act on and a supervisor can audit.

The artefacts that discharge the obligation

• A decision record per decision. Inputs used, model and version, output, the threshold or rule applied, timestamp, and any human review. This is the single most valuable artefact and it is an engineering task, not a research one.
• Reasons in a controlled vocabulary. A ranked set of factors drawn from a fixed list, mapped to plain language. Stable, auditable, and comparable across cases — which a free-text explanation is not.
• Counterfactuals where they can be computed. The most useful thing you can give a person, and it does not require interpreting the model — only the ability to re-run it with a changed input.
• System documentation. Purpose, data, performance across relevant groups, known limitations, the human oversight arrangement. A model card is a reasonable starting structure for this.
• Evidence of human involvement. Much of the regulatory weight falls on whether a decision was made solely by automated means. Meaningful human review, with a record showing it was meaningful, changes which obligations attach at all.

Why research explainability does not fit

Three reasons, and they are structural rather than a matter of maturity.

Faithfulness is not established. An obligation to give an accurate explanation is not discharged by a method you cannot verify. The randomisation sanity checks on saliency maps are directly relevant: an explanation that survives randomising the model is not evidence about the decision, whatever it looks like.

The unit is wrong. A circuit explains a behaviour across a distribution. The obligation is about one decision about one person. Even a perfect mechanistic account does not automatically yield the sentence the applicant is owed.

The audience is wrong. “Attention head 9.6 attended to your postcode” is not meaningful information to a lay reader, and meaningfulness to the recipient is an explicit criterion in more than one instrument.

What to build

1. Determine whether the decision is in scope: does it have legal or similarly significant effect, and is it made solely by automated means? Most systems are out of scope, and knowing that saves building the rest.
2. Log the decision record from day one. Retrofitting it is the expensive path and the gaps are unrecoverable.
3. Define the reason vocabulary before choosing the model. If a model cannot produce reasons from that vocabulary, that is a fact about whether you can use it here.
4. Prefer a simpler model where the obligation is strict. A constrained model whose factors are legible may be worth more than a stronger model you cannot explain — this is a real trade-off, and pretending post-hoc explainability closes it is how organisations end up out of compliance with a good-looking dashboard.
5. Where a complex model is genuinely necessary, generate counterfactuals and validate them by re-running the model on the counterfactual input. An explanation you have tested is worth more than an attribution you have not.