Model Interpretability: SHAP, LIME and Their Limits

An attribution tells you what the model used. It does not tell you what causes the outcome, it does not tell you the model is right, and — as one well-known paper demonstrates — it does not even guarantee that a model behaving badly on real data will look bad in the explanation.

Two questions people conflate

• What did this model use to produce this output? A question about the model. Attribution methods answer it, subject to assumptions.
• What causes the outcome in the world? A question about reality. Attribution methods do not answer it and cannot, because the model only ever saw correlations.

The gap between them is where interpretability does damage. If postcode is a proxy for income and the model leans on postcode, SHAP will correctly report postcode as important. Reading that as “postcode causes default” is a category error, and it is the error that gets repeated in a slide deck to a stakeholder who cannot see the distinction.

SHAP: the axioms and the price

Shapley values come from cooperative game theory (Shapley, 1953). The setup: a coalition of players produces a payoff, and you want to divide the payoff fairly. Map features to players and the model’s prediction to the payoff, and you have SHAP — Lundberg and Lee (NeurIPS 2017).

The reason it is not just one method among many is a uniqueness theorem. Shapley values are the only attribution satisfying all four of:

• Efficiency. The attributions sum exactly to the prediction minus a baseline. Nothing is unexplained and nothing is invented.
• Symmetry. Two features that contribute identically in every coalition receive identical credit.
• Dummy. A feature that never changes the output gets zero.
• Additivity. Attributions for an ensemble are the sum of the attributions for its members — which is what makes TreeSHAP over a boosted ensemble coherent.

The price is combinatorial: exact Shapley values require evaluating every subset of features, 2ᵈ of them. TreeSHAP computes them in polynomial time for tree ensembles, which is why SHAP is ubiquitous in exactly the setting where boosted trees are; for other models KernelSHAP samples, and the result is an estimate with variance that people routinely quote to three decimal places.

Two assumptions to keep visible. The baseline is a choice — attributions are always relative to something (the dataset mean, a specific reference row), and changing it changes every number. And “removing” a feature is not something a model supports, so it is simulated by marginalising over a background distribution; the common implementation does this assuming feature independence, which is false whenever features are correlated, which is nearly always. Correlated features can therefore split credit in ways that look arbitrary, because they are.

LIME: a local surrogate

Ribeiro, Singh and Guestrin (KDD 2016) take a different route: perturb the input around the point of interest, get the black-box model’s predictions on the perturbations, and fit a sparse linear model weighted by proximity to the original point. The coefficients of that little linear model are the explanation.

It is model-agnostic and it is cheap, and both of its knobs are load-bearing. The perturbation distribution decides what “nearby” means — for tabular data that is a sampling choice, for text it is usually deleting words, for images it is toggling superpixels. The kernel width decides how local “local” is. Neither has a principled default, and explanations are known to move when they change, so a LIME output without its settings recorded is not reproducible.

What an explanation is not evidence of

The sharpest result here is worth knowing by name. Slack, Hilgard, Jia, Singh and Lakkaraju (AIES 2020, “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods”) exploit the fact that both methods query the model on perturbed inputs, which do not look like real data. An adversary can therefore build a classifier that detects whether it is being asked about a real record or a perturbed one, behave however it likes on real records, and behave innocuously on the perturbed queries the explainer uses. The explanation comes back clean.

The lesson is not that the methods are useless. It is that an explanation is an artefact produced by querying a model, and anything that can distinguish those queries from real traffic can influence the artefact. An attribution is not an audit, and it is not a compliance control.

Which is the argument Rudin makes directly (Nature Machine Intelligence, 2019, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”): where a decision must be justified, use a model whose logic is the explanation — a scoring rule, a short decision list, a sparse linear model — rather than an opaque model plus a post-hoc story about it.

Using them for what they are good at

• Debugging, and specifically leak-hunting. The best use by far. A feature with implausibly high importance is the standard signature of target leakage, and attributions find it in minutes.
• Sanity-checking against domain knowledge. If an expert says the model should care about tenure and it does not, something is wrong with the features or the label — the explanation has generated a hypothesis, which is its proper output.
• Comparing two models’ behaviour. Stable across a retrain, or completely different? A model whose attributions reshuffle every retrain is telling you something about its variance.
• Not as a justification to an affected person. “Your loan was declined, and here is a Shapley value” is not a reason. It is a description of a computation.

The language-model analogue is worth flagging, since the same confusion has been imported wholesale: a chain of thought is generated text, not a log of the computation that produced the answer, and treating it as an explanation assumes a faithfulness that the research does not support unconditionally. Token-level attributions have the same status as SHAP values here — evidence about the model, not about the world, and not a substitute for testing the output.