Saliency Maps in Vision Models
11 min read · updated August 4, 2026
A saliency map highlights the pixels that supposedly drove a prediction. In 2018 Adebayo and colleagues showed that several popular methods produce nearly the same map when the model’s weights are randomised — which means those maps were reporting on the image, not on the model. That result is the most useful thing to know about this literature.
The family of methods
| Method | Description |
|---|---|
| vanilla gradient | The gradient of the class score with respect to the input. A first-order sensitivity map: how much would this pixel change the score. Extremely noisy, because gradients of a deep ReLU network are locally erratic. |
| gradient × input | Multiply the gradient by the input value. Reduces noise and gives an attribution rather than a sensitivity — an approximation of each pixel's contribution rather than its influence. |
| integrated gradients | Sundararajan and colleagues, 2017. Integrate the gradient along a straight path from a baseline image to the real one. Satisfies stated axioms including completeness — attributions sum to the score difference from the baseline. The baseline choice matters enormously and is usually left unstated. |
| SmoothGrad | Smilkov and colleagues, 2017. Average the gradient over many noisy copies of the input. A visual improvement that makes maps look cleaner; whether it makes them more faithful is a separate question and not settled by how they look. |
| guided backpropagation | Modifies the backward pass through ReLUs to suppress negative signals. Produces strikingly sharp, edge-like images. Failed the sanity checks — see below. |
| CAM / Grad-CAM | Selvaraju and colleagues, 2017. Weight the final convolutional feature maps by gradient-derived importance and upsample. Coarse but class-discriminative: change the target class and the map genuinely moves. |
| occlusion / LIME / SHAP | Perturbation-based. Cover regions, re-run the model, measure the change. Model-agnostic and expensive; results depend heavily on the patch size and on how you fill the occluded region. |
The two sanity checks
The 2018 paper “Sanity Checks for Saliency Maps” proposed two tests, and their design is the reason they bite. Both are necessary conditions: passing does not make a method good, but failing means it cannot be doing what it claims.
The model randomisation test. Take your trained model, produce a saliency map for an image. Now randomise the model’s weights — either all of them, or progressively from the last layer backwards — and produce the map again. An explanation of the model’s reasoning must change, drastically, because the model no longer computes anything. If the map is substantially unchanged, the method is a function of the input and the architecture, not of what the model learned.
The data randomisation test. Train the same architecture on the same images with the labels randomly permuted. The resulting model has memorised noise and cannot have learned meaningful class features. Its saliency maps should look nothing like those of the properly trained model. If they look the same, again, the method is not reporting on what the model learned.
What failed, and why that is the useful part
The reported outcome: guided backpropagation and guided Grad-CAM produced maps that were largely invariant to model randomisation. The sharp, convincing, edge-outlined images those methods are known for were substantially a function of the input, produced by an operation that behaves somewhat like an edge detector regardless of what the network had learned. Gradient-based methods and Grad-CAM behaved better under the tests.
The reason this matters more than a ranking of methods: the failing maps were the most convincing ones. They looked like explanations. Sharpness, alignment with object boundaries and visual plausibility are exactly the properties a human reviewer uses to judge an explanation, and they were uncorrelated with whether the explanation was about the model.
Running the checks yourself
These are cheap and you should run them on whatever attribution method you are relying on, including one somebody else validated — the result depends on the architecture and the task.
- Fix a set of about fifty images and compute the attribution map for each with the trained model. Store them.
- Re-initialise the model’s weights layer by layer from the output backwards, recomputing the maps at each stage. Full randomisation at once is the blunt version; the cascade tells you which layers the method is actually sensitive to.
- Compare each randomised map to the original with more than one metric — rank correlation over pixels, structural similarity, and the overlap of the top-k pixel sets. These disagree, and disagreement is informative.
- Establish a floor: compare the original map to the map of a different image. That is what “unrelated” scores. A randomised-model similarity near your original-versus-original score is a failure; near the unrelated-image score is a pass.
- For the data randomisation test, permute the training labels, retrain to convergence on the training set, and repeat the comparison. More expensive, and the more damning of the two when a method fails it.
Report the numbers rather than showing the pictures. The entire lesson of the 2018 result is that side-by-side images are not how you judge this.
The other failure modes
- Baseline dependence. Integrated gradients requires a baseline input representing “absence”. A black image makes genuinely black pixels unattributable by construction; a blurred image, a random image and a dataset mean all give different attributions for the same prediction. The baseline is a modelling assumption presented as a default.
- Input-shift sensitivity. Kindermans and colleagues showed that adding a constant to every input — a transformation a network can trivially absorb into its first bias, leaving predictions identical — changes the attributions produced by several methods. An explanation that moves when the model’s behaviour does not is reporting something other than the model.
- Manipulability. Subsequent work has shown that a model can be fine-tuned to produce a chosen attribution map while keeping its predictions essentially unchanged. If explanations can be optimised independently of behaviour, an explanation cannot be used as evidence of good behaviour by anyone who does not trust the party producing it — which is a serious problem for regulatory uses of explainability.
The same problems in text models
Token-level attribution inherits all of it and adds one. In an image, neighbouring pixels are correlated and a map is judged as a region. In text, the units are discrete tokens, the baseline is even more arbitrary — what is the “absence” of a word? — and a single tokenisation boundary can split the word your attribution is about into three pieces with three different scores.
The one advantage text has is that intervention is cheap and natural. Deleting a token and re-running is a well-defined counterfactual, and for text a leave-one-out ablation is often more trustworthy than any gradient method, at the cost of one forward pass per token.
What to use, and for what
For debugging a vision model: Grad-CAM, checked against the randomisation test on your own architecture, and treated as a hypothesis generator. Its best use is catching a model that has latched onto a spurious background cue — a coarse, robust signal that a coarse method can deliver.
For a faithfulness claim: perturbation-based evaluation. Remove what the method says was important, measure how much the prediction changes, and compare against removing random regions of the same size. This tests the attribution against the model’s actual behaviour instead of against your eye.
For explaining a decision to a person affected by it: not a saliency map. A counterfactual — what would have had to be different for the outcome to change — is both more actionable and more robust, and it is closer to what people asking for an explanation actually want.