Detecting a Damaged or Occluded Object in an Image

Most of the recall you lose to occlusion is lost in the evaluation and the post-processing, not in the network. Two mechanisms account for it, and both are arithmetic you can work out before you touch a model.

Two different boxes for the same object

An occluded object has two defensible bounding boxes. The modal box encloses only the visible pixels. The amodal box encloses the object’s full extent, including the part hidden behind whatever is in front of it. They are different boxes and they answer different questions, and a large fraction of “the detector is bad at occlusion” is really the annotation convention disagreeing with the training target or with the evaluator.

The datasets that take this seriously annotate both. CrowdHuman — Shao and colleagues, “CrowdHuman: A Benchmark for Detecting Human in a Crowd” (2018) — gives every instance three boxes: head, visible region, and full body, across 15,000 training images with about 470,000 human instances and an average of 22.6 people per image. KITTI and CityPersons carry occlusion or visibility flags on each instance for the same reason. If your ground truth is amodal and your model is trained on visible extents, every occluded object scores badly even when the detector is behaving exactly as intended.

Which convention you want depends on the downstream use. A robot reaching for a partly hidden mug needs the amodal extent to plan a grasp. A system measuring how much of a pallet is visible for a loading check needs the modal extent. A tracker generally wants amodal, because the amodal box is stable through the occlusion while the modal box shrinks and grows. Pick one, then make annotation, loss and evaluation agree — that single alignment resolves more occlusion complaints than any architectural change.

What occlusion does to IoU, exactly

Suppose the ground truth is amodal, of area A, and the detector predicts precisely the visible region, of area v * A, where v is the visible fraction. The visible region is contained in the full extent, so the intersection is v * A and the union is A. Then IoU is exactly v. Not approximately: the containment makes it an identity, and it holds for any shape.

IoU = (v * A) / A = v        for a modal prediction inside an amodal box

v = 0.80  -> IoU 0.80
   true positive at IoU 0.50 and at IoU 0.75
   contributes at 7 of COCO's 10 thresholds (0.50 .. 0.80)

v = 0.60  -> IoU 0.60
   true positive at IoU 0.50, false positive at IoU 0.75
   contributes at 3 of 10 thresholds

v = 0.45  -> IoU 0.45
   below 0.50: counted as a false positive AND a false negative
   contributes at 0 of 10 thresholds

The step from 60% visible to 45% visible is where the recall cliff is, and note what happens there: a single correct-looking detection is charged twice, once as a prediction matching nothing and once as an object nobody found. Precision and recall both fall from the same event. That is the same double charge that makes panoptic scores look low; see how Panoptic Quality treats a near miss.

NMS deletes the second person

Greedy non-maximum suppression takes the highest-scoring box, deletes every remaining box whose IoU with it exceeds a threshold, and repeats. It exists to remove duplicate detections of one object. In a crowd it removes correct detections of different objects.

Two people standing one behind the other can easily have amodal boxes overlapping at IoU 0.65. With the common NMS threshold of 0.5, the lower-scoring one is deleted no matter how confident the model was about it. The model detected both; the post-processing threw one away. Raising the threshold to 0.7 keeps it and admits duplicate boxes everywhere else in the image, and there is no single global value that is right for both a sparse car park and a crowd — that trade is structural, not a tuning failure.

Soft-NMS — Bodla and colleagues, “Soft-NMS: Improving Object Detection With One Line of Code” (2017) — replaces deletion with a decay. Instead of removing an overlapping box, it multiplies that box’s score by a function of the overlap, typically a Gaussian:

greedy NMS   : s_i <- 0                      if IoU(M, b_i) > threshold
Soft-NMS     : s_i <- s_i * exp(-IoU(M, b_i)^2 / sigma)

A strongly overlapping true detection survives with a reduced score and is recovered by lowering the operating point, rather than being irrecoverable. It requires no retraining and is a drop-in change at inference time, which makes it the first thing to try.

Set-prediction detectors of the DETR family remove the problem differently: a fixed set of learned queries is matched one-to-one to the ground truth by a Hungarian assignment during training, so duplicate suppression is learned and there is no NMS at inference at all. The cost is a fixed query budget — a scene with more objects than queries loses some of them without any signal that it happened, which matters for exactly the dense scenes you adopted it for.

What changes in training

• Repulsion loss. Wang and colleagues (2018) add terms that push a predicted box away from neighbouring ground-truth objects and away from other predictions, so a box does not drift onto the adjacent person. It attacks the same crowding that NMS then has to clean up, at the point where it is created.
• Visibility-aware heads. Occlusion-aware detectors predict per-part visibility alongside the box and weight the pooled features by it, so an occluded region contributes less evidence rather than contributing misleading evidence.
• Erasing augmentation. Random erasing and cutout train the model to commit on partial evidence. This is the cheapest intervention on the list and generally worth doing regardless.
• Amodal supervision. Datasets with amodal masks train the model to complete the hidden extent. That is genuinely useful for grasping and tracking and actively harmful when you wanted the visible pixels — another reason to settle the convention first.

What to do without retraining

1. Confirm which convention your ground truth uses and which your model emits. Measure IoU on a handful of occluded instances by hand. If they systematically sit just under threshold, the convention is the bug.
2. Switch to Soft-NMS, or raise the NMS IoU threshold and lower the score threshold together, then remove the resulting duplicates using evidence NMS does not have — track continuity, or a depth/geometry check.
3. Use a second viewpoint. Occlusion is a property of the line of sight, not of the object; a camera thirty degrees away is very often unoccluded, and adding one is cheaper than any modelling work.
4. In video, track. An object hidden for eight frames is the same track, and a motion-model association carries identity through the gap using the pre-occlusion trajectory. This converts an occlusion problem into an interpolation problem.
5. When the scene is dense enough that boxes stop being meaningful, change the output. Density-map regression gives a count without ever resolving individuals; see counting objects in an image.
6. For people and articulated objects, keypoints degrade more gracefully than boxes because each keypoint carries its own visibility flag; see human pose estimation.

Damage is the same problem with a different cause. A crushed box, a torn label or a partially disassembled part removes exactly the evidence the model learned to rely on, and it removes it in ways not represented in the training distribution. If damaged instances are the thing you care about, they need their own class and their own examples; expecting a model trained on intact objects to flag a damaged one is asking it to generalise in the one direction it was never supervised in. The general shape of that failure is in why a model that tests well fails in production.