Semantic and Instance Segmentation: The Difference That Matters

The difference is not a matter of precision. Semantic segmentation and instance segmentation return different data structures, and the one that merges two touching objects cannot be repaired into the one that separates them.

The two output shapes

Semantic segmentation returns one integer per pixel: an H × W array where each entry is a class index. For a 640×480 image and a 20-class problem, that is a single 640× 480 map of small integers, produced from a 640×480×20 tensor of per-class scores by an argmax over the class axis. Every pixel gets exactly one label, and there is no field anywhere in that structure that could hold an object identity.

Instance segmentation returns a list. Each entry is a binary mask, a class label and a score — the same list shape a detector returns, with a mask where the box was. Mask R-CNN produces exactly this by adding a small mask head to a detector, so it inherits the whole detection pipeline including non-max suppression, and inherits its failure modes too. The list can be empty. It can have two overlapping masks that both claim the same pixel, which is legal in this representation and impossible in the other.

The annotation cost differs in the same direction and by more than people expect. A semantic label can be produced by painting regions, and an annotator who paints the whole flock of sheep as one region has produced a correct label. An instance label requires deciding where one animal ends and the next begins, which is slower per image, needs a written convention for ambiguous cases, and produces disagreement between annotators that has to be adjudicated. If you commission semantic labels and later decide you needed instances, you are commissioning the work again from scratch — the existing masks contribute almost nothing, for exactly the reason the next section gives.

Two touching objects, on a grid

Here is a 12×6 patch containing two sheep standing flank to flank, with . for background and S for sheep. This is what a semantic model returns — a class per pixel and nothing else:

semantic map, class ids (0 = background, 7 = sheep)

  0 0 0 0 0 0 0 0 0 0 0 0
  0 0 7 7 7 7 7 7 7 7 0 0
  0 7 7 7 7 7 7 7 7 7 7 0
  0 7 7 7 7 7 7 7 7 7 7 0
  0 0 7 7 7 7 7 7 7 7 0 0
  0 0 0 0 0 0 0 0 0 0 0 0

count of pixels labelled 7 : 34
number of sheep in the image: 2
number of sheep derivable from this array: unknown

The two animals touch at column 6. There is no boundary pixel between them, because a boundary is not a class and the model was never asked to emit one. Instance segmentation returns something structurally different for the same patch:

instance 1: class sheep, score 0.94, mask =

  0 0 0 0 0 0 0 0 0 0 0 0
  0 0 1 1 1 1 1 0 0 0 0 0
  0 1 1 1 1 1 1 0 0 0 0 0
  0 1 1 1 1 1 1 0 0 0 0 0
  0 0 1 1 1 1 1 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0

instance 2: class sheep, score 0.88, mask =

  0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 1 1 1 0 0
  0 0 0 0 0 0 0 1 1 1 1 0
  0 0 0 0 0 0 0 1 1 1 1 0
  0 0 0 0 0 0 0 1 1 1 1 0
  0 0 0 0 0 0 0 1 1 1 0 0

union of the two masks = the semantic map above
count of sheep = len(instances) = 2

Note the direction of the arrow. You can always flatten instances into a semantic map by unioning the masks per class. You cannot go the other way, and that asymmetry is the entire practical content of the distinction.

Why you cannot recover instances afterwards

The usual first idea is connected-component labelling: find the connected blobs of class 7 and call each one an object. Run it on the map above and it returns 1, because the two sheep form a single 4-connected region. It is not a weak heuristic that could be tuned; the information required to split them was never in the array.

Watershed on a distance transform does slightly better and fails differently: it will split a region at its narrowest waist whether or not that waist is an object boundary, so it separates two touching sheep and also bisects one sheep photographed with its head turned. On any workload where counting matters — livestock, cells under a microscope, boxes on a pallet — you are choosing between two different wrong answers, which is a bad position to be in when the right model was available at training time.

The rule that follows is simple and worth applying before you pick an architecture: if a downstream consumer will ever ask “how many”, or attach an identity, or track something between frames, the task is instance segmentation. If the consumer only asks “how much of the image is road”, semantic is the correct and cheaper choice — it has no box head, no proposals, and no suppression step to tune. Where you need both, that is what panoptic segmentation is: instance masks for countable things and a semantic label for uncountable stuff, in one output with no pixel claimed twice.

mIoU and mask AP measure different things

Semantic segmentation is scored with mean IoU: for each class, compute the intersection of predicted and ground-truth pixel sets over their union, then average over classes. Two properties bite. First, it is computed over the whole dataset per class rather than per image, so a rare class present in ten images can dominate the mean as heavily as the sky. Second, it says nothing about objects, so a model that merges your two sheep perfectly scores 1.0 for the sheep class.

Instance segmentation is scored with mask AP, using the same machinery as box AP but with mask IoU as the overlap measure — in COCO, averaged over IoU thresholds 0.50 to 0.95 in steps of 0.05, as set in pycocotools’ cocoeval.py with iouType set to segm. Under mask AP the merged prediction scores badly, because one mask covering two ground-truth instances matches at most one of them and the other becomes a false negative. The metric and the task shape agree, which is the point.

Where both fail: boundaries and thin structures

Both tasks are dominated by interior pixels, and both metrics are area ratios, so a model can be badly wrong at every edge and still score well. A 512×512 object has roughly 260,000 interior pixels and about 2,000 boundary pixels; getting every boundary pixel wrong by one costs under 1% of IoU. If your application cares about the boundary — measuring a part, cutting out a product photo — mIoU is the wrong headline number and boundary IoU or a trimap band around the edge is what to report.

Thin structures fail for a related reason. Networks that downsample by 32× before upsampling cannot represent a two-pixel-wide cable at the bottleneck at all, and the mask head’s own resolution is often much smaller than the box — Mask R-CNN’s default is a 28×28 mask upsampled to fit, so a 400-pixel-tall person has their silhouette described by 28 rows. That is enough for a torso and not nearly enough for fingers, and it is a resolution limit rather than a training problem, so more data does not move it.