Panoptic Segmentation: Combining Semantic and Instance Masks

Panoptic segmentation is usually introduced as “semantic plus instance”, which makes it sound like a union of two outputs. It is the opposite: it is a single output with a constraint neither of the other two has, and the constraint is the whole point.

What it actually outputs

A panoptic prediction assigns every pixel in the image exactly one pair: a semantic class, and an instance id. Not zero pairs, not two. Semantic segmentation gives you a class per pixel but cannot tell two adjacent people apart. Instance segmentation gives you separable objects but its masks may overlap, and any pixel belonging to no detected object — road, sky, wall — is simply absent from the output. Panoptic is the total, non-overlapping assignment: a partition of the image.

The COCO panoptic format stores that partition as a PNG rather than as polygons, because polygons cannot express “exactly once”. Each segment gets an integer id, and the id is encoded in the pixel colour as id = R + 256 * G + 256 * 256 * B. A sidecar JSON file carries a segments_info array mapping each id to its category_id, area, bbox and iscrowd flag. Pixels with id 0 are void: unlabelled regions that are excluded from evaluation entirely rather than counted as background.

# reading a COCO panoptic PNG
import numpy as np
from PIL import Image

rgb = np.array(Image.open("000000139.png"), dtype=np.uint32)
ids = rgb[:, :, 0] + 256 * rgb[:, :, 1] + 256 * 256 * rgb[:, :, 2]

# ids is now H x W of segment ids; segments_info in the JSON gives each
# id its category. np.unique(ids) is the segment list for this image.

That encoding matters practically. Reading the PNG with a lossy decoder, or resizing it with anything other than nearest-neighbour, corrupts ids into segments that never existed — interpolating between id 7 and id 9 produces id 8, which is a different object. The same trap applies to any label map; see how the annotation formats differ on disk.

Things, stuff, and why the split exists

Classes are divided into things and stuff. Things are countable and have instances: person, car, bottle, dog. Stuff is amorphous and has no meaningful instance count: sky, road, grass, wall. In COCO panoptic there are 80 thing classes and 53 stuff classes, 133 in total.

The distinction is not philosophical, it is operational. A thing class contributes one segment per object in the image; a stuff class contributes at most one segment per image, no matter how many disconnected regions of sky the buildings cut it into. That is why you cannot evaluate panoptic output by counting connected components: two separate patches of visible sky are one segment by definition, and a model that splits them into two is penalised.

The split also explains the two sub-scores you will see reported. Panoptic Quality is normally given as PQ, PQ^Th (things only) and PQ^St (stuff only), because the two behave very differently: stuff segments are large and easy to overlap well but hard to bound precisely, things are small and easy to miss entirely. A single PQ number hides which of those you have a problem with.

Panoptic Quality, decomposed

Kirillov, He, Girshick, Rother and Dollár defined both the task and its metric in “Panoptic Segmentation” (2018). The metric first has to decide which predicted segment corresponds to which ground-truth segment, and the matching rule is a single threshold: a predicted segment and a ground-truth segment of the same class match if their IoU is strictly greater than 0.5.

That threshold is doing more work than it looks. Because panoptic segments do not overlap, two predicted segments cannot both have IoU above 0.5 with the same ground-truth segment — their intersections with it would sum to more than the segment’s own area. So the matching is provably unique, and there is no assignment problem to solve, no greedy tie-breaking, no score-ordering. The non-overlap constraint from the first section is what buys that. It is the reason the task is defined the way it is.

With matches fixed, every segment falls into one of three buckets: a true positive (a matched pair), a false positive (a predicted segment with no match), or a false negative (a ground-truth segment with no match). Then

PQ  =  sum of IoU over matched pairs
       -----------------------------------
       |TP|  +  0.5 * |FP|  +  0.5 * |FN|

    =  SQ * RQ

SQ  =  (sum of IoU over matched pairs) / |TP|          segmentation quality
RQ  =  |TP| / (|TP| + 0.5 * |FP| + 0.5 * |FN|)         recognition quality

RQ is exactly the F₁ score over segments, and SQ is the mean IoU of the segments you did find. That factorisation is the useful part of the metric: a low PQ with a high SQ means you are missing or hallucinating whole objects, while a low PQ with a high RQ means you find everything and bound none of it well. Those need different fixes. PQ is computed per class and then averaged over classes, so a class that appears in three images counts as much as road, which appears in all of them.

A worked PQ calculation

Take one image whose ground truth has six segments: three people (things), plus road, sky and building (stuff). A model predicts six segments as well. Suppose the overlaps come out as follows — these are illustrative inputs chosen to exercise every bucket, not a measurement of any model.

• person A, IoU 0.92 with a ground-truth person — match
• person B, IoU 0.81 — match
• person C, IoU 0.44 with the third ground-truth person — not a match, because 0.44 is below 0.5
• road, IoU 0.95 — match
• sky, IoU 0.97 — match
• a spurious car segment overlapping nothing — no match
• the building in the ground truth has no predicted counterpart at all

So |TP| = 4, with IoU sum 0.92 + 0.81 + 0.95 + 0.97 = 3.65. The unmatched predictions are person C and the car, so |FP| = 2. The unmatched ground truth is the third person and the building, so |FN| = 2. Note that person C contributed to both: one near-miss segment is charged twice, once as a prediction nobody wanted and once as an object nobody found.

SQ = 3.65 / 4                     = 0.9125
RQ = 4 / (4 + 0.5*2 + 0.5*2)      = 4 / 6      = 0.6667
PQ = 0.9125 * 0.6667                            = 0.6083

direct form, as a check:
PQ = 3.65 / (4 + 0.5*2 + 0.5*2)   = 3.65 / 6   = 0.6083

A model that segments beautifully — mean IoU above 0.91 on everything it found — scores 0.61. That gap is the single most common surprise when a team moves from mIoU to PQ, and it is not a harsher metric being unfair. It is the double charge on near misses, and the fact that missing one building costs the same as missing one person. If you want the number to move, the lever is usually recognition, not boundary quality.

Where the two heads disagree

The architectures that produce panoptic output mostly do not produce it directly. A two-branch model — an instance branch giving scored, possibly overlapping masks, and a semantic branch giving a class per pixel — has to be fused into a legal partition, and the fusion is a heuristic:

1. Sort the instance masks by score, highest first, and paint them onto an empty canvas in that order. A pixel already claimed by a higher-scoring instance is not overwritten.
2. Discard any instance whose remaining unclaimed area, after the painting, falls below a fraction of its original area — typically around half. An instance almost entirely covered by a better-scoring one is assumed to be a duplicate.
3. Fill every still-unclaimed pixel from the semantic branch’s argmax, keeping only stuff classes.
4. Delete stuff segments below a minimum area, marking those pixels void.

Each step has a failure you can see in the output. A correct but low-scoring instance behind a high-scoring wrong one is painted over and then discarded by the overlap rule. The minimum-area threshold silently deletes small distant objects, which is why distant pedestrians vanish from panoptic output more often than from the detector that fed it. And because the semantic branch fills the gaps, a thing-class pixel the instance branch missed becomes whatever stuff class ranked highest there — a missed person turns into road, not into nothing.

Set-prediction models remove the heuristic. A transformer decoder with a fixed set of queries predicts one mask per query and is trained with a one-to-one Hungarian matching against ground truth, so duplicate suppression and the thing/stuff decision are learned rather than applied afterwards. That trades one failure mode for another: the query budget is fixed, so a scene with more segments than queries loses some, quietly. The general treatment of the two upstream tasks is in semantic and instance segmentation, and the output shapes side by side are in what each task actually returns.