What Classification, Detection and Segmentation Each Actually Output

The three tasks are usually distinguished by what they “do”. It is more useful to look at what comes back, because the shape of the output decides what you can compute from it and what you must annotate to train it.

One image, three questions

Take a single 1920×1080 street photograph containing two people, one dog, a parked car, road across the bottom and sky across the top. Every output below is that same image through a different head. The question each answers is not a matter of emphasis; it is a matter of which quantities are recoverable from the returned tensor at all.

Classification: a vector

A classifier returns logits of shape (1, C). For an ImageNet-pretrained backbone C is 1000. A softmax converts them to probabilities that sum to one across the classes, and the answer is the argmax with its score.

logits.shape          -> (1, 1000)
softmax(logits)[0][:3] -> street_sign 0.31, minivan 0.22, Labrador 0.11

The forced competition is the defining property. Our image contains a dog and a car and two people, and softmax requires the scores to sum to one, so the presence of the car actively lowers the dog’s score. Multi-label classification replaces softmax with a per-class sigmoid and cross-entropy with binary cross-entropy; the outputs no longer sum to one, each class gets its own threshold, and the calibration problem becomes per-class. That is a different model, not a different way of reading the same one.

There is also no way for the vector to say “none of these”. Softmax always sums to one over the classes it has, so a photograph of something entirely outside the label set still produces a confident top-1. If your production stream contains inputs the training set has no class for — a blurred frame, an empty conveyor, a hand in front of the lens — that category has to exist as a class, or be rejected upstream by a separate check. It cannot be inferred from the scores.

What is unrecoverable: location, extent, count. If the question is “how many people”, no post-processing of a (1, 1000) vector answers it. Annotation, correspondingly, is one label per image — the cheapest supervision in vision by a wide margin, which is why classification is the right first attempt whenever the question genuinely is a single label per image. On what the score means, see confidence calibration.

Detection: a variable-length list

A detector returns a list of records, one per object: a box, a class id, a score. The list length varies per image, which is the awkward part — it is not a fixed-shape tensor, so the raw head output and the final output look nothing alike.

raw head, YOLOv8 at 640x640 input:
  (1, 84, 8400)
    84   = 4 box coordinates + 80 class scores
           (anchor-free; there is no separate objectness channel)
    8400 = 80*80 + 40*40 + 20*20, the stride 8 / 16 / 32 grids

after score filtering and non-maximum suppression:
  person  (612, 341, 705, 690)  0.94
  person  (688, 352, 771, 681)  0.88
  dog     (742, 588, 861, 692)  0.81
  car     (1104, 402, 1502, 612) 0.79

Two things in that transformation cause most integration bugs. First, box coordinates leave the network as centre-x, centre-y, width, height normalised to the network input, and the network input was produced by letterboxing — scaling to fit and padding to square. Rescaling by a plain width ratio instead of inverting the letterbox offsets every box by the padding. Second, the conventions on disk disagree with each other: COCO JSON stores [x, y, width, height] in absolute pixels from the top-left corner, Pascal VOC XML stores xmin, ymin, xmax, ymax in absolute pixels, and a YOLO label file stores class index followed by normalised centre-x, centre-y, width, height. Feeding one into a reader expecting another produces boxes that are silently wrong rather than an error. That ground is covered in the annotation formats.

The score is a per-class confidence after a sigmoid, not a probability that the detection is correct, and mAP integrates over every threshold so a model can rank well and still have no good single operating point. The list is also capped. Every detection implementation carries a maximum-detections parameter, commonly 100 or 300, applied after sorting by score. In a sparse scene it never binds; in a dense one it silently truncates the tail, which is exactly the regime where you were relying on the list length as a count. NMS is a further post-processing choice with its own threshold, and it is where genuinely overlapping objects get deleted — see what happens with occluded and overlapping objects. Annotation cost is one box per object, perhaps ten to twenty times a class label per image.

Segmentation: a map, or a stack of masks

Semantic. The head returns per-pixel logits of shape (1, C, H, W). A DeepLabv3 model on the 21-class Pascal VOC label set at a 513-pixel input returns (1, 21, 513, 513); argmax over dimension 1 gives a (513, 513) integer label map, which is then resized to 1920×1080 with nearest-neighbour interpolation. Bilinear resizing of a label map is meaningless — the average of class 7 and class 9 is class 8, a different object.

logits.shape                 -> (1, 21, 513, 513)
labels = logits.argmax(dim=1) -> (1, 513, 513)  int64

both people are one connected region of class "person":
  count is not recoverable
  area is: (labels == PERSON).sum() gives pixels directly

That memory shape is also why segmentation heads run at reduced resolution. A full-resolution 1000-class output would be 1000×1080×1920 floats, about 8.3 GB in fp32 for one image. Predict small, upsample after.

Instance. A Mask R-CNN style head produces, per detected box, a 28×28 mask logit map, which is sigmoided, thresholded at 0.5 and resized into that box’s pixel extent. The output is N boolean masks of shape (H, W) plus a class and a score each. Masks may overlap, and pixels belonging to no detection are simply not represented. The 28×28 origin is why instance masks look blocky on large objects: the boundary detail was never computed.

Panoptic. One (H, W) map where every pixel carries both a class and an instance id, exactly once, with no overlaps and no gaps. It is the only one of the three from which you can read both “how many people” and “how much of the frame is road” without ambiguity. The mechanism and its metric are in panoptic segmentation; the difference between the semantic and instance variants is in semantic versus instance segmentation.

Which output your problem needs

• “Is there a defect on this part?” Classification. One label per image, cheapest annotation, and the answer is directly the output.
• “Where are the defects and how many?” Detection. Counting is the list length; location is the box.
• “What fraction of the surface is corroded?” Semantic segmentation. The answer is a pixel count over a label map, and boxes cannot give it because they overlap and include background.
• “How many separate lesions and how large is each?” Instance segmentation. You need both separability and per-object area, which is exactly the pair neither of the other two provides.
• “Label every pixel once for a downstream consumer.” Panoptic, because the downstream consumer needs the partition property rather than the labels.

Annotation cost climbs in the same order — a label, a box, a polygon — and for a fixed budget that ordering is usually the real constraint rather than model quality. A classifier trained on twenty thousand labelled images will often beat a segmentation model trained on the eight hundred polygons the same effort buys, when the question can be phrased as a label at all.