Point Cloud Segmentation Explained

Classification asks what a cloud is. Segmentation asks what every point in it is, which sounds like the same problem run N times and is not: the moment the label is per-point, a global descriptor is no longer an answer, and the metric you were using stops working.

What actually changes

A classifier collapses the set to one vector and predicts once. That collapse is exactly what a segmenter cannot do, because the pooled vector has no per-point information left in it — by construction, it is the same vector no matter which point you ask about.

The standard repair is to keep both. Compute the global descriptor as usual, then concatenate it back onto each point’s own local feature and run a second shared MLP over the combined vector. Each point now sees itself plus a summary of everything else, which is enough to say “I am a flat horizontal patch, and the scene I am in is a corridor, therefore floor”. That trick is the segmentation branch in the original PointNet and it survives, in some form, in most things since.

Encoder-decoder architectures take it further: downsample to a coarse set of points with wide context, then upsample back to the full resolution, interpolating features from the coarse level and adding skip connections from the matching fine level. It is the same U-shape as image segmentation, with farthest point sampling standing in for strided pooling and inverse-distance-weighted interpolation standing in for transposed convolution. The reason the shape recurs is that the underlying tension is the same — a per-element label needs both fine spatial resolution and a wide view, and one network stage cannot supply both.

Semantic, instance and panoptic

• Semantic segmentation gives every point a class: ground, building, vegetation, vehicle. Two adjacent cars are one connected blob of “vehicle” points, which is fine for a map and useless for counting.
• Instance segmentation separates the objects: this group of points is car #1, that group is car #2. Usually done by predicting an embedding per point and clustering it, or by predicting an offset from each point toward its object’s centre and then clustering the shifted points — the second is more robust because the shifted points of one object collapse into a tight ball while embeddings can drift.
• Panoptic segmentation does both, and draws the distinction that matters operationally: countable objects (“things”: cars, poles, people) get instance IDs, while unbounded regions (“stuff”: ground, vegetation, road surface) get only a class. Asking for instance IDs on ground points is not a harder problem, it is an ill-posed one.

The receptive-field problem

A point on a flat horizontal surface is locally indistinguishable whether it belongs to a road, a table top, or a roof. The information that resolves it lives metres away, and the network only gets it if the neighbourhood structure carries it there.

That makes the neighbourhood definition the most consequential hyperparameter in the pipeline, and there are two families with different failure modes. A k-nearest-neighbour graph always returns k points, so it adapts to density automatically — but in a sparse region those k points may span ten metres and cross three objects. A fixed-radius ball query has a fixed physical meaning, which is what you usually want — but it returns 300 points near the sensor and 2 points at the far edge of the scan, so the feature statistics differ across the same cloud.

Neither is correct in general. What is correct is to know which one your model uses and to check the two extremes of your data against it: run a histogram of neighbourhood point counts at several ranges before training, not after the results look strange.

Worked: why accuracy lies and mIoU does not

Outdoor scans are dominated by ground. Suppose a small labelled scan of 20 points: 14 ground, 4 building, 2 vegetation. A model that predicts “ground” for every single point scores:

overall accuracy = 14 / 20 = 70 %

per-class IoU = TP / (TP + FP + FN)
  ground:      TP 14, FP 6, FN 0  ->  14 / 20 = 0.700
  building:    TP  0, FP 0, FN 4  ->   0 /  4 = 0.000
  vegetation:  TP  0, FP 0, FN 2  ->   0 /  2 = 0.000

mean IoU = (0.700 + 0.000 + 0.000) / 3 = 0.233

Seventy per cent accuracy for a model that has learned nothing. Now a real prediction on the same scan: 13 of the 14 ground points correct with 1 building point wrongly called ground, 3 of 4 building points correct, and 1 of 2 vegetation points correct with the other called building.

ground:      TP 13, FP 1, FN 1  ->  13 / 15 = 0.867
building:    TP  3, FP 1, FN 1  ->   3 /  5 = 0.600
vegetation:  TP  1, FP 0, FN 1  ->   1 /  2 = 0.500

mean IoU = (0.867 + 0.600 + 0.500) / 3 = 0.656
overall accuracy = 17 / 20 = 85 %

Accuracy moved from 70% to 85% — a 15-point gain that undersells the difference. mIoU moved from 0.233 to 0.656, because it refuses to let the majority class carry the score and it penalises false positives and false negatives on the same footing. This is why every serious 3D segmentation benchmark reports mIoU and why per-class IoU should always be read alongside it: a mean of 0.65 built from 0.95, 0.95 and 0.05 is a model with a hole in it.

Almost all the error is at boundaries

Plot the per-point errors of a trained segmenter and they concentrate in thin shells: the join between a wall and the floor, the base of a tree trunk where it meets grass, the edge of a parked car against the road. Interior points are easy and there are many of them, which is another reason aggregate accuracy is uninformative.

Two mechanisms produce this. Neighbourhood features at a boundary mix two surfaces, so the input itself is ambiguous — a normal estimated over a ball that straddles a wall-floor join points at 45° and belongs to neither surface. And ground-truth annotation is least reliable in exactly those places, so the training signal there is noisiest. If your downstream task is sensitive to boundaries, measure a boundary-restricted IoU rather than trusting the global one.

What to do before reaching for a network

A large share of segmentation work in survey and construction data never needs a learned model. Ground is a plane or a smoothly varying surface and comes out with a RANSAC plane fit or a morphological filter, and removing it first typically deletes the majority of points and disconnects the rest into components you can label by size and shape. Vertical planes fall out the same way. Euclidean clustering on what remains separates objects with no training data at all.

Use a network when the classes are genuinely semantic — distinguishing a car from a rubbish bin of similar size, or a hedge from a fence — and not merely geometric. Where the distinction is geometric, a classical pipeline is faster, deterministic, has no domain-shift problem when the sensor changes, and can be debugged by looking at one number. The global-descriptor mechanism behind the learned option is worked through on the classification page.