Point Cloud Classification Explained

Classifying a point cloud means putting one label on a whole set of coordinates: this scan is a chair, that one is a lamp. The hard part is not the geometry. It is that the same object arrives as N! different matrices depending on the order the scanner happened to write the points in, and a network must give all of them the same answer.

The problem is the ordering, not the geometry

An image has a fixed grid. Pixel (0,0) is always the top left, its neighbours are always the same four pixels, and a convolution can rely on both facts. A point cloud has neither. It is an unordered set of (x, y, z) triples, often with intensity or colour attached, and the storage order is an artefact of the sensor’s sweep or of whatever tool wrote the file last.

Feed that set into a multilayer perceptron by flattening it into a vector of length 3N and you have built a model of the write order. Sort the points by x and you get one answer; sort by scan line and you get another. Neither is wrong in the data, so the model has learned something that is not a property of the object.

There is also no fixed N. One scan of a chair has 2,048 points and another has 30,000. A flattened input layer needs a fixed width, so you would have to resample every cloud to the same count before the model saw it, throwing away detail on the dense ones and inventing it on the sparse.

A symmetric function is the only fix

The requirement can be written down exactly: the network f must satisfy f(p1, ..., pN) = f(pσ(1), ..., pσ(N)) for every permutation σ. A function with that property is called symmetric, and the useful ones are few: max, sum, mean, and things built from them.

PointNet, introduced by Charles Qi, Hao Su, Kaichun Mo and Leonidas Guibas at CVPR 2017, takes the most direct possible route. Apply the same small MLP to every point independently, lifting each (x, y, z) to a high-dimensional feature vector; then collapse the set with a single element-wise max across points. The per-point stage cannot see order because it never sees more than one point. The pooling stage cannot see order because max does not care. Everything after the pool is an ordinary classifier on one fixed-length vector.

The paper also proves the shape is not a compromise: any continuous set function can be approximated arbitrarily well by this per-point-MLP-then-max form, given enough width. That result is what makes the architecture worth understanding rather than merely remembering. The PointNet paper states the universal approximation result and the critical-set bound.

Worked: pooling four points

Take four points and suppose the per-point MLP has already lifted each to a three-dimensional feature. Three dimensions instead of the usual 1,024, so the arithmetic fits on a line:

point        feature after the shared MLP
p1 (2,0,0)   [0.90, 0.10, 0.40]
p2 (0,2,0)   [0.20, 0.80, 0.30]
p3 (0,0,2)   [0.30, 0.20, 0.90]
p4 (1,1,1)   [0.50, 0.60, 0.50]

max over points, channel by channel:
  channel 0: max(0.90, 0.20, 0.30, 0.50) = 0.90   from p1
  channel 1: max(0.10, 0.80, 0.20, 0.60) = 0.80   from p2
  channel 2: max(0.40, 0.30, 0.90, 0.50) = 0.90   from p3

global descriptor = [0.90, 0.80, 0.90]

Now write the four points in any other order. Each still goes through the same MLP and produces the same feature, and the three maxima are taken over the same four numbers. The descriptor is identical. That is the whole invariance argument, and it is worth doing once by hand because it also exposes the next thing.

The critical point set

Look at which point won each channel. Channel 0 came from p1, channel 1 from p2, channel 2 from p3. Nothing came from p4. Delete p4 from the cloud entirely and the global descriptor does not change by a single digit — and neither does the classification, because the classifier only ever sees the descriptor.

This generalises. With K output channels, at most K points can contribute to the pooled vector, so a cloud of 30,000 points running through a 1,024-channel pool is summarised by at most 1,024 of them. PointNet calls these the critical point set, and it is the reason the architecture is unusually robust to missing data: dropping points that were not critical is provably a no-op. It is also the reason the descriptor saturates — adding points beyond the critical set cannot make the representation richer, only differently distributed.

Max is chosen over sum for exactly this. Sum pooling is equally symmetric but every point moves the result, so the descriptor drifts with point count and a denser scan of the same chair lands somewhere else in feature space. Mean pooling fixes the scale but blurs: a hundred points on a flat seat outvote the four on a thin armrest, and the armrest is what distinguishes the class.

What flat pooling throws away

The shared MLP sees one point at a time. That means no point ever knows what its neighbours look like, and the only place information combines is the single global pool. For distinguishing a chair from a lamp on a clean CAD model that is enough. For anything with fine local structure it is not, because local geometry — the curvature of an edge, the spacing of a railing — never gets computed anywhere.

PointNet++, from the same group later in 2017, is the standard answer: apply PointNet to small neighbourhoods, pool those into a smaller set of points with richer features, and repeat. Neighbourhoods come from a ball query of fixed radius around centroids chosen by farthest point sampling, which gives a hierarchy analogous to a convolutional network’s receptive field growth. The radius is the parameter that matters, and it is in metres: too small and it catches noise, too large and it averages across an object boundary.

Later work replaces the ball query with a learned graph. DGCNN’s EdgeConv builds a k-nearest-neighbour graph in feature space and recomputes it each layer, so points that are far apart geometrically but similar semantically end up as neighbours. The symmetric-function requirement never goes away — the aggregation over each neighbourhood is still a max — but where the neighbourhood comes from becomes a design choice.

Where it breaks

• Rotation. Nothing above is rotation invariant. A chair rotated 90° about the vertical produces different coordinates, different features and a different descriptor. PointNet bolts on a small network that predicts an alignment matrix applied to the input, but it is learned, not guaranteed, and it does not survive an arbitrary rotation the model never saw. Canonical alignment or heavy rotation augmentation is doing most of the real work in practice.
• Scale and units. The model reads raw coordinates, so a cloud in millimetres and the same cloud in metres are different inputs. Normalisation into a unit sphere is standard and is also a decision: it discards absolute size, which is often the single most discriminative feature between a real chair and a doll’s house chair.
• Density variation. A LiDAR scan is dense nearby and sparse far away, and a network trained on uniformly sampled CAD models has never seen that. Neighbourhood-based methods are worse here than flat pooling, because a fixed-radius ball contains 200 points at 5 m and 4 points at 50 m. Multi-scale grouping exists precisely to blunt this.
• Whole-cloud labels stop being useful early. A scan of a street is not one class. The moment you want per-point answers the problem changes shape — see point cloud segmentation, where the global descriptor gets concatenated back onto every point rather than replacing them, and the voxel-grid alternative, which buys back convolution at a cost you can compute.