3D Object Detection From LiDAR Point Clouds

A LiDAR detector outputs boxes with seven or more numbers each: centre, size, heading. Almost every architectural decision in the pipeline is driven by one measurement — the fraction of the detection volume that contains any points at all, which is a few per cent.

Why an image detector does not transfer

Three properties of a sweep break a 2D detector at once. The data is unordered, so there is no tensor to convolve. It is sparse and irregular, so a dense 3D convolution spends most of its arithmetic on empty space. And it is metric: a car is 4.5 m long at 5 m and at 50 m, whereas in an image it shrinks, which is the entire reason image detectors need multi-scale feature pyramids and LiDAR detectors mostly do not.

The metric property is a gift and the sparsity is the bill. Object size priors are exact in 3D, so an anchor box for “car” can be a single fixed size rather than a scale pyramid. But the input has to be made regular before a convolution can touch it, and how that is done is what separates the named architectures.

Voxelisation, and a worked occupancy count

Voxelisation crops the sweep to a detection volume and divides it into a regular grid. Take the volume that the KITTI-style automotive configurations use — roughly 0 to 70.4 m ahead, ±40 m to the sides, and −3 m to +1 m vertically — with a voxel of 0.2 m × 0.2 m × 0.4 m:

grid dimensions
  x: 70.4 m / 0.2 m = 352
  y: 80.0 m / 0.2 m = 400
  z:  4.0 m / 0.4 m =  10
  total voxels = 352 x 400 x 10 = 1,408,000

points available
  a 64-beam spinning sensor at 10 Hz produces on the order of
  120,000 points per sweep, and a large share of those fall
  outside the crop (behind the vehicle, above +1 m, beyond 70 m)

upper bound on occupancy
  120,000 / 1,408,000 = 8.5 %   if every point landed in its own voxel

what actually happens
  points arrive in scan lines, so many share a voxel; occupied
  voxels typically land in the low single-digit percentages

That number is the design constraint. A dense 3D convolution over 1,408,000 cells does 100% of the work for a few per cent of the signal, and it does it at every layer. VoxelNet, from Yin Zhou and Oncel Tuzel at CVPR 2018, established the encode-then-convolve shape — a small PointNet inside each non-empty voxel produces a fixed-length feature, then 3D convolutions run over the resulting grid — and it was also the demonstration that the dense version is too slow for a vehicle.

Sparse convolution and the pillar shortcut

Two fixes followed, and both are still in use. SECOND replaced the dense 3D convolution with a sparse one that stores only occupied voxels in a hash map and computes outputs only where inputs exist. This keeps the 3D grid but makes the cost proportional to occupancy rather than to volume — the difference between 1.4 million cells and perhaps 20,000.

PointPillars, from Alex Lang and colleagues at CVPR 2019, removes the third dimension instead. Set the voxel height to the full vertical extent so each cell is a vertical pillar; encode the points in each pillar with a small PointNet; scatter the results back to a 2D bird’s-eye-view pseudo-image. With 0.16 m pillars over a similar crop the grid is roughly 432 × 496 cells with no z axis at all, and everything downstream is an ordinary 2D convolutional backbone — the same kind of network, and the same optimised kernels, that image detectors already use. The PointPillars paper describes the pillar encoder and the fixed-size padding it needs. Because the number of non-empty pillars varies per sweep, the implementation caps it and zero-pads, which is a small detail with a real consequence: in a very dense scene, pillars beyond the cap are dropped.

What the detection head predicts

The output is an oriented 3D box: centre (x, y, z), extent (l, w, h), and a yaw angle, plus a class and a confidence. Two details cause most of the confusion.

• Regression is relative to an anchor. Anchor-based heads place fixed-size boxes for each class at every BEV cell and regress offsets, which is why the class size priors matter so much. Anchor-free heads predict a centre heatmap instead and regress size directly, which removes the anchor-matching threshold as a tuning knob.
• Yaw is angular and loss functions are not. A box at 0° and the same box at 180° occupy the same space, so a naive L1 loss on the angle punishes a correct-but-flipped prediction enormously. The usual fixes are to regress sin and cos of the angle, or to regress the angle modulo π and classify the direction separately.
• Non-maximum suppression happens in BEV. Overlap is computed on the ground-plane rectangle, not on the full 3D box, because vehicles do not stack and the 2D rotated IoU is far cheaper.

Why recall falls off with range

Angular sampling makes distant objects sparse in a way that is worth computing rather than hand-waving. A 64-beam sensor covering about 27 degrees of vertical field of view has a beam spacing near 0.42°, which is 0.0073 radians. The number of beams that strike a target of height h at range r is approximately h / (r × 0.0073):

target: a car, 1.5 m tall
  at 10 m:  1.5 / (10 x 0.0073) = 20.5 -> about 20 scan lines
  at 30 m:  1.5 / (30 x 0.0073) =  6.8 -> about 7 scan lines
  at 60 m:  1.5 / (60 x 0.0073) =  3.4 -> about 3 scan lines

point count on the object falls roughly as 1/r^2, because both
the vertical and the horizontal sampling thin at the same rate

Three scan lines is not enough geometry to fit a heading to. This is why detection metrics are almost always reported split by range and by difficulty, why long-range detection is treated as a separate problem, and why fusing a camera — which does not thin with range in the same way — is worth the calibration effort. See LiDAR and camera fusion for the projection that makes that possible, and density and scan resolution trade-offs for the same arithmetic applied to survey work.

How it is scored, and what that hides

Detection is scored with average precision at a 3D IoU threshold — commonly 0.7 for vehicles and 0.5 for pedestrians and cyclists. Two things about that are easy to miss. A 0.7 3D IoU on a 4.5 m box is a tight tolerance: a 20 cm centre error and a few degrees of yaw error can drop a correct detection below threshold. And the threshold difference between classes is not arbitrary — a pedestrian box is small enough that the same absolute error costs far more IoU, so an equal threshold would make the pedestrian number meaningless rather than merely hard.

What the headline number hides is the distribution. A model can hold its aggregate AP while losing most of its recall past 50 m, or while failing on the heavily occluded subset, because those cases are a small fraction of the labelled objects. Read the range-split and occlusion-split tables before believing a single figure, and treat any benchmark score as evidence about that benchmark’s sensor and city rather than about your own.