Fusing LiDAR and Camera Data for Perception

Fusion is a coordinate transform followed by a great deal of care about the transform. The mathematics is three matrix multiplications; the engineering is calibration, time synchronisation, and knowing which points are hidden from the camera even though they project inside the image.

Why fuse at all

The two sensors fail in complementary ways, and the complementarity is structural rather than incidental. LiDAR measures range directly and accurately, is unaffected by darkness, and gives metric geometry with no scale ambiguity. Its density falls with the square of distance — three scan lines on a car at 60 m — and it carries no colour, no text, and nothing that distinguishes a red traffic light from a green one.

A camera has high angular resolution that does not thin with distance, full colour, and everything semantic. It has no range at all, only the statistical guess of monocular depth, and it degrades in low light and in glare. Fusing gives the geometry of one and the semantics of the other, and the price is that you now have two sensors that must agree about where things are, to a fraction of a degree, forever.

It is worth being precise about what fusion buys, because the two most common motivations want different things. If the goal is better detection, the camera is supplying semantics that geometry cannot provide and a coarse association is enough — a point being assigned to roughly the right object is sufficient to paint it with a class score. If the goal is a coloured or textured point cloud, the requirement is much stricter: a point must land on the correct pixel, because a one-pixel error at a colour boundary puts a fence post’s colour on the sky behind it and the result is visible to anyone looking at it. The calibration tolerance for the second is roughly an order of magnitude tighter than for the first, and pipelines routinely inherit a calibration that was good enough for the first and is being used for the second.

Three coordinate frames

A projection passes through three frames and two transforms, and most fusion bugs are a mix-up between them.

• The LiDAR frame, where points arrive. Automotive convention is x forward, y left, z up.
• The camera frame. Computer vision convention is x right, y down, z forward — a different axis assignment entirely, not merely a rotation of degrees. Getting this wrong produces a projection that looks plausible and is transposed or mirrored.
• The image plane, in pixels, with the origin at the top left.

The transform between the first two is the extrinsic calibration: a rotation R and a translation t, six numbers, measured once with a target visible to both sensors. The transform from camera frame to pixels is the intrinsic matrix K, containing focal lengths fx, fy in pixel units and the principal point cx, cy, obtained from a checkerboard calibration along with the lens distortion coefficients.

Worked: projecting one point

a LiDAR return at (10.0, 2.0, -0.5) metres
  10 m ahead, 2 m to the left, 0.5 m below the sensor

step 1 — rotate into camera axes
  camera x = -(lidar y)   right      = -2.0
  camera y = -(lidar z)   down       =  0.5
  camera z =  (lidar x)   forward    = 10.0

step 2 — translate by the extrinsic offset
  assume the camera sits 0.08 m above and 0.27 m behind
  the LiDAR, i.e. t = (0.00, -0.08, -0.27) in camera axes

  p_cam = (-2.00, 0.42, 9.73)

step 3 — check the point is in front of the camera
  z = 9.73 > 0   ok
  (skip this and points BEHIND the camera project to
   perfectly valid-looking pixels, mirrored. it is the
   single most common fusion bug.)

step 4 — apply the intrinsics
  assume fx = fy = 720 px, cx = 610, cy = 175

  u = fx * (x / z) + cx = 720 * (-2.00 / 9.73) + 610
    = 720 * (-0.2055) + 610 = -147.9 + 610 = 462.1

  v = fy * (y / z) + cy = 720 * ( 0.42 / 9.73) + 175
    = 720 * ( 0.0432) + 175 =   31.1 + 175 = 206.1

step 5 — bounds check
  (462, 206) lies inside a 1242 x 375 image. keep it.

Steps 3 and 5 are not formalities. Roughly half of a spinning sensor’s returns are behind the camera, and without the depth check they land on top of the foreground as a mirrored ghost cloud.

What a small calibration error costs

Extrinsic rotation error is amplified by range and rotation error in pixels is amplified by focal length. Both are worth computing once so the tolerance is a number rather than a feeling.

a 0.5 degree extrinsic rotation error = 0.00873 radians

lateral error in the world
  at 10 m:  10 x 0.00873 = 0.087 m   (8.7 cm)
  at 30 m:  30 x 0.00873 = 0.262 m   (26 cm)
  at 60 m:  60 x 0.00873 = 0.524 m   (52 cm)

error in the image, with fx = 720 px
  720 x tan(0.5 deg) = 720 x 0.00873 = 6.3 px
  — constant in pixels, regardless of range

a 2 cm translation error
  at 10 m: subtends 0.02 / 10 = 0.002 rad = 1.4 px
  at 60 m: subtends 0.02 / 60 = 0.00033 rad = 0.24 px
  — translation error matters up close, rotation at range

Half a degree is not a large miscalibration and 26 cm at 30 m is enough to put a point that belongs to a pedestrian onto the road behind them. Rotation error dominates at range and translation error dominates near the sensor, which tells you where to put your calibration targets: at the distances you care about, not at the convenient one.

Calibration also drifts. Thermal cycling, vibration and a mounting bracket that was knocked in a workshop all move the extrinsics, so a calibration is a maintenance item. The practical monitor is cheap: project points onto detected image edges and track the mean distance between them over time. A slowly rising number is drift; a step change is something that got hit.

Timing, and the sweep that is not an instant

A spinning LiDAR at 10 Hz takes 100 ms to complete one rotation, so points at the start and the end of a “frame” were measured a tenth of a second apart. The camera exposes at one instant. Treating the sweep as simultaneous with the image is an error whose size is straightforward:

ego vehicle at 15 m/s (54 km/h), 10 Hz LiDAR

  time spread within one sweep     = 100 ms
  worst-case ego motion in a sweep = 15 x 0.100 = 1.50 m
  typical camera-to-point offset   = up to 50 ms
                                   = 0.75 m of ego motion

an oncoming vehicle at 15 m/s closes an additional
  15 x 0.050 = 0.75 m in that same 50 ms

The fix is deskewing: use an ego-motion estimate — wheel odometry, an IMU, or the pose from a SLAM front end — to transform every point to a common reference instant using its own timestamp. This requires per-point timestamps, which is why they exist in the point record and why a pipeline that discards them during conversion has thrown away the ability to do fusion properly. Deskewing corrects for the ego vehicle’s motion; it does not correct for other objects moving, which remains a residual error proportional to their relative speed.

Where in the network to fuse

• Early, at the point level. Project each point into the image, sample the image there, and attach the result to the point — RGB values, or better, the per-pixel class scores from a semantic segmentation network. PointPainting, from Sourabh Vora and colleagues at CVPR 2020, does exactly this and feeds the decorated points into an unmodified LiDAR detector. It is the cheapest fusion to implement and it inherits every calibration and timing error directly.
• Mid, at the feature level. Encode each modality into a shared space — usually a bird’s-eye-view grid — and fuse the feature maps. This is what recent architectures do, and it is more tolerant of small misalignment because a feature map is spatially smoothed relative to individual points.
• Late, at the detection level. Run two independent detectors and associate their outputs. Weakest in accuracy, strongest in engineering: each sensor degrades independently, one can fail entirely without taking the system with it, and you can ship the two pipelines on different schedules.

One failure cuts across all three. LiDAR and camera are at different positions, so a LiDAR point on a background wall can project into a pixel occupied by a foreground object — the camera cannot see through the object, but the projection does not know that. Fusing naively paints background points with foreground semantics. The fix is a visibility test: render the points to a depth buffer at image resolution and discard any point whose projected depth is substantially behind the nearest point in the same pixel.