Estimating Depth From a Single Image

A single photograph cannot determine depth. This is not a limitation of current models; it follows from the projection equations, and the correct way to read a monocular depth network is as a very good guess about which scene is most likely, not as a measurement.

Why it is ill-posed, exactly

A pinhole camera maps a world point to a pixel by u = f X / Z + cx and v = f Y / Z + cy. Scale the entire scene by a factor k — every X, Y and Z multiplied by k — and the ratios X/Z and Y/Z are unchanged, so every pixel lands in exactly the same place. A doll’s house photographed close up and a real house photographed from far away produce identical images. No algorithm can distinguish them from the pixels alone, because the information is not in the pixels.

Focal length compounds this. If f is unknown, a wide-angle shot of a small nearby scene and a telephoto shot of a large distant one are also identical. This is why models that claim metric depth take camera intrinsics as an explicit input, and why running one on an image whose focal length you do not know returns a number in units that do not exist.

What a model does instead is learn a prior. Among all scenes consistent with these pixels, which is most likely given millions of photographs of the world? Doors are about two metres tall, floors are flat, people are not forty metres wide. The output is a maximum-likelihood scene under that prior, which is genuinely useful and is a categorically different object from a triangulated measurement — see structure from motion, where two viewpoints make depth observable rather than inferred.

The cues a model can actually use

• Ground contact. The strongest cue in most images and the one models lean on hardest. If the camera is at a known height above a flat floor, the image row where an object meets the ground determines its distance directly. This is why depth predictions degrade sharply on images with no visible ground plane, and why cropping the bottom off an image can change the depth of everything in it.
• Relative size within a known category. Cars are a consistent size, so a small car in the frame is a distant car. This fails precisely and predictably on scale models, toys, and photographs of photographs.
• Occlusion and T-junctions. Where one contour terminates against another, the terminating surface is behind. This is a purely ordinal cue — it says which is nearer, never by how much — and it is the most reliable cue in the list because it is geometrically valid rather than statistical.
• Linear perspective. Parallel lines converge to a vanishing point at a rate set by their distance. Strong in corridors, roads and buildings; absent outdoors in nature.
• Texture gradient. A repeating pattern — brickwork, gravel, a lawn — compresses with distance, and the compression rate encodes surface slant as well as depth.
• Aerial perspective and defocus. Distant objects are hazier and lower in contrast; objects away from the focal plane are blurred. Both are real physical cues and both are weak, camera dependent, and easily removed by post-processing before the image reaches the model.

What the model outputs is not depth

The widely used zero-shot models do not predict metres. MiDaS, published by René Ranftl and colleagues, trained across a mixture of datasets whose ground truth was captured by incompatible means — stereo, laser, structured light, even 3D films — with no consistent scale between them. The training loss is therefore made invariant to scale and shift, and the output is inverse depth (disparity) defined only up to an unknown positive scale a and offset b:

predicted = a * (1 / true_depth) + b     with a, b unknown

consequences:
  - larger predicted value = nearer, always
  - ratios of predicted values mean nothing
  - a and b are re-estimated per image, so two frames of a
    video are not on the same scale even if nothing moved

The last line is the one that catches people building on these models. Running an affine-invariant network frame by frame over a video produces depth that flickers and breathes, not because the model is unstable but because each frame has its own a and b. Temporal consistency has to be imposed afterwards, or a video-specific model used. The MiDaS paper describes the scale-and-shift-invariant loss and the dataset mixing it exists for. Depth Anything, from Lihe Yang and colleagues at CVPR 2024, follows the same output convention while scaling the training set with pseudo-labelled unlabelled images.

Getting to metres

Two routes, with different requirements.

Fit the affine transform against a reference. If you have any sparse true depths for the image — a handful of LiDAR returns, points from a structure-from-motion reconstruction, one measured distance — you can least-squares fit a and b so the prediction agrees with them, then apply that transform to the whole map. Two known depths are the minimum for the two unknowns; more is better because the fit is then overdetermined and you can inspect the residual. This is the standard way a monocular network is used alongside a sparse sensor, and it is the mechanism behind depth completion.

Or use a model trained for metric output, which requires being told the camera intrinsics. These exist and work within the range of cameras and scenes they were trained for; treat a metric prediction on an unusual lens or an unusual scene scale as untrusted until checked against something real.

How it is scored

Three metrics dominate, and they measure different things.

AbsRel  = mean over pixels of |d_pred - d_true| / d_true
          relative error, so a 1 m error at 50 m counts far
          less than a 1 m error at 2 m

RMSE    = sqrt(mean of (d_pred - d_true)^2)
          absolute, in metres, so it is dominated by the
          far field and by a handful of large mistakes

delta<1.25 = fraction of pixels where
             max(d_pred/d_true, d_true/d_pred) < 1.25
          a threshold accuracy: what share of pixels are
          within 25 % either way. delta<1.25^2 and
          delta<1.25^3 loosen it to 56 % and 95 %

A model can win on AbsRel and lose on RMSE by being excellent up close and poor at range, which is exactly the trade a depth network makes, because the near field has the strongest cues. Read both. And note that all three are computed only where ground truth exists — sky, mirrors and absorbing surfaces have no ground truth from a depth sensor, so the places where models fail worst are usually excluded from the metric.

Where it fails, and why each failure is that cue

• Mirrors and glass. The model predicts the depth of the reflected or transmitted scene, because every pictorial cue in that region belongs to it. There is no cue in the image that says “this is a surface at 1.2 m”, and a human gets it right using context the network does not have.
• Sky. Infinite depth is not representable, so models clamp it to a maximum value that varies between implementations. Any code consuming a depth map needs an explicit sky mask rather than trusting the number.
• Thin structures. Wires, railings, chair legs and foliage occupy a pixel or two and get smoothed into the background. This is partly the architecture’s resolution and partly the training data, since laser ground truth on a wire is itself unreliable.
• Unusual scale. Miniatures, close-up macro work, aerial imagery, and any scene where the size prior is wrong. The model is not confused; it is confidently answering the question “what is the most likely scene” and your scene is not the likely one.
• Anything safety-relevant. A monocular depth map is adequate for background blur, relighting, compositing, coarse occlusion ordering in AR, and as a prior fed into a system that also measures. It is not adequate on its own for obstacle avoidance or for dimensioning, and fusing it with even a sparse real range sensor — see LiDAR and camera fusion — changes it from a guess into a measurement with a guess filling the gaps.