Autonomous Vehicles: How to Read a Claim About Where the Field Is

Almost every public number about self-driving is either not comparable across programmes or not statistically capable of supporting the claim made from it. That is not cynicism — it follows from how the metrics are defined and from how rare the events being measured are.

The levels, and the words that matter more

SAE J3016 defines six levels, 0 through 5, and they are widely misquoted. The distinction that carries all the weight is between Level 2 and Level 3: at Level 2 the human is driving and the system assists, so the human is responsible for monitoring continuously; at Level 3 the system is driving within its conditions and the human need only be available to take over when asked. Levels 4 and 5 remove the fallback human entirely, differing only in whether the conditions are limited.

But the level is the less informative half of any claim. The other half is the operational design domain — the specific conditions under which the system is designed to function. Road types, geographic area, weather, lighting, speed range, traffic conditions. A Level 4 system with an ODD of one mapped district in fair weather and a Level 4 system with an ODD of all divided highways in a state are both “Level 4”, and they are not the same engineering achievement or the same product.

So the rule for reading any autonomy claim is: the level tells you who is responsible, the ODD tells you what was accomplished, and a claim without an ODD is not a claim.

Why disengagement rates do not compare

Disengagement reporting — how often a safety driver took control, normalised by distance — is the most-cited public number in the field and it is close to useless for ranking programmes. Deliberately, no figures appear on this page; the reason is that the numbers are not measuring the same thing from one filer to the next. At least six incompatibilities stack up:

• The denominator is chosen. Miles driven where and when? A programme that tests in dense, complex traffic accumulates hard miles. One that accumulates highway miles at low density looks better on the same underlying competence. Nothing in the metric normalises for difficulty.
• The numerator is defined by the filer. What counts as a disengagement — any takeover, only safety-relevant ones, only ones the system requested? Definitions differ, and conservative definitions inflate the count relative to permissive ones.
• Safety-driver policy differs. A team instructed to take over at the first hint of discomfort generates far more disengagements than one instructed to let the system continue unless a collision is plausible. This is a training decision, not a capability difference.
• Testing strategy differs. A programme deliberately driving its hardest unsolved scenarios to gather data will report worse figures than one demonstrating a mature route, and the first programme is doing the more useful work.
• Self-reporting. The figures are compiled by the entity they reflect, under a definition that entity applies.
• Regime change. Once a programme removes the safety driver, disengagements stop existing as a category and are replaced by remote-assistance events, which are reported differently or not at all — so the metric disappears exactly at the point of most interest.

The general lesson is one this library makes elsewhere about model leaderboards: a metric that is cheap to report and expensive to standardise gets optimised as a metric. See why a benchmark number does not transfer to your setting — the mechanism is identical.

The mileage problem

Even with a perfect, honest, comparable metric, there is a statistical obstacle that no amount of reporting discipline removes. The events that matter most — fatalities and serious injuries — are extremely rare per mile in human driving, and demonstrating that a system is better than a very low rate requires observing enough miles for the difference to be distinguishable from noise.

Kalra and Paddock at RAND set this out in 2016 in Driving to Safety, and the conclusion is the durable part: establishing with statistical confidence that autonomous vehicles have a lower fatality rate than human drivers would require driving hundreds of millions of miles, and in some formulations billions — which under realistic fleet sizes and testing rates is a span of years or decades. The argument is simply the arithmetic of rare events, and it does not depend on any assumption about the technology.

Two consequences follow and both are visible in how the field actually operates. First, safety cases are built from decomposition and simulation rather than from road mileage alone — argue about specific hazards, show coverage, use scenario libraries and replay. Second, the rare-event problem means public confidence will be driven by incidents rather than by statistics, because incidents are observable and rates are not.

The rate structure of the stack

It is worth seeing where the timing constraints sit, because they are the same layered structure as everywhere else in this cluster and the consequences are stark at speed.

perception     camera/lidar frame rate, ~10-30 Hz
prediction     per perception cycle
planning       ~10 Hz typical
control        ~100 Hz (steering, throttle, brake)

distance covered while a decision is pending:
  30 m/s (~108 km/h) x 100 ms = 3.0 m
  30 m/s             x 300 ms = 9.0 m
  13 m/s (~47 km/h)  x 300 ms = 3.9 m

a full-stack latency of 300 ms at motorway speed is
most of a car length of committed motion.

This is why the deceleration and fallback behaviour is designed explicitly rather than left as a timeout: a minimal risk manoeuvre is a specified action, executed by a layer that does not depend on the slow part of the stack being available. It is the automotive instance of the general principle in splitting a slow planner from a fast controller, with the added constraint that stopping is not always the safe option.

What to ask instead

• What is the ODD, stated precisely? Geography, weather, speeds, road types, times of day.
• Is there a human in the loop, where, and at what rate? Remote assistance is legitimate and common; the interesting figure is events per vehicle-hour and what an assistant is permitted to do.
• What is the published safety case? A structured argument with claims, evidence and hazard analysis says more than any aggregate rate. Standards like UL 4600 exist to give that argument a shape.
• What is reported to regulators, and by what definition? Mandatory crash-reporting regimes have their own known biases — chief among them that reporting duty varies by system type — but they are at least defined externally rather than by the filer.
• What happens when a component fails? Redundancy in sensing, compute, steering and braking is where automotive functional safety practice lives, and it is a more informative question than any capability demonstration.