Safety Standards for Physical AI: Where Certification Meets a Stochastic Policy

Functional safety assumes a specification: you say what the system must do, you argue that it does it, and an assessor checks the argument. A learned policy’s specification is a dataset, and you cannot audit a dataset the way you audit a state machine. The resolution is architectural, and it is well established.

Why this is not ordinary software safety

Start with what makes a physical failure different in kind. A wrong token produces a bad sentence that somebody reads and discards. A wrong torque moves mass, and the relationship between mass, velocity and injury is not a matter of opinion — it is mechanics, and it is calculable in advance, which is why the standards can specify limits rather than intentions.

The three properties that break the usual verification story are worth separating, because they fail for different reasons:

• The behaviour is not enumerable. A conventional controller has a transfer function you can analyse and bounds you can prove. A network has a mapping defined by a hundred million parameters over an input space nobody has characterised, and “it did the right thing on the test set” is not a proof about the input you have not seen.
• The failure is not fail-stop. Software safety is largely built around detecting a fault and stopping. A policy that is confidently wrong does not raise an exception, does not crash, and does not look different from a policy that is right. There is nothing for a watchdog to catch.
• The specification is inductive. The system was not told what to do; it was shown. Changing behaviour means changing data and retraining, and the relationship between a data change and a behaviour change is empirical. That is a poor fit for a regime built on traceability from requirement to implementation to test.

The standards, and what each is for

The framework is older and more concrete than most software engineers expect. Describing it accurately matters, because the design pattern that follows falls out of it.

The containment pattern

The resolution to “you cannot certify a neural network” is not to certify it. It is to build a machine that is safe regardless of what the network outputs, and then let the network drive it.

Concretely: the policy does not command actuators. It commands setpoints into a layer that is small, deterministic, analysable and built to the appropriate Performance Level, and that layer enforces limits the policy has no way to reach past. The limits are the certified artefact; the policy is untrusted input to it.

   [ learned policy ]        untrusted, unbounded, slow
            |  setpoints (pose, velocity, force target)
            v
   [ limiter / arbiter ]      simple, deterministic, testable
            |  clamped commands
            v
   [ motion controller ]      real-time, deadline-bound
            |
            v
   [ safety-rated hardware ]  dual-channel, diagnosed, PL-rated
     speed limit, force limit, safe torque off, e-stop

each layer down is faster, simpler and more trusted.
the top layer can be wrong without the machine being unsafe.

What the enforcing layers can check is deliberately mundane, and that mundanity is the point — every item is expressible as an inequality:

• Workspace limits. Safety-rated position monitoring that will not permit the tool centre point outside a defined volume, regardless of what was commanded.
• Speed and separation monitoring. Measured distance to a person, from a safety-rated sensor, driving a speed limit that approaches zero as separation approaches the protective distance.
• Power and force limiting. Contact forces held below the ISO/TS 15066 biomechanical limits for the body region concerned, through mechanical design, torque sensing and limited inertia.
• Rate and jerk limits. A policy that emits a wild setpoint gets a bounded acceleration towards it rather than a step — which also happens to protect the mechanism from the policy.
• Safe torque off. A hardware function that removes the ability to produce torque through a dedicated, diagnosed channel that does not route through the application processor at all.

Note how well this composes with the timing structure in the latency page: the layers that must be fast and the layers that must be trusted turn out to be the same layers, for related reasons. Simple enough to hold a deadline and simple enough to certify are close to the same property.

The standard written for exactly this problem

ISO 21448 deserves its own treatment because it is the standards world’s acknowledgement that the classical model was incomplete. Classical functional safety asks: what happens when a component fails? SOTIF asks the different question: what happens when nothing fails and the system is still unsafe, because its intended functionality has performance limitations in a situation nobody anticipated?

That is the learned-policy failure mode stated in the vocabulary of a published standard. A perception system that does not recognise an object it never saw has not failed in the 61508 sense — every component worked to specification. It is a SOTIF hazard.

The framework organises scenarios into four areas: known-safe, known-unsafe, unknown-safe and unknown-unsafe. The engineering work is moving scenarios out of the unknown-unsafe region — by analysis, by simulation, by field testing, by scenario libraries — and demonstrating that the residual is acceptably small. It does not make the problem easy. It makes the problem legible, and it gives you a structure for arguing about it, which is what a safety case needs.

What it means for how you build

• Never let the model command torque. The interface between learned and certified is a setpoint with bounds, and the bounds are enforced somewhere the model cannot reach. This is the single design rule from which most of the rest follows.
• Write the degraded behaviour first. What the machine does when the policy is late, incoherent or absent is a specification with a stop category attached, not an exception handler. Design it before the happy path.
• Make the safety layer boring and separate. Different code, different review, different change cadence, ideally different hardware. Every feature added to it is verification debt, and its value comes from being small enough to reason about completely.
• Log for the investigation you will have to do. Inputs, outputs, limiter interventions and timing, retained long enough to reconstruct an incident. When something goes wrong, the question will be what the policy saw and what the limiter did about it, and if that was not recorded the answer does not exist.
• Do not treat a prompt or a system message as a control. Instructions shape a distribution; they do not bound it. Everything in this library about validating at the boundary rather than asking nicely applies with more force when the output is a physical motion.