Safe RL and Constrained Optimisation

There are two ways to stop an agent doing something: make it expensive, or make it impossible. They fail differently, they are not interchangeable, and most systems need both because each covers the other’s failure.

Two ways to say no

The distinction is whether the constraint lives inside the optimisation or outside it. Inside, it is subject to optimisation pressure like everything else. Outside, it is not — and it is also not learned, not adaptive, and not aware of anything you did not enumerate.

When a penalty stops binding

A penalty is a price, and an agent maximising return will pay any price lower than what the violation is worth. The condition is arithmetic:

the agent violates whenever   R_gain  >  lambda

R_gain  = extra return obtainable by violating
lambda  = penalty applied per violation

example: a shortcut that saves 20 steps at -0.02 per step
  R_gain = 0.40
  lambda = 0.10  ->  violates every time    (net +0.30)
  lambda = 1.00  ->  never violates         (net -0.60)

so the penalty must exceed the largest gain the violation can produce,
and that gain is a property of the environment you have not enumerated.

Two things make this worse than it first appears. You are setting lambda against a maximum gain you do not know, because if you knew every way the constraint could be profitably violated you would not need the learning system. And a very large lambda is not a free solution: it dominates the gradient, the agent becomes overwhelmingly averse to anything near the boundary, and it may never explore the region where the good policies live.

There is a third problem specific to language models. Reward model scales are unidentified — only differences are constrained by the training data — so a penalty calibrated against one reward model is not calibrated against the next one you train. The safety margin drifts with a component nobody thinks of as safety-relevant.

Constrained MDPs and the budget nobody notices

The principled version of a penalty is a constrained MDP (Altman, 1999): maximise return subject to an expected cost staying below a budget d. Lagrangian methods solve it by adapting the penalty coefficient during training — raise it when the cost exceeds the budget, lower it when there is slack. Constrained Policy Optimization (Achiam et al., 2017) and the Safety Gym benchmarks (Ray et al., 2019) are the standard references.

This is a genuine improvement on a hand-set penalty, because the coefficient is tuned by the data rather than guessed. Two properties of it are routinely missed.

• The budget is a violation rate, and it is not zero. A constraint of “expected cost below d” permits violations at rate d by construction. If your requirement is “never”, this formulation cannot express it: at d = 0 the feasible set has no interior and the Lagrange multiplier grows without bound.
• It is a constraint in expectation. An agent that is perfectly safe on 999 episodes and catastrophic on the thousandth satisfies an expected-cost constraint. Where the tail is what you care about, the objective does not describe your requirement, and risk-sensitive formulations that constrain a quantile rather than a mean are the thing to look for.

Hard constraints, and their four failures

A shield sits between the policy and the environment and blocks disallowed actions. It gives an actual guarantee, which is more than any penalty offers, and it fails in four ways that are easy to underestimate.

1. It only forbids what you enumerated. A shield is a list. Everything not on the list is permitted, including the thing that goes wrong. This is the same enumeration problem as a deny-list firewall and it has the same track record.
2. It constrains actions, not outcomes. Blocking individually unsafe actions does not block unsafe sequences of individually safe ones. Any constraint on what the world ends up like, rather than on what was done, needs a model of consequences, and that model is a second thing that can be wrong.
3. The agent learns to rely on it. If the shield always intervenes, the policy is trained in an environment where the dangerous action is free, so it never learns to avoid it. Remove the shield — a deployment change, a new code path, a version that forgot — and the underlying policy is unsafe. The shield has to be treated as part of the trained system, not as an add-on to it.
4. It can make the task unsolvable. A sufficiently conservative shield can trap the agent in a region containing no good policy, and the symptom is an agent that appears not to learn. The diagnostic is to log intervention rates: if the shield fires on a large fraction of proposed actions, it is not a safety layer, it is the policy.

The problem specific to learning

Both approaches share a difficulty with no clean solution: an agent learns that something is bad by doing it. A penalty teaches avoidance only after the violation has been experienced, several times, with enough repetitions to separate its effect from noise — see credit assignment for how many.

For anything with real consequences this is unacceptable, and the standard responses are all forms of not doing it in the real world: train in simulation and transfer; train offline against logged data, with the caveats on offline RL; or use a hard constraint during training so the violation is prevented rather than learned from — which brings back the reliance failure above, because a violation that never happens produces no learning signal about itself.

The honest position is that these are trade-offs and not solutions. García and Fernández (2015) surveyed the field and the structure of the problem has not changed since: you can learn from consequences or avoid consequences, and doing both requires an accurate model of what the consequences would have been.

The same two shapes in an LLM system

Every part of this maps onto language model deployment, and recognising which shape you have explains the failure you are seeing.

• Refusal training is a penalty. It is a learned tendency traded against helpfulness inside one objective, which is why it degrades under sufficient pressure from the other side and why it generalises imperfectly to phrasings unlike the training data.
• A content filter is a hard constraint. It is enumerable, it does not negotiate, and it fails exactly on the categories nobody listed. See input versus output guardrails for where in the pipeline it should sit.
• Tool permissions are a shield. An agent that cannot call the delete endpoint will not delete anything, regardless of what it has been argued into. This is the strongest safety mechanism available in an agent system, it is ordinary engineering, and it is underused relative to how well it works.
• Spending limits are a constraint on outcomes. A per-key budget bounds the cost of a loop that has gone wrong, regardless of why it went wrong — which is precisely the property the action-level constraints lack.

The design rule that falls out of the whole page: put a hard constraint on anything whose worst case you cannot accept, and use penalties for the things you want less of rather than none of. A penalty on an unacceptable outcome is a statement about its price, and the agent will eventually find out what that price buys.