Safe Reinforcement Learning via Shielding

2 min read

A shield is a small reactive system, synthesised before training from a safety specification in temporal logic and a finite, conservative abstraction of the environment. The paper puts it in the RL loop in two ways. A preemptive shield hands the agent the list of safe actions at each step. A post-posed shield watches the agent’s chosen action and replaces it only if it would violate the specification. Synthesis is a safety game between the shield and the environment abstraction; the shield picks the least restrictive moves inside the winning region.

Findings

  • Shields are small to build: 2 s for a 9x9 grid world with bombs, 0.6 s for a 15x9 grid world with a moving opponent, 2 s for a self-driving car around a block, under 1 s for a water tank.
  • Shielded agents never violate the specification in any of the four domains, and in most of them learn faster. In the car example the unshielded DQN still crashes at the end of training while the shielded one never does.
  • In Seaquest the shield (surface before oxygen runs out; do not surface early without divers) did not change the score but removed all violations.
  • The paper states the conditions under which a shield preserves the convergence guarantees of the underlying learner.

Limits

  • The authors admit the main cost: you need an approximate model of which actions are unsafe in which states. The guarantee holds for every MDP that the abstraction covers, and no further.
  • Only the safety fragment of LTL is enforced. A shield cannot force progress, so “never crash” is easy and “eventually finish the task” is out of scope.
  • Shield inputs (wall distance, oxygen level) are read from the simulator state. A wrong or spoofed sensor feeding the shield is outside the model.

Relevance

Shielding is the RL version of Simplex: let the untrusted policy act, and override only when the model says it must. It builds on shield synthesis; CBF safety filters do the same job in continuous control. The open question for runtime monitoring and safe control is what a shield should do when its abstraction is wrong, since the paper treats an observed violation of the abstraction as a win for the system.