Innovations in Systems and Software Engineering · 2022 · 25 citations · 17 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningMarkov Decision ProcessesPolicy LearningReinforcement LearningControl ActionsOptimal PoliciesReinforcement Learning TheorySafe Reinforcement LearningSafety Requirement SpecificationShieldingTemporal Logic Specification
Abstract Besides the recent impressive results on reinforcement learning (RL), safety is still one of the major research challenges in RL. RL is a machine-learning approach to determine near-optimal policies in Markov decision processes (MDPs). In this paper, we consider the setting where the safety-relevant fragment of the MDP together with a temporal logic safety specification is given, and many safety violations can be avoided by planning ahead a short time into the future. We propose an approach for online safety shielding of RL agents. During runtime, the shield analyses the safety of each available action. For any action, the shield computes the maximal probability to not violate the safety specification within the next k steps when executing this action. Based on this probability and a given threshold, the shield decides whether to block an action from the agent. Existing offline shielding approaches compute exhaustively the safety of all state-action combinations ahead of time, resulting in huge computation times and large memory consumption. The intuition behind online shielding is to compute at runtime the set of all states that could be reached in the near future. For each of these states, the safety of all available actions is analysed and used for shielding as soon as one of the considered states is reached. Our approach is well-suited for high-level planning problems where the time between decisions can be used for safety computations and it is sustainable for the agent to wait until these computations are finished. For our evaluation, we selected a 2-player version of the classical computer game Snake . The game represents a high-level planning problem that requires fast decisions and the multiplayer setting induces a large state space, which is computationally expensive to analyse exhaustively.
17
Reinforcement Learning: An Introduction
Richard S. Sutton, Andy Barto · IEEE Transactions on Neural Networks · 1998 · 27.1K citations
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison et al. · Nature · 2016 · 15.8K citations
The temporal logic of programs
Amir Pnueli · 1977 · 5.7K citations
Safe Reinforcement Learning via Shielding
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers et al. · 2018 · 620 citations
Richard Cheng, Gábor Orosz, Richard M. Murray et al. · Proceedings of the AAAI Conference on Artificial Intelligence · 2019 · 533 citations · Full text