A comprehensive survey on safe reinforcement learning

TLDR

Safe Reinforcement Learning is the process of learning policies that maximize expected return while ensuring reasonable system performance and respecting safety constraints during learning and deployment. This survey categorizes and analyzes two Safe Reinforcement Learning approaches and uses the classification to review existing literature and propose future research directions. The first approach modifies the optimality criterion by incorporating a safety factor into the classic discounted horizon, while the second alters exploration by integrating external knowledge or a risk metric.

Abstract

Safe Reinforcement Learning can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes. We categorize and analyze two approaches of Safe Reinforcement Learning. The first is based on the modification of the optimality criterion, the classic discounted finite/infinite horizon, with a safety factor. The second is based on the modification of the exploration process through the incorporation of external knowledge or the guidance of a risk metric. We use the proposed classification to survey the existing literature, as well as suggesting future directions for Safe Reinforcement Learning.