Trust Region Policy Optimization

TLDR

The article proposes a method for optimizing control policies with guaranteed monotonic improvement. The authors approximate a theoretically justified scheme to create the practical Trust Region Policy Optimization (TRPO) algorithm. TRPO effectively optimizes large nonlinear policies, achieving robust performance on robotic locomotion and Atari games while maintaining monotonic improvement with minimal hyperparameter tuning.

Abstract

In this article, we describe a method for optimizing control policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified scheme, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.