Operations Research · 2021 · 49 citations · 13 references
ComputingComputer ScienceMachine LearningMarkov Decision ProcessesPolicy EvaluationExploration-exploitation TradeoffOptimal ControlPolicy LearningReinforcement LearningReinforcement Learning FoundationsGlobal Convergence AnalysisMarkov Decision ModelsNatural Policy GradientOptimization AlgorithmsPolicy GradientsPolicy OptimizationRegularizationRegularized Policy Learning
Preconditioning and Regularization Enable Faster Reinforcement Learning Natural policy gradient (NPG) methods, in conjunction with entropy regularization to encourage exploration, are among the most popular policy optimization algorithms in contemporary reinforcement learning. Despite the empirical success, the theoretical underpinnings for NPG methods remain severely limited. In “Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization”, Cen, Cheng, Chen, Wei, and Chi develop nonasymptotic convergence guarantees for entropy-regularized NPG methods under softmax parameterization, focusing on tabular discounted Markov decision processes. Assuming access to exact policy evaluation, the authors demonstrate that the algorithm converges linearly at an astonishing rate that is independent of the dimension of the state-action space. Moreover, the algorithm is provably stable vis-à-vis inexactness of policy evaluation. Accommodating a wide range of learning rates, this convergence result highlights the role of preconditioning and regularization in enabling fast convergence.
13
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al. · Nature · 2015 · 30K citations
Advances in neural information processing systems 7
Krzysztof J. Cios, Mark E. Shields · Neurocomputing · 1997 · 22.3K citations
Natural Gradient Works Efficiently in Learning
Neural Computation · 1998 · 2.7K citations