arXiv (Cornell University) · 2019 · 19 citations · 63 references
ComputingComputer ScienceMathematicsArtificial IntelligenceMachine LearningOptimizationDeep LearningNeural NetworksOptimal ControlPolicy LearningReinforcement LearningReinforcement Learning FoundationsConvergenceNatural Policy GradientOptimal PoliciesOptimality ConditionsOverparameterized ModelsPolicy Convergence AnalysisPolicy GradientsStationary Points
Policy gradient methods with actor-critic schemes demonstrate tremendous\nempirical successes, especially when the actors and critics are parameterized\nby neural networks. However, it remains less clear whether such "neural" policy\ngradient methods converge to globally optimal policies and whether they even\nconverge at all. We answer both the questions affirmatively in the\noverparameterized regime. In detail, we prove that neural natural policy\ngradient converges to a globally optimal policy at a sublinear rate. Also, we\nshow that neural vanilla policy gradient converges sublinearly to a stationary\npoint. Meanwhile, by relating the suboptimality of the stationary points to the\nrepresentation power of neural actor and critic classes, we prove the global\noptimality of all stationary points under mild regularity conditions.\nParticularly, we show that a key to the global optimality and convergence is\nthe "compatibility" between the actor and critic, which is ensured by sharing\nneural architectures and random initializations across the actor and critic. To\nthe best of our knowledge, our analysis establishes the first global optimality\nand convergence guarantees for neural policy gradient methods.\n
63
Reinforcement Learning: An Introduction
IEEE Transactions on Neural Networks · 2005 · 25.8K citations
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison et al. · Nature · 2016 · 15.8K citations
Mastering the game of Go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan et al. · Nature · 2017 · 9.2K citations
Engineering, Educational Psychology, Cognitive Psychology +7