Neural Policy Gradient Methods: Global Optimality and Rates of\n Convergence

Lingxiao Wang, Zhuoran Yang, Zhaoran Wang

arXiv (Cornell University) · 2019 · 19 citations · 63 references

DOIFull text

Open access

Concepts

Abstract

Policy gradient methods with actor-critic schemes demonstrate tremendous\nempirical successes, especially when the actors and critics are parameterized\nby neural networks. However, it remains less clear whether such "neural" policy\ngradient methods converge to globally optimal policies and whether they even\nconverge at all. We answer both the questions affirmatively in the\noverparameterized regime. In detail, we prove that neural natural policy\ngradient converges to a globally optimal policy at a sublinear rate. Also, we\nshow that neural vanilla policy gradient converges sublinearly to a stationary\npoint. Meanwhile, by relating the suboptimality of the stationary points to the\nrepresentation power of neural actor and critic classes, we prove the global\noptimality of all stationary points under mild regularity conditions.\nParticularly, we show that a key to the global optimality and convergence is\nthe "compatibility" between the actor and critic, which is ensured by sharing\nneural architectures and random initializations across the actor and critic. To\nthe best of our knowledge, our analysis establishes the first global optimality\nand convergence guarantees for neural policy gradient methods.\n

References

63