Addressing Function Approximation Error in Actor-Critic Methods

TLDR

In value‑based reinforcement learning, function approximation errors cause overestimated values and suboptimal policies. The study shows that these errors also affect actor‑critic methods and proposes mechanisms to reduce their impact on both actor and critic. The algorithm extends Double Q‑learning by using the minimum of two critics to curb overestimation, links target networks to bias, and recommends delaying policy updates to lower per‑update error. Evaluation on OpenAI Gym tasks demonstrates that the method outperforms the state of the art in every environment tested.

Abstract

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.