Double Q-learning

Hado van Hasselt

Neural Information Processing Systems · 2010 · 393 citations · 20 references

Concepts

TL;DR

In stochastic environments, Q‑learning suffers from large overestimations of action values caused by a positive bias from using the maximum action value as an approximation. The authors propose an alternative method to approximate the maximum expected value for any set of random variables. They apply this double‑estimator approach to Q‑learning, yielding the Double Q‑learning algorithm. Double Q‑learning converges to the optimal policy and outperforms standard Q‑learning in settings where overestimation degrades performance.

Abstract

In some stochastic environments the well-known reinforcement learning algorithm Q-learning performs very poorly. This poor performance is caused by large overestimations of action values. These overestimations result from a positive bias that is introduced because Q-learning uses the maximum action value as an approximation for the maximum expected action value. We introduce an alternative way to approximate the maximum expected value for any set of random variables. The obtained double estimator method is shown to sometimes underestimate rather than overestimate the maximum expected value. We apply the double estimator to Q-learning to construct Double Q-learning, a new off-policy reinforcement learning algorithm. We show the new algorithm converges to the optimal policy and that it performs well in some settings in which Q-learning performs poorly due to its overestimation.

References

20