Neural Information Processing Systems · 2010 · 393 citations · 20 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningMarkov Decision ProcessesBias-variance TradeoffPolicy LearningReinforcement LearningReinforcement Learning FoundationsOff-policy LearningOff-policy Policy ImprovementOff-policy Reinforcement LearningOff-policy Value EstimationQ-learningReinforcement
In stochastic environments, Q‑learning suffers from large overestimations of action values caused by a positive bias from using the maximum action value as an approximation. The authors propose an alternative method to approximate the maximum expected value for any set of random variables. They apply this double‑estimator approach to Q‑learning, yielding the Double Q‑learning algorithm. Double Q‑learning converges to the optimal policy and outperforms standard Q‑learning in settings where overestimation degrades performance.
In some stochastic environments the well-known reinforcement learning algorithm Q-learning performs very poorly. This poor performance is caused by large overestimations of action values. These overestimations result from a positive bias that is introduced because Q-learning uses the maximum action value as an approximation for the maximum expected action value. We introduce an alternative way to approximate the maximum expected value for any set of random variables. The obtained double estimator method is shown to sometimes underestimate rather than overestimate the maximum expected value. We apply the double estimator to Q-learning to construct Double Q-learning, a new off-policy reinforcement learning algorithm. We show the new algorithm converges to the optimal policy and that it performs well in some settings in which Q-learning performs poorly due to its overestimation.
20
Reinforcement Learning: An Introduction
IEEE Transactions on Neural Networks · 2005 · 25.8K citations
Monte Carlo sampling methods using Markov chains and their applications
W. Keith Hastings · Biometrika · 1970 · 15.2K citations
Tree-Based Batch Mode Reinforcement Learning
Damien Ernst, Pierre Geurts, Louis Wehenkel · 2005 · 866 citations · Full text