arXiv (Cornell University) · 2015 · 81 citations · 23 references
BusinessDecision SciencesDecision TheoryMachine LearningPolicy EvaluationBias-variance TradeoffPolicy ImprovementReinforcement LearningValue FunctionsBandit AlgorithmsOff-policy EvaluationOff-policy LearningOff-policy Policy ImprovementOff-policy Reinforcement LearningOff-policy Value EstimationSafe Policy ImprovementSafe Policy Learning
We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.
23
Statistics and Causal Inference
Paul W. Holland · Journal of the American Statistical Association · 1986 · 5.1K citations
Kybernetes · 1998 · 1.6K citations
Approximately Optimal Approximate Reinforcement Learning
Sham M. Kakade, John Langford · 2002 · 595 citations