2020 · 32 citations · 21 references
ComputingComputer ScienceDecision SciencesMachine LearningMarkov Decision ProcessesPolicy EvaluationMarkov ModelingPartially Observable Markov Decision ProcessesProbabilistic Machine LearningReinforcement LearningReinforcement Learning FoundationsBatch Policy LearningOff-policy EvaluationOff-policy LearningOff-policy Reinforcement LearningPartial ObservabilityPartial Observability ModelsPartially Observable Reinforcement Learning
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Markov Decision Processes (POMDPs) and establish what we believe is the first off-policy evaluation result for POMDPs. In addition, we formulate a model in which observed and unobserved variables are decoupled into two dynamic processes, called a Decoupled POMDP. We show how off-policy evaluation can be performed under this new model, mitigating estimation errors inherent to general POMDPs. We demonstrate the pitfalls of off-policy evaluation in POMDPs using a well-known off-policy method, Importance Sampling, and compare it with our result on synthetic medical data.
21
Reinforcement Learning: An Introduction
Richard S. Sutton, Andy Barto · IEEE Transactions on Neural Networks · 1998 · 27.1K citations
Causality: models, reasoning, and inference
Choice Reviews Online · 2010 · 1.1K citations