arXiv (Cornell University) · 2019 · 65 citations · 50 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningUncertainty QuantificationExploration-exploitation TradeoffMeta-learningPolicy LearningReinforcement LearningReinforcement Learning FoundationsVariational InferenceAction Selection ModelsApproximate Bayesian InferenceBayesian OptimizationDeep Reinforcement LearningExplorationTask UncertaintyVariational BayesVariational Bayes Inference
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.
50
Auto-Encoding Variational Bayes
Diederik P. Kingma, Max Welling · UvA-DARE (University of Amsterdam) · 2013 · 15.6K citations · Full text
Automatic differentiation in PyTorch
Adam Paszke, Sam Gross, Soumith Chintala et al. · 2017 · 11.1K citations
MuJoCo: A physics engine for model-based control
Emanuel Todorov, Tom Erez, Yuval Tassa · 2012 · 4.4K citations
Engineering, Mechanical Engineering, Robotics Engineering +17