arXiv (Cornell University) · 2019 · 41 citations · 20 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningMarkov Decision ProcessesMeta-learningMulti-task LearningPolicy LearningReinforcement LearningReinforcement Learning FoundationsValue FunctionsCredit AssignmentMeta-learning TheoryModel-free AlgorithmsReinforcement Learning TransferReward Shaping
Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple similar tasks to solve. In this paper, we consider reward shaping on a distribution of tasks, and propose a general meta-learning framework to automatically learn the efficient reward shaping on newly sampled tasks, assuming only shared state space but not necessarily action space. We first derive the theoretically optimal reward shaping in terms of credit assignment in model-free RL. We then propose a value-based meta-learning algorithm to extract an effective prior over the optimal reward shaping. The prior can be applied directly to new tasks, or provably adapted to the task-posterior while solving the task within few gradient updates. We demonstrate the effectiveness of our shaping through significantly improved learning efficiency and interpretable visualizations across various settings, including notably a successful transfer from DQN to DDPG.
20
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al. · Nature · 2015 · 30K citations
Reinforcement Learning: An Introduction
IEEE Transactions on Neural Networks · 2005 · 25.8K citations
The Behavior of Organisms: An Experimental Analysis
F. Nowell Jones, B. F. Skinner · The American Journal of Psychology · 1939 · 4.4K citations
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
David Silver, Thomas Hubert, Julian Schrittwieser et al. · Science · 2018 · 3.5K citations · Full text