arXiv (Cornell University) · 2018 · 44 citations · 21 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningRobot LearningImitation LearningLearning From DemonstrationPolicy IterationPolicy LearningReinforcement LearningReinforcement Learning FoundationsFast LearningMirror DescentMirror Descent Policy IterationSimulated Environments
Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.
21
Playing Atari with Deep Reinforcement Learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al. · arXiv (Cornell University) · 2013 · 5.1K citations · Full text
Robust Stochastic Approximation Approach to Stochastic Programming
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan et al. · SIAM Journal on Optimization · 2009 · 2.1K citations
High-Dimensional Continuous Control Using Generalized Advantage Estimation
John Schulman, Philipp Moritz, Sergey Levine et al. · arXiv (Cornell University) · 2015 · 1.8K citations · Full text