Journal of Machine Learning Research · 2018 · 57 citations · 39 references
ComputingComputer ScienceRobotics EngineeringMachine LearningRobot ControlRoboticsAutonomous ExplorationExploration-exploitation TradeoffOptimal ControlPolicy LearningReinforcement LearningStochastic ControlDeep Reinforcement LearningExperience ReplayLong-term UtilityOff-policy LearningOff-policy Reinforcement LearningPrior Related KnowledgeReinforcement Learning For RoboticsReplay
Experience replay is a technique that allows off-policy reinforcement-learning methods to reuse past experiences. The stability and speed of convergence of reinforcement learning, as well as the eventual performance of the learned policy, are strongly dependent on the experiences being replayed. Which experiences are replayed depends on two important choices. The first is which and how many experiences to retain in the experience replay buffer. The second choice is how to sample the experiences that are to be replayed from that buffer. We propose new methods for the combined problem of experience retention and experience sampling. We refer to the combination as experience selection. We focus our investigation specifically on the control of physical systems, such as robots, where exploration is costly. To determine which experiences to keep and which to replay, we investigate different proxies for their immediate and long-term utility. These proxies include age, temporal difference error and the strength of the applied exploration noise. Since no currently available method works in all situations, we propose guidelines for using prior knowledge about the characteristics of the control problem at hand to choose the appropriate experience replay strategy.
39
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al. · Nature · 2015 · 30K citations
Bootstrap Methods: Another Look at the Jackknife
B. Efron · The Annals of Statistics · 1979 · 17.4K citations · Full text
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison et al. · Nature · 2016 · 15.8K citations
Mastering the game of Go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan et al. · Nature · 2017 · 9.2K citations
Engineering, Educational Psychology, Cognitive Psychology +7
Continuous control with deep reinforcement learning
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel et al. · arXiv (Cornell University) · 2015 · 5.4K citations · Full text
Computing, Computer Science, Physics +17