2017 · 127 citations · 31 references
ComputingComputer ScienceRobotics EngineeringMachine LearningRobot ControlRoboticsManipulation ControlManipulation LearningModel-based Robot LearningOptimal ControlPolicy LearningReinforcement LearningDiscontinuous DynamicsFeedback PoliciesGeneralization In ManipulationModel-free AlgorithmsPath-integral ControlPick-and-placePick-and-place ManipulationStochastic Optimal Control
We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discontinuous contact dynamics. We build on a prior technique called guided policy search (GPS), which iteratively optimizes a set of local policies for specific instances of a task, and uses these to train a complex, high-dimensional global policy that generalizes across task instances. We extend GPS in the following ways: (1) we propose the use of a model-free local optimizer based on path integral stochastic optimal control (PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), which enables us to learn local policies for tasks with highly discontinuous contact dynamics; and (2) we enable GPS to train on a new set of task instances in every iteration by using on-policy sampling: this increases the diversity of the instances that the policy is trained on, and is crucial for achieving good generalization. We show that these contributions enable us to learn deep neural network policies that can directly perform torque control from visual input. We validate the method on a challenging door opening task and a pick-and-place task, and we demonstrate that our approach substantially outperforms the prior LQR-based local policy optimizer on these tasks. Furthermore, we show that on-policy sampling significantly increases the generalization ability of these policies.
31
Continuous control with deep reinforcement learning
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel et al. · arXiv (Cornell University) · 2016 · 6.8K citations · Full text
Computing, Computer Science, Physics +17
Trust Region Policy Optimization
John Schulman, Sergey Levine, Philipp Moritz et al. · arXiv (Cornell University) · 2015 · 3.1K citations · Full text
End-to-end training of deep visuomotor policies
Sergey Levine, Chelsea Finn, Trevor Darrell et al. · Journal of Machine Learning Research · 2016 · 1.7K citations
End-to-End Training of Deep Visuomotor Policies
Sergey Levine, Chelsea Finn, Trevor Darrell et al. · arXiv (Cornell University) · 2015 · 1.4K citations · Full text