arXiv (Cornell University) · 2018 · 131 citations · 0 references
ComputingComputer ScienceRobotics EngineeringMachine LearningRobot ControlRoboticsManipulation ControlManipulation LearningPolicy ImprovementPolicy IterationPolicy LearningReinforcement LearningCost ModelsLinear Quadratic RegulationLinear Quadratic RegulatorModel-based Reinforcement LearningModel-free ControlReinforcement Learning For ManipulationReinforcement Learning For RoboticsStructured Representations
Model‑based reinforcement learning is data‑efficient for control tasks but struggles with complex observations such as images. The authors propose a representation learning approach that optimizes for simple dynamics and cost models to enable iterative model‑based policy improvement in systems with complex dynamics and image observations. By learning such representations, they apply a linear‑quadratic regulator–based model‑based RL to image‑observed robotics tasks, including real‑world arm manipulation, and evaluate its performance. The method achieves substantially better final performance than other model‑based RL baselines while being significantly more efficient than model‑free RL.
Model-based reinforcement learning (RL) has proven to be a data efficient approach for learning control tasks but is difficult to utilize in domains with complex observations such as images. In this paper, we present a method for learning representations that are suitable for iterative model-based policy improvement, even when the underlying dynamical system has complex dynamics and image observations, in that these representations are optimized for inferring simple dynamics and cost models given data from the current policy. This enables a model-based RL method based on the linear-quadratic regulator (LQR) to be used for systems with image observations. We evaluate our approach on a range of robotics tasks, including manipulation with a real-world robotic arm directly from images. We find that our method produces substantially better final performance than other model-based RL methods while being significantly more efficient than model-free RL.