Learning Attractor Landscapes for Learning Motor Primitives

Auke Jan Ijspeert, Jun Nakanishi, Stefan Schaal

Infoscience (Ecole Polytechnique Fédérale de Lausanne) · 2002 · 557 citations · 7 references

Full text

Open access

Concepts

TL;DR

Control problems in continuous state‑action spaces, such as robotic manipulators, often require finding trajectories to reach goal states, but reinforcement learning is limited in high‑dimensional continuous domains. This paper proposes to learn complex control policies by transforming existing simple canonical policies instead of learning from scratch. Canonical policies are expressed as differential equations with attractor properties, and nonlinear transformations via nonparametric regression generate new stable policies while preserving stability. The approach enables rapid acquisition of movement skills, online adaptation to dynamic environments, classification of learned skills, and demonstrates feasibility and robustness on a 30‑DOF humanoid robot in simulation and real experiments.

Abstract

Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach.

References

7