Teacher algorithms for curriculum learning of Deep RL in continuously\n parameterized environments

Abstract

We consider the problem of how a teacher algorithm can enable an unknown Deep\nReinforcement Learning (DRL) student to become good at a skill over a wide\nrange of diverse environments. To do so, we study how a teacher algorithm can\nlearn to generate a learning curriculum, whereby it sequentially samples\nparameters controlling a stochastic procedural generation of environments.\nBecause it does not initially know the capacities of its student, a key\nchallenge for the teacher is to discover which environments are easy, difficult\nor unlearnable, and in what order to propose them to maximize the efficiency of\nlearning over the learnable ones. To achieve this, this problem is transformed\ninto a surrogate continuous bandit problem where the teacher samples\nenvironments in order to maximize absolute learning progress of its student. We\npresent a new algorithm modeling absolute learning progress with Gaussian\nmixture models (ALP-GMM). We also adapt existing algorithms and provide a\ncomplete study in the context of DRL. Using parameterized variants of the\nBipedalWalker environment, we study their efficiency to personalize a learning\ncurriculum for different learners (embodiments), their robustness to the ratio\nof learnable/unlearnable environments, and their scalability to non-linear and\nhigh-dimensional parameter spaces. Videos and code are available at\nhttps://github.com/flowersteam/teachDeepRL.\n