Hierarchical Reinforcement Learning with Advantage-Based Auxiliary\n Rewards

Abstract

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving\nlong-horizon problems with sparse and delayed rewards. Many existing HRL\nalgorithms either use pre-trained low-level skills that are unadaptable, or\nrequire domain-specific information to define low-level rewards. In this paper,\nwe aim to adapt low-level skills to downstream tasks while maintaining the\ngenerality of reward design. We propose an HRL framework which sets auxiliary\nrewards for low-level skill training based on the advantage function of the\nhigh-level policy. This auxiliary reward enables efficient, simultaneous\nlearning of the high-level policy and low-level skills without using\ntask-specific knowledge. In addition, we also theoretically prove that\noptimizing low-level skills with this auxiliary reward will increase the task\nreturn for the joint policy. Experimental results show that our algorithm\ndramatically outperforms other state-of-the-art HRL methods in Mujoco domains.\nWe also find both low-level and high-level policies trained by our algorithm\ntransferable.\n