Learning to Interrupt: A Hierarchical Deep Reinforcement Learning Framework for Efficient Exploration

Abstract

To achieve scenario intelligence, humans must transfer knowledge to robots by developing goal-oriented algorithms, which are sometimes insensitive to dynamically changing environments. While deep reinforcement learning achieves significant success recently, the long and difficult training process limits its application. In this paper, we propose a hybrid structure named Option-Interruption in which human knowledge is embedded into a hierarchical reinforcement learning framework. Our architecture has two key components: options, represented by existing human-designed methods, can significantly speed up the training process and interruption mechanism, based on learnable termination functions, enables our system to timely terminate the current option according to the external environment. To implement this architecture, we derive a set of update rules based on policy gradient methods and present a complete training process. In the experiment part, our method is evaluated in two simulated tasks, Four-room navigation and exploration task, which shows the efficiency and flexibility of our framework.