Distributed Hierarchical Deep Reinforcement Learning for Large-Scale Grid Emergency Control

Abstract

Reliable and fast emergency control technologies are essential to guarantee the transient stability of power systems. In recent years, deep reinforcement learning (DRL) has been widely adopted in emergency controls for its high-dimensional feature extraction ability and fast response speed. However, conventional DRL methods still suffer from computational inefficiency and convergence difficulties when directly applied in large-scale power systems for their inherent defect in high-dimensional action space. In this paper, a novel DRL-based framework named hierarchical reduction reinforcement learning (HR2L) is developed for emergency controls to fill these gaps. HR2L achieves an efficient and accurate action space reduction based on self-supervised algorithm, significantly alleviating the computation complexity. Moreover, an experiences sharing-based (ES-based) distributed architecture is tailored for HR2L to further enhance its scalability. Numerical simulations on IEEE 39-bus and 300-bus systems demonstrate that compared with other state-of-the-art (SOTA) DRL methods, HR2L shows better convergence speed, solution quality, training robustness, as well as adaptability to large-scale power systems.