Offline Meta-Reinforcement Learning for Active Pantograph Control in High-Speed Railways

Abstract

Previous reinforcement learning (RL) methods suffer significant performance degradation or collapse when deployed to the real world due to the huge sim-real gap. This article proposes a hybrid offline-and-online meta-RL (HOMRL) algorithm that leverages prior task experience to learn and adapt to new pantograph active control tasks in real-world applications. The policy learning process consists of three phases: offline meta-policy pretraining, online adaptation, and fine-tuning. First, we construct an offline meta-RL approach that learns from the massive and heterogeneous static training datasets, eliminating online interaction's high cost and hazard. Second, we combine context-based meta-RL with online fine-tuning to generalize to challenging tasks, while high safety and success rates are critical in railway applications. Finally, the proposed environment-sensitive task encoder (TE) and well-trained agent can adapt to new tasks quickly and efficiently, even in unseen tasks and nonstationary environments. If the new task is similar to the prior data, the contextual meta-learner adapts immediately. If it is too different, it gradually adapts through fine-tuning.