AAV Swarm Cooperative Search Based on Scalable Multiagent Deep Reinforcement Learning With Digital Twin-Enabled Sim-to-Real Transfer

Abstract

Cooperative target search (CTS) technology is highly desirable in various multi-autonomous aerial vehicle (AAV) applications. However, searching for unknown targets in a dynamic threatening environment is a challenging problem, especially for AAVs with limited sensing range and communication capabilities. Besides, traditional searching methods lack scalability and efficient collaboration among the AAV swarm in dynamic environments. In this work, a digital twin (DT)-enabled distributed CTS approach was presented for AAV swarms and achieving sim-to-real transfer. Specifically, a new scalable multi-agent reinforcement learning (MARL) based algorithm called SAMARL is adopted to improve effectiveness and adaptability, combining a multi-head attention mechanism. In SAMARL, a scalable observation space with graph representation and an environmental cognition map is designed to thoroughly consider the target search rate, area coverage, and safety assurance. Then, a DT-driven training framework is proposed to facilitate the continuous evolution of MARL models and address the tradeoff between training speed and environment fidelity. Furthermore, we innovatively develop a distributed AAV swarm digital twin cooperative target search validation system, including real flight control, communication simulation tools, and a 3D physics engine. Extensive simulations validate its superiority compared to state-of-the-art strategies. More importantly, we also conduct real-world flight experiments on different scale mission areas and AAV swarms, further demonstrating the generalization and scalability of trained models.