Multi-AUV Cooperative Underwater Multi-Target Tracking Based on Dynamic-Switching-Enabled Multi-Agent Reinforcement Learning

Abstract

In recent years, autonomous underwater vehicle (AUV) swarms are gradually becoming popular and have been widely promoted in ocean exploration or underwater tracking, etc. In this paper, we propose a multi-AUV cooperative underwater multi-target tracking algorithm especially when the real underwater factors are taken into account. We first give normally modelling approach for the underwater sonar-based detection and the ocean current interference on the target tracking process. Then, based on software-defined networking (SDN), we regard the AUV swarm as a underwater ad-hoc network and propose a hierarchical software-defined multi-AUV reinforcement learning (HSARL) architecture. Based on the proposed HSARL architecture, we propose the “Dynamic-Switching” mechanism, it includes “Dynamic-Switching Attention” and “Dynamic-Switching Resampling” mechanisms which accelerate the HSARL algorithm's convergence speed and effectively prevents it from getting stuck in a local optimum state. Additionally, we introduce the reward reshaping mechanism for further accelerating the convergence speed of the proposed HSARL algorithm in early phase. Finally, based on a proposed AUV classification method, we propose a cooperative tracking algorithm called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b>ynamic-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b>witching-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b>ased <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b>ARL (DSBM)-driven tracking algorithm. Evaluation results demonstrate that our proposed DSBM tracking algorithm can perform precise underwater multi-target tracking, comparing with many of recent research products in terms of various important metrics.