IEEE Transactions on Wireless Communications · 2019 · 395 citations · 31 references
ComputingAerospace EngineeringComputer ScienceGame TheoryPath PlanningUnmanned Aircraft SystemsInterference ManagementMobile NetworkingMobile NetworksNetwork GamesPolicy LearningReinforcement LearningCellular Interference ManagementCellular Network OptimizationDeep Reinforcement LearningEnergy-aware Path PlanningInterferenceReinforcement Learning In GamesSubgame Perfect Nash EquilibriumUser-level Interference
The study proposes an interference‑aware path‑planning scheme for cellular‑connected UAVs that balances energy efficiency, latency, and interference. The authors model the problem as a dynamic game and solve it with a deep reinforcement learning algorithm using echo‑state‑network cells that map network observations to actions, enabling each UAV to learn optimal paths, power levels, and cell associations. The algorithm converges to a subgame‑perfect Nash equilibrium, with altitude bounds reducing complexity, and simulations demonstrate lower UAV latency and higher ground‑user rates than a shortest‑distance heuristic while adapting optimal altitude to network density and data‑rate demands.
In this paper, an interference-aware path planning scheme for a network of cellular-connected unmanned aerial vehicles (UAVs) is proposed. In particular, each UAV aims at achieving a tradeoff between maximizing energy efficiency and minimizing both wireless latency and the interference level caused on the ground network along its path. The problem is cast as a dynamic game among UAVs. To solve this game, a deep reinforcement learning algorithm, based on echo state network (ESN) cells, is proposed. The introduced deep ESN architecture is trained to allow each UAV to map each observation of the network state to an action, with the goal of minimizing a sequence of time-dependent utility functions. Each UAV uses ESN to learn its optimal path, transmission power level, and cell association vector at different locations along its path. The proposed algorithm is shown to reach a subgame perfect Nash equilibrium (SPNE) upon convergence. Moreover, an upper and lower bound for the altitude of the UAVs is derived thus reducing the computational complexity of the proposed algorithm. Simulation results show that the proposed scheme achieves better wireless latency per UAV and rate per ground user (UE) while requiring a number of steps that is comparable to a heuristic baseline that considers moving via the shortest distance towards the corresponding destinations. The results also show that the optimal altitude of the UAVs varies based on the ground network density and the UE data rate requirements and plays a vital role in minimizing the interference level on the ground UEs as well as the wireless transmission delay of the UAV.
31
Reinforcement Learning: An Introduction
Richard S. Sutton, Andy Barto · IEEE Transactions on Neural Networks · 1998 · 27.1K citations