Stress Testing of UAS Traffic Management Decision Making Systems

Abstract

The emergence of new operations, such as package delivery and air taxis, that could co-exist in the current airspace has been a topic of great interest in recent years. A big challenge associated with the introduction of these new operations is ensuring that the complex system-of-systems responsible for these operations known as Unmanned Traffic Management (UTM) meet the high safety standards of aviation. In this work, we present a general purpose framework for validating decision making systems, protocols, and algorithms that may exist in the UTM ecosystem. We propose a novel, simulation driven approach for validation in UTM that can automatically discover the failure modes of a decision making system, and optimize the parameters that configure the system to improve its performance. We apply this approach to Urban Air Mobility (UAM) and package delivery use cases, and examine in detail the failure modes and the potential improvements of two UTM services critical for strategic deconfliction: trajectory planning and optimal scheduling. Using simulation, we demonstrate that our algorithm is able to discover failure modes in the system that would be challenging for humans to find and fix, and we show how the algorithm can learn from these failure modes to improve the performance of the UTM services in question. We also demonstrate the significant sample efficiency improvements that our algorithm has over naive stress testing approaches that rely on uniform sampling.