A Closer Look at Invalid Action Masking in Policy Gradient Algorithms

TLDR

Deep reinforcement learning has excelled in complex strategy games, yet actions sampled from the full policy distribution frequently violate game rules, so invalid actions are typically masked—a practice whose consequences remain largely unexplored. This study aims to theoretically justify action masking, empirically demonstrate its necessity as the invalid action space grows, and analyze various masking regimes, including disabling masking after training. The authors provide a theoretical analysis of masking effects, conduct experiments varying the size of invalid action spaces, and evaluate different masking strategies such as post‑training removal.

Abstract

In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved state-of-the-art performance in many challenging strategy games. Because these games have complicated rules, an action sampled from the full discrete action distribution predicted by the learned policy is likely to be invalid according to the game rules (e.g., walking into a wall). The usual approach to deal with this problem in policy gradient algorithms is to “mask out” invalid actions and just sample from the set of valid actions. The implications of this process, however, remain under-investigated. In this paper, we 1) show theoretical justification for such a practice, 2) empirically demonstrate its importance as the space of invalid actions grows, and 3) provide further insights by evaluating different action masking regimes, such as removing masking after an agent has been trained using masking.