Counterfactual Multi-Agent Policy Gradients

TLDR

Cooperative multi‑agent systems, exemplified by network routing and autonomous vehicle coordination, require efficient decentralised reinforcement learning methods. The authors introduce COMA, a counterfactual multi‑agent policy gradient method, to address decentralised policy learning. COMA employs a centralised critic to estimate Q‑values, decentralised actors to optimise policies, and a counterfactual baseline that marginalises a single agent’s action while fixing others, enabling efficient computation in a single forward pass and evaluation on StarCraft micromanagement with partial observability. COMA outperforms other multi‑agent actor‑critic methods on StarCraft micromanagement and rivals state‑of‑the‑art centralised controllers with full state access.

Abstract

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.