Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep\n Multi-Agent Reinforcement Learning

Abstract

QMIX is a popular $Q$-learning algorithm for cooperative MARL in the\ncentralised training and decentralised execution paradigm. In order to enable\neasy decentralisation, QMIX restricts the joint action $Q$-values it can\nrepresent to be a monotonic mixing of each agent's utilities. However, this\nrestriction prevents it from representing value functions in which an agent's\nordering over its actions can depend on other agents' actions. To analyse this\nrepresentational limitation, we first formalise the objective QMIX optimises,\nwhich allows us to view QMIX as an operator that first computes the\n$Q$-learning targets and then projects them into the space representable by\nQMIX. This projection returns a representable $Q$-value that minimises the\nunweighted squared error across all joint actions. We show in particular that\nthis projection can fail to recover the optimal policy even with access to\n$Q^*$, which primarily stems from the equal weighting placed on each joint\naction. We rectify this by introducing a weighting into the projection, in\norder to place more importance on the better joint actions. We propose two\nweighting schemes and prove that they recover the correct maximal action for\nany joint action $Q$-values, and therefore for $Q^*$ as well. Based on our\nanalysis and results in the tabular setting, we introduce two scalable versions\nof our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW)\nQMIX and demonstrate improved performance on both predator-prey and challenging\nmulti-agent StarCraft benchmark tasks.\n