Publication | Open Access
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
429
Citations
0
References
2020
Year
Artificial IntelligenceEngineeringMachine LearningData ScienceDeep Reinforcement LearningMulti-agent SystemsMulti-agent ReinforcementStarcraft Multi-agent ChallengeMixing NetworkDistributed Ai SystemComputer ScienceDistributed Artificial IntelligenceRobot LearningIntelligent SystemsDeep LearningMulti-agent LearningMulti-agent Planning
Real‑world multi‑agent systems require decentralized coordination, yet centralized training with global state can be leveraged; however, how to derive decentralized policies from joint action‑value functions remains unclear. The study introduces QMIX, a value‑based method for training decentralized policies centrally, and proposes SMAC as a benchmark to assess its performance. QMIX employs a monotonic mixing network with non‑negative weights to combine per‑agent value functions, guaranteeing consistency between centralized joint‑action values and decentralized policies during end‑to‑end training. QMIX significantly outperforms existing multi‑agent reinforcement learning methods on challenging SMAC scenarios.
In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.