Multiagent Federated Reinforcement Learning for Secure Incentive Mechanism in Intelligent Cyber–Physical Systems

Abstract

Federated learning (FL) is an emerging technology for empowering various applications that generate large amounts of data in intelligent cyber–physical systems (ICPS). Though FL can address users’ concerns about data privacy, its maintenance still depends on efficient incentive mechanisms. For long-term incentivization to participants in data federation under dynamic environments, deep reinforcement learning as a promising technology has been extensively studied. However, the nonstationary problem caused by the heterogeneity of ICPS devices results in a serious effect on the convergence rate of existing single-agent reinforcement learning. In this article, we propose a multiagent learning-based incentive mechanism to capture the stationarity approximation in FL with heterogeneous ICPS. First, we formulate the secure communication and data resource allocation problem as a Stackelberg game in FL with multiple participants. Then, to tackle the heterogeneous problem, we model this multiagent game as a partially observable Markov decision process. In particular, a multiagent federated reinforcement learning algorithm is proposed to learn the allocation policies efficiently by dwindling variances in policy evaluation caused by interaction among multiple devices without the requirement of sharing privacy information. Moreover, the proposed algorithm is proved to attain convergence at an expected rate. Finally, extensive experimental results demonstrate that our proposed algorithm significantly outperforms baseline approaches.