Enhancing Grid-Interactive Buildings Demand Response: Sequential Update-Based Multiagent Deep Reinforcement Learning Approach

Abstract

Demand response (DR) programs aim to improve grid reliability, reduce customer costs, and manage building energy demand. However, coordinating autonomous demand response in grid-interactive buildings faces challenges due to uncertainties in system model parameters, complex energy demands, and interdependent operational constraints. Specifically, the energy management problem in grid-interactive buildings is formulated as a Partially Observable Markov Decision Process (POMDP), where each building has only local information and no knowledge of other buildings’ behavior. This makes achieving consistent coordinated energy management challenging. To address these issues, this paper proposes a novel approach called Sequential Update-based Multi-Agent Deep Reinforcement Learning (SUMADRL). This approach updates the policy of each building sequentially, considering the updates already made by previous buildings. This ensures that the joint policy update of buildings is always highly aligned with the overall energy optimization goal and improves coordinated energy management among buildings. Additionally, the sequential update scheme provides theoretical guarantees for achieving monotonically improving rewards and converging to Nash equilibrium, enhancing credibility and stability. Experimental results show that the proposed method significantly reduces energy costs, achieves efficient load shaping, and demonstrates superior effectiveness and stability in demand response compared to the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.