2022 · 16 citations · 19 references
ComputingComputer ScienceComputer SystemsArtificial IntelligenceMachine LearningMarkov Decision ProcessesConstrained Markov Decision ProcessesExploration-exploitation TradeoffImitation LearningPolicy LearningReinforcement LearningData Center CoolingDeep Reinforcement LearningImitation Learning SafetyMarkov Decision ModelsOffline Imitation LearningReward ShapingSafe Reinforcement LearningState Space Exploration
Deep reinforcement learning (DRL) has shown good performance in tackling Markov decision process (MDP) problems. As DRL opti-mizes a long-term reward, it is a promising approach to improving the energy efficiency of data center cooling. However, enforcement of thermal safety constraint during DRL's state exploration is a main challenge. The widely adopted reward shaping approach adds negative reward when the exploratory action results in unsafety. Thus, it needs to experience sufficient unsafe states before it learns how to prevent unsafety. In this paper, we propose a safety-aware DRL framework for single-hall data center cooling control. It applies offline imitation learning and online post-hoc rectification to holis-tically prevent thermal unsafety during online DRL. In particular, the post-hoc rectification searches for the minimum modification to the DRL-recommended action such that the rectified action will not result in unsafety. The rectification is designed based on a thermal state transition model that is fitted using historical safe operation traces and able to extrapolate the transitions to unsafe states ex-plored by DRL. Extensive evaluation for chilled water and direct expansion cooled data centers in two climate conditions shows that our approach saves 22.7% to 26.6% total data center power compared with conventional control, reduces safety violations by 94.5% to 99% compared with reward shaping.
19
Mastering the game of Go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan et al. · Nature · 2017 · 9.2K citations
Engineering, Educational Psychology, Cognitive Psychology +7
Continuous control with deep reinforcement learning
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel et al. · arXiv (Cornell University) · 2016 · 6.8K citations · Full text
Computing, Computer Science, Physics +17
EnergyPlus: Energy simulation program
Drury B. Crawley, Linda K. Lawrie, C.O. Pedersen et al. · 2000 · 729 citations
Engineering, Computational Engineering, Mechanical Engineering +12
Safe Exploration in Continuous Action Spaces
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík et al. · arXiv (Cornell University) · 2018 · 276 citations · Full text