Thermal and mechanical analysis and design of the IBM Power 775 water cooled supercomputing central electronics complex

Abstract

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. Water cooling enabled a compute node with 34% greater performance (Flops), resulted in a processor temperature 20-30°C lower than that typically provided with air cooling, and reduced the power consumed in the data center to transfer the IT heat to the outside ambient by as much as 45%. The next generation of this platform, the Power 775 Supercomputing node/system, is a significant leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 95% of its heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an inter-board air-to-water heat exchanger. This paper will detail key thermal and mechanical design issues associated with the Power 775 server drawer or central electronics complex (CEC). Topics to be addressed include processor and optical I/O Hub Module thermal design (including thermal interfaces); water cooled memory design; module cold plate designs; CEC level water distribution; module level structural analyses for thermal performance; module/board land grid array (LGA) load distribution; effect of load distribution on module thermal interfaces; and the effect of cold plate tubing on module (LGA) loading.