The document discusses liquid and air cooling in data centers. It notes that for most data centers, liquid and air cooling will not be mutually exclusive. It provides examples of different types of liquid cooling solutions from mainframe to immersion and discusses some of the physics around why liquid cooling can be more effective than air cooling in certain scenarios. It also summarizes that liquid cooling is the removal of heat by liquid and gives an example of a data center utilizing both liquid and air cooling methods.
2. #DATACENTERWORLD | DATACENTERWORLD.COM
Data Center World – Certified Vendor Neutral
Each presenter is required to certify that their presentation will
be vendor-neutral.
As an attendee you have a right to enforce this policy of
having no sales pitch within a session by alerting the speaker
if you feel the session is not being presented in a vendor
neutral fashion. If the issue continues to be a problem, please
alert Data Center World staff after the session is complete.
3. #DATACENTERWORLD | DATACENTERWORLD.COM
Speaker Background
• Thought leader and recognized expert on data center
optimization
• Currently serves as the Senior Engineer and Company
Science Officer of Upsite Technologies
• Certified U.S. Department of Energy Data Center Energy
Practitioner (DCEP) HVAC Specialist
• Previous AFCOM Presentations:
• How IT Decisions Impact Facilities: The Benefit of Mutual Understanding
• Designing, Deploying, and Managing Efficient Data Centers
• Myths of Data Center Containment
• Understanding the Science Behind Data Center Airflow Management
• Data Center Cooling Efficiency: Understanding the Science of the 4 Delta T’s
4. #DATACENTERWORLD | DATACENTERWORLD.COM
Adoption of Liquid Cooling
~1970-1995
• Mainframes
1990-2000
• Gaming
• Custom
built PCs
2010-2015
• HPC driven
liquid
cooling in
the data
center
2005-2010
• Chilled
door
2015-
• Direct
contact
• Full
immersion
5. #DATACENTERWORLD | DATACENTERWORLD.COM
• Sandia National Laboratory is completing a purpose built
(primarily) liquid cooled data center in New Mexico
• 14,000 sq. ft.
• 10MW
• 85% warm water liquid cooled
• Augmented air cooling of 1.5MW (15%)
Sample Case Study
8. #DATACENTERWORLD | DATACENTERWORLD.COM
• Density
• HPC – latency
• Performance
• Higher clock speeds (research, modeling, finance)
• Economy
• Efficiency
• IT fans represent ~20% of IT load. Affects all aspects of power
distribution (switch gear, UPS, PDU, etc.)
• Warm water cooling can produce significant CapEx and OpEx savings
Drivers for Liquid Cooling
9. #DATACENTERWORLD | DATACENTERWORLD.COM
• Density
• HPC – latency
• Performance
• Higher clock speeds (research, modeling, finance)
• Economy
• Efficiency
• IT fans represent ~20% of IT load. Affects all aspects of power
distribution (switch gear, UPS, PDU, etc.)
• Warm water cooling can produce significant CapEx and OpEx savings
• More accurate PUE
• Server fans are on the IT load side of the PUE equation when they are
actually part of the cooling infrastructure
Drivers for Liquid Cooling
10. #DATACENTERWORLD | DATACENTERWORLD.COM
• Air is “understood,” familiar
• Care and handling paradigm shift – immersion
• White space – immersion
• Risk – liquid in cabinet, electrocution, outages
• No standards or many options
• Capital cost
• Complexity of running two systems
• Resistance due to hydrophobia
Resistance to Liquid Cooling
11. #DATACENTERWORLD | DATACENTERWORLD.COM
• “Power in heat out always.” – Ken Brill
• Every kW of electricity becomes a kW of heat that needs to be
removed from the building
• kW of IT consumption
• kW of fan motor consumption
• kW of transformer losses
• kW of distribution losses (resistance)
Definition of Liquid Cooling
12. #DATACENTERWORLD | DATACENTERWORLD.COM
• “Power in heat out always.” – Ken Brill
• Every kW of electricity becomes a kW of heat that needs to be
removed from the building
• kW of IT consumption
• kW of fan motor consumption
• kW of transformer losses
• kW of distribution losses (resistance)
• Liquid Cooling: Removal of heat by a liquid
• When the first loop that takes the heat away from the IT components is
liquid, then it is liquid cooling
• Many loops are often required to remove heat from the building
Definition of Liquid Cooling
19. #DATACENTERWORLD | DATACENTERWORLD.COM
• Rear door cooling
• Active (fans), or passive
• In row cooling
• Over row cooling
• In the top of cabinet cooling
Types of “Liquid” Cooling (Close Coupled Air Cooling)
20. #DATACENTERWORLD | DATACENTERWORLD.COM
• Touch Cooling
• Direct Liquid Cooling (DLC) to components, “plumbing”
Types of Liquid Cooling
CoolIT Lenovo
AsetekChilldyne
22. #DATACENTERWORLD | DATACENTERWORLD.COM
• Immersion
• Single Phase: using ElectroSafe, a formulated dielectric oil
• Green revolution
• Pump energy to keep moving
• 2 Phase: using dielectric fluids (Novec, Fluorinert)
• ICEOTOPE, LiquidCool
Types of Liquid Cooling
23. #DATACENTERWORLD | DATACENTERWORLD.COM
Liquid Cooling
Close Coupled Air Cooling
Immersion Touch
Immersion
Direct Liquid
Cooling
Fixed Cold Plate Rear Door In Row
% of Heat
Captured by
Liquid
100% ~65% to 75% ~93% 0% N/A
% of Heat
Captured by Air
0% ~25% to 35% ~7% 100% N/A
Effectiveness of Various Types of Liquid Cooling
Cabinet Level
24. #DATACENTERWORLD | DATACENTERWORLD.COM
Air Touch Immersion
Compute/Processing
Y Y Y
Switching (lots of cords/ports) Y Difficult / In development Difficult / In development
Solid state storage Y Y Y
Spinning media Y Y
Spinning media if sealed
(HDD)
Power supplies Y In development Y
What Does Air and Liquid Cool Well and Not Cool Well
25. #DATACENTERWORLD | DATACENTERWORLD.COM
• Space available for air and water are different
• Liquid is often running through small tubing
• 1/4” – 0.05 sq. in.
• 1/8” – 0.012 sq. in.
• Air is likely to have several square inches 3.0 sq. in. (60x – 1/4” tube)
• Density of air ~0.074 lbs. / ft3 (1 lbs. – 12.4 ft3)
• Density of water ~ 62.3 lbs. / ft3 (1 lbs. – 0.016 ft3) (842x better
than air)
• Specific heat of air ~993 J/kg °C
• Specific heat of water ~4200 J/kg °C (only 4x better than air)
Physics of Liquid Cooling
26. #DATACENTERWORLD | DATACENTERWORLD.COM
• Convection heat transfer from a surface to an fluid is governed
by the equation: q = hA(Tw - Tf)
• h = the convection heat transfer coefficient
• A = the surface area
• Tw = the temperature of the surface
• Tf = the temperature of the fluid
• Coefficient of heat transfer for moderate speed air is~100 and
moderate flow water is ~ 3000
• Water is 30x better than air
Physics of Liquid Cooling
27. #DATACENTERWORLD | DATACENTERWORLD.COM
• Convection heat transfer from a surface to an fluid is governed
by the equation: q = hA(Tw - Tf)
• h = the convection heat transfer coefficient
• A = the surface area
• Tw = the temperature of the surface
• Tf = the temperature of the fluid
• Coefficient of heat transfer for moderate speed air is~100 and
moderate flow water is ~ 3000
• Water is 30x better than air
• Surface area available
• Water flowing over chip - 1 sq. in.
• Air flowing over heat sink – 10 sq. in.
• Taking surface area into consideration water is 3x better than
air
Physics of Liquid Cooling
28. #DATACENTERWORLD | DATACENTERWORLD.COM
• Liquid cooling is the removal of heat by a liquid
• Depending on the component and solution, only a portion of the
heat can be removed by a liquid
• Not all equipment will be liquid cooled
• The question is not which is better, but how can you use both
liquid and air effectively?
• Liquid has always been used in the data center, just how close
• Liquid and air cooling will not be mutually exclusive
Key Takeaways
29. #DATACENTERWORLD | DATACENTERWORLD.COM
• Sandia National Laboratory is completing a purpose built
(primarily) liquid cooled data center in New Mexico
• 14,000 sq. ft.
• 10MW
• 85% warm water liquid cooled
• Augmented air cooling of 1.5MW (15%)
Sample Case Study
30. #DATACENTERWORLD | DATACENTERWORLD.COM
Questions & Answers
Lars Strong, P.E.
Senior Engineer, Upsite Technologies
lds@upsite.com
Follow Upsite for the latest news and information on data center airflow optimization.
Facebook InstagramLinkedInTwitter YouTube Upsite Blog