Gary Bernstein, Director of Network and Communications Services (NCS) at McGill University, presented these slides as part of the Cybera Summit 2010 session "Ultra-Efficient Data Centres: Design and Applications". For more information, visit http://www.cybera.ca/ultra-efficient-data-centres-design-and-applications

1. Ultra Efficient HPC Data
Center
Natural Low Energy Cooling Conceptual Design
10/8/10 1

3. Project Funding
 Canada's Advanced Research and Innovation
Network (CANARIE)
 Canada California Strategic Innovation Partnership
(CCSIP)
• ISTP Canada
• University of California
• McGill University

4. Site Selection
 Three candidate locations in Quebec
• McDonald Campus of McGill University in St. Anne de Bellevue
• Campus of the Institut de recherche d’Hydro-Quebec (IREQ) in
Varennes
• IREQ campus in Shawinigan
 McDonald Campus of McGill University selected as
site for project
 All enjoy
 Cold climate
 Renewable energy resource (hydroelectric)
 Inexpensive electricity

5. The System Approach: An Overview
 Goals: Most Efficient Class One Data Center
 Climate Evaluation
 Define Loads and How to Best Serve Them
• Water cooled equipment
• Medium temperature chilled water (65F, 75F)
 Optimize Heat Rejection for Climate and Loads
• Evaporative free cooling – Primary cooling
• Seasonal ice storage – Top up cooling
 Backup Approach
 Results

6. Goals
 Provide ASHRAE TC9.9 Class 1 Datacenter
 No compressor based cooling
 Lower construction cost
 Lower operating cost
 Best efficiency
 Environmentally friendly
 Construction materials
 Recycle heat, water

7. Proposed Annual Electrical Costs
Comparison
$10,000,000
$9,000,000
$8,000,000
$7,000,000
$5 Million/yr Annual
$6,000,000 Savings Target
$5,000,000
$4,000,000
$3,000,000
$2,000,000
$1,000,000
$0
San Diego (1.35 PUE) Montreal (1.06 PUE)
at $0.09/kWh at $0.05/kWh

8. Aerial Perspective – “Farm” at
McDonald Campus
Cooling Towers and
Mechanical Infrastructure 20,000 SF Phase 1
8 MW IT Load
VA
Hospital
Fuel Tanks
Cooling Ice Pond
Stormwater Detension
Office, Shipping/
Receiving Electrical
Infrastructure

9. Climate: Free Cooling Analysis with 65F CHWS
0.030
Data Source: Government of Canada - National Climate Data & Information Archive
0.028
Data Set: WMO #71627, Montreal/Pierre Elliott Trudeau Airport, Typical Year
0.026
Elevation: 118 feet
Humidity Ratio (lbs H2O per lbs dry air)
0.024
Air Pressure: 14.633224 psia
0.022
0.020 Auxillary Cooling
80 hrs/yr
0.018
0.016 Partial Free Cooling
1234 hrs/yr
0.014
0.012
0.010
0.008
Full Free Cooling
0.006 7446 hrs/yr
0.004
0.002
0.000
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Dry Bulb Temperature (F)

10. Climate: Free Cooling Analysis With 75F CHWS
0.030
Data Source: Government of Canada - National Climate Data & Information Archive
0.028
Data Set: Montreal/Pierre Elliott Trudeau Intl Airport, Typical Year
0.026 Elevation: 118 feet Auxillary Cooling
Air Pressure: 14.633224 psia 0 hrs/yr
Humidity Ratio (lbs H2O per lbs dry air)
0.024
0.022
0.020 Partial Free Cooling
0.018 114 hrs/yr
0.016
0.014
0.012
0.010
0.008
Full Free Cooling
0.006 8646 hrs/yr
0.004
0.002
0.000
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Dry Bulb Temperature (F)

11. Load Strategies
 Climate analysis shows higher temperature chilled
water offers many more hours of free cooling
 Highly concentrated heat loads
• A single high density rack
can put off as much
waste heat as a
VW Beetle (40kW)
• Air exiting racks
typically exceeds
90F

12. Load Heat Collection Strategies
 Higher temperature Chilled Water Supply (CHWS)
offers many more hours of free cooling: Design to
use 75F and 65F CHWS
• Direct water based cooling most efficient
• Hot aisle / cold aisle for minority of load

13. Primary Cooling Strategies: Medium
Temp. Cooling Water and Free Cooling

14. Primary Cooling Strategies: Medium
Temp. Cooling Water and Free Cooling
 Design to cool with 65F/18C and 75F/24C water
• 90% of 65F load served with cooling tower provided free cooling;
99.3% of 75F load
• 590,000 ton-hrs (2,100 MWh) Top up Cooling Required

15. Supplemental Cooling: Seasonal Ice
Storage Slush Pond System
 Fill in winter with plowed snow collection
 Melt water cools data center

17. Slush Pond
 Paved collection basin, 75,000 ft3 (2,100 m3)
 Drive-in slope on one side for plow loading
 Lightweight, waterproof insulating cover or roof to protect
from warm rains
 Extensive drain system to collect meltwater
 Berms for sides, or dig into ground

18. Sundsvall, Sweden, Snow Storage - Empty

19. Sundsvall, Sweden, Snow Storage - Full

20. Snow dump overruns in Montreal, 2008-09

21. Slush Pond System – Pumping and
Filtration
 Mature waste water handling technology
 Remove gravel, wood, grit from melt water
 Remove oils and road chemicals prior to release as required
 Filter
 Select heat exchangers for highly corrosive fluid
 Maintain complete separation between pond water and
building loop water
 Integrate settling tank to also serve as emergency
storage

22. Key approaches
 Keep storage pond simple
 Leverage local snow removal program if possible
 Collect snow dumpage fees?
 Provide appropriate maintenance
 Provide for pile grooming, drain clearing, filter cleaning, etc
 Use in lieu of chillers to save cost
 Consider emergency chiller rental for backup
 Design properly
 Simple concept but careful design required

23. Office Approaches
 Much lower load
 Design for comfort and optimal use of medium
temperature water

24. Backup – Do Not Invest in Chillers 'Just
in Case'!
Pay for it only
when (if) you
ever need it
Design for
portable
air-cooled
chillers
to connect in
an emergency

25. Results
McGill-USCD HPC Data Center PUE Itemization
Fans; 1.5%
CRAH Fans; 0.0%
Humidifier; 0.0%
CHW Plant; 2.1%
Transformer Loss;
0.5%
UPS Loss; 0.6%
Racks; 94.0% PDU Loss; 1.0%
Data Center
Lights; 0.2%
Power Usage Effectiveness (PUE) = Total Energy / Rack Energy = 1.06

26. Results
Supply Temperatures Annual Energy Use Mechanical Cooling Needed Water Usage
Hours of Free
Cooling / year PUE Additional Load at
Air Water Cost Evaporation +
Cooled Cooled
Energy
( $0.058/kWh) Hours Extreme Weather Carry Over
per Year (wetbulb = 68.7°F)
°C °F °C °F hrs/yr % of yr MWh/yr $ tons gallons
23.9 75.0 23.9 75.0 8,532 97% 1.06 74,567 $4,325,000 228 0 30,100,000