1. BrainServe Datacenter: the high-density choice
Swiss Big Data User Group Meeting – 24.06.2013

2. Agenda
 BrainServe in very brief
 The high-density choice: back in 2008
 Cooling technologies: low-density and high-density
 Water-based cooling
 BrainServe example
 High-density and energy efficiency
 Q&A
Swiss Big Data User Group Meeting – 24.06.2013

3. BrainServe in very brief
 Swiss based independent datacenter in the northwestern part of Lausanne
 Physical housing space for IT equipment
 Collaboration with IT providers and telecom operators
 Dedicated plot owned by BrainServe
 Location with a low risk profile
 Datacenter design and building « from scratch » (2008 – 2010)
 Highly secure and resilient environment (2’000 m2 of housing areas)
 1’000 m2 of «classical» air cooling
 1’000 m2 of high-density liquid cooling
Swiss Big Data User Group Meeting – 24.06.2013

4. The high-density choice: back in year 2008
 Emergence of data-centric universe
 Already more data, and still increasing
 Storage, processing needs and access requirements are increasing accordingly
 Hardware evolution: power densification
 Historically (1970 – 2005): smaller transistors used less electricity to operate
 exponentially better performance & density for a constant power envelope (“free energy”)
 Since 2005: continue to make transistors smaller, but they use similar electricity to operate
 chip energy consumption is shooting up (end of “free energy”)
 Electrical input = heat output
 increase in computing power = increase in energy consumption = increase in heat dissipation
 It is often not possible nor economical to spread the load across the space
 increase in computing power = increase in heat density  requires high-density cooling
Swiss Big Data User Group Meeting – 24.06.2013

5. Cooling technologies: low-density
Swiss Big Data User Group Meeting – 24.06.2013
 Traditionally: air is the primary heat transport media
 Heat exchanger are away from the IT racks
 Air is blown into the plenum under the raised
floor and comes out in front of the racks
(preferably on a cold aisle containment)
 In standard configuration (traditional
room height): 4 kW/rack on average
 Technical and operational constraints
may lead to very disparate densities and
hot spots that are difficult to handle

6. Cooling technologies: high-density
 High-density applications require typically
10 kW to 20 kW per rack
 Different manufacturers and designs for high-
density cooling: in-row cooling, in-rack cooling,
overhead cooling, rear door cooling
 Identical central feature: bring the heat
exchanger closer to the server rack
 Reduced volume of air to be moved
 Shorter and more predictable air flows
 Handling of hot spots easier
 Better use of the cooling capacities
 Most of the products require connections
to a chilled water loop
Swiss Big Data User Group Meeting – 24.06.2013

7. Water-based cooling
 Why do we use water?
 Air is abundant and easy to handle, but not a very good heat conductor
 To transfer more heat we need to dramatically increase the volume of air
transported
 Water is a much better heat conductor
Swiss Big Data User Group Meeting – 24.06.2013
Source:AndréOppermann,SWINOG-26

8. BrainServe example (1)
 1’000 m2 (50% of IT space) equipped for high-density
 N+N redundancy (production, distribution and cooling units)
 Necessary to cope with rise of temperature in case of failure
 In-row cooling with hot aisle containment
 Designed for 10 kW/rack on average,
easily scalable to 50 kW/rack
Swiss Big Data User Group Meeting – 24.06.2013

9. BrainServe example (2)
 Great modularity
 Positioning of the rack cooling matches the specific needs
 Proportions of IT racks and cooling units are tightly coupled
 Automatic chilled water network isolation
in case of major failure
 Water leak detection system
 10 MVA electrical feed
 Power distribution with 400 A and 630 A busbars
 Fire detection and automatic extinction
 BrainServe is an industrial partner for the EXTREME
(Energy- and thermal-aware design of many-core
heterogeneous datacenters) research project with
the EPFL that look for future energy efficient design
for IT and datacenters
Swiss Big Data User Group Meeting – 24.06.2013

10. High-density and energy efficiency
 Energy efficiency is a major concern for datacenters
 Among others, high-density is a factor of efficiency improvement
 A datacenter is most efficient when it is fully loaded, and the IT load has the
biggest impact on the global efficiency (bigger than e.g. the outside temperature)
 Densification of the IT load improves the energy efficiency: the more energy we use, the more
energy we save
 By bringing the heat exchanger closer to the heat source, we keep the air volume
to be moved as low as possible
 Power consumption is proportional to the cube of the fan speed
 By increasing the ΔT on the heat exchanger (e.g. with hot aisle containment), we
lower the air volume to be moved for a given load
 High-density does not mean lower temperature in the IT room
 ASHRAE recommended environmental envelope is independent from the power density
 High-density means more re-usable heat
Swiss Big Data User Group Meeting – 24.06.2013

11. Gabriel Boissonnard
T +41 21 637 69 04
M g.boissonnard@brainserve.ch
W www.brainserve.ch
THANK YOU FOR YOUR ATTENTION!
Swiss Big Data User Group Meeting – 24.06.2013