The document discusses air flow measurements and thermal responses in data centers to help sustainability. It describes experiments conducted in the CEETHERM data center using a 3D PIV system and server simulator to measure air flows under different rack configurations and thermal loads. High perforated tile flow rates were found to cause issues like adverse pressure gradients and hot pockets forming in racks. Temperature mapping showed the impact of tile flow on cold and hot aisle temperatures. The research aims to better understand air flows to help optimize data center design and efficiency.

1. Characterizing Air Flows and
Thermal Responses to Help
Sustainable Data Centers
Yogendra Joshi
G.W. Woodruff School of Mechanical Engineering
Georgia Institute of Technology
Atlanta, GA 30332, U.S.A.
Acknowledgements:
Intel, IBM, Department of Energy, National Science Foundation (NSF), Triad Tiles, APC
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 1

2. Measuring Air Flows in Data Centers
Designed and built by McKenney’s
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

3. Details of the CEETHERM data center and the
location of the server simulator
42 U
0.09m.
3 1/2" Downflow Upflow Upflow Downflow
CRAC 1 CRAC 2 CRAC 3 CRAC 4
13
14
0.44m. Compute rack Compute rack
10 U 1'-5 1/2"
12
11
Empty rack Empty rack Compute rack Compute rack
Power
distribution
10
Test rack Test rack Compute rack Compute rack
9
unit
Server Compute rack Compute rack
Test rack
7
0.44m.
Simulatror
8.75m
8
1'-5 1/2"
10 U
Test rack Test rack Compute rack Compute rack
5
6
1.98m. Power
Empty rack Empty rack Compute rack Compute rack
3
4
distribution Power
(6'-6") unit distribution
unit
Compute rack Compute rack
1
2
Note:
0.44m. Perforated tile dampers
10 U Storage Network
1'-5 1/2" on tiles 1-4, 7,11-14 are PIV measurement plane
closed during the
D experiments D
O O
O O
R R
Downflow CRAC 6 Downflow CRAC 5
0.44m.
10 U 1'-5 1/2" 6.4m 6.4m
Experimental section Computational section
Only Tiles 5,6,8,9,10 are open during the experiment. All
0.53m. other tiles are closed.
1'-9"
CRAC 1 and CRAC 6 are simultaneously operated
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

4. Experimental Facilities
Fan speed
and Heat
setting
dials
3-D PIV system
• 3-D Stereoscopic PIV (Particle Image Velocimetry
system for room level air flow mapping.
Server Simulator
• 22.8 kW Server Simulator with adjustable fan and Perforated floor
heater settings to simulate a variety of heat loads. tiles with dampers
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 4

5. How much air flow rate is required to cool a 20kW rack?
Typical Raised floor air
distribution system Server Air requirements
Server cold ∆T=20°C Server hot A 20kW Server requires ~ 0.996 m3/s
air exit
air inlet (2,110 CFM for a ∆T=20°C)
Tin=27°C Tout =47°C
A 2x2 (4 ft2, 0.3716 m2) opening with no
tile V ~ 2.68 m/s (~6 mph)
Only tile (no Damper) 56%opening; Area
= 0.208 m2 V ~ 4.788 m/s (~10.7
Servers mph)
Tile with dampers (26% open); Area =
0.0977 m2 V ~ 10.2 m/s (~22.8 mph)
Perforated floor Server pressure drop
tile
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

6. Rack air flow boundaries for a 22kW rack (0.754 m3/s of tile air flow)
5 Velocity (m/s)
4
0 1.25 2.5 3.75 5 62% of the air requirement;
Velocity (m/s)
3
about double at air inlet temp.
Aisle Top
2
1
0
of 30 oC
0 100 200 300 400 500 600 700 0 150 300 450 600
-1 Distance measured from the centre of cold aisle (mm)
Distance measured from the center of the cold
aisle (mm) Top aisle boundary
U u-component v-component 1950 2000
1950
2000
1800 2000
1800
1950 1800
1800
Aisle top
Height of the server simulator measured from the perforated floor tile (mm)
1800
Height of the server simulator measured from the perforated floor tile (mm)
Height of the the server simulator measured from the perforated tile (mm) (mm)
1650 1650 1800
1800
1650
Velocity (m/s) 1600
65% of perf. tile air flow is unutilized
Height of server simulator measured from the perforated floor floor tile
1600 1500 Velocity (m/s) 1500 6 1650
Velocity (m/s)
0.8 1500 5.5 1600
5.5 Velocity (m/s)
1400
1350 0.792 1350
5
1500
1350 and escapes from the top.
5.496
5.5
5.038
1400
1200
0.726
4.5 1350 Tile surface 5.496
Ascending column of air is pushed
1400
Rack Height (mm)
0.66 1200 1200
4.58
5.038 1200
1200
Rack Height (mm)
4
Rack Height (mm)
0.594 4.122
4.58
1050 1200 1200
1000
900
0.528
0.462
1050 3.5
3
1050
1050
towards the uniform flow.
Highly non rack inlet.
3.664
4.122
3.206
3.664
1000
900 900 1000
800 750
0.396
0.33
2.5
Compression due to entrainment
900
3.206
2.748
2.748 800
Reversed flow at the ends result in
2 750 2.29
750
800
600
600
0.264
0.198
1.5 causes a steady increase in v-comp
750
600
2.29
1.832
1.832 600
velocity sucked into plenum
air being
600 1 1.374
600 1.374 600
450 0.132 0.916
0.5 450
400 0.066 450 0.916
0.458
300 0 450 400
0.458 400
200
150
0
300 Peaks and valleys in velocity field
300
300
0
0
150
200
0 0
150
relate to damper geometry
150
200
Aisle Center 0
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 0 0
0 Rack Inlet 0
Velocity (m/s)
Aisle center 150 300 450 600 Rack Inlet -2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5
-2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5
U
6
u-component v-component Distance measured from the centre of cold aisle (mm)
Velocity (m/s) Rack Velocity (m/s)
Velocity (m/s)
Velocity (m/s) U
U u-component
u-component v-component
v-component
5 inlet
Aisle center
4 0 .5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Rack inlet 0 .5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Velocity (m/s)
3
Tile Surface
u-component suggests entrainment
Tile Surface
2
Deviation from normal to slanting
1
Entrainment predominant at 150 tile surface and subsides at the aisle top
0
the 300 450 600
Air flow at the bottom half of the rackcentre ofaffected by the perf. tile flow
-1 0
-2 150
100
300
is cold aisle (mm)
200
450
300
600
Distance measured from the center of the cold
Distance measured from the
400 500 600 700
Distance measured from the centre of cold aisle (mm)
25% reduction in flowdue to reversed flow at the tile surface
-ve v component is compared to surface aisle (mm)
U Tile case where of no air supply from tile
u-component v-component
Tile surface
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 6

7. Comparison of cold aisle air distribution for high and low density racks
1950
Height of the server simulator measured from the perforated floor tile (mm)
1950
1800
1800
Height of the server simulator measured from the perforated floor tile (mm)
Velocity (m/s)
1650
1650 3
Velocity (m/s)
2.8 1500 6
1500
2.6 5.5
1350
1350 2.4 5
2.2 1200 4.5
1200 2 4
1050 3.5
1050 1.8
1.6 3
900
900 1.4 2.5
750 2
750 1.2
1.5
1 600 1
600 0.8
0.5
0.6 450
450 0
0.4
300
300 0.2
150
150
0
0 150 300 450 600
0 150 300 450 600 Distance measured from the centre of cold aisle (mm)
Distance measured from the centre of cold aisle (mm)
650 CFM (6.5 kW rack) 2500 CFM (22 kW rack)
 Study suggests that there is a limit to the amount of useful air discharged from
the perf tile.
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 7

8. Grid based Temperature Measurement
Consist of 256 thermo couple
sensors
Type T Copper Constantan
Thermocouples
Measurement accuracy of
±0.2°C
Response time of 4ms/channel
Complete temperature mapping
of the cold/hot aisle.
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 8

9. 6
5
Complete 3-D temperature mapping
4
3
2
1
3 2 1
6 planes along the
height
10 planes along the
width of the cold aisle 5 planes along the
depth of the rack
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 9

10. Effect of high perf. tile flow (0.667 m3/s, 1414 CFM)) on cold and hot aisle
temperature profiles (TCRAC Supply= 12.3°C)
Velocity profile at Aisle top
Hot aisle
Cold aisle temperature
temperature map
map
• High velocity discharge • Hot pocket in the
cause adverse pressure lower half of the
gradients at rack inlet rack due to reduced
• Pocket of hot air due to air intake and
reversed flow in the cold recirculating flows.
aisle. Velocity profile at perf. tile
surface
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 10

11. Effect of low perf. tile flow (0.227 m3/s, 586 CFM)) on cold and hot aisle
temperature profiles (TCRAC Supply= 12°C)
Velocity profile at Aisle top
Hot aisle
Cold aisle temperature
temperature map
map
• Higher temperature
• Absence of reversed flows gradients in the hot aisle
• Hot air pocket has vanished • Temperatures increase in
• Hot air influx into the aisle at the upper half of the rack
the top
Velocity profile at perf. tile
surface
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 11

12. Real time server measurements
12
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012 12

13. Rack Rear view with fan and the power supply details
Rack
Blade Center
BC-6 • Each rack accommodates
6 Blade Center
• Each Blade Center has 14
BC-5 Blades
• Each Blade has 2
processors (total 168
BC-4 DC
processors per rack)
Processor 1
and 2 • Processor temperature is
BC-3 measured for each
processor
• Air inlet temperature is
BC-2 measured for each blade
center
0 2S L
BC-1 Blade
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

14. Effect of Perforated Tile Flow on CPU Temperature and Fan Speed
100 % CRAC VFD setting (~12,000 CFM)
ComparisonRackD6.wmv
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

15. Effect of Perforated Tile Flow on CPU Temperature and Fan Speed
60 % CRAC VFD setting (~7,000 CFM)
CRAC_60_Celcius.wmv
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012

16. Multi-University NSF I/UCRC E3S Center: Compact Models of Contained
Aisles
K=13.2
Velocity Static
(m/s) Pressure
(Pa)
16
TAG/GT Sustainable HPC Infrastructure Workshop, Atlanta GA, Feb. 24, 2012