This document discusses energy efficiency in data centers. It begins by outlining the large and growing energy consumption of data centers, noting they account for 1.3% of worldwide energy production and will exceed 400 GWh/year by 2015. It then discusses how data centers are used for traditional applications like web services as well as emerging applications in areas like smart cities that generate huge amounts of data. The document outlines various strategies for optimizing energy efficiency at different levels, from workload scheduling at the chip/server level to thermal-aware resource management and floorplanning. It stresses the need for holistic optimization across all levels from chips to data centers to minimize total energy consumption.

1. Energy Efficiency in Data Centers
Marina Zapater
Marina Zapater | Going Green1
GreenLSI – Integrated Systems Lab
Electronic Engineering Dept
Green

2. Green
Marina Zapater | Going Green 2
Data Centers

3. Green
Marina Zapater | Going Green 3
Outline
• Why Data Centers (DC) in
this Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Conclusions

4. Green
Marina Zapater | Going Green 4
Outline
• Why Data Centers (DC) in
this Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Conclusions

5. Green
Marina Zapater | Going Green 5
US EPA 2007 Report to Congress on Server and Data Center Energy Efficiency
Why DC in this Workshop?
Motivation

6. Green
Marina Zapater | Going Green 6
Motivation
• Energy consumption of data centers
– 1.3% of worldwide energy production in 2010
– USA: 80 mill MWh/year in 2011 = 1,5 x NYC
– 1 data center = 25 000 houses
• More than 43 Million Tons of CO2 emissions per
year (2% worldwide)
• More water consumption than many industries
(paper, automotive, petrol, wood, or plastic)
Jonathan Koomey. 2011. Growth in Data center electricity use 2005 to 2010

7. Green
Marina Zapater | Going Green 7
Motivation
José M.Moya | Madrid (Spain), July 27, 2012 7
• It is expected for total data
center electricity use to
exceed 400 GWh/year by
2015.
• The required energy for
cooling will continue to be at
least as important as the
energy required for the
computation.
• Energy optimization of future
data centers will require a
global and multi-disciplinary
approach.
0
5000
10000
15000
20000
25000
30000
35000
2000 2005 2010
Worldserverinstalledbase
(thousands)
High-end servers
Mid-range servers
Volume servers
0
50
100
150
200
250
300
2000 2005 2010
Electricityuse
(billionkWh/year)
Infrastructure
Communications
Storage
High-end servers
Mid-range servers
Volume servers
5,75 Million new servers per year
10% unused servers (CO2 emissions
similar to 6,5 million cars)

8. Green
Marina Zapater | Going Green 8
What about urban DC?
• 50% of urban DC have already or will soon reach the
maximum capacity of the power grid

9. Green
Marina Zapater | Going Green 9
Tier 4 Data Center

10. Green
Marina Zapater | Going Green 10
Outline
• Why Data Centers (DC) in
this Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Our vision and future trends

11. Green
Marina Zapater | Going Green 11
The DC in next generation
applications
• Traditional uses of Data Centers:
– Webmail, Web search, Databases, Social networking or distributed
storage, High-performance computing (HPC), Cloud computing
• Next-generation applications:
– Population monitoring applications: e-Health, Ambient Assisted Living
– Smart cities
• Next-generation applications generate huge amounts of data
• Need to store, analize and generate knowledge

12. Green
Marina Zapater | Going Green 12
Global energy optimization
• Solution: GoingGreen!
• How: Global energy optimization strategies
– Proposal of a holistic energy optimization framework
– Minimizing overall power consumption
– Multi-level optimization: WBSN, Personal Servers and Data Centers

13. Green
Marina Zapater | Going Green 13
Global energy optimization
• Executing part of the workload in the Personal Servers
– Classifying tasks depending on their demand
– Resource management techniques based on fast runtime allocation
algorithms executed on the Personal Servers
– Executing some tasks in Personal Servers instead of forwarding load to DC.
– Up to 10% in energy savings and 15% execution time savings

14. Green
Marina Zapater | Going Green 14
Outline
• Why Data Centers (DC) in this
Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Conclusions

15. Green
Marina Zapater | Going Green 15
Energy Consumption at the DC
What is really a Data Center?
http://cesvima.upm.es
WORKLOAD Scheduler Resource
Manager
Execution

16. Green
Marina Zapater | Going Green 16
Energy Consumption at the DC
How does cooling work?
• Typical raised-floor air-cooled Data Center:

17. Green
Marina Zapater | Going Green 17
Energy Consumption at the DC
Power consumption breakdown
• The major contributors to electricity costs are:
– Cooling (around 50%)
– Servers (around 30-40%)
• The most common metric to measure efficiency in
Data Centers is PUE (Power Usage Effectiveness)

18. Green
Marina Zapater | Going Green 18
Power Usage Effectiveness
(PUE)
• Average PUE ≈ 2
• State of the Art: PUE ≈ 1,2
– The important part is IT energy consumption
– Current work in energy efficient data centers is focused in
decreasing PUE
– Decreasing PIT does not decrease PUE, but it has in impact
on the electricity bill

19. Green
Marina Zapater | Going Green 19
“Traditional” approaches
What would Google do?
PUE = 1.2

20. Green
Marina Zapater | Going Green 20
Research trends
Abstraction level
• Higher levels of
abstraction bring
more benefits
• Some areas have
brought more
benefits than
others
Solutions proposed by the State of the Art

21. Green
Marina Zapater | Going Green 21
Outline
• Why Data Centers (DC) in this
Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Conclusions

22. Green
Marina Zapater | Going Green 22
Our approach
• Global strategy to allow the
use of multiple information
sources to coordinate
decisions in order to reduce
the total energy consumption
• Use of knowledge about the
energy demand
characteristics of the
applications, and
characteristics of computing
and cooling resources to
implement proactive
optimization techniques

23. Green
Marina Zapater | Going Green 23
Energy Optimization:
Holistic Approach
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

24. Green
Marina Zapater | Going Green 24
Resource Management at
the Room level
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

25. Green
Marina Zapater | Going Green 25
Resource Management at the Room level
Leveraging heterogeneity – IT perspective
• Use heterogeneity to minimize energy consumption from a
static/dynamic point of view
– Static: Finding the best data center set-up, given a number of
heterogeneous machines
– Dynamic: Optimization of task allocation in the Resource Manager
• We show that the best solution implies an heterogeneous data
center
– Most data centers are heterogeneous (several generations of
computers)
– 5 to 22% energy savings for static solution
– 24% to 47% energy savings for dynamic solution
M. Zapater, J.M. Moya, J.L. Ayala. Leveraging Heterogeneity for
Energy Minimization in Data Centers, CCGrid 2012

26. Green
Marina Zapater | Going Green 26
Resource Management at the Room level
Leveraging heterogeneity – IT perspective
• Energy profiling of tasks of the SPEC CPU 2006 benchmark
• Usage of MILP algorithms to schedule tasks in servers where
they consume less energy
• Implemented in a real resource manager (SLURM)

27. Green
Marina Zapater | Going Green 27
Resource Management at the Room level
IT + Cooling perspective
• Generating a thermal model for
the data room:
– Data Center environmental
monitoring to gather temperature,
humidity, differential pressure
– Predict server temperature and
room temperature
• Optimum resource
management attending to
cooling and IT power
– Real environment with
heterogeneous servers
– SLURM resource manager

28. Green
Marina Zapater | Going Green 28
Resource Management at
the Server level
Chip Server Rack Room Multi-
room
Sched & alloc 2 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

29. Green
Marina Zapater | Going Green 29
Resource Management at the Server level
Leakage-temperature tradeoffs - Cooling
• Exploring the leakage-temperature tradeoffs at the server level
– At higher temperatures, CPU increases power consumption due to
leakage
– To decrease CPU temperature, fan speed raises, increasing server
cooling consumption.
M. Zapater, J.L. Ayala., J.M. Moya, K. Vaidyanathan, K. Gross, and A. K. Coskun, “Leakage and
temperature aware server control for improving energy efficiency in data centers”, DATE 2013

30. Green
Marina Zapater | Going Green 30
Resource Management at the Server level
Leakage-temperature tradeoffs - Cooling
• Implemented fan speed controllers that reduce server power
consumption by 10%.

31. Green
Marina Zapater | Going Green 31
Resource Management at
the Chip level
Chip Server Rack Room Multi-
room
Sched & alloc 2 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

32. Green
Marina Zapater | Going Green 32
Scheduling and resource allocation policies
in MPSoCs
A. Coskun , T. Rosing , K. Whisnant and K. Gross "Static and dynamic temperature-
aware scheduling for multiprocessor SoCs", IEEE Trans. Very Large Scale Integr. Syst.,
vol. 16, no. 9, pp.1127 -1140 2008
Fig. 3. Distribution of thermal hot spots, with DPM (ILP).
A. Static Scheduling Techniques
We next provide an extensive comparison of the ILP based
techniques. We refer to our static approach as Min-Th&Sp.
As discussed in Section III, we implemented the ILP for min-
Fig. 4. Distribution of spatial gradients, with DPM (ILP).
hot spots. While Min-Th reduces the high spatial differentials
above 15 C, we observe a substantial increase in the spatial
gradients above 10 C. In contrast, our method achieves lower
and more balanced temperature distribution in the die.
Fig. 3. Distribution of thermal hot spots, with DPM (ILP).
A. Static Scheduling Techniques
We next provide an extensive comparison of the ILP based
techniques. We refer to our static approach as Min-Th&Sp.
As discussed in Section III, we implemented the ILP for min-
Fig. 4. Distribution of spatial gradients, with DPM (ILP).
hot spots. While Min-Th reduces the high spatial differentials
above 15 C, we observe a substantial increase in the spatial
gradients above 10 C. In contrast, our method achieves lower
and more balanced temperature distribution in the die.
UCSD – System Energy Efficiency Lab

33. Green
Marina Zapater | Going Green 33
Scheduling and resource allocation
policies in MPSoCs
• Energy characterization of applications allows to
define proactive scheduling and resource allocation
policies, minimizing hotspots
• Hotspot reduction allows to raise cooling
temperature
+1oC means around 7% cooling energy savings

34. Green
Marina Zapater | Going Green 34
Energy Optimization:
Holistic Approach
Chip Server Rack Room Multi-
room
Sched & alloc 2 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

35. Green
Marina Zapater | Going Green 35
JIT Compilation in Virtual
Machines
• Virtual machines compile
(JIT compilation) the
applications into native code
for performance reasons
• The optimizer is general-
purpose and focused in
performance optimization

36. Green
Marina Zapater | Going Green 36
Back-end
JIT compilation for
energy minimization
• Application-aware compiler
– Energy characterization of applications and transformations
– Application-dependent optimizer
– Global view of the data center workload
• Energy optimizer
– Currently, compilers for high-end processors oriented to performance
optimization
Front-end
Optimizer Code generator

37. Green
Marina Zapater | Going Green 37
Energy saving potential for the
compiler (MPSoCs)
T. Simunic, G. de Micheli, L. Benini, and M. Hans. “Source code optimization and
profiling of energy consumption in embedded systems,” International Symposium on
System Synthesis, pages 193 – 199, Sept. 2000
– 77% energy reduction in MP3 decoder
Fei, Y., Ravi, S., Raghunathan, A., and Jha, N. K. 2004. Energy-optimizing source code
transformations for OS-driven embedded software. In Proceedings of the International
Conference VLSI Design. 261–266.
– Up to 37,9% (mean 23,8%) energy savings in
multiprocess applications running on Linux

38. Green
Marina Zapater | Going Green 38
Global Management of
Low Power Modes
Chip Server Rack Room Multi-
room
Sched & alloc 2 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

39. Green
Marina Zapater | Going Green 39
Global Management of
Low-power modes (DVFS)
• DVFS (Dynamic Voltage and Frequency Scaling) is based upon:
– As suppy voltage decreases, power decreases quadratically
– But delay increases (performance decreases) only linearly
– The maximum frequency also decreases linearly
• Currently, low-power modes, if used, are activated by
inactivity of the server operating system
• To minimize energy consumption, changes between modes
should be minimized
• On the other hand, workload knowledge allows to globally
schedule low-power modes without any impact in
performance

40. Green
Marina Zapater | Going Green 40
Global Management of
Low-power modes (DVFS)
• By using a thermal model,
we can predict the
behaviour of a workload
under each power mode
• We can use resource
management algorithms
to change DVFS on
runtime, adapting to our
workload.

41. Green
Marina Zapater | Going Green 41
Temperature-aware
floorplanning of MPSoCs
Chip Server Rack Room Multi-
room
Sched & alloc 2 2 1
Application
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5

42. Green
Marina Zapater | Going Green 42
Temperature-aware
floorplanning of MPSoCs

43. Green
Marina Zapater | Going Green 43
Potential energy savings
with floorplanning
– Up to 21oC reduction of maximum temperature
– Mean: -12oC in maximum temperature
– Better results in the most critical examples
Y. Han, I. Koren, and C. A. Moritz. Temperature Aware Floorplanning. In Proc. of the
Second Workshop on Temperature-Aware Computer Systems, June 2005

44. Green
Marina Zapater | Going Green 44
Temperature-aware
floorplanning in 3D chips
• 3D chips are getting interest due to:
– Scalability: reduces 2D
equivalent area
– Performance: shorter wire
length
– Reliability: less wiring
• Drawback:
– Huge increment of hotspots
compared with 2D equivalent designs

45. Green
Marina Zapater | Going Green 45
Temperature-aware
floorplanning in 3D chips
• Up to 30oC reduction per layer in a 3D chip with 4 layers and
48 cores

46. Green
Marina Zapater | Going Green 46
Outline
• Why Data Centers (DC) in this
Workshop?
• The DC in next-generation
applications
• Energy consumption at the
Data Center
• Insight on optimization
strategies
• Conclusions

47. Green
Marina Zapater | Going Green 47
There is still much more
to be done
• Smart Grids
– Consume energy when everybody else does not
– Decrease energy consumption when everybody
else is consuming
• Reducing the electricity bill
– Variable electricity rates
– Reactive power coefficient
– Peak energy demand

48. Green
Marina Zapater | Going Green 48
Conclusions
• Reducing PUE is not the same than reducing energy
consumption
– IT energy consumption dominates in state-of-the-art data
centers
• Application and resources knowledge can be effectively
used to define proactive policies to reduce the total energy
consumption
– At different levels
– In different scopes
– Taking into account cooling and computation at the same time
• Proper management of the knowledge of the data center
thermal behavior can reduce reliability issues
• Reducing energy consumption is not the same than
reducing the electricity bill

49. Green
Marina Zapater | Going Green 49
Thank you for your attention
Marina Zapater
marina@die.upm.es
http://greenlsi.die.upm.es
(+34) 91 549 57 00 x-4227
ETSI de Telecomunicación, B105
Avenida Complutense, 30
Madrid 28040, Spain
Thanks to our collaborators: