Full paper: http://ieeexplore.ieee.org/abstract/document/7996505/ As more organizations rapidly adopt cloud services, energy consumption in data centers (DCs) is increasing such that today Information and Communication Technology (ICT) has become a major consumer of energy. A large portion of ICT energy consumption is used to power servers running in DCs and the network they use to communicate. In this study, we consider that, often, energy cost at a particular DC is related to the electricity price regulated by Independent System Operators / Regional Transmission Organizations (ISOs/RTOs). As these prices vary in time and depend on the geographical locations of the DCs, recent studies have shown that the spatio-temporal variations of electricity price can be exploited to reduce electricity cost. While most prior works consider a quasi-static scenario with known workload patterns, our study proposes a dynamic workload-aware algorithm that exploits the spatio-temporal variations of electricity costs with the goal to minimize the energy cost in ICT. Our algorithm uses dynamic request rerouting and live virtual machine (VM) migration to move workloads to DCs with lower electricity cost. We consider VM migration cost (including electricity cost at optical backbone network nodes), bandwidth constraints for migration, VM consolidation, constraints from Service Level Agreement (SLA), and administrative overhead of VM migration. Our simulation studies show that the proposed algorithm reduces operational cost and improves energy efficiency of data centers significantly.

1. Dynamic Workload Migration
Over Optical Backbone Network
To Minimize Data Center Electricity Cost
Sabidur Rahman*, Abhishek Gupta*, Massimo Tornatore*†, and Biswanath Mukherjee*
*University of California, Davis, USA †Politecnico di Milano, Italy
ONS-2: Optical Data Center Networking
5/26/20171

2. Agenda
• Introduction to problem
• Motivation
• Electricity market
• Formal statement
• Power consumption model
• Proposed algorithm
• Dynamic Workload-Aware VM Placement and Migration
• Results
• Summary and future work
5/26/20172
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

3. 5/26/20173
Geographically distributed data centers
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
Source: http://royal.pingdom.com/2008/04/11/map-of-all-google-data-center-locations/

4. Annual electricity cost
5/26/20174
A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, and B. Maggs, “Cutting the electric bill for internet-scale systems,” SIGCOMM ’09,
vol. 39, no. 4, pp. 123–134, Oct. 2009.
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

5. Electricity market
5/26/20175
Source: http://www.isorto.org/about/default
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
• 7 major ISOs/RTOs in USA
• Electricity cost varies over:
• time
• location
(mostly due to characteristics
of power sources and supply
/demand behavior)
Independent System Operator (ISO)
Regional Transmission Organization (RTO)

6. Variable electricity cost
5/26/20176
A. Gupta, U. Mandal, P. Chowdhury, M. Tornatore and B. Mukherjee, “Cost-efficient live VM migration based on varying
electricity cost in optical cloud networks”, Photonic Network Communications, Sep 2015
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

7. Key concepts
• Exploit spatio-temporal variation of electricity prices for
geographically-distributed data centers
• Live VM migration
• Service request re-routing (considering SLA!)
• Solution for dynamic scenarios
• Most existing work on static/quasi-static scenarios
• Power model
• Backbone network power consumption (due to VM migration)
5/26/20177
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

8. Problem statement
5/26/20178
Dynamic
Optimization
Dynamic
Optimization
Where to „serve‟
the request, or
where to
„migrate‟ the
running service
Where to „serve‟
the request, or
where to
„migrate‟ the
running service
DCs‟ current
capacities
DCs‟ current
capacities
Network state and link
capacities
Network state and link
capacities
Electricity price dataElectricity price data Service SLAService SLA
Service requests
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

9. DC power model
5/26/20179
Server
Rack
Total IT equipment
Total DC
VM migration power model
Network nodes
DC + Network
VM migration
Heating, cooling,
ventilation, lighting,
and maintenance.
Heating, cooling,
ventilation, lighting,
and maintenance.
Administrative
overhead of
managing VM
migration
Administrative
overhead of
managing VM
migration
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
(u= server utilization)
(n= total # of bits, Ci,j cost of electricity)

10. Algorithm
Dynamic Workload-Aware VM Placement and Migration (DWVPM)
Step I (initial placement)
where to place new requests?
5/26/201710
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
StartStart
At arrival of a new
service, find lowest
cost DC available
At arrival of a new
service, find lowest
cost DC available
Place the incoming
service and update
DC status
Place the incoming
service and update
DC status
Step II (migration of services)
where to migrate running services?
For already running service, calculate
cost of migration to candidate DCs
For already running service, calculate
cost of migration to candidate DCs
Epoch
expired?
Epoch
expired?
Migration
saves cost?
Migration
saves cost?
Move service to lowest cost DC availableMove service to lowest cost DC available
Done with all
running
services?
Done with all
running
services?
No
Yes
No
Yes
Yes No

11. Novelties of approach
• Epoch makes the migration frequency variable
• epoch dynamically adjusts migration frequency
• Dynamic service arrival and duration
• We use practical values from DC workload traces studied in prior works
• Combination of backbone network and server power consumption
• Per-Rack VM consolidation in DCs which further reduces the electricity cost
5/26/201711

12. Simulation setup
5/26/201712
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
[15] A. K. Mishra,et al.,
“Towards Characterizing
Cloud Backend Workloads:
Insights from Google
Compute Clusters,” ACM
SIGMETRICS Performance
Evaluation Review, 2010.
[16] T. Paul, et al., “The
User Behavior in Facebook
and its Development from
2009 until 2014,” arXiv,
2015.

13. Results: normalized cost vs. load
5/26/201713
Higher load,
lower cost savings
Higher load,
lower cost savings
VM migration helps to
minimize cost
VM migration helps to
minimize cost
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

14. Results: impact of DC capacity
(for fixed transport capacity)
5/26/201714
More VMs to migrate,
lower cost savings
(bandwidth capacity limit)
More VMs to migrate,
lower cost savings
(bandwidth capacity limit)
Higher load,
lower cost savings
Higher load,
lower cost savings
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
Nr of VMs per DC

15. Results: 24-hour variation on cost savings
5/26/201715
Cost savings over the day
varies with the load and
electricity price
Cost savings over the day
varies with the load and
electricity price
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

16. Results: impact of epoch
5/26/201716
Frequently executing the algorithm has more
cost savings in lower loads than higher loads
Frequently executing the algorithm has more
cost savings in lower loads than higher loads
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

17. Summary and future work
• Spatio-temporal variation of electricity prices can help to minimize DC
electricity cost significantly
• DWVPM optimizes DC electricity cost in dynamic scenarios
• Savings in the orfer of 20-30%
• Future work:
• Use of new virtualization platforms such as „docker containers‟
• A-priori identification of the right “dynamicity”
5/26/201717
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost

18. 5/26/201718
Massimo Tornatore: Dynamic Workload Migration over Optical Backbone Network to Minimize Data Center Electricity Cost
massimo.tornatore@polimi.it