XeMPUPiL@IBM - June 2, 2017

1. XeMPUPiL
A performance-aware power capping
orchestrator for the Xen hypervisor
Marco Arnaboldi, author
marco1.arnaboldi@mail.polimi.it
xx June 2017

2. 2
Bird’s eye view
A performance-aware power capping
orchestrator for the Xen hypervisor

3. 3
Bird’s eye view
A performance-aware power capping
orchestrator for the Xen hypervisor
Instrumentation-free
workload monitoring

4. 4
Bird’s eye view
A performance-aware power capping
orchestrator for the Xen hypervisor
Instrumentation-free
workload monitoring
Power management techniques
HW vs. SW

5. 5
Bird’s eye view
A performance-aware power capping
orchestrator for the Xen hypervisor
Instrumentation-free
workload monitoring
Open Source virtualization layer adopted
by many fortune companies
Power management techniques
HW vs. SW

6. 6
Outline
I. Introduction
II. Problem definition
III. State of the Art
IV. Proposed Solution
V. Experimental results
VI.Conclusion

7. 7
Introduction
Power consumption trends in Data Centers[1]
[1] US Department of Energy, Lawrence Berkeley National Laboratory

8. 8
Introduction
Power consumption trends in Data Centers[1]
[1] US Department of Energy, Lawrence Berkeley National Laboratory
US energy price
January 2017
10.15 Cents/Kilowatt-hour
=~ 6 Billion USD/year

9. 9
State of the Art
SOFTWARE APPROACH
✓ efficiency
✖ timeliness
MODEL BASED
MONITORING [3]
THREAD
MIGRATION [2]
RESOURCE
MANAGMENT DVFS [4]
RAPL [1]
CPU
QUOTA
HARDWARE APPROACH
✖ efficiency
✓ timeliness
[1] H. David, E. Gorbatov, U. R. Hanebutte, R. Khanna, and C. Le. Rapl: Memory power estimation and capping. In International Symposium on Low Power Electronics and Design (ISPLED), 2010.
[2] R. Cochran, C. Hankendi, A. K. Coskun, and S. Reda. Pack & cap: adaptive dvfs and thread packing under power caps. In International Symposium on Microarchitecture (MICRO), 2011.
[3]M. Ferroni, A. Cazzola, D. Matteo, A. A. Nacci, D. Sciuto, and M. D. Santambrogio. Mpower: gain back your android battery life! In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication, pages 171–
174. ACM, 2013.
[4] T. Horvath, T. Abdelzaher, K. Skadron, and X. Liu. Dynamic voltage scaling in multitier web servers with end-to-end delay control. In Computers, IEEE Transactions. IEEE, 2007.

10. 10
State of the Art
RESOURCE
MANAGMENT
CPU
QUOTA
HYBRID APPROACH [5]
✓ efficiency
✓ timeliness
SOFTWARE APPROACH
✓ efficiency
✖ timeliness
HARDWARE APPROACH
✖ efficiency
✓ timeliness
[1] H. David, E. Gorbatov, U. R. Hanebutte, R. Khanna, and C. Le. Rapl: Memory power estimation and capping. In International Symposium on Low Power Electronics and Design (ISPLED), 2010.
[2] R. Cochran, C. Hankendi, A. K. Coskun, and S. Reda. Pack & cap: adaptive dvfs and thread packing under power caps. In International Symposium on Microarchitecture (MICRO), 2011.
[3]M. Ferroni, A. Cazzola, D. Matteo, A. A. Nacci, D. Sciuto, and M. D. Santambrogio. Mpower: gain back your android battery life! In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication, pages 171–
174. ACM, 2013.
[4] T. Horvath, T. Abdelzaher, K. Skadron, and X. Liu. Dynamic voltage scaling in multitier web servers with end-to-end delay control. In Computers, IEEE Transactions. IEEE, 2007.
[5] H. Zhang and H. Hoffmann. Maximizing performance under a power cap: A comparison of hardware, software, and hybrid techniques. In International Conference on Architectural Support for Programming Languages and Operating Systems
(ASPLOS), 2016.
MODEL BASED
MONITORING [3]
THREAD
MIGRATION [2]
DVFS [4]
RAPL [1]

11. 11
Proposed Solution
Xen
Hypervisor
Hardware
DomainU
Workload
DomainU
Workload
Dom0
Decide
Observe
XeMPower
Hardware
Events
Counters
PUPiL
CLI
buffers
Hypercall manager
RAPL
interface
XL
Act

12. u Server setup (aka Sandy)
u 2.8-GHz quad-core Intel Xeon E5-1410 processor, no HT enabled (4 physical
core)
u 32GB of RAM
u Xen hypervisor version 4.4
u paravirtualized instance of Ubuntu 14.04 as Dom0, pinned on the first pCPU and
with 4GB of RAM
12
Experimental Setup
u Benchmarking
u Embarrassingly Parallel (EP) [1]
u IOzone [3]
u cachebench [2]
u Bi-Triagonal solver (BT) [1]
EP IOzone cachebench BT
CPU-bound YES NO NO YES
IO-bound NO YES NO YES
memory-bound NO NO YES YES
[1] Nas parallel benchmarks. http://www.nas.nasa.gov/publications/npb. html#url. Accessed: 2017-04-01.
[2] Openbenchmarking.org. https://openbenchmarking.org/test/pts/ cachebench. Accessed: 2017-04-01.
[3] Iozone filesystem benchmark. http://www.iozone.org. Accessed: 2017- 04-01.

13. 13
Experimental Results
0
0.2
0.4
0.6
0.8
1.0
NO RAPL
RAPL 40
RAPL 30
RAPL 20NormalizedPerformance
0
0.2
0.4
0.6
0.8
1.0
EP cachebench IOzone BT
Baseline Definition via RAPL

14. 14
Experimental Results
Baseline Definition via RAPL
0
0.2
0.4
0.6
0.8
1.0
NO RAPL
RAPL 40
RAPL 30
RAPL 20NormalizedPerformance
0
0.2
0.4
0.6
0.8
1.0
EP cachebench IOzone BT
CPU-intensive
benchmarks
suffer
processor
frequency
reduction

15. 15
Experimental Results
Baseline Definition via RAPL
0
0.2
0.4
0.6
0.8
1.0
NO RAPL
RAPL 40
RAPL 30
RAPL 20NormalizedPerformance
0
0.2
0.4
0.6
0.8
1.0
EP cachebench IOzone BT
Other
benchmarks suffer
processor voltage
reduction

16. 16
Experimental Results
0
0.5
1.0
PUPiL 40
RAPL 40
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 30
RAPL 30
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 20
RAPL 20
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
XeMPUPiL
results
compared to the
baseline

17. 17
Experimental Results
XeMPUPiL
results
compared to the
baseline
XeMPUPiL
outperforms pure
RAPL
for IO-, MEM-, and
mix-bound
benchmarks
0
0.5
1.0
PUPiL 40
RAPL 40
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 30
RAPL 30
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 20
RAPL 20
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT

18. 18
Experimental Results
XeMPUPiL
results
compared to the
baseline XeMPUPiL suffers
pure CPU-bound
benchmarks, due
to Xen developer-
transparent
optimization
0
0.5
1.0
PUPiL 40
RAPL 40
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 30
RAPL 30
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT
0
0.5
1.0
PUPiL 20
RAPL 20
Normalizedperformance
0
0.5
1.0
EP cachebench IOzone BT

19. 19
Future Works
u (Integrating || Moving) orchestrator logic into
scheduler
u Exploit new RAPL version on Haswell family
u Explore new policies regarding:
u Decision
u Resource assignment

20. 20
Thank you!!!
XeMPUPiL
“Towards a performance-aware
power capping orchestrator for the
Xen hypervisor” @ EWiLi’16,
October 6th, 2016, Pittsburgh, USA

21. 21
ODA Details
ACTDECIDEOBSERVE
u Exploration in the space of
all possible resource
configuration, based on
binary search tree
u Policy in order to distribute
the virtual resources on the
physical ones.
u Enforce power cap via
RAPL
u Define a cpu pool for the
workload
u Launch the workload on the
pool
u Change the number of the
resource on the pool
accordingly with the
decision phase
u Pin workload’s vCPU over
pCPU accordingly with the
map decided
The decision phase is similar to the one implemented in
PUPiL. The major changes are in how we evaluate the metrics
gathered in the previous phase and in how we assign the
physical resources to each virtual domain.
The evaluation criterion is based on the average IR rate,
given a certain time window: this allows the workload to adapt
to the actual configuration before a new decision is taken.
For what concerns the allocation of resources to each
domains, we chose to work at a core-level granularity: on the
one hand, each domain owns a set virtual CPUs (vCPUs),
while, on the other hand, we have a set of physical CPUs
(pCPU) present on the machine. Each vCPU is mapped on a
pCPU for a certain amount of time, while it may happen that
even multiple vCPUs can be mapped on the same pCPU.
We wanted our allocation policy to be as fair as possible,
covering the whole set of pCPUs if possible; given a workload
with M virtual resources and an assignment of N physical
resources, to each pCPUi we assign:
vCPUs(i) =
2
6
6
6
6
6
M
X
0<j<i
vCPUs(j)
N i
3
7
7
7
7
7
(1)
where i is a number between 0 and N 1, i.e., it spans over
the set of pCPUs.
C. Act
The act phase essentially consists in: 1) setting the chosen
power cap and 2) actuating the selected resource configuration.
2Source code available at: https://bitbucket.org/necst/xempower
written to set a limit on the po
CPU socket.
In a virtualized environment,
accessible by the virtual doma
tenant Dom0. However, this li
invoking custom hypercalls th
derlying hardware. To the bes
hypervisor does not natively
interact with the RAPL inter
implemented our custom hype
der to be enough generic, we
"xempower_rdmsr" and "x
one allows to reads, while the
specified MSR from Dom0.
Each hypercall needs to be
the hypervisor, that runs bare
kernel keeps track of the list o
input parameters they accept.
function has to be declared and
by the kernel at runtime: our im
Xen build-in functions to safely
i.e., wrms_safe and rdmsr_
if something goes wrong in ac
critical problems to happen at
We then implemented
Interface (CLI) tools to ac
Dom0: xempower_RaplS
xempower_RaplPowerMoni
consumption of the socket.
value of power cap and the p
are passed through the whole
u Instruction retired per
domain metric
u Data gathered from
xempowermon
u Use of HPC and Xen
scheduler in order to map
the IR to the respective
domain

22. 22
RAPL Details
MSR
INTEL RAPL INTERFACE
HYPERCALL MANAGER
BUFFER
XEMPOWER
CLI TOOL
u Two tools based on xc native tool: XEMPOWER_RAPLSETPOWER and
XEMPOWER_RAPLPOWERMONITOR
u Tools divided into two parts
u FRONTEND: manage users command and gather information ad
privileges about the session. Pass the user parameters to the backend
u BACKEND: bake the hyperbola, declaring an hypercall structure and
filling it with the user parameters. The invoke the just defined hypercall
u Used to map user space memory to kernel memory, in order to perform
“pass by reference like” mechanism inside hyperbola
u Declaration of two custom hypercall: XEMPOWER_RDMSR and
XEMPOWER_WRMSR
u Implementation of the routines that will manage the two custom hyperbolas
u Accessed by the routines, that write to and read from RAPL specific MSR
register, in order to set the power cap and to retrive metrics on the socket
power consumption
u Three registers are accessed:
u RAPL_PWR_INFO
u RAPL_PK_POWER_LIMIT
u RAPL_PK_POWER_INFO