Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur

Profiling Application Performance
and Resource Utilization
in Virtualized Environments
UmeshBellur
PuruKulkarni
http://www.cse.iitb.ac.in/synerg
Systems and Networks Research Group
Department of Computer Science and Engineering
Indian Institute of Technology Bombay

Feb 16th, 2011
2
Virtualization based Provisioning
Abstract physical resources as virtual objects
Virtual machines execute on virtual systems
Virtual machine monitor multiplexes physical resources across virtual resources
Benefits
Direct: Rapid provisioning, Interoperability, Performance Isolation, Increased resource utilization
Derived: Server Consolidation, Flexible resource provisioning, Efficient replication, backup and migration

Feb 16th, 2011
3
Virtualisation-based Provisioning Issues
Flexible Resource Provisioning
SLA-based migration
Power-aware consolidation
Cloud bursting
Physical to Virtual
Resource requirements mapping and modelling
Capacity planning
VM placement policies
Deployment & Manageability
Resource accounting in Clouds
Prototype Testbed setups
VM Optimisations
Shared memory based I/O for co-located VMs
Network I/O guarantees
VMs for resource-constrained platforms

Feb 16th, 2011
4
Provisioning & Capacity Planning in Virtual Environments
Some questions of interest
Application behaviour and resource usage under different workloads?
Throughput, response time, network, CPU, memory ...
With finite resources which SLAs can be satisfied?
How will the application behave with varying resources?
What is the impact of VM movement on application performance and resource usage levels?

Dom0
DomU
Xen Hypervisor
Hardware
Why separate VM-specific tool?
Existing tools not virtualisation aware
All I/O through privileged VM
Impact of VM migration
Workload generation and
app. performance separate
from resource monitoring
5
01/12/10

Feb 16th, 2011
6
State of the art for virtual environments
Resource monitoring tools
xenmon, xentop, sysstat, Hyperic etc.
Load generation benchmarks
SPEC, TPCW ...
Isolated from profiling framework
Load generation and control
Resource configuration control
User-level performance metrics
Require an integrated tool (for VM worlds)
Load generation + Resource configuration + Measurement and monitoring

Feb 16th, 2011
7
Problem Definition
Profile performance and resource utilization levels of applications in virtual environments
Multi-tier, multi-VM based (web-based) application
Implications on
Capacity planning
Migration and Consolidation Strategies
virtPerf*
Application profiling tool for virtualized environments
Built on top of AutoPerf** – a similar tool for physical environments
Current scope: Xen and kvm
* The tool forms the thesis of a current MTech student - PrajaktaPatil.
** S. Shirodkar, “Autoperf: An automated load generator and performance measurement tool for multi-tier software systems,”, Master’s thesis, IIT Bombay, Mumbai, India, June 2007

Feb 16th, 2011
8
virtPerf slave
virtPerf master
profiling
agent
requests
DomU
Load
generator
responses
DomU
Physical machine 1
Controller
DomU
Physical machine 2
Dom0
virtPerf Architecture
Workload generator + Measurement tool

Feb 16th, 2011
9
DomU
Dom0
Load Generator and VM Initialisation
- Multi-client load
- Warm-up detection
requests
Load
generator
responses
load
level
ready
Initialize
Controller
Load characteristics
Transaction details
- Set resource configurations
- IP to VM id mapping
Warmup => Stable response time
Thread startup, caching etc.
Agents initialize VMs according to resource profile

Feb 16th, 2011
10
DomU
Dom0
Run Termination and Measurements
requests
Load
generator
responses
DONE
STOP
profiling
Controller
Resource usage summary
Load characteristics
Transaction details
<network, disk, cpu,
memory, service demand>
...
- Collate performance metrics
- Select next load level or VM resource configuration
xentop at Host (Dom0)
sysstat and ps at Guest (DomU)

Feb 16th, 2011
MSD of a Profiling Run

Feb 16th, 2011
12
Metrics Collected
Resource usage metrics (averaged over duration of run)
CPU utilization
Network & disk I/O
Memory usage
Performance metrics
Average response time
Throughput
Service demand

Feb 16th, 2011
13
virtPerf Features
Load control
#clients, think time distributions, total number of requests issued
AutoPerf provides automatic load level section, warmup detection
VM resource configuration
#cores, CPU allocation (min, max), CPU pinning
To both Dom0 and each of the DomUs individually
Impact of Migration
Resource requirements, Performance hit (drop in throughput, increase in response time)
VM Collocation
Resource contention impact & VM Collocation benefits

Feb 16th, 2011
virtPerf Validation
Metrics reported by virtPerf for resource consumptions and service performance known apriori.
CPU Utilization:
A service that can consume a specific amount of CPU with a knob to vary this from 10-90%.
virtPerf reports utilization within 3% of this at any level.
Network I/O:
A service will read/write a known amount of I/O at a specified rate.
virtPerf reports back this number with a 400 byte overhead per request caused by HTTP and the virtPerf control overhead
Disk I/O:
Similar to Network I/O and results are within 2% of expected values

Feb 16th, 2011
15
Experimental Setup
PMs: Quad Core Intel i5 Intel(R) Core(TM) i5 CPU, 760 @ 2.80GHz 3 GB RAM
Virtual platforms: Xen
WebCalendar & RUBiS - 2 tier applications with a MySQL backend and a HTTP-based front-end

Feb 16th, 2011
16
Throughput Saturation
webcal saturates at 64 req/s
virtPerf can automatically detect saturation
CPU utilization of VM (DomU) ~400% (4 core m/c)

Feb 16th, 2011
17
Close to service demand =>high arrival rate
Throughput saturates with ~12 users
Think time >> service demand =>Low arrival rate
Throughput saturates with~72 users
Sensitivity to Think time distribution
Think time distribution: Poisson
Mean: <80ms, 150 ms, 800 ms>

Feb 16th, 2011
18
1 core vs 4 cores
Maximum Throughput 16 req/s : 65 req/s
Response time 900 ms : 3000 ms
Performance metrics ratio 1 : 4
VM CPU Utilization
~80 % : 375 %
VM Configuration (#cores)

Feb 16th, 2011
19
Throughput
c1 >> c2 >> c3
Response time
c1 << c2 << c3
VM Configuration (placement)
3 configurations
Single VM app (c1)
2 VMs on 1 PM (c2)
2 VMs on 2 PMs(c3)

Feb 16th, 2011
20
Response time approximately
half with 300% CPU
Achieved Throughput doubles
(25 req/s to 50 req/s)
WebCalendar
VM Resource Configuration (CPU availability)
CPU for DomU (guest VM): 150% and 300%

Feb 16th, 2011
VM Resource Configuration for RUBiS
Impact of increasing CPU for the Web Tier felt ONLY AFTER DB
Tier has sufficient CPU (260%). Else DB Tier is the bottleneck
Throughput goes from 90 to 156 requests/sec at the DB tier
receiving upto 340% and the web tier going from 10 to 30%

Feb 16th, 2011
VM Migration while under Load

Feb 16th, 2011
Effects of Migration

Feb 16th, 2011
Core Pinning
Peak throughput almost doubles when the DB
Tier is given it’s own core compared to when
all the cores are shared amongst the VMs – Dom0
and 2 app VMs sharing 2 cores.
VirtPerf can pin VMs on specific
cores of a machine.
Makes a difference to application
performance.

Feb 16th, 2011
25
virtPerf Summary
Workload generator + Application profiler
Feature set (profiling with ...)
Load distribution control
VM resource configuration and control
Performance and resource metrics with migration
Profiling with VM interference (to do)
Demonstration of virtperf capabilities and correctness via lab set up with a couple of multi tier applications

Feb 16th, 2011
26
Future Work
virtperf features addition
Session, probabilistic request workloads
Profiling with interfering VMs
Evaluation and testing with kvm
Experimentation
Testing at scale
Black-box vs. Gray-box profiling - Remove profiling within VM
Bottleneck identification with multi-tier applications
Open-source release
http://www.cse.iitb.ac.in/synerg

Feb 16th, 2011
27
Other Virtualisation Related Projects@synerg
Flexible resource provisioning
Affinity-aware resource profiling and prediction
Benchmarking, estimating and tuning cost of migration
p2v placement tool
Design and evaluation of virtualisation-aware placement techniques
vSim
A simulator for virtualisation-based systems
Network I/O optimisations for collocated VMs

Feb 16th, 2011
Questions?umesh, puru@cse.iitb.ac.in
Acknowledgements:
Prof. VarshaApte who seeded the idea with AutoPerf
Past MTech students ShrirangShirodkar, VaibhavSelot

Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur

by Yahoo! Developer Network

Accessibility

Upload Details

Usage Rights

2 Embeds 149

Statistics

Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur Presentation Transcript

http://d.hatena.ne.jp	148
http://doryokujin.hatenablog.jp	1