Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur

  • 1,652 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,652
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
41
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Profiling Application Performance
    and Resource Utilization
    in Virtualized Environments
    UmeshBellur
    PuruKulkarni
    http://www.cse.iitb.ac.in/synerg
    Systems and Networks Research Group
    Department of Computer Science and Engineering
    Indian Institute of Technology Bombay
  • 2. Feb 16th, 2011
    2
    Virtualization based Provisioning
    • Abstract physical resources as virtual objects
    • 3. Virtual machines execute on virtual systems
    • 4. Virtual machine monitor multiplexes physical resources across virtual resources
    Benefits
    Direct: Rapid provisioning, Interoperability, Performance Isolation, Increased resource utilization
    Derived: Server Consolidation, Flexible resource provisioning, Efficient replication, backup and migration
  • 5. Feb 16th, 2011
    3
    Virtualisation-based Provisioning Issues
    Flexible Resource Provisioning
    SLA-based migration
    Power-aware consolidation
    Cloud bursting
    Physical to Virtual
    Resource requirements mapping and modelling
    Capacity planning
    VM placement policies
    Deployment & Manageability
    Resource accounting in Clouds
    Prototype Testbed setups
    VM Optimisations
    Shared memory based I/O for co-located VMs
    Network I/O guarantees
    VMs for resource-constrained platforms
  • 6. Feb 16th, 2011
    4
    Provisioning & Capacity Planning in Virtual Environments
    Some questions of interest
    • Application behaviour and resource usage under different workloads?
    • 7. Throughput, response time, network, CPU, memory ...
    • 8. With finite resources which SLAs can be satisfied?
    • 9. How will the application behave with varying resources?
    • 10. What is the impact of VM movement on application performance and resource usage levels?
  • Dom0
    DomU
    Xen Hypervisor
    Hardware
    Why separate VM-specific tool?
    • Existing tools not virtualisation aware
    • 11. All I/O through privileged VM
    • 12. Impact of VM migration
    • 13. Workload generation and
    app. performance separate
    from resource monitoring
    5
    01/12/10
  • 14. Feb 16th, 2011
    6
    State of the art for virtual environments
    • Resource monitoring tools
    • 15. xenmon, xentop, sysstat, Hyperic etc.
    • 16. Load generation benchmarks
    • 17. SPEC, TPCW ...
    • 18. Isolated from profiling framework
    • 19. Load generation and control
    • 20. Resource configuration control
    • 21. User-level performance metrics
    Require an integrated tool (for VM worlds)
    • Load generation + Resource configuration + Measurement and monitoring
  • Feb 16th, 2011
    7
    Problem Definition
    • Profile performance and resource utilization levels of applications in virtual environments
    • 22. Multi-tier, multi-VM based (web-based) application
    • 23. Implications on
    • 24. Capacity planning
    • 25. Migration and Consolidation Strategies
    • 26. virtPerf*
    • 27. Application profiling tool for virtualized environments
    • 28. Built on top of AutoPerf** – a similar tool for physical environments
    • 29. Current scope: Xen and kvm
    * The tool forms the thesis of a current MTech student - PrajaktaPatil.
    ** S. Shirodkar, “Autoperf: An automated load generator and performance measurement tool for multi-tier software systems,”, Master’s thesis, IIT Bombay, Mumbai, India, June 2007
  • 30. Feb 16th, 2011
    8
    virtPerf slave
    virtPerf master
    profiling
    agent
    requests
    DomU
    Load
    generator
    responses
    DomU
    Physical machine 1
    Controller
    DomU
    Physical machine 2
    Dom0
    virtPerf Architecture
    • Workload generator + Measurement tool
  • Feb 16th, 2011
    9
    DomU
    Dom0
    Load Generator and VM Initialisation
    - Multi-client load
    - Warm-up detection
    requests
    Load
    generator
    responses
    load
    level
    ready
    Initialize
    Controller
    Load characteristics
    Transaction details
    - Set resource configurations
    - IP to VM id mapping
    • Warmup => Stable response time
    • 31. Thread startup, caching etc.
    • 32. Agents initialize VMs according to resource profile
  • Feb 16th, 2011
    10
    DomU
    Dom0
    Run Termination and Measurements
    requests
    Load
    generator
    responses
    DONE
    STOP
    profiling
    Controller
    Resource usage summary
    Load characteristics
    Transaction details
    <network, disk, cpu,
    memory, service demand>
    ...
    - Collate performance metrics
    - Select next load level or VM resource configuration
    • xentop at Host (Dom0)
    • 33. sysstat and ps at Guest (DomU)
  • Feb 16th, 2011
    MSD of a Profiling Run
  • 34. Feb 16th, 2011
    12
    Metrics Collected
    • Resource usage metrics (averaged over duration of run)
    • 35. CPU utilization
    • 36. Network & disk I/O
    • 37. Memory usage
    • 38. Performance metrics
    • 39. Average response time
    • 40. Throughput
    • 41. Service demand
  • Feb 16th, 2011
    13
    virtPerf Features
    • Load control
    • 42. #clients, think time distributions, total number of requests issued
    • 43. AutoPerf provides automatic load level section, warmup detection
    • 44. VM resource configuration
    • 45. #cores, CPU allocation (min, max), CPU pinning
    • 46. To both Dom0 and each of the DomUs individually
    • 47. Impact of Migration
    • 48. Resource requirements, Performance hit (drop in throughput, increase in response time)
    • 49. VM Collocation
    • 50. Resource contention impact & VM Collocation benefits
  • Feb 16th, 2011
    virtPerf Validation
    • Metrics reported by virtPerf for resource consumptions and service performance known apriori.
    • 51. CPU Utilization:
    • 52. A service that can consume a specific amount of CPU with a knob to vary this from 10-90%.
    • 53. virtPerf reports utilization within 3% of this at any level.
    • 54. Network I/O:
    • 55. A service will read/write a known amount of I/O at a specified rate.
    • 56. virtPerf reports back this number with a 400 byte overhead per request caused by HTTP and the virtPerf control overhead
    • 57. Disk I/O:
    • 58. Similar to Network I/O and results are within 2% of expected values
  • Feb 16th, 2011
    15
    Experimental Setup
    • PMs: Quad Core Intel i5 Intel(R) Core(TM) i5 CPU, 760 @ 2.80GHz 3 GB RAM
    • 59. Virtual platforms: Xen
    WebCalendar & RUBiS - 2 tier applications with a MySQL backend and a HTTP-based front-end
  • 60. Feb 16th, 2011
    16
    Throughput Saturation
    • webcal saturates at 64 req/s
    • 61. virtPerf can automatically detect saturation
    • 62. CPU utilization of VM (DomU) ~400% (4 core m/c)
  • Feb 16th, 2011
    17
    Close to service demand =>high arrival rate
    Throughput saturates with ~12 users
    Think time >> service demand =>Low arrival rate
    Throughput saturates with~72 users
    Sensitivity to Think time distribution
    Think time distribution: Poisson
    Mean: <80ms, 150 ms, 800 ms>
  • 63. Feb 16th, 2011
    18
    1 core vs 4 cores
    Maximum Throughput 16 req/s : 65 req/s
    Response time 900 ms : 3000 ms
    Performance metrics ratio 1 : 4
    VM CPU Utilization
    ~80 % : 375 %
    VM Configuration (#cores)
  • 64. Feb 16th, 2011
    19
    VM Configuration (placement)
    • 3 configurations
    • 68. Single VM app (c1)
    • 69. 2 VMs on 1 PM (c2)
    • 70. 2 VMs on 2 PMs(c3)
  • Feb 16th, 2011
    20
    Response time approximately
    half with 300% CPU
    Achieved Throughput doubles
    (25 req/s to 50 req/s)
    WebCalendar
    VM Resource Configuration (CPU availability)
    • CPU for DomU (guest VM): 150% and 300%
  • Feb 16th, 2011
    VM Resource Configuration for RUBiS
    • Impact of increasing CPU for the Web Tier felt ONLY AFTER DB
    Tier has sufficient CPU (260%). Else DB Tier is the bottleneck
    • Throughput goes from 90 to 156 requests/sec at the DB tier
    receiving upto 340% and the web tier going from 10 to 30%
  • 71. Feb 16th, 2011
    VM Migration while under Load
  • 72. Feb 16th, 2011
    Effects of Migration
  • 73. Feb 16th, 2011
    Core Pinning
    Peak throughput almost doubles when the DB
    Tier is given it’s own core compared to when
    all the cores are shared amongst the VMs – Dom0
    and 2 app VMs sharing 2 cores.
    • VirtPerf can pin VMs on specific
    cores of a machine.
    • Makes a difference to application
    performance.
  • 74. Feb 16th, 2011
    25
    virtPerf Summary
    • Workload generator + Application profiler
    • 75. Feature set (profiling with ...)
    • 76. Load distribution control
    • 77. VM resource configuration and control
    • 78. Performance and resource metrics with migration
    • 79. Profiling with VM interference (to do)
    • 80. Demonstration of virtperf capabilities and correctness via lab set up with a couple of multi tier applications
  • Feb 16th, 2011
    26
    Future Work
    • virtperf features addition
    • 81. Session, probabilistic request workloads
    • 82. Profiling with interfering VMs
    • 83. Evaluation and testing with kvm
    • 84. Experimentation
    • 85. Testing at scale
    • 86. Black-box vs. Gray-box profiling - Remove profiling within VM
    • 87. Bottleneck identification with multi-tier applications
    • 88. Open-source release
    http://www.cse.iitb.ac.in/synerg
  • 89. Feb 16th, 2011
    27
    Other Virtualisation Related Projects@synerg
    • Flexible resource provisioning
    • 90. Affinity-aware resource profiling and prediction
    • 91. Benchmarking, estimating and tuning cost of migration
    • 92. p2v placement tool
    • 93. Design and evaluation of virtualisation-aware placement techniques
    • 94. vSim
    • 95. A simulator for virtualisation-based systems
    • 96. Network I/O optimisations for collocated VMs
  • Feb 16th, 2011
    Questions?umesh, puru@cse.iitb.ac.in
    Acknowledgements:
    Prof. VarshaApte who seeded the idea with AutoPerf
    Past MTech students ShrirangShirodkar, VaibhavSelot