Profiling Application Performance <br />and Resource Utilization <br />in Virtualized Environments<br />UmeshBellur<br />P...
Feb 16th, 2011<br />2<br />Virtualization based Provisioning<br /><ul><li>Abstract physical resources as virtual objects
Virtual machines execute on virtual systems
Virtual machine monitor multiplexes physical resources across virtual resources</li></ul>Benefits<br />Direct: Rapid provi...
Feb 16th, 2011<br />3<br />Virtualisation-based Provisioning Issues<br />Flexible Resource Provisioning<br /> SLA-based mi...
Feb 16th, 2011<br />4<br />Provisioning & Capacity Planning in Virtual Environments<br />Some questions of interest<br /><...
Throughput, response time, network, CPU, memory ...
With finite resources which SLAs can be satisfied?
How will the application behave with varying resources?
What is the impact of VM movement on application performance and resource usage levels?</li></li></ul><li>Dom0<br />DomU<b...
All I/O through privileged VM
Impact of VM migration
Workload generation and </li></ul>  app. performance separate <br />  from resource monitoring<br />5<br />01/12/10<br />
Feb 16th, 2011<br />6<br />State of the art for virtual environments<br /><ul><li>Resource monitoring tools
xenmon, xentop, sysstat, Hyperic etc.
Load generation benchmarks
SPEC, TPCW ...
Isolated from profiling framework
Load generation and control
Resource configuration control
User-level performance metrics</li></ul>Require an integrated tool (for VM worlds)<br /><ul><li>Load generation + Resource...
Multi-tier, multi-VM based (web-based) application
Implications on
Capacity planning
Migration and Consolidation Strategies
virtPerf*
Application profiling tool for virtualized environments
Built on top of AutoPerf** – a similar tool for physical environments
Upcoming SlideShare
Loading in...5
×

Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur

1,707

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,707
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur"

  1. 1. Profiling Application Performance <br />and Resource Utilization <br />in Virtualized Environments<br />UmeshBellur<br />PuruKulkarni<br />http://www.cse.iitb.ac.in/synerg<br />Systems and Networks Research Group<br />Department of Computer Science and Engineering<br />Indian Institute of Technology Bombay<br />
  2. 2. Feb 16th, 2011<br />2<br />Virtualization based Provisioning<br /><ul><li>Abstract physical resources as virtual objects
  3. 3. Virtual machines execute on virtual systems
  4. 4. Virtual machine monitor multiplexes physical resources across virtual resources</li></ul>Benefits<br />Direct: Rapid provisioning, Interoperability, Performance Isolation, Increased resource utilization<br />Derived: Server Consolidation, Flexible resource provisioning, Efficient replication, backup and migration<br />
  5. 5. Feb 16th, 2011<br />3<br />Virtualisation-based Provisioning Issues<br />Flexible Resource Provisioning<br /> SLA-based migration<br /> Power-aware consolidation<br /> Cloud bursting<br />Physical to Virtual <br /> Resource requirements mapping and modelling<br /> Capacity planning<br /> VM placement policies<br />Deployment & Manageability <br /> Resource accounting in Clouds<br /> Prototype Testbed setups<br />VM Optimisations<br /> Shared memory based I/O for co-located VMs<br /> Network I/O guarantees <br />VMs for resource-constrained platforms<br />
  6. 6. Feb 16th, 2011<br />4<br />Provisioning & Capacity Planning in Virtual Environments<br />Some questions of interest<br /><ul><li>Application behaviour and resource usage under different workloads?
  7. 7. Throughput, response time, network, CPU, memory ...
  8. 8. With finite resources which SLAs can be satisfied?
  9. 9. How will the application behave with varying resources?
  10. 10. What is the impact of VM movement on application performance and resource usage levels?</li></li></ul><li>Dom0<br />DomU<br />Xen Hypervisor<br />Hardware<br />Why separate VM-specific tool?<br /><ul><li>Existing tools not virtualisation aware
  11. 11. All I/O through privileged VM
  12. 12. Impact of VM migration
  13. 13. Workload generation and </li></ul> app. performance separate <br /> from resource monitoring<br />5<br />01/12/10<br />
  14. 14. Feb 16th, 2011<br />6<br />State of the art for virtual environments<br /><ul><li>Resource monitoring tools
  15. 15. xenmon, xentop, sysstat, Hyperic etc.
  16. 16. Load generation benchmarks
  17. 17. SPEC, TPCW ...
  18. 18. Isolated from profiling framework
  19. 19. Load generation and control
  20. 20. Resource configuration control
  21. 21. User-level performance metrics</li></ul>Require an integrated tool (for VM worlds)<br /><ul><li>Load generation + Resource configuration + Measurement and monitoring</li></li></ul><li>Feb 16th, 2011<br />7<br />Problem Definition<br /><ul><li>Profile performance and resource utilization levels of applications in virtual environments
  22. 22. Multi-tier, multi-VM based (web-based) application
  23. 23. Implications on
  24. 24. Capacity planning
  25. 25. Migration and Consolidation Strategies
  26. 26. virtPerf*
  27. 27. Application profiling tool for virtualized environments
  28. 28. Built on top of AutoPerf** – a similar tool for physical environments
  29. 29. Current scope: Xen and kvm</li></ul>* The tool forms the thesis of a current MTech student - PrajaktaPatil. <br />** S. Shirodkar, “Autoperf: An automated load generator and performance measurement tool for multi-tier software systems,”, Master’s thesis, IIT Bombay, Mumbai, India, June 2007 <br />
  30. 30. Feb 16th, 2011<br />8<br />virtPerf slave<br />virtPerf master<br />profiling<br />agent<br />requests<br />DomU<br />Load<br />generator<br />responses<br />DomU<br />Physical machine 1<br />Controller<br />DomU<br />Physical machine 2<br /> Dom0<br />virtPerf Architecture<br /><ul><li>Workload generator + Measurement tool</li></li></ul><li>Feb 16th, 2011<br />9<br />DomU<br /> Dom0<br />Load Generator and VM Initialisation<br /> - Multi-client load <br /> - Warm-up detection<br />requests<br />Load<br />generator<br />responses<br />load<br />level<br />ready<br />Initialize<br />Controller<br />Load characteristics<br />Transaction details<br /> - Set resource configurations<br /> - IP to VM id mapping<br /><ul><li>Warmup => Stable response time
  31. 31. Thread startup, caching etc.
  32. 32. Agents initialize VMs according to resource profile</li></li></ul><li>Feb 16th, 2011<br />10<br />DomU<br />Dom0<br />Run Termination and Measurements<br />requests<br />Load<br />generator<br />responses<br />DONE<br />STOP<br />profiling<br />Controller<br />Resource usage summary<br />Load characteristics<br />Transaction details<br /><network, disk, cpu, <br />memory, service demand><br />...<br /> - Collate performance metrics<br /> - Select next load level or VM resource configuration<br /><ul><li>xentop at Host (Dom0)
  33. 33. sysstat and ps at Guest (DomU)</li></li></ul><li>Feb 16th, 2011<br />MSD of a Profiling Run<br />
  34. 34. Feb 16th, 2011<br />12<br />Metrics Collected<br /><ul><li>Resource usage metrics (averaged over duration of run)
  35. 35. CPU utilization
  36. 36. Network & disk I/O
  37. 37. Memory usage
  38. 38. Performance metrics
  39. 39. Average response time
  40. 40. Throughput
  41. 41. Service demand</li></li></ul><li>Feb 16th, 2011<br />13<br />virtPerf Features<br /><ul><li>Load control
  42. 42. #clients, think time distributions, total number of requests issued
  43. 43. AutoPerf provides automatic load level section, warmup detection
  44. 44. VM resource configuration
  45. 45. #cores, CPU allocation (min, max), CPU pinning
  46. 46. To both Dom0 and each of the DomUs individually
  47. 47. Impact of Migration
  48. 48. Resource requirements, Performance hit (drop in throughput, increase in response time)
  49. 49. VM Collocation
  50. 50. Resource contention impact & VM Collocation benefits</li></li></ul><li>Feb 16th, 2011<br />virtPerf Validation<br /><ul><li>Metrics reported by virtPerf for resource consumptions and service performance known apriori.
  51. 51. CPU Utilization:
  52. 52. A service that can consume a specific amount of CPU with a knob to vary this from 10-90%.
  53. 53. virtPerf reports utilization within 3% of this at any level.
  54. 54. Network I/O:
  55. 55. A service will read/write a known amount of I/O at a specified rate.
  56. 56. virtPerf reports back this number with a 400 byte overhead per request caused by HTTP and the virtPerf control overhead
  57. 57. Disk I/O:
  58. 58. Similar to Network I/O and results are within 2% of expected values</li></li></ul><li>Feb 16th, 2011<br />15<br />Experimental Setup<br /><ul><li>PMs: Quad Core Intel i5 Intel(R) Core(TM) i5 CPU, 760 @ 2.80GHz 3 GB RAM
  59. 59. Virtual platforms: Xen</li></ul>WebCalendar & RUBiS - 2 tier applications with a MySQL backend and a HTTP-based front-end <br />
  60. 60. Feb 16th, 2011<br />16<br />Throughput Saturation<br /><ul><li>webcal saturates at 64 req/s
  61. 61. virtPerf can automatically detect saturation
  62. 62. CPU utilization of VM (DomU) ~400% (4 core m/c)</li></li></ul><li>Feb 16th, 2011<br />17<br />Close to service demand =>high arrival rate<br /> Throughput saturates with ~12 users<br />Think time >> service demand =>Low arrival rate<br /> Throughput saturates with~72 users<br />Sensitivity to Think time distribution<br />Think time distribution: Poisson<br />Mean: <80ms, 150 ms, 800 ms><br />
  63. 63. Feb 16th, 2011<br />18<br />1 core vs 4 cores<br />Maximum Throughput 16 req/s : 65 req/s<br />Response time 900 ms : 3000 ms<br /> Performance metrics ratio 1 : 4 <br />VM CPU Utilization<br />~80 % : 375 %<br />VM Configuration (#cores)<br />
  64. 64. Feb 16th, 2011<br />19<br /><ul><li>Throughput
  65. 65. c1 >> c2 >> c3
  66. 66. Response time
  67. 67. c1 << c2 << c3</li></ul>VM Configuration (placement)<br /><ul><li> 3 configurations
  68. 68. Single VM app (c1)
  69. 69. 2 VMs on 1 PM (c2)
  70. 70. 2 VMs on 2 PMs(c3)</li></li></ul><li>Feb 16th, 2011<br />20<br />Response time approximately<br />half with 300% CPU<br />Achieved Throughput doubles<br />(25 req/s to 50 req/s)<br />WebCalendar<br />VM Resource Configuration (CPU availability)<br /><ul><li>CPU for DomU (guest VM): 150% and 300%</li></li></ul><li>Feb 16th, 2011<br />VM Resource Configuration for RUBiS<br /><ul><li> Impact of increasing CPU for the Web Tier felt ONLY AFTER DB</li></ul> Tier has sufficient CPU (260%). Else DB Tier is the bottleneck<br /><ul><li> Throughput goes from 90 to 156 requests/sec at the DB tier </li></ul> receiving upto 340% and the web tier going from 10 to 30%<br />
  71. 71. Feb 16th, 2011<br />VM Migration while under Load<br />
  72. 72. Feb 16th, 2011<br />Effects of Migration<br />
  73. 73. Feb 16th, 2011<br />Core Pinning<br />Peak throughput almost doubles when the DB<br />Tier is given it’s own core compared to when<br />all the cores are shared amongst the VMs – Dom0<br />and 2 app VMs sharing 2 cores.<br /><ul><li> VirtPerf can pin VMs on specific</li></ul> cores of a machine.<br /><ul><li> Makes a difference to application</li></ul> performance.<br />
  74. 74. Feb 16th, 2011<br />25<br />virtPerf Summary<br /><ul><li>Workload generator + Application profiler
  75. 75. Feature set (profiling with ...)
  76. 76. Load distribution control
  77. 77. VM resource configuration and control
  78. 78. Performance and resource metrics with migration
  79. 79. Profiling with VM interference (to do)
  80. 80. Demonstration of virtperf capabilities and correctness via lab set up with a couple of multi tier applications</li></li></ul><li>Feb 16th, 2011<br />26<br />Future Work<br /><ul><li>virtperf features addition
  81. 81. Session, probabilistic request workloads
  82. 82. Profiling with interfering VMs
  83. 83. Evaluation and testing with kvm
  84. 84. Experimentation
  85. 85. Testing at scale
  86. 86. Black-box vs. Gray-box profiling - Remove profiling within VM
  87. 87. Bottleneck identification with multi-tier applications
  88. 88. Open-source release</li></ul>http://www.cse.iitb.ac.in/synerg<br />
  89. 89. Feb 16th, 2011<br />27<br />Other Virtualisation Related Projects@synerg<br /><ul><li>Flexible resource provisioning
  90. 90. Affinity-aware resource profiling and prediction
  91. 91. Benchmarking, estimating and tuning cost of migration
  92. 92. p2v placement tool
  93. 93. Design and evaluation of virtualisation-aware placement techniques
  94. 94. vSim
  95. 95. A simulator for virtualisation-based systems
  96. 96. Network I/O optimisations for collocated VMs</li></li></ul><li>Feb 16th, 2011<br />Questions?umesh, puru@cse.iitb.ac.in<br />Acknowledgements:<br />Prof. VarshaApte who seeded the idea with AutoPerf<br />Past MTech students ShrirangShirodkar, VaibhavSelot<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×