Your SlideShare is downloading. ×
Building vSphere Perf Monitoring Tools
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Building vSphere Perf Monitoring Tools

7,342

Published on

Balaji and Ravi present on how to build vSphere monitoring tools using the vSphere APIs - this is a must view for anyone managing a large complex environment. For vSphere SDKs, API visit …

Balaji and Ravi present on how to build vSphere monitoring tools using the vSphere APIs - this is a must view for anyone managing a large complex environment. For vSphere SDKs, API visit http://developer.vmware.com Blogs, Forums, Sample Code

Published in: Sports
2 Comments
13 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,342
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
520
Comments
2
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Developing Performance Monitoring Tools Using vSphere APIs: Techniques and Best Practices TEXIBP 1005 Balaji Parimi & Ravi Soundararajan, VMware Inc. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 2. Disclaimer This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. “These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.” Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 2 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 3. Follow Us Join the conversation Facebook Get the latest scoop Search for VMware Partner Network Be a part of the network Twitter t Search for VMware_Partners Linkedin Subscribe to RSS Search for VMware Partner Network http://blogs.vmware.com/powerofpartnership/ Follow this event LIVE! Help us help you! Please fill out the survey #VMwarePEX2010 at the end of the course. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 3 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 4. Who is our Target Audience? System Administrators: • Monitor the performance of their virtual infrastructure • Understand the bottlenecks in the datacenter to help reconfigure their storage or network topology • Plan for future growth in the datacenter VMware ISV Partners: • Collect the relevant/important performance counters and hand them off to a performance analysis tool • Create a wizard/tool that helps system administrators troubleshoot their datacenter Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 4 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 5. Agenda Use Cases Building a Performance Monitoring Application Q&A Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 5 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 6. Use Cases Use Cases: What do I look for and why? • CPU • Memory • Disk • Network Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 6 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 7. CPU ESX is designed to run Virtual Machines Schedulable entity = “world” • Virtual Machines are composed of worlds (mks, vCPUs) • Service Console (agents like vpxa, hostd) (Classic ESX) Proportional-share scheduler for resource management • Limits, Shares, and Reservations World states (simplified view): • ready = ready-to-run but no physical CPU free • run = currently active and running • wait = blocked on I/O Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 7 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 8. So, How Do I Spot CPU Performance Problems? One common issue is high CPU ready time • High ready time possible contention for CPU resources among VMs • Many possible reasons CPU over commitment (high %rdy + high %used) Workload variability Limit set on VM • No fixed threshold, but > 20% for a VCPU Investigate further Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 8 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 9. CPU: Useful Metrics Per-Host Metric (Client) Metric Metric (sdk) Description (esxtop) Usage (%) %USED cpu.usage.average CPU used over the collection interval (%) Usage (MHz) n/a cpu.usagemhz.average CPU used over the collection interval (MHz) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 9 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 10. CPU: Useful Metrics Per-VM Metric (Client) Metric Metric (SDK) Description (esxtop) Usage (%) %USED cpu.usage.average CPU used over the collection interval Used (ms) %USED cpu.used.summation CPU used over the collection interval)* Ready (ms) %RDY cpu.ready.summation CPU time spent in ready state* Swap wait time %SWPWT cpu.swapwait.summation CPU time spent (ms) [ESX4.0 waiting for host- hosts] level swap-in * Units different between esxtop and vSphere client Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 10 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 11. Spotting CPU Overcommitment in esxtop • 2-CPU box, but 3 active VMs (high %used) • High %rdy + high %used can imply CPU overcommitment Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 11 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 12. Caveat on Ready Time: Workload Variability (screenshot from VI Client) Some caveats on ready time Used time Used time ~ ready time: may signal contention. However, might not be overcommitted due to workload variability In this example, we have Ready time ~ used time periods of activity and idle periods: CPU isn’t overcommitted all the time Ready time < used time Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 12 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 13. Further ready time examination High Ready Time High MLMTD: there is a limit on this VM… High ready time not always because of overcommitment When you see high ready time, double-check if limit is set Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 13 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 14. Further Ready Time Investigation: No Fixed Threshold…Why? ODBC Sniff traffic vSphere Probe VM vSphere VM DB ESX 1 ESX 2 vSphere communicates with DB Probe VM monitors vSphere-to-DB traffic The more traffic, the more work done by Probe VM Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 14 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 15. Why Isn’t There A Fixed Ready Time Threshold? Ready time jump from 12.5% (idle DB) to 20% (busy DB) – didn’t notice until responsiveness suffered! Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 15 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 16. Summary of Possible Reasons for High Ready Time CPU overcommitment • Possible solution: add more CPUs or VMotion the VM Workload variability • A bunch of VMs wake up all at once • Note: system may be mostly idle: not always overcommitted Limit set on VM • 4x2GHz host, 2 vcpu VM, limit set to 1GHz (VM can consume 1GHz) • Without limit, max is 2GHz. With limit, max is 1GHz (50% of 2GHz) • CPU all busy: %USED: 50%; %MLMTD & %RDY = 150% [total is 200%, or 2 CPUs] Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 16 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 17. Where do we go from here? Use Cases: What do look for and why • CPU • Memory • Disk • Network Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 17 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 18. Memory ESX must balance memory usage for all worlds • Virtual machines, Service Console, and vmkernel consume memory • Page sharing to reduce memory footprint of Virtual Machines • Ballooning to relieve memory pressure in a graceful way • Host swapping to relieve memory pressure when ballooning insufficient ESX allows overcommitment of memory • Sum of configured memory sizes of virtual machines can be greater than physical memory if working sets fit Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 18 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 19. Ballooning vs. Swapping (1) Ballooning: Memctl driver grabs pages and gives to ESX Guest OS choose pages to give to memctl (avoids “hot” pages if possible): either free pages or pages to swap - Unused pages are given directly to memctl - Pages to be swapped are first written to swap partition within guest OS and then given to memctl VM1 VM2 F memctl 2. Reclaim 3. Redistribute Swap partition w/in Guest OS 1. Balloon ESX Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 19 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 20. Ballooning vs. Swapping (2) Swapping: ESX reclaims pages forcibly Guest doesn’t pick pages…ESX may inadvertently pick “hot” pages ( possible VM performance implications) Pages written to VM swap file VM1 VM2 Swap Partition ESX 1. Force Swap VSWP (w/in guest) 2. Reclaim (external to guest) 3. Redistribute Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 20 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 21. Ballooning vs. Swapping (3) Bottom line: • Ballooning may occur even when no memory pressure just to keep memory proportions under control • Ballooning is vastly preferably to swapping Guest can surrender unused/free pages - With host swapping, ESX cannot tell which pages are unused or free and may accidentally pick “hot” pages Even if balloon driver has to swap to satisfy the balloon request, guest chooses what to swap - Can avoid swapping “hot” pages within guest Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 21 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 22. Ok, So Why Do I Care About Memory Usage? If running VMs consume too much host memory… • Some VMs do not get enough host memory • This forces either ballooning or host swapping to satisfy VM demands • Host swapping or excessive ballooning reduced VM performance If I do not size a VM properly (e.g., create Windows VM with 128MB RAM) • Within the VM, swapping occurs, resulting in disk traffic • VM may slow down • But…don’t make memory too big! (High overhead memory) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 22 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 23. Important Memory Metrics (Per VM) Metric (Client) Metric Metric (SDK) Description (esxtop) Swap in rate SWR/s mem.swapinRate.average Rate at which mem is (ESX4.0 Hosts) swapped in from disk Swap out rate SWW/s mem.swapoutRate.average Rate at which mem is (ESX4.0 Hosts) swapped out to disk Swapped SWCUR mem.swapped.average (level 2 ~swap out – swap in counter) Swap in n/a mem.swapin.average Mem swapped in from (cumulative) disk Swap out n/a mem.swapout.average Mem swapped out to (cumulative) disk One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 23 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 24. Important Memory Metrics (Per Host, sum of VMs) Metric (Client) Metric Metric (SDK) Description (esxtop) Swap in rate SWR/s mem.swapinRate.average Rate at which mem is (ESX4.0 Hosts) swapped in from disk Swap out rate SWW/s mem.swapoutRate.average Rate at which mem is (ESX4.0 Hosts) swapped out to disk Swap used SWCUR mem.swapused.average (level ~swap out – swap in 2 counter) Swap in n/a mem.swapin.average Mem swapped in from (cumulative) disk Swap out n/a mem.swapout.average Mem swapped out to (cumulative) disk One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 24 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 25. Example of Swapping Increased swap activity may be a sign of over-commitment No swapping Lots of swapping Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 25 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 26. A Stacked Chart (per VM) of Swapping Lots of swapping No swapping Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 26 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 27. Where do we go from here? Use Cases: What do look for and why • CPU • Memory • Disk • Network Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 27 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 28. Disk Performance Problems 101 What should I look for to figure out if disk is an issue? • Am I getting the IOPs I expect? • Am I getting the bandwidth (read/write) I expect? • Are the latencies higher than I expect? • Where is time being spent? What are some things I can do? • Make sure devices are configured properly (caches, queue depths) • Use multiple adapters and multipathing • Check networking settings (for iSCSI/NAS) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 28 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 29. Useful Disk Metrics Metric Metric Metric (SDK) Description (Client) (Esxtop) Commands CMDS/s disk.commands.summation commands issued in the sampling interval Read rate MBREADS/s disk.read.average KB/s read* Write rate MBWRTN/s disk.write.average KB/s written* Device latency DAVG/cmd disk.deviceLatency.average Average latency at device Kernel latency KAVG/cmd disk.kernelLatency.average Average latency in vmkernel Command GAVG/cmd disk.totalLatency.average Total latency for latency command * Units different between esxtop and vSphere client Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 29 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 30. Disk Performance Example: vSphere Client SAN cache enabled: High Write Throughput SAN cache disabled: Poor throughput Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 30 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 31. Another Disk Example: Slow VM Power On Trying to Power on a VM • Sometimes, powering on VM would take 5 seconds • Other times, powering on VM would take 5 minutes! Where to begin? • Powering on a VM requires disk activity on host Check disk metrics for host Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 31 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 32. Let’s look at the vSphere client… Max Disk Latencies range from 100ms to 1100ms…very high! Why? (counter name: disk.maxTotalLatency.latest) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 32 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 33. Using Esxtop to Examine Slow VM Power On Note very large DAVG/cmd and GAVG/cmd Rule of thumb: GAVG/cmd > 50ms = high latency! What does this mean? • Latency when command reaches device is high • Latency as seen by the guest is high • Low KAVG/cmd: command isn’t queuing in VMkernel • What’s up? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 33 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 34. High Disk Latency: Mystery Solved Host events: disk has connectivity issues high latencies! Bottom line: monitor disk latencies; issues may not be related to virtualization! Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 34 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 35. A More Complex Disk Example Ethernet VM1 VM2 VM3 VM4 … VM15 NAS ESX Disk Traffic VMs communicate with NAS (network traffic) NAS writes to disk (disk traffic) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 35 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 36. Combining Metrics: A More Complex Disk Example Setup: • Group of Virtual Machines running on a host • Each Virtual Machine talks to a Virtual Machine serving as a NAS device Problem: • Suddenly, I cannot log in to any of the Virtual Machines (really slow!) Initial Speculation • Virtual Machines are saturated in some resource Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 36 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 37. A More Complicated Disk Example, Part I (CPU) Predictable CPU usage, Host not saturated “Chaotic” CPU usage, Host saturated Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 37 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 38. Complicated Disk Example, Part 2: Disk Usage Predictable, balanced disk usage Uneven, reduced disk usage Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 38 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 39. Complicated Disk Example, Part 3: Write Rate Read and write traffic Increased write traffic, zero read traffic Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 39 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 40. Complicated Example: Putting it all Together Each App Virtual Machine reads to & writes from same NAS Something caused excessive writes from each Virtual Machine • Increased CPU usage per Virtual Machine • Increased write traffic per Virtual Machine • Ton of writes on NAS VM! Issue • Bug in application within Virtual Machine caused error condition • Error condition caused excessive writes to same NAS - Network traffic for application VMs, disk traffic on NAS VM • Each Virtual Machine is so busy writing that it never reads Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 40 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 41. Where do we go from here? Use Cases: What do look for and why • CPU • Memory • Disk • Network Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 41 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 42. Network Performance Problems 101 What should I look for to figure out if network is an issue? • Am I getting the packet rate that I expect? • Am I getting the bandwidth (read/write) I expect? • Is all traffic on one NIC, or spread across many NICs? • [more advanced…not available through counters]: out-of-order packets? What are some things I can do? • Check host networking settings (full-duplex/half-duplex, 10Gig network vs 100Mb network?, firewall settings) • Check VM settings: all VMs on proper networks? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 42 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 43. Useful networking metrics Metric (Client) Metric Metric (SDK) Description (esxtop) Packets PKTTX/s net.transmitted.average Packets transmitted (in transmitted in sampling interval) sampling interval Packets received PKTRX/s net.received.average Packets received (in sampling in the sampling interval) interval Data transmit MbTX/s net.transmitted.average Amount of data rate (KBps) transmitted per second* Data receive rate MbRX/s net.received.average Amount of data (KBps) received per second* * Units different between esxtop and vSphere client Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 43 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 44. Network Performance Troubleshooting Customer complains about slow network She’s running netperf on a GigE Link She sees only 200Mbps Why? I bet it’s that VMware stuff!! - Note to reader: Please don’t blame VMware first ☺ Where do we start? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 44 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 45. Where do we begin? Check VM Bandwidth. Measure VM Bandwidth (net.transmitted.average) 200 Mb/s Screenshot from the vSphere client Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 45 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 46. Check Host Bandwidth Measure Host Bandwidth (net.transmitted.average) Host sees around 900Mbps…why is VM at 200Mbps? Hmm…are we sharing this NIC with multiple VMs? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 46 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 47. All VMs using same NIC (VM Network) All VMs using “VM Network” and sharing 1 physical NIC Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 47 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 48. All Traffic is Going Through One Nic! Measure per-physical-NIC traffic All traffic through one NIC on this host Hmm…all VM traffic is going through 1 NIC Let’s split the VMs across NICs Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 48 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 49. Split VMs Across Multiple NICs. Bingo! Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 49 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 50. So Complicated! Where do I begin? So much to track… But I have different needs! For example… Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 50 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 51. What if I want esxtop for multiple VMs at once? vTop: A stats display similar to esxtop, but showing multiple metrics per VM in one screen Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 51 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 52. What if I want to see if DRS is ‘working’? Highlight the effectiveness of DRS by overlaying stats and events data Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 52 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 53. What if I Want to Compare Guest/Host Metrics on One Chart? Overlay stats data from the ESX and the GOS Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 53 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 54. What If I Want To Compare the Throughput of 2 RPs? Aggregate stats data by resource pools, clusters, group of VMs(Windows, Linux), vApps, etc. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 54 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 55. Sounds Great…but how do I do this stuff? Basic Steps 1. Collect the Stats you Need 2. Interpret the Stats 3. Present the stats Observations 1. Stats collection is similar across customer environments 2. Interpretation/Presentation is specific to customer Stats collection is a major pain point 1. For customers: so many stats, complex API 2. For us: we get the same questions over and over again So…we’ve created a reference application for Stats Collection Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 55 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 56. Agenda Use Cases Building a Performance Monitoring Application Q&A Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 56 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 57. Building a Performance Monitoring Application Monitoring Applications consist of a few standard parts: • Selecting entities to monitor (hosts, VMs, etc.) • Selecting metrics to monitor (CPU, Memory, etc.) • Choosing collection frequency • Keeping abreast of inventory changes • Collecting stats • Presenting data in a meaningful way Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 57 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 58. StatsFeeder: A Reference Monitoring Application Present data in a meaningful way Collect statistics (note: many sources!) Keep track of inventory changes Select Entities and Metrics to monitor and set collection frequency Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 58 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 59. A Brief Walkthrough of StatsFeeder 1. Select Entities/Metrics to Monitor 2. Keep Track of Inventory Changes 3. Collect Statistics 4. Present Statistics Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 59 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 60. Selecting Entities/Metrics to Monitor Relevant Modules in StatsFeeder 1. Counter Info Initializer 2. User Input Reader 3. Entity MoRef Retriever 4. Entity Metrics and Performance Provider Summary Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 60 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 61. Counter Info Initializer Retrieves available counters. Tip: • Counter information is static: retrieve only once. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 61 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 62. Counter Info Initializer: Code Snippet The PerformanceManager managed object contains a property with the name ‘perfCounter’ It is an array of type PerfCounterInfo. Use PropertyCollector to retrieve this property and get the supported counters. PropertySpec pSpec = new PropertySpec(); pSpec.setType(“PerformanceManager”); pSpec.setPathSet(new String[]{“perfCounter”}); ObjectSpec oSpec = new ObjectSpec(); oSpec.setObj(serviceContent.getPerfManager()); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 62 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 63. User Input Reader Reads the configuration file for: • The entities for which the performance statistics are being retrieved. Specify individual entities by name Specify the name of the container and the child type to monitor the entities in that container • The performance statistics to retrieve. • The frequency at which the performance data should be retrieved. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 63 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 64. User Input Reader: Sample Configuration File Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 64 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 65. Entity MoRef Retriever Retrieves MoRefs of entries in configuration file. Tricks: • MoRef does not change once an item is added to the vCenter inventory. • MoRef does change if entity is removed and re-added. Therefore: • Maintain cache of MoRefs • Monitor changes to the inventory • Use Views to retrieve MoRefs Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 65 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 66. Entity MoRef Retriever Retrieve the MoRef of individual entities using PropertyCollector or SearchIndex. Retrieve the MoRefs of entities in a container using ContainerView: • Obtain the MoRef of the container • Create the ContainerView • ContainerView has a property with the name ‘view’ and it is an array of type ManagedObjectReference. This contains the MoRefs of all the types specified during the ContainerView creation. vimPort.createContainerView( serviceContent.getViewMgr, containerMoRef, new String[]{“VirtualMachine”},//MO type true //recursively look in the container ); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 66 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 67. Entity Metrics and Performance Provider Summary Retrieve the metric IDs for all the entities. • E.g.: For dual CPU VM, metric IDs are 0 and 1 for CPU 0 and CPU 1. • Caveat: Can’t get this for powered off VMs and host systems, disconnected hosts and orphaned VMs Tricks: • Performance Metric IDs for entities do not change if hardware does not change. Therefore: • Cache Metric IDs for entities. Tip: • Use wild card for metric IDs Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 67 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 68. Entity Metrics vimPort.queryAvailablePerfMetric ( serviceContent.getPerfManager(), entityMoRef, null, null, intervalID //If not specified, defaults to //available metrics for historical stats. ); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 68 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 69. Entity Metrics and Performance Provider Summary Performance Provider Summary provides information about: • The type of statistics it supports (real- time or historical) • The refresh rate for statistics generation Performance Provider Summary data is the same for managed objects of the same type. Tricks: • You don’t have to retrieve this data for all entities. • Make sure the interval between two performance data queries is equal to at least the refresh rate to avoid getting the same value. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 69 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 70. Performance Provider Summary vimPort.queryPerfProviderSummary( serviceContent.getPerfManager(), entityMoRef, ); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 70 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 71. A Brief Walkthrough of StatsFeeder 1. Select Entities/Metrics to Monitor 2. Keep Track of Inventory Changes 3. Collect Statistics 4. Present Statistics Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 71 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 72. Keeping Track of Inventory Changes Relevant Module in StatsFeeder: Inventory/Entity Properties Monitor Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 72 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 73. Inventory/Entity Properties Monitor Tracks changes to the inventory (to keep EntityMoRefCache up to date) Tracks changes to the virtual hardware of the VMs (to keep Entity Metrics cache up to date) Use Views to monitor these changes Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 73 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 74. Inventory/Entity Properties Monitor To monitor for changes to inventory: • Create InventoryView • Open the inventory view folders to monitor for changes • Monitor for changes to ‘view’ property, as it contains the MoRefs of all the managed objects in the inventory vimPort.openInventoryViewFolder( inventoryViewMoRef, new ManagedObjectReference[]{}//MoRefs of //all the containers to expand ); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 74 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 75. Inventory/Entity Properties Monitor To monitor for changes to VM hardware: • Create ListView with the MoRefs of all the VMs • Create the appropriate TraversalSpec to start from the created ListView • Monitor for changes to VM hardware using PropertyCollector vimPort.createListView ( serviceContent.getViewManager(), new ManagedObjectReference[]{}//MoRefs of //all the VMs ); TraversalSpec tSpec = new TraversalSpec(); tSpec.setType(“ListView”); tSpec.setPath(“view”); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 75 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 76. Inventory/Entity Properties Monitor PropertySpec pSpec = new PropertySpec(); pSpec.setType(“VirtualMachine”); pSpec.setPathSet(propertyArr);//Props String array ObjectSpec oSpec = new ObjectSpec(); oSpec.setObj(viewObj);//View object oSpec.setSelectSet(new SelectionSpec[]{tSpec}); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 76 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 77. A Brief Walkthrough of StatsFeeder 1. Select Entities/Metrics to Monitor 2. Keep Track of Inventory Changes 3. Collect Statistics 4. Present Statistics Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 77 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 78. Collecting Statistics Relevant Modules in StatsFeeder 1. PerfQuerySpec Cache 2. Stats Reader 3. Stats Receiver Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 78 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 79. PerfQuerySpec Architecture To grab counters: QueryPerf(PerfQuerySpec[] querySpec) PerfQuerySpec: Specifies which counters to grab Entity Format MetricId StartTime EndTime IntervalID maxSample (host, VM) (CSV, (20s, 300s) normal) PerfQuerySpec[]: [pQs1, pQs2, pQs3, …] • Array of PerfQuerySpec objects pQs1, pQs2, pQs2 • Can grab multiple stats using single QueryPerf call Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 79 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 80. Complexities of QueryPerf How Does vSphere Process QueryPerf(querySpec[])? 1. vCenter receives queryPerf request with querySpec[] 2. vCenter takes each querySpec one at a time 3. vCenter gets data for each querySpec before processing next one Options for querySpec[]: 1. 1 entry 1 stat or set of stats for a single entity (e.g., all CPU) 2. Multiple entries. Examples: pQs1 pQs2 pQs3 • Each entry for a different entity VM1,cpu.* VM2,cpu.* H3,mem.* … • Each entry for a different stat type, same entity VM1,cpu.* VM1,net.* VM1,mem.* Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 80 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 81. Implications of QuerySpec Format of QuerySpec Allows Multiple Client Options 1. Grab each stat one at a time 2. Grab a group of stats per entity at once 3. Grab all stats for all entities at once 4. Grab stats for a subset of entities at once Some Tradeoffs: 1. Network processing (large result sets vs. small result sets) 2. Client aggregation overhead 3. vCenter processing (Each QueryPerf handled in a single thread) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 81 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 82. PerfQuerySpec Cache Reuse PerfQuerySpec, as long as there are no changes to the inventory and changes to the hardware of the corresponding entity. Tricks: • Keep the PerfQuerySpec cache up to date using the monitoring module. • Use wild card for metric IDs to automatically account for hardware changes. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 82 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 83. PerfQuerySpec PerfQuerySpec pqSpec = new PerfQuerySpec(); pqSpec.setIntervalId(intervalId); pqSpec.setMaxSample(maxSamples); perfQuerySpec.setFormat(“csv”);//Set format to CSV PerfMetricId pmid = new PerfMetricId(); pmid.setCounterId(counterId); pmid.setInstance("*");//Wild card pqSpec.setMetricId(new PerfMetricId[]{pmid}); pqSpec.setEntity(entityMor); Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 83 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 84. Stats Reader Retrieves the performance data from the vCenter Server and various guest specific performance tools like xperf, perfmon, or top etc. Tip: • Use parallelism Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 84 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 85. Stats Reader - Using Parallelism class StatsReader implements Runnable { private PerfQuerySpec[] perfQuerySpecArr = null; public StatsReader(PerfQuerySpec[] pqsArr) { perfQuerySpecArr = pqsArr; } public void run() { if (perfQuerySpecArr != null) { vimPort.queryPerf (serviceContent.getPerfManager(), perfQuerySpecArr); } } } Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 85 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 86. Stats Receiver This is a simple interface to process the output data from the StatsReader. • By default CSVFileStatsReceiver and DefaultStatsReceiver are provided. CSVFileStatsReceiver – Writes the performance data in CSV format to a file DefaultStatsReceiver – Writes the performance data to the console. Users can simply implement this interface to create their own StatsReceivers to: • Write the performance data to a database. • Publish the data to a message queue. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 86 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 87. Stats Receiver /* All the receivers need to implement an IStatsReceiver interface that has ‘receiveStats’ method */ public void receiveStats(String entityName, PerfEntityMetricBase[] pembArr) { writeStatsToDB(entityName, pembArr); } Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 87 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 88. A Brief Walkthrough of StatsFeeder 1. Select Entities/Metrics to Monitor 2. Keep Track of Inventory Changes 3. Collect Statistics 4. Present Statistics Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 88 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 89. Present Data in Meaningful Way Relevant modules in StatsFeeder 1.Stats Interpreter 2.Stats Presenter Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 89 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 90. Stats Interpreter Interprets the performance data to identify potential problems. You can write your own rules engine to interpret the performance data and plug it in easily. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 90 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 91. Stats Presenter Interpret the data using your own rules and present it in your own way. • E.g.: Present the data in graphical format using histograms, charts etc. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 91 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 92. Stats Presenter Sample Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 92 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 93. Summary of Pieces Present data Sample Rules Engine Store data (CSV, DB, etc.) Collect Data Watch for inventory changes Maintain cache of querySpecs Retrieves metadata for entities to be monitored and stats to be collected Retrieves MoRefs of entities in config file Get entities to monitor, stats to retrieve, and collection interval from config file What counters are supported? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 93 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 94. Conclusion: Performance API Tips and Tricks Collect static data once • CounterIDs, metricIDs, MoRefs etc. • Use Views to keep this data up to date. • Reuse PerfQuerySpec as much as possible Use CSV format • Reduces serialization cost and the size of metadata Choose metrics and query intervals carefully • Query the real-time stats at a slower rate than the refresh rate • Query over small time increments • Choose correct stats levels Use parallelism (multi-threaded clients) Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 94 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 95. Resources for Developers StatsFeeder Source Code: http://communities.vmware.com/servlet/JiveServlet/downloadBody/ 11933-102-1-11789/StatsFeeder.zip Developer Community: http://communities.vmware.com/community/developer/forums/mana gementapi Code Samples: http://communities.vmware.com/community/developer/codecentral Blogs: http://blogs.vmware.com/developer Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 95 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 96. Q&A • Questions ??? • Please complete the survey. Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 96 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 97. Ready. Set. Go! Take Action GO! Get Educated • Visit Partner University for information on partner programs and resources • Access to VMware Partner Exchange presentations GO! Get Competent • Enroll in training and certification that counts toward earning competencies GO! Register Your Deals to Earn More Margin • Visit Partner Central to learn about VMware’s registration programs Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 97 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
  • 98. Any Questions? Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware 98 products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

×