This document discusses VMware performance troubleshooting. It covers topics like root cause analysis, performance characteristics of CPU, memory, disk and networking, and tools like ESXTop, vm-support and the service console. It provides guidelines on capacity planning, virtual machine optimization and design best practices.

1. VMware Performance TroubleshootingPresented by Chris Kranz

2. Topics CoveredIntroductionRoot Cause AnalysisPerformance CharacteristicsCPUNetworkingMemoryDiskVirtual Machine optimisationESXTopvm-supportService ConsoleResource GroupsDesign GuidelinesCapacity Planner limitations and cautionsConclusionReference Articles

3. IntroductionMultiple layers of virtualisation are used to increase service levels, availability and manageabilityHowever, multiple layers of virtualisation often mask performance and configuration issues making it more of a challenge to troubleshoot and correctThe worst out come is that performance issues after a virtualisation project lead to the perception that VMware results in reduced performance and future confidence in VMware can be affected

4. Virtual Machine ResourcesCPUMemoryDiskNetworkingPerformance Basics

5. Resource Maximumshttp://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

6. Typical HostTypical 3 VMs per core, 24VM’s per HostEach has 2GB of RAM = 48GB of RAM

7. Root Cause Analysishttp://www.vmware.com/resources/techresources/10066

8. Root Cause ...

9. Do not rely on guest tools, butCan show high CPU, & Memory UtilisationMeasurement of Latency & throughput of Disk & Network InterfacesUse the virtualisation layer, to diagnose cause:Guest is unaware of virtualisation workloadThe way in which guest OS’s account time is differentNo visibility of available resourcesMonitoring Performance

10. esxtop (service console only)resxtop (remote command line utilities)Performance graphs in vCentrePerformance Analysis Tools

11. esxtop can be run:Interactively Batch (eg. esxtop -a -b > analysis.csv)Load batch into windows perfmon or MS ExcelTwo keys to rememberH : helpF : fields to displayesxtop

12. esxtop basicsHost ResourcesName of Resource Pool, Virtual Machine or WorldNumber of Worlds

13. Performance CharacteristicsCPUNetworkingMemoryDiskSlow ProcessingHigh CPU WaitPacket LossSlow NetworkSlow ProcessingDisk SwappingLog StallsDisk QueueSlow Application PerformanceReduced User ExperienceData Loss and Corruption

14. CPUESX SchedulerBasic World StatesRead / Run / WaitCPU StatesReady / Usage / WaitServiceConsoleVirtualMachineLimits / Shares / Reservations

15. CPUHigh %RDY + High %User can imply over commitmentesxtopPCPU(%): CPU utilization

16. %USED: Utilization

17. %RDY: Ready Time

18. %RUN: Run Time

19. %WAIT: Wait and idling timeCPUVI-ClientUsed Time > Ready Time: Possible CPU over-committmentUsed TimeReady Time

20. CPUFurther Investigation%MLMTD shows this VM has been limited

21. CPUFurther InvestigationHigh ready time caused by CPU resource limit

22. VMware Memory Management Transparent Page Sharing

23. VMware Tools Balloon Driver to force the VM to swap to disk

24. Virtual Machine Page FileMemoryBallooning vs. SwappingBallooning driver causes the host to swap pages that it chooses to diskESX Swapping will swap any pages to disk.

25. Ballooning can be disabled (0 value) or controlled on a per Virtual Machine basis using:sched.mem.maxmemctlDefault is set to 65%, can be controlled at host level.Only is an issue in resource contention scenarios. (or VM’s with low latency eg Citrix)Memory

26. Memory - HostVI Client shows memory usage of the host. This is calculated as “consumed + overhead memory + Service Console”.Performance charts are a very good way of showing the Virtual Machine memory breakdown. Consumed Memory

27. Ballooned Memory

28. Shared Memory

29. Swapped MemoryMemory - GuestHost Memory = Consumed + Overhead MemoryGuest Memory = Active Memory for Guest OS

30. Memory – Guest Overhead

31. MemoryVirtual Machine Memory Metrics – VI Client

32. MemoryHost Memory Metrics – VI Client

33. MemoryPMEM: Total physical memory breakdownVMKMEM: Memory managed by vmkernelCOSMEM: Service Console memory breakdownPSHARE: Page sharing statisticsSWAP: Swap statisticsMEMCTL: Balloon driver dataesxtop

34. Memoryesxtop / VI Client metrics : Virtual Machines

35. Memoryesxtop / VI Client metrics : Host Usage

36. MemoryVI Client memory usage graph

37. MemoryTroubleshooting Memory usage issues

38. NetworkingSwitch Assisted Teaming (IP Hash)

39. VLAN Trunking

40. Flow Control (full)

41. Speed & Duplex (1000Mb / Full)

42. Port Fast

43. BPDU Disabled

44. STP Disabled

45. Link State Tracking

46. Jumbo FramesNetwork configuration is more likely to blame than resource contention

47. NetworkingesxtopTransmit and Receive in Mb/sTransmit and Receive in Packets

48. NetworkingesxtopDropped Packets TransmitDrop Packets Received

49. DiskVarying Factors File system performance

50. Disk subsystem configuration (SAN, NAS, iSCSI, local disk)

51. Disk caching

52. Disk formats (thick, sparse, thin)ESX Storage StackDifferent latencies for different disks

53. Queuing within the kernelK: KernelD: DeviceG: Guest

54. DiskVI Client statisticsQuite Coarse Statistics Disk read / write rate (KB/s)