VMware vSphere Performance Troubleshooting

vSphere Performance Monitoring and Troubleshooting
Overview
What?
CPU, Memory, Disk, Network
How?
Use available tools and a systematic methodology
Why?
Need to build confidence in virtualizing critical and high demand applications

Top Issues
Top Issues:
Storage "performance capacity" oversubscription
Memory oversubscription
SMP overuse
Firmware & driver issues

What tools do we have at our disposal?
Top tools for information collection:
vCenter - Performance charts and alarms
Guest OS* - Task Manager/Resource Monitor and PerfMon
ESX Host - esxtop and vscsiStats
vSpherePowerCLI
*Guest based monitoring is subject to inaccuracy

Prepare vCenter Settings

Prepare custom vCenter alerts:
Host Console Swap In Rate  512KBps Warning, 1024 KBps Alert
Host Console Swap Out Rate  512KBps Warning, 1024 KBps Alert
VM CPU Ready  1000ms Warning, 2000ms Alert
VM Disk Latency  20ms Warning, 50ms Alert

Prepare esxtop
ESXTOP realtime monitoring:
esxtop(run command from SSH or tech-support mode)
s 2 (refresh view every 2 seconds)
V (View VMs only)
h(for quick in-tool command reference)
Batch Mode for a 5 minute capture of all stats:
esxtop-b -a -d 2 -n 150 > esxtop_capture.csv

Prepare PowerCLI
Run PowerCLI:
Tip: Run as Administrator
Set-ExecutionPolicyremotesigned
Connect-VIServer -Server <host> -Protocol https -User <user> -Password <pass>
<host> can be IP address or name of ESX server or vCenter
Get-VM
Get-Stat -common -realtime

Where do we get started?

Network Overview

Network
Troubleshooting Guidance:
1. Physical Issues - A bad cable, a failing switch port or NIC, or an incompatible/flawed firmware or device driver (use VMXNET3 whenever possible)
2. Configuration Issues - Inconsistent configuration of vSwitches, Port Groups, or upstream VLAN trunks
3. Capacity Issues - Too many VMs on a single NIC; inadequate switch backplane or uplink capacity; sharing “unmanaged” network infrastructure for storage and data
4. Thresholds – Bandwidth saturation, dropped packets

Network – What can we see?

Network
vCenter Metrics:
Receive packets dropped
Transmit packets dropped

Network
ESXTOP Metrics:

Network
ESXTOP Commands:
esxtop
s 2
n
f

Network
ESXTOP Example:

Network
PowerCLI Commands:
Get-Stat -net -realtime
Get-Stat -Entity <Host> -stat net.droppedRx.summation
Get-Stat -Entity <Host> -stat net.droppedTx.summation

Network – What can’t we see?

Network
Possible resources for external monitoring:
Native Telnet/SSH/HTTP-based interface counters and stats
Third-party SNMP, NetFlow and ICMP tools

CPU Overview

CPU
1. Physical Issues - Rare and always catastrophic (e.g. obvious)
2. Configuration Issues - Too many / too few vCPUs per VM; SMP/HAL mismatch; incorrect CPU affinity settings
3. Capacity Issues - CPU saturation at the guest or host level; CPU starvation due to high IO or other system level ops
4. Thresholds – Waiting for CPU cycles (due to co-scheduling, swapping, high IO)

CPU – What can we see?

CPU
vCenter Metrics:
Host/Guest Saturation
Stacked Graph (per VM)
Usage

CPU
vCenter Metrics:
Guest
Ready (value/20=n%)
Swap Wait

CPU
ESXTOP Metrics:

CPU
ESXTOP Commands:
esxtop
s 2
V
c
e GID (expand/contract a VM world)

CPU
ESXTOP Example:
Excessive vCPUs

CPU
ESXTOP Example:
Now with fewer vCPUs

CPU
ESXTOP Example:
SMP impacting multiple VMs

CPU
PowerCLI Example
Get-Stat -cpu
Get-Stat -Entity <VM> -stat cpu.ready.summation -realtime
Very cool script code at:
http://www.peetersonline.nl/index.php/vmware/examine-vmware-cpu-ready-times-with-powershell/

CPU – Not much else to see…

CPU
Possible resources for external monitoring:
Vendor specific systems management tools,
MS System Center, etc.
http://www.peetersonline.nl/index.php/vmware/examine-vmware-cpu-ready-times-with-powershell/

Memory Overview

Memory
1. Physical Issues - Rare and usually catastrophic
2. Configuration Issues - Memory overcommit; incorrect configuration of shares, reservations or limits
3. Capacity Issues - Physical memory exhaustion
4. Thresholds – Active memory swapping

Memory – What can we see?

Memory
vCenter Metrics
Swap in rate
Swap out rate
Swap used

Memory
ESXTOP Metrics:

Memory
ESXTOP Commands:
esxtop
s 2
V
m
f

Memory
ESXTOP Example:
m – Heavy swapping and ballooning

Memory
PowerCLI Commands:
Get-Stat -mem
Get-Stat -Entity <VM> -stat mem.swapoutRate.average -realtime
Get-Stat -Entity <VM> -stat mem.swapinRate.average -realtime
Get-Stat -Entity <VM> -stat mem.vmmemctl.average -realtime
Get-Stat -Entity <Host> -stat mem.swapused.average -realtime

Memory – The occasional DIMM failure…

Memory
Possible external monitoring options:
Vendor specific systems management tools, MS System Center, etc.
Don’t forget vCenter ‘Hardware Status’ reporting

Storage Overview

Storage
1. Physical Issues - A bad cable, a failing switch port or HBA/NIC, or an incompatible/flawed firmware or device driver (use LSI Logic Parallel/SAS as appropriate)
2. Configuration Issues - Inconsistent or incorrect configuration of LUN masking, zoning, or multi-pathing; inappropriate resource provisioning; aligning queue depth with storage type
3. Capacity Issues - Too many VMs or VMDKs on a LUN; too much IO load for an array or RAID group
4. Thresholds – Latency and queuing

Storage – What can we see?

Storage
vCenter Metrics:
Datastore
Read latency
Write latency

Storage
ESXTOP Metrics:

Storage

Storage
ESXTOP Commands (HBA/LUN):
esxtop
s 2
V
d
f
e vmhba#

Storage
ESXTOP Commands(LUN/Datastore):
esxtop
s 2
V
u
L 38
f
e <devname>

Storage
ESXTOP Commands (VM/VMDK):
esxtop
s 2
V
v
f
e GID

Storage
ESXTOP Examples:
d - Multipathing / Expand adapter to view targets

Storage
ESXTOP Examples:
u - Queuing, Disk or Kernel?

Storage
ESXTOP Examples:
v - Identify the IO consumer

Storage
vscsiStatsCommand:
[root@host ~]# cd /usr/lib/vmware/bin
./vscsiStats -l
./vscsiStats -s -w <worldid>
./vscsiStats -w <worldid> -p all -c > /path/vscsistats.csv
./vscsiStats -x

Storage
vscsiStatsExample:

Storage
vscsiStatsExample:
http://dunnsept.wordpress.com/2010/03/11/new-vscsistats-excel-macro/

Storage
vscsiStatshistograms:

Storage
PowerCLI Commands:
Get-Stat -disk
Get-Stat -stat disk.totalLatency.average -realtime
Get-Stat -stat disk.deviceLatency.average -realtime
Get-Stat -stat disk.kernelLatency.average -realtime

Storage – What can’t we see?

Storage – More of what we can’t see

Storage
Possible external monitoring solutions:
Vendor specific SAN and fabric/network tools, native Telnet/SSH/HTTP-based tools for most networks, third-party SNMP-based tools

Working with PowerCLI
PowerCLI Tips:
For a complete list of stat objects:
Get-StatType -Entity <Host/VM>
Pipe the outputs to a file:
Get-Stat -stat <stat> -realtime | ft -autosize > c:temp<filename>.csv
Import the CSV file data to a spreadsheet with fixed width parameters
Build pretty graphs

Working with PowerCLI

Way More Information
ESXTOP / vscsiStats / PowerCLI:
http://www.yellow-bricks.com/esxtop/ Special thanks to Duncan Epping!
http://communities.vmware.com/docs/DOC-3930
http://www.vmware.com/support/developer/PowerCLI/PowerCLI41/html/Get-Stat.html
http://www.lucd.info/2009/12/30/powercli-vsphere-statistics-part-1-the-basics/
http://simongreaves.co.uk/blog/esxtop-guide
http://dunnsept.wordpress.com/2010/03/11/new-vscsistats-excel-macro/

Easy button?
What is the problem with these tools?
Limited alerting mechanisms, no collection automation or historical data for comparison, and no correlation of events!
vCenter Operations Standard / Enterprise

http://blog.lewan.com	757
http://feeds.feedburner.com	7
http://translate.googleusercontent.com	2
http://paper.li	1

VMware vSphere Performance Troubleshooting

by dbrinkmann on Aug 25, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 767

Statistics

VMware vSphere Performance Troubleshooting — Presentation Transcript