Getting the Most out of Your VMware Environment with Performance Tuning Mike Burke Virtual Infrastructure Practice Director January 2009

What is Performance Tuning?

What is Performance Tuning? Proper planning and design Configuration best practices Operational awareness X

Proper planning and design

Configuration best practices

Operational awareness

Start with the “Four Core” But first!

A Few Questions… How many CPUs per host? How much memory per host? Ratio of memory to CPU? 4:1 ? 2:1 ? What’s the “Best Practice”?

How many CPUs per host?

How much memory per host?

Ratio of memory to CPU?

A Simple Answer “ Best Practices” are subjective Tuning is not a replacement for proper planning Don’t guess – gather the data Know your workloads Are you CPU-bound or memory-bound? Spec hardware based on need

“ Best Practices” are subjective

Tuning is not a replacement for proper planning

Don’t guess – gather the data

Know your workloads

Are you CPU-bound or memory-bound?

Spec hardware based on need

Optimizing CPU More cores are generally better Use vSMP sparingly Consider workload needs Should it be virtualized? Consider vSMP scheduling constraints ESX must schedule all VCPUs at once Re-evaluate 4-CPU candidates

More cores are generally better

Use vSMP sparingly

Consider workload needs

Should it be virtualized?

Consider vSMP scheduling constraints

ESX must schedule all VCPUs at once

Re-evaluate 4-CPU candidates

Memory Optimization Generally, more host RAM = better performance Configure a VM’s memory based on the actual need. Tune migrated systems accordingly Be careful over-allocating RAM No swapping!

Generally, more host RAM = better performance

Configure a VM’s memory based on the actual need.

Tune migrated systems accordingly

Be careful over-allocating RAM

No swapping!

Advanced Memory Parameters Mem.ShareScanTime Time interval for TPS scanning (60 min) Mem.ShareScanGhz Amount of memory pages to scan per 1Ghz idle cycles (4MB/sec per 1Ghz) Mem.CtlMaxPercent Limits memory reclamation by ballooning (vmmemctl) (65%)

Mem.ShareScanTime

Time interval for TPS scanning (60 min)

Mem.ShareScanGhz

Amount of memory pages to scan per 1Ghz idle cycles (4MB/sec per 1Ghz)

Mem.CtlMaxPercent

Limits memory reclamation by ballooning (vmmemctl) (65%)

Advanced Memory Parameters Per VM: sched.mem.maxmemctl Max memory reclamation by ballooning (vmmemctl) sched.mem.pshare.enable Enable /disable memory sharing (TPS) sched.swap.persist vSwap persists after power-off

Per VM:

sched.mem.maxmemctl

Max memory reclamation by ballooning (vmmemctl)

sched.mem.pshare.enable

Enable /disable memory sharing (TPS)

sched.swap.persist

vSwap persists after power-off

Optimizing Storage Use shared storage Fiber-Channel SAN iSCSI NFS ESX supports dynamic failover… MRU only for Active/Passive … but manual load balancing Provision alternately/accordingly

Use shared storage

Fiber-Channel SAN

iSCSI

NFS

ESX supports dynamic failover…

MRU only for Active/Passive

… but manual load balancing

Provision alternately/accordingly

Optimizing Storage RAID Types Balance performance vs. cost Know your workload Use iSCSI HBA’s on production Offload CPU time to hardware Jumbo frames

RAID Types

Balance performance vs. cost

Know your workload

Use iSCSI HBA’s on production

Offload CPU time to hardware

Jumbo frames

Storage Recommendations Tier storage based on need Tier 1 – DMX/Fiber Tier 2 – iSCSI/SATA ISO/Templates – NFS, iSCSI or SATA Tier storage based on environment Split PROD and DEV/UAT Split based on DR plans System, data, swap sched.swap.dir

Tier storage based on need

Tier 1 – DMX/Fiber

Tier 2 – iSCSI/SATA

ISO/Templates – NFS, iSCSI or SATA

Tier storage based on environment

Split PROD and DEV/UAT

Split based on DR plans

System, data, swap

sched.swap.dir

Storage Recommendations LUN sizing VMs per LUN Average size of disk Remember vSwap and headroom One VMFS per LUN Align VMFS volumes More, smaller LUNs vs. less, larger LUNs

LUN sizing

VMs per LUN

Average size of disk

Remember vSwap and headroom

One VMFS per LUN

Align VMFS volumes

More, smaller LUNs vs. less, larger LUNs

VMFS Recommendations Limit hosts per LUN Locking issues Do not use extents Use SMotion Use RDM’s for disks over a certain size Draw a line in the sand Unload VMFS2 drivers vmkload_mod –u vmfs2

Limit hosts per LUN

Locking issues

Do not use extents

Use SMotion

Use RDM’s for disks over a certain size

Draw a line in the sand

Unload VMFS2 drivers

vmkload_mod –u vmfs2

Networking Gigabit only Negotiate auto/auto in most instances Know your settings Four NICs minimum Enable spanning-tree PortFast

Gigabit only

Negotiate auto/auto in most instances

Know your settings

Four NICs minimum

Enable spanning-tree PortFast

Networking Dedicate NICs to iSCSI/NFS Second Service Console interface Software iSCSI load-balancing Only possible with EtherChannel

Dedicate NICs to iSCSI/NFS

Second Service Console interface

Software iSCSI load-balancing

Only possible with EtherChannel

Networking Always more than one NIC per vSwitch Different physical switches Different transceivers Check PCI addressing for VMNIC ordering Combine Service Console and VMkernel LB at the port group level Add third dedicated to failover Ensure consistency in port usage Security conscious: Isolate VMotion

Always more than one NIC per vSwitch

Different physical switches

Different transceivers

Check PCI addressing for VMNIC ordering

Combine Service Console and VMkernel

LB at the port group level

Add third dedicated to failover

Ensure consistency in port usage

Security conscious: Isolate VMotion

Networking Place high-I/O VMs on the same: Host vSwitch Port group

Place high-I/O VMs on the same:

Host

vSwitch

Port group

HA Recommendations Ensure the Service Console has redundant paths PINGable gateway Alternate source Watch memory over-commitment Understand admission control No strict admission control in 2-node clusters All names/IPs registered in DNS

Ensure the Service Console has redundant paths

PINGable gateway

Alternate source

Watch memory over-commitment

Understand admission control

No strict admission control in 2-node clusters

All names/IPs registered in DNS

HA Advanced Parameters das.isolationaddress Address to PING for failure detection das.defaultfailoverhost Which host should be the target of the failover das.failuredetectiontime How long to wait to initiate failure detection das.failuredetectioninterval How long between heartbeats

das.isolationaddress

Address to PING for failure detection

das.defaultfailoverhost

Which host should be the target of the failover

das.failuredetectiontime

How long to wait to initiate failure detection

das.failuredetectioninterval

How long between heartbeats

DRS Recommendations Remember – CPU and Memory only Use affinity rules Specify exempt VMs Place appropriate VMs together Separate conflicting VMs Go Fully-Automated with HA But, conservative migration threshold

Remember – CPU and Memory only

Use affinity rules

Specify exempt VMs

Place appropriate VMs together

Separate conflicting VMs

Go Fully-Automated with HA

But, conservative migration threshold

A Word on DRS and HA HA relies on reservations to determine available capacity Set at the VM level das.vmMemoryMinMB Catch-all memory reservation to consider when failover occurs das.vmCpuMinMHz Catch-all CPU reservation to consider when failover occurs X

HA relies on reservations to determine available capacity

Set at the VM level

das.vmMemoryMinMB

Catch-all memory reservation to consider when failover occurs

das.vmCpuMinMHz

Catch-all CPU reservation to consider when failover occurs

Shares, Reservations and Limits Reservations are hard commits Shares are relative priority Limits create sandboxes

Reservations are hard commits

Shares are relative priority

Limits create sandboxes

General Host Recommendations Mix workloads Better overall utilization Leverage reservations for critical systems Particularly in HA scenarios Limit 3 rd party tools in the Service Console Antivirus If you must, remove /vmfs from scan list Configure NTP

Mix workloads

Better overall utilization

Leverage reservations for critical systems

Particularly in HA scenarios

Limit 3 rd party tools in the Service Console

Antivirus

If you must, remove /vmfs from scan list

Configure NTP

General VM Recommendations Always load VMware Tools Disable peripherals note used COM, LPT, Floppy, etc. Time sync VMware Tools or NTP Disable Sync driver on Terminal Server and Database VMs (vmmemctl) “ Can” screen savers and backgrounds Unmap unused CD-ROMs/ISOs

Always load VMware Tools

Disable peripherals note used

COM, LPT, Floppy, etc.

Time sync

VMware Tools or NTP

Disable Sync driver on Terminal Server and Database VMs (vmmemctl)

“ Can” screen savers and backgrounds

Unmap unused CD-ROMs/ISOs

Questions?

Thanks! Mike Burke Virtual Infrastructure Practice Director [email_address] January 2009

Key Counters CPU Usage (% and MHz) % Ready/CPU Ready ESXTOP/VC, per VM Memory Usage % Memory Consumed Memory Granted Memory Balloon Memory Swap Used Memory Shared

CPU Usage

(% and MHz)

% Ready/CPU Ready

ESXTOP/VC, per VM

Memory Usage %

Memory Consumed

Memory Granted

Memory Balloon

Memory Swap Used

Memory Shared

Key Counters Disk Usage (KBps) VMFS Volume Free Space VC database or ESX host directly Network Usage (KBps) Host and per-VMNIC

Disk Usage (KBps)

VMFS Volume Free Space

VC database or ESX host directly

Network Usage (KBps)

Host and per-VMNIC

ESX Host Monitoring Virtual Center Changing graph timescale changes sample size Change collection intervals Increase thread count esxtop Be sure to change modes (c - cpu, d - disk, m - memory) %RDY is a red flag VM’s are spending time in a queue waiting to be scheduled Swapping stats Out of RAM? Consistent CPU times > 80% Need another host? Disk queue lengths Read vs. Write characteristics What disk I/O loads are in competition on the SAN?

Virtual Center

Changing graph timescale changes sample size

Change collection intervals

Increase thread count

esxtop

Be sure to change modes (c - cpu, d - disk, m - memory)

%RDY is a red flag

VM’s are spending time in a queue waiting to be scheduled

Swapping stats

Out of RAM?

Consistent CPU times > 80%

Need another host?

Disk queue lengths

Read vs. Write characteristics

What disk I/O loads are in competition on the SAN?

Virtual Machine Monitoring Establish a baseline with Performance Monitor Understand what “normal” is before trouble starts What are your SLA’s? What is expected/acceptable performance? Average/Peak CPU Utilization Processor Queue Length Memory Usage Peak Memory Usage Page Faults I/O Reads vs. Write I/O Bytes (Read and Write) Disk Queue Lengths Network Bytes Received/Sent

Establish a baseline with Performance Monitor

Understand what “normal” is before trouble starts

What are your SLA’s?

What is expected/acceptable performance?

Average/Peak CPU Utilization

Processor Queue Length

Memory Usage

Peak Memory Usage

Page Faults

I/O Reads vs. Write

I/O Bytes (Read and Write)

Disk Queue Lengths

Network Bytes Received/Sent

Generally Speaking… CPU and Memory will be constrained Disk and network saturation uncommon Keep overall CPU utilization under 65% Keep Memory utilization under: 100% for production systems 125% for development/test systems

CPU and Memory will be constrained

Disk and network saturation uncommon

Keep overall CPU utilization under 65%

Keep Memory utilization under:

100% for production systems

125% for development/test systems