This document provides an overview of using esxtop and resxtop tools for advanced performance troubleshooting on VMware ESX hosts. It discusses esxtop counters, how they are derived from raw VMkernel counters, and how counter values are displayed. It also covers esxtop batch mode for generating perfmon compatible logs, using resxtop to connect to remote ESX hosts, and how esxtop can consume non-trivial CPU resources on large environments. Specific examples are given for troubleshooting CPU and memory problems using the tools.
Master VMware Performance and Capacity ManagementIwan Rahabok
12 Sep 2016 update: See this http://virtual-red-dot.info/operationalize-sddc-program-2/ for details.
-------------
Based on the book http://virtual-red-dot.info/performance-and-capacity-management/
Master performance and capacity management of VMware SDDC
Master VMware Performance and Capacity ManagementIwan Rahabok
12 Sep 2016 update: See this http://virtual-red-dot.info/operationalize-sddc-program-2/ for details.
-------------
Based on the book http://virtual-red-dot.info/performance-and-capacity-management/
Master performance and capacity management of VMware SDDC
How to Optimize Microsoft Hyper-V Failover Cluster and Double PerformanceStarWind Software
High availability in a virtualized workload may require to sacrifice failover cluster performance. Using an optimized for virtualization approach on data storage and virtual machines placement and protection will give you desired boost of performance.
The presentation shows how to:
- Achieve true Hyper-V cluster high availability with just 2 Hyper-V hosts and zero storage hardware
- Boost Hyper-V cluster performance by configuring automatic dynamic optimization
- Effectively track VMs resources usage
- Save an extra 30% of Hyper-V cluster resources by utilizing agentless antivirus
webinar vmware v-sphere performance management Challenges and Best PracticesMetron
With the majority of businesses using internal Cloud Services, whether it be Software as a Service (SaaS), Platform as a Service (PaaS) or Infrastructure as a Service (IaaS) in a VMware vSphere environment, this presentation gives an insight into how to manage the gathering Storm Clouds. After an introduction to VMware's Virtual Infrastructure 4 (vSphere) environment andCloud Computing, we discuss how Capacity Management provides the means to spot potential Storm Clouds far in advance and more specifically, how you can avoid them.
Delving deeper we look at IaaS and how to identify potential capacity on demand issues. Discussion focuses on topics such as:
•identifying whether virtual machines are under or over provisioned
•the advantages/disadvantages of application sizing
•how to minimize SLA impact
•whether to scale the infrastructure out, up or in and ultimately how to get it right.
Typically organizations have adopted a "silo mentality" whereby they ring fence IT systems and don’t share resources through lack of trust and confidence. We look at the advantages virtualization brings in terms of flexibility, scalability, cost reduction (monetary and environmental) and how we can protect our 'loved ones' through resource pools, shares, reservations and limits.
With all this in mind, join us to find out what information and processes we recommend you need to have and implement to avoid an Internal Storm and ensure that Brighter Outlook!
Introduction - vSphere 5 High Availability (HA)Eric Sloof
VMware HA clusters enable a collection of ESXi hosts to work together so that, as a group, they provide higher levels of availability for virtual machines than each ESXi host could provide individually. When you plan the creation and usage of a new VMware HA cluster, the options you select affect the way that cluster responds
to failures of hosts or virtual machines.
Авторский учебный курс от Архитектора Microsoft Алексея Кибкало.
Что такое Hyper-V
Версии Windows Server 2012 Hyper-V
Аппаратные требования к Windows Server 2012 Hyper-V
Установка Hyper-V
Сетевые возможности Windows Server 2012 Hyper-V
Что такое Live Migration
Высокодоступные кластеры Windows Server 2012 Hyper-V
Аварийное восстановление и Hyper-V Replica
Азы управления при помощи System Center
При поддержке "Звезды и С" www.stars-s.ru
Hypervisors and Virtualization - VMware, Hyper-V, XenServer, and KVMvwchu
With co-presenter Maninder Singh, delivered a presentation about hypervisors and virtualization technology for an independent topic study project for the Operating System Design (EECS 4221) course at York University, Canada in October 2014.
Virtualization, briefly, is the separation of resources or requests for a service from the underlying physical delivery of that service. It is a concept in which access to a single underlying piece of hardware is coordinated so that multiple guest operating systems can share a single piece of hardware, with no guest operating system being aware that it is actually sharing anything at all.
How to Optimize Microsoft Hyper-V Failover Cluster and Double PerformanceStarWind Software
High availability in a virtualized workload may require to sacrifice failover cluster performance. Using an optimized for virtualization approach on data storage and virtual machines placement and protection will give you desired boost of performance.
The presentation shows how to:
- Achieve true Hyper-V cluster high availability with just 2 Hyper-V hosts and zero storage hardware
- Boost Hyper-V cluster performance by configuring automatic dynamic optimization
- Effectively track VMs resources usage
- Save an extra 30% of Hyper-V cluster resources by utilizing agentless antivirus
webinar vmware v-sphere performance management Challenges and Best PracticesMetron
With the majority of businesses using internal Cloud Services, whether it be Software as a Service (SaaS), Platform as a Service (PaaS) or Infrastructure as a Service (IaaS) in a VMware vSphere environment, this presentation gives an insight into how to manage the gathering Storm Clouds. After an introduction to VMware's Virtual Infrastructure 4 (vSphere) environment andCloud Computing, we discuss how Capacity Management provides the means to spot potential Storm Clouds far in advance and more specifically, how you can avoid them.
Delving deeper we look at IaaS and how to identify potential capacity on demand issues. Discussion focuses on topics such as:
•identifying whether virtual machines are under or over provisioned
•the advantages/disadvantages of application sizing
•how to minimize SLA impact
•whether to scale the infrastructure out, up or in and ultimately how to get it right.
Typically organizations have adopted a "silo mentality" whereby they ring fence IT systems and don’t share resources through lack of trust and confidence. We look at the advantages virtualization brings in terms of flexibility, scalability, cost reduction (monetary and environmental) and how we can protect our 'loved ones' through resource pools, shares, reservations and limits.
With all this in mind, join us to find out what information and processes we recommend you need to have and implement to avoid an Internal Storm and ensure that Brighter Outlook!
Introduction - vSphere 5 High Availability (HA)Eric Sloof
VMware HA clusters enable a collection of ESXi hosts to work together so that, as a group, they provide higher levels of availability for virtual machines than each ESXi host could provide individually. When you plan the creation and usage of a new VMware HA cluster, the options you select affect the way that cluster responds
to failures of hosts or virtual machines.
Авторский учебный курс от Архитектора Microsoft Алексея Кибкало.
Что такое Hyper-V
Версии Windows Server 2012 Hyper-V
Аппаратные требования к Windows Server 2012 Hyper-V
Установка Hyper-V
Сетевые возможности Windows Server 2012 Hyper-V
Что такое Live Migration
Высокодоступные кластеры Windows Server 2012 Hyper-V
Аварийное восстановление и Hyper-V Replica
Азы управления при помощи System Center
При поддержке "Звезды и С" www.stars-s.ru
Hypervisors and Virtualization - VMware, Hyper-V, XenServer, and KVMvwchu
With co-presenter Maninder Singh, delivered a presentation about hypervisors and virtualization technology for an independent topic study project for the Operating System Design (EECS 4221) course at York University, Canada in October 2014.
Virtualization, briefly, is the separation of resources or requests for a service from the underlying physical delivery of that service. It is a concept in which access to a single underlying piece of hardware is coordinated so that multiple guest operating systems can share a single piece of hardware, with no guest operating system being aware that it is actually sharing anything at all.
VMworld 2015: Virtualize Active Directory, the Right Way!VMworld
Active Directory Domain Services (ADDS) allows organizations to deploy a scalable and secure directory service for managing users, resources and applications. Virtualization of ADDS has been supported for many years now, however has required careful management to avoid pitfalls around replication, time management, and access. Windows Server 2012 provides greater support for virtualization by including virtualization-safe features and support for rapid domain controller deployment.
VMworld 2017 - Top 10 things to know about vSANDuncan Epping
In this session Cormac Hogan and I go over the top 10 things to know about vSAN. This is based on two years of questions/answers from our field and customers. Useful for any VMware vSAN customer!
#STO1264BU #STO1264BE
A look at the new enhancements to core storage in vSphere 6.5, including VMFS6, Automated UNMAP, I/O Filters, and much more, as delivered by Cormac Hogan and Cody Hosterman
VMworld 2015: Extreme Performance Series - vSphere Compute & MemoryVMworld
In this session we'll dive deep into how the vSphere compute and memory schedulers work to provide the same level of performance as bare metal. Hosted by two outstanding performance engineers, they will review concepts like how and when vSphere schedules vCPUs, how virtual machines are idles, understand virtual machine memory overhead and how large memory pages help or hurt performance. If you want to understand what vSphere does at an atomic level you don't want to miss this advanced session.
The needs for immediate responsiveness of VMs in the virtualized environments have been on the rise. Several services in SKT also require soft realtime support for virtual machines to substitute the physical machines to achieve high utilization and adaptability. However, consolidated multiple OSes and irregular external events might render the hypervisor infringe on a VM's promptitude. As a solution of this problem, we are improving Xen's credit scheduler by introducing the RT_PRIORITY that guarantees a VM's running at any given point in time as long as credits remains to be burn. It would increase the quality of service and make a VM's behavior predictable on the consolidated environment. In addition, we extend our suggestion to the multi-core environment and even a large number of physical machines by using live migrations.
VMworld 2013
Peter Boone, VMware
Seongbeom Kim, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Amazon EC2 provides a broad selection of instance types to deliver high performance for a diverse mix of applications. In this session, we overview the drivers of system performance and discuss in depth how Amazon EC2 instances deliver system performance while also providing elasticity and complete control over your infrastructure. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Linux power management: are you doing it right?Chris Simmonds
Devices operating on battery power need effective power management: anything you can do to reduce power usage will increase battery life. Even for devices running on mains power, better power managements has benefits in reducing the need for cooling and lower energy costs. This presentation describes the four principles of power management: don't rush if you don't have to; don't be ashamed of being idle; turn off things you are not using; and sleep when there is nothing else to do. Each of these has a counterpart in the Linux kernel.
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
The Roman Empire A Historical Colossus.pdfkaushalkr1407
The Roman Empire, a vast and enduring power, stands as one of history's most remarkable civilizations, leaving an indelible imprint on the world. It emerged from the Roman Republic, transitioning into an imperial powerhouse under the leadership of Augustus Caesar in 27 BCE. This transformation marked the beginning of an era defined by unprecedented territorial expansion, architectural marvels, and profound cultural influence.
The empire's roots lie in the city of Rome, founded, according to legend, by Romulus in 753 BCE. Over centuries, Rome evolved from a small settlement to a formidable republic, characterized by a complex political system with elected officials and checks on power. However, internal strife, class conflicts, and military ambitions paved the way for the end of the Republic. Julius Caesar’s dictatorship and subsequent assassination in 44 BCE created a power vacuum, leading to a civil war. Octavian, later Augustus, emerged victorious, heralding the Roman Empire’s birth.
Under Augustus, the empire experienced the Pax Romana, a 200-year period of relative peace and stability. Augustus reformed the military, established efficient administrative systems, and initiated grand construction projects. The empire's borders expanded, encompassing territories from Britain to Egypt and from Spain to the Euphrates. Roman legions, renowned for their discipline and engineering prowess, secured and maintained these vast territories, building roads, fortifications, and cities that facilitated control and integration.
The Roman Empire’s society was hierarchical, with a rigid class system. At the top were the patricians, wealthy elites who held significant political power. Below them were the plebeians, free citizens with limited political influence, and the vast numbers of slaves who formed the backbone of the economy. The family unit was central, governed by the paterfamilias, the male head who held absolute authority.
Culturally, the Romans were eclectic, absorbing and adapting elements from the civilizations they encountered, particularly the Greeks. Roman art, literature, and philosophy reflected this synthesis, creating a rich cultural tapestry. Latin, the Roman language, became the lingua franca of the Western world, influencing numerous modern languages.
Roman architecture and engineering achievements were monumental. They perfected the arch, vault, and dome, constructing enduring structures like the Colosseum, Pantheon, and aqueducts. These engineering marvels not only showcased Roman ingenuity but also served practical purposes, from public entertainment to water supply.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
2. Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
“THESE FEATURES ARE REPRESENTATIVE OF FEATURE AREAS UNDER
DEVELOPMENT. FEATURE COMMITMENTS ARE SUBJECT TO CHANGE, AND
MUST NOT BE INCLUDED IN CONTRACTS, PURCHASE ORDERS, OR SALES
AGREEMENTS OF ANY KIND. TECHNICAL FEASIBILITY AND MARKET
DEMAND WILL AFFECT FINAL.”
2
5. esxtop counters
1. esxtop does not create performance metrics
• esxtop derives performance metrics from raw counters exported in the
VMkernel System Info nodes (VSI nodes)
• esxtop can show new counters on older ESX system if the raw counters
are present in VMKernel
5
6. esxtop counters
2. Counter values
• Many raw counters have static values that do no change with time – esxtop
displays them as it is
• Many counters increment monotonically, esxtop reports the delta for these
for the given refresh interval – for instance CMDS/sec, packets
transmitted/sec etc
• %USED and %RUN - CPU occupancy delta between successive
snapshots
6
7. Refresh interval
3. Graphs will look different depending on the refresh interval
• Many counters values are dependent on refresh interval
• Larger refresh interval smoothens spikes and troughs
2 second refresh interval 10 second refresh interval
7
8. esxtop counters
4. Counter normalization
• By default counters are shown for the group
• In group view counters values are cumulative
• In expanded view, counters are normalized per entity
Cumulative
stats
Pressing ‘e’ key expands a group
vcpu world
consumes CPU
8
9. esxtop counters
5. %USED can exceed 100
• Turbo boost can increase the processor clock speed
• Asynchronous work can be happening on a different core on behalf of the
VM
VM on a NFS datastore running I/O intensive workload
9
10. esxtop batch mode
6. Batch mode (-b)
• Produces windows perfmon compatible CSV file
• CSV file compatibility requires fixed number of columns on every row -
statistics of VMs/worlds instances that appear after starting the batch mode
are not collected because of this reason
• Only counters that are specified in the configuration file are collected, (-a)
option collects all counters
• Counters are named slightly differently
10
15. I/O Latencies
7. IO latencies
• IO latencies are measured per SCSI command so it is not affected by
refresh interval
• Reported latencies are average values for all the SCSI commands issued
within the refresh interval window
• Reported average latencies can be different on different screens (adapter,
LUN, VM), since each screen accounts for different group of I/Os
15
16. resxtop – remote esxtop
8. You can use resxtop to connect to different ESX hosts
• Newer version of resxtop will connect to older ESX hosts
9. You don’t need root access to view esxtop counters
• resxtop can authenticate using vCenter credentials
16
17. esxtop CPU usage
10. esxtop can consume non-trivial amount of CPU
• When you have very large inventory (VMs, LUNs, virtual disks, virtual NICs
etc)
CPU consumption on a host with 512 VMs CPU usage when using resxtop
• You can limit the amount of data collected by limiting the fields (columns)
and entities (rows), you can also reduce CPU consumption by locking
entities, (-l) option
CPU consumption with esxtop -l
17
19. esxtop screens
Screens
• c: cpu (default)
• m: memory
VM VM VM VM
• n: network
• d: disk adapter
• u: disk device (added in ESX 3.5) CPU Memory Virtual
vSCSI
• v: disk VM (added in ESX 3.5) Scheduler Scheduler Switch
• i: Interrupts (new in ESX 4.0) c, i, p m n d, u, v
• p: power management (new in ESX
4.1)
VMkernel
19
21. CPU Constrained
SMP VM
High CPU
utilization
Both the
virtual CPUs
CPU
constrained
21
22. CPU Contention
4 CPUs, VMs don’t
3 SMP get to run %ready
all at VMs all the time accumulates
100%
22
23. CPU Limit
Max
Limited
CPU
Limit AMAX = -1 : Unlimited
23
24. Mis-configured SMP VM
vCPU 1 not Incorrect (UP) Kernel/HAL inside the
used by the guest or the application inside the
VM guest is single threaded
24
25. Power management – CPU frequency scaling
C states: C0 – busy, C1 – halted, C2 – deep halt
P states: P0 – Highest clock frequency, P11 – Lowest clock frequency
25
26. VM Power Usage
Experimental feature, not enabled by default.
VMkernel advanced setting: Power.ChargeVMs
26
27. CPU clock frequency scaling
VM is running all
the time but uses
only 75% of the
clock frequency
%USED: CPU usage with reference to base clock frequency
%UTIL: CPU utilization with reference to current clock frequency
%RUN: CPU scheduled time
27
28. Hyperthreading
Two VMs running
on different cores
Two VMs sharing
the same core
%LAT_C counter
shows the time de-
scheduled due to
core sharing
28
31. New metrics in CPU screen
%LAT_C : %time the VM was not scheduled due to CPU resource issue
%LAT_M : %time the VM was not scheduled due to memory resource issue
%DMD : Moving CPU utilization average in the last one minute
EMIN : Minimum CPU resources in MHZ that the VM is guaranteed to get
when there is CPU contention
31
33. esxtop memory screen (m)
Possible states:
high, soft, hard
and low
PMEM – Total Physical memory
VMKMEM - Memory managed by VMKernel
COSMEM - Memory used by Service Console
33
34. Not able to power-on a new VM
Memory reservation
820 MB
4G memory reservation
reservation requested
Overhead
memory
needs to be
reserved
34
35. Granted Memory
Granted Memory = Memory touched by the guest
Windows and FreeBSD Guests touches (zeroes) all its memory during boot
Linux Guests touches memory when it first uses it
35
36. Ballooning versus Swapping
VM with Swapped in
the past but Swap target is
Memory MCTL: N - Balloon Balloon
not actively more for the VM
Hog driver not active, tools driver swaps
swapping without the balloon
VMs probably not installed less
now driver
36
37. Memory Compression Stats
COWH : Copy on Write Pages hints – amount of memory in MB that are
potentially shareable
CACHESZ: Compression Cache size
CACHEUSD: Compression Cache currently used
ZIP/s, UNZIP/s: Memory compression/decompression rate
37
38. Wide NUMA - CPU
2 NUMA
nodes with
~6G each
NUMA home
node not assigned
4G, can fit into
a single node
6-vcpu VM –
cannot fit into
a NUMA node
size of 4
CPUs
38
39. NUMA affinity not set
NUMA machine
with 2 nodes
NHN: NUMA Home Node
All the memory in
remote node NLMEM: Memory in local node
NRMEM: Memory in remote node
CPU affinity set to
wrong NUMA node
39
40. Wide NUMA - Memory
2 NUMA
nodes with
~6G each
NUMA home
node not
assigned
VM cannot be
fit into a single
NUMA node
40
42. vSwitch active uplink
TEAM-PNIC : The uplink that the virtual switch port is currently using
42
43. Dropped packets at vSwitch
Packet drops usually happens when the traffic has
no flow control (UDP/Multicast/Broadcast packets)
43
44. Multicast/Broadcast stats
PKTTXMUL/s – Multicast packets transmitted per second
PKTRXMUL/s – Multicast packets received per second
PKTTXBRD/s – Broadcast packets transmitted per second
PKTRXBRD/s – Broadcast packets received per second
44
45. NFS stats
DAVG and KAVG is not available for network backed storage
GAVG – gives the end to end latency
45
47. Disk I/O latency
Host bus adapters (HBAs) - Latency stats from the
includes SCSI, iSCSI, RAID, Device, Kernel and the
and FC-HBA adapters Guest
DAVG/cmd - Average latency (ms) from the Device (LUN)
KAVG/cmd - Average latency (ms) in the VMKernel
GAVG/cmd - Average latency (ms) in the Guest
47
48. Problem with the disk subsystem
Good
throughput
Low device
Latency
Bad
throughput
Device Latency is
high - cache disabled
48
50. FC bottleneck
‘v’ – VM view
‘u’ – device view
‘d’ – adapter view
50
51. vStorage API for Array Integration (VAAI) stats
CLONE_RD, CLONE_WR: Number of Clone read/write requests
CLONE_F: Number of Failed clone operations
MBC_RD/s, MBC_WR/s – Clone read/write MBs/sec
ATS – Number of ATS commands
ATSF – Number of failed ATS commands
ZERO – Number of Zero requests
ZEROF – Number of failed zero requests
MBZERO/s – Megabytes Zeroed per second
51
52. VAAI - virtual disk creation example
vStorage API for Array Integration (VAAI)
52
55. Other diagnostic tools (1 of 2)
sched-stats and schedtrace
• vm-support -s/-S flag captures sched-stats
• vm-support -c flag captures scheduler trace – takes lot of disk space
memstats
• Provides detailed memory usage stats with resource pool hierarchy
ft-stats
• FT Virtual Machine stats
• Collected with vm-support –s/S flag
55
56. Other diagnostic tools (2 of 2)
swatchStats
• Stopwatch stats for VMFS, SCSI events
vscsiStats
• Virtual machine SCSI disk I/O stats
• Provides histogram information for latency, IO size, inter-arrival time and
outstanding I/Os
56
57. vscsiStats
# vscsiStats -l
World group Virtual Machine
leader id Name
Virtual scsi disk
handle ids -
unique across
virtual machines
57
58. vscsiStats – latency histogram
# vscsiStats -p latency -w 118739 -i 8205
Latency in
I/O microseconds
distribution
count
58