This document provides answers to frequently asked questions about Spectre and Meltdown security vulnerabilities for VMware ESXi administrators. It discusses available patches for ESXi, the need to apply microcode and guest operating system updates, and how to check that new CPU capabilities are properly exposed in virtual machines. Applying patches requires updating vCenter, ESXi hosts, and coordinating reboots of guest VMs, which can be a significant effort for large environments. Enabling Extended CPU Features in vCenter helps maintain VM compatibility during the update process.

1. VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Spectre/Meltdown security
vulnerabilities FAQ
Typical questions and practical
answers for IT Infrastructure
Practitioners and vSphere
Administrators
David Pasek, VMware, Staff TAM
June 4, 2018
v.04

2. ‹#› 2VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
What is
Spectre/Meltdown?
Security vulnerabilities referenced as following CVE’s:
CVE-2017-5753 (Spectre Variant 1) - Branch target injection
CVE-2017-5715 (Spectre Variant 2) - Bounds check bypass
CVE-2017-5754 (Variant 3 - Meltdown) - Rogue data cache load
CVE-2018-3640 (Spectre Variant 3a) - Rogue System Register Read
CVE-2018-3639 (Spectre Variant 4) - Speculative Store Bypass
Sources: https://www.us-cert.gov/ncas/alerts/TA18-004A and
https://www.us-cert.gov/ncas/alerts/TA18-141A

3. 3VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: Do exist all ESXi patches for Spectre/Meltdown vulnerabilities?
A: VMware released ESXi patches for Spectre variants 1, 2 but patches for new Spectre
variants 3a and 4, as of June 4th, 2018, are on hold until Intel has released updated
microcode which has been tested by VMware. Meltdown (Variant 3) vulnerability
remediation is done only at Guest OS level, therefore it does not need ESXi patches.
Details:
• VMware patches for Spectre variant 1 (CVE-2017-5753) exist. See. VMSA-2018-0002.
• VMware patches for Spectre variant 2 (CVE-2017-5715) exist. See. VMSA-2018-0002 and
VMSA-2018-0004.
• VMware patches for Variant 3 aka Meltdown (CVE-2017-5754) are not required. This
remediation is dependent solely on Operating System patches.
• VMware patches for Spectre variant 3a (CVE-2018-3640) and variant 4 (CVE-2018-3639)
are, as of June 4th, 2018, on hold until Intel has released updated microcode which has been
tested by VMware. See VMSA-2018-0012.

4. 4VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: What does this all mean for vSphere administrators?
A: Well, actually nothing new. It is, “just” a security update of the infrastructure. However,
the update process is more complicated than usual because the infrastructure has to be
patched end-to-end.
VMware vSphere administrator must apply following update procedure.
1. Update vCenter. Note: Patches add new CPU feature masks (IBRS, IBPB, STIBP) into the
existing EVC baselines and only enable them when all ESXi hosts within vSphere Cluster have
updated their CPU microcode.
2. (optional) Enable EVC on vSphere Clusters if you will need a long time to upgrade hosts / need
to vMotion VMs after power cycle across different version hosts. Note, this is not necessary if
you won’t power cycle VMs while you bring up the cluster to the same patch level.
3. Update to the latest BIOS with patched CPU microcode. Note: VMware delivers updated CPU
microcode for most CPU models with an ESXi patch but most HW vendors recommend to
update the BIOS, refer to their requirements.
4. Apply ESXi security patches
Steps continue on next slide.

5. 5VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: What does this all mean for vSphere administrators?
5. Validate VM hardware is at least in version 9 (for Spectre 2 OS mitigation). For better
performance (Meltdown mitigation), VM hardware 11 is recommended (enables INVPCID,
Haswell CPU required). There are some performance improvements for VM hardware 9 with
Linux (PCID), not for Windows though.
6. Apply all applicable security patches for your Guest OS which have been made available from
the OS vendor (not critical to do this last but it makes sense to align the required reboot with
the power cycle). Note: Power Off / Power On of VM is required. VM restart is not sufficient.
… continuing from the previous slide

6. 6VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: What is the reason to consider enablement of EVC within vSphere Cluster?
A: When EVC is not enabled, vCenter Spectre specific patches are not in use and you can
experience vMotion issues. The issue would not occur on running VMs but on Power
Cycled or newly deployed VMs as the new CPU features would be exposed into VM thus
identified as VM running on vMotion not compatible ESXi host.
Details:
• An ESXi host that is running a patched vSphere hypervisor with updated microcode will see
new CPU features that were not previously available. These new features will be exposed to all
Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these virtual
machines now see additional CPU features, vMotion to an ESXi host lacking the microcode or
hypervisor patches applied will be prevented.
• The vCenter patches enable vMotion compatibility to be retained within an EVC cluster. In
order to maintain this compatibility the new features are hidden from guests within the cluster
until all hosts in the cluster are properly updated.
• At that time, the cluster will automatically upgrade its capabilities to expose the new features.
Unpatched ESXi hosts will no longer be admitted into the EVC cluster.
Source: https://kb.vmware.com/kb/52085

7. 7VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: What is the reason to consider enablement of EVC within vSphere Cluster?
Supplemental screenshot
You would see this vMotion issue when EVC is not enabled, the source ESXi host is patched thus new
CPU features are exposed to VM, and the target ESXi host is not patched.

8. 8VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: What is the reason to consider enablement of EVC within vSphere Cluster?
Supplemental screenshot
When EVC is enabled, vMotion works perfectly even from patched to unpatched ESXi host because
new features are hidden from guests within the cluster until all hosts in the cluster are properly updated.

9. 9VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: Why have a lot of VMware customers still not applied patches?
A: Well, there are usually two main reasons.
1. Security update requires huge update effort especially in large environments
2. Security update has performance impact on applications
Details:
• First things first, the performance impact is the same or similar on bare metal. The
hypervisor performance overhead is negligible (below 2%)
• The negative performance impact is hard to predict as it is workload specific. That’s the
reason the application owners should evaluate the specific impact on their application.
• IT management is afraid of the unpredictable performance impact, lack of computing
resources and tremendous impact on capacity planning.
• In case a VM hardware upgrade is required, a maintenance window with application owners
has to be planned. Note: A Virtual Machine hardware upgrade can bring a certain risk
because you are practically changing the motherboard.
• Power Off / Power On VMs is required, which is another reason the maintenance window
must be planned with or by application owners.

10. 10VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: Is it necessary to update VMware Tools to remediate Spectre vulnerabilities?
A: No. VMware Tools are irrelevant to Spectre remediation. VM hardware (aka
Compatibility) is what matters. You should manage VMware Tools as usual. The best
practice is to use the latest VMware Tools version but it is not a strict requirement.
Details:
• Even though VMware Tools are irrelevant to Spectre, here are some details:
• If you use VMTools 10.x then it is independent of the ESXi version thus independent of
VM hardware.
• If you use VMTools 9.x then you should use the latest release for your particular ESXi
version.*
• VM hardware 9+ is the only requirement to mitigate Spectre 2 (OS). Minimal requirement is
VM hardware version 9 as it is necessary to pass-through IBRS,IBPB and STIBP.
• For better Meltdown mitigation performance, VM hardware version 11 is recommended as it
supports INVPCID (Invalidate Process-Context Identifier, requires Haswell or newer CPU).
• Some improvements on Linux are already available for VM hardware 9 (PCID), 11 (INVPCID)
further improves on that whereas for Windows, 11 is the minimum for any HW assisted
performance improvement

11. 11VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: A Guest OS reboot is not sufficient. How can vSphere Admin schedule an
automated VM Power Cycle during Guest OS reboot?
A: As a vSphere Admin, you cannot schedule a VM Power Cycle during a regular Guest
OS reboot. The only exception is when you schedule a VM Compatibility Upgrade (aka
VM hardware upgrade). This special use case is covered on the next slide.
Details:
• If you have only basic vSphere, you have to schedule specific app maintenance windows
with the application owners and work with them in tandem. This can be a very time
consuming effort especially in environments with thousands of VMs. Alternatively, you can
consider a scheduled VM Compatibility Upgrade. Please, note that it cannot be used for VMs
that are already on the latest VM hardware version which is supported on your ESXi release.
• If you have Cloud Management Platforms like vRealize Automation or vCloud Director then a
Power Cycle can be done by server/application owners by themselves.

12. 12VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: Do I need Power Cycle VM after scheduled VM hardware upgrade on
Guest OS Reboot?
A: No. If you schedule a VM Compatibility Upgrade (aka VM Hardware Upgrade) for your
VMs, it will actually Power-Off/Power-On VM on next Guest OS reboot as it is a part of the
VM hardware upgrade workflow.
Details:
• Cannot be used for VMs already using the latest VM hardware version supported by your
ESXi version.
• Please note that a VM hardware upgrade is the same operation as changing the
motherboard in physical computer. It is a significant change for the guest OS which may or
may not be successful, therefore there are some associated risks.
• You should have a backup of your VMs to recover from in case of any issues.
• The full procedure is documented in VMware documentation “Schedule a Compatibility
Upgrade for Virtual Machines”.
• You or the app owner should test if new CPU capabilities are exposed inside the Guest OS
and that the security mitigations will work.
• The final decision, if you want to use this feature (scheduled VM hardware upgrade) is up to
you and you have to understand all potential impacts and associated risks. It usually works
without issues, but as discussed, you change the motherboard for an already installed and
configured OS.

13. 13VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: Do I need Power Cycle VM after scheduled VM hardware upgrade on
Guest OS Reboot?
Supplemental screen shots

14. 14VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: How can I check new CPUs capabilities exposure in to Guest OS?
A: The best is to test it by Spectre/Meltdown checkers within Guest OS.
Details:
• To use MS-Windows checker read Microsoft Article at https://support.microsoft.com/en-
us/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in.
• PowerShell 5.0 is required
• Install-Module SpeculationControl
• Set-ExecutionPolicy RemoteSigned -Scope Currentuser
• Get-SpeculationControlSettings
• To use Linux checker read the article at https://github.com/speed47/spectre-meltdown-checker
where is a shell script to tell if your system is vulnerable.
• If you want “Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI” read
William Lam’s blog post at https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-
guest-mitigation-spectre-patches-using-powercli.html and leverage PowerCLI script available at
https://github.com/lamw/vghetto-scripts/blob/master/powershell/VerifyESXiMicrocodePatch.ps1

15. 15VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: How I can check new CPUs capabilities exposure in to Guest OS?
Supplemental screen shots – MS-Windows and Linux vulnerabilities checkers
MS-Windows checker
Linux checker

16. 16VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Q: How I can check new CPUs capabilities exposure in to Guest OS?
Supplemental screen shots – William Lam’s PowerCLI functions
Verify-ESXiMicrocodePatchAndVM
Verify-ESXiMicrocodePatch

17. 17VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Conclusion
• All patches for Spectre variants 1,2 and meltdown are released and the update process
should be well understood.
• Patches for Spectre variants 3a and 4 are on hold until Intel has released updated
microcode.
• Spectre/Meltdown patching is definitely not a simple project, especially in large
organizations where collaboration among multiple teams and departments is required.
• OS Patches have negative performance impact on certain applications.
• Some of these OS remediations can be disabled in operating systems, therefore
hardware and vSphere layers can be patched and application owner can choose
between security or performance.
• Please note that depending on the vector and OS, it might not be possible to disable all
remediations. Contact your OS vendor for further information.

18. VMware and TAM Customer Confidential │ ©2018 VMware, Inc.
Other questions?
Contact your TAM or
E-mail dpasek@vmware.com