Downloaded 209 times
Uploaded byThe Linux Foundation
Nested Virtualization Update from Intel
The document provides an update on Intel's nested virtualization technology, highlighting recent enhancements, architecture, and features related to nested VMX. Key topics include stability improvements, virtual EPT (Extended Page Tables), and VT-d (Virtual Technology for Directed I/O) capabilities to enhance I/O performance for nested guests. The document concludes with a call to action for support from various virtualization management systems and mentions upcoming features in interrupt virtualization.
More Related Content
Similar to Nested Virtualization Update from Intel
![XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems](https://cdn.slidesharecdn.com/ss_thumbnails/xendevsummit2019-babchuk-op-tee-190812095541-thumbnail.jpg?width=640&height=640&fit=bounds)
Nested Virtualization Update from Intel
- 1. Nested Virtualization UpdateFrom Intel Xiantao Zhang, Eddie Dong Intel Corporation
- 2. Legal Disclaimer INFORMATION INTHIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2012 Intel Corporation.
- 3. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancement −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 3
- 4. Motivation and Goals •Why nested virtualization? − Ordinary OS are adopting VMX now −Windows 7 XP compatibility mode −Windows 8 Hyper-V − Other Commercial VMMs requires VMX for Guest better performance − vmware vmm − Anti-virus software depends on VMX Guest − McAfee Deep Defender • What is the goal ? VMM − To make VMX-based system software run smoothly in a Xen guest. VMM Hardware Platform with VMX Enabled 4
- 5. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 5
- 6. Nested VMX Architecture VM entry/exit Virtual VM entry/exit Nested Guest L2 Dom0 HVM Guest Nested L1 vVMCS VMM Shadowing Shadow Virtual Xen VMM VMCS VMCS EPT L0 Virtual Virtual VMX VT-d VMX-Enabled Platform 6(VT-d, EPT etc.)
- 7. History • Nested VMXupdate @ Xen Summit Asia (Nov. 2009) − Nested VMX design is presented − Showed Initial Status −Nested guest can boot up to BIOS early stage with limitations − single vCPU/single nested guest/ No vCPU migration • Refined nested VMX support was pushed into upstream − Support multiple nested guests − Also includes supporing SMP nested guests • However, experimental & preliminary support − Very limited configurations can work −“KVM on Xen”, Linux guest can successfully boot up −“Xen on Xen” does not work − No virtual VT-d, virtual EPT 7
- 8. Previous Status −Only one combination can work L2 Guest OS L0-VMM L1-VMM 32Bit PAE OS 64Bit OS Win2012 Ubuntu RHEL6.0 RHEL5.4 Win7 RHEL6.0 Win7 Server 12.04 Xen X X X X X X X Xen KVM X √ X X X X X 8
- 9. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 9
- 10. Stability Enhancement − Greatly enhanced stability, with several critical bugs fixed! L2 Guest OS L2 Guest OS(SMP) L0-VMM L1-VMM 32Bit PAE OS 64Bit OS Win2012 Ubuntu RHEL6.0 RHEL5.4 Win7 RHEL6.3 RHEL6.0 Win7 Server 12.04 Xen X √ X √ X √ X √ X √ X √ X √ Xen KVM X √ √ X √ X √ X √ X √ X √ 10
- 11. Performance Without Optimizations 1 L2 Guest2(Xen on Xen) 0.9 0.8 L1 Guest 0.7 0.6 Native 0.5 0.4 Platform: SNB-EP 0.3 OS: RHEL6.3 Guest MEM:4GB Memory 0.2 CPU: 2 VCPU 0.1 0 SpecJBB Unixbench Kernel Build CPU-INT 11
- 12. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 12
- 13. VM entry/exit VirtualEPT Architecture Virtual VM entry/exit L2 L2 Guest L1 vVMCS Shadowing L2-GPA-> L1 GPA L1 VMM L1 EPT Shadowing L0 VMCS-L1 sVMCS-L2 L0 EPT L1-GPA-> HPA Shadow EPT L0 VMM L2-GPA->HPA Switch to Shadow EPT @ virtual vmentry 13
- 14. Virtual EPT: UsingEPT Shadowing • No write-protection to L1-EPT (Guest EPT paging structure) − Flexibility is good. • Trap-and-emulate guest’s INVEPT − Update the shadow EPT entries • Better SMP Scalability − No global lock is required • Requires page-level INVEPT − Individual address invalidation 14
- 15. Enhanced INVEPT Instructionfor Virtual EPT • INVEPT limitations − No Individual address invalidation −Only single context and all context invalidation • Little performance impact, however, hurt nested performance sharply! −Has to drop shadow EPT table for L1’s each INVEPT(with single context) • Performance loss if frequent INVEPT in VMM • For example, KVM • Enhance it in Software Way − Add Individual address invalidation for virtual EPT −Expose it to nested VMM through PV approach − Need to enhance VMMs −Easy implementation for Xen and VMM • Benefits − Reduce frequent shadow EPT paging structure flush 15
- 16. Performance Evaluation ForVirtual EPT 7 6 5 4 w/o virtual EPT 3 with virtual EPT 2 1 0 Kernel Build SpecJBB Netperf CPU-INT 16
- 17. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 17
- 18. Virtual VT-d: ExposeVT-d Capability to L1VMM • I/O performance for L2 guest is very slow − Due to extremely long device emulation path through all the way to L1 & L0 VMMs • How to fix that? − Present virtual VT-d engine to L1 VMM − So, device can be directly assigned to L2 guest −High I/O performance, because of minimum VMM intervention. • Must-to-have features in Virtual VT-d − DMA Remapping & Queue Invalidation: Exposed − Interrupt remapping: Not Exposed 18
- 19. Virtual VT-d Architecture Nested Guest L2 Domain 0 Guest view of VT-d DMA Engine L1 Bus 0 DevQ,Funy Dev Q: Qemu device Qemu Dev P: PT device HVM DevP,Funn loader Nested VMM VT-d page table Hypercall Hypercall Shadowing Virtual VT-d Hardware VT-d vDMAR vQI IO TLB Bus 0 Xen L0 Dev M FunN Bus N Device ATC VMM Shadow VT-d page table Hypervisor 19
- 20. Two types ofguest devices • Pass through device −DMA (IOVA->GPA) is handled by hardware VT-d engine −Remap guest root/context structure −Use physical remapping table to emulate guest remapping table − IOVA -> L0 HPA, + audit (use a dummy page for Out of Bound gpn) − Maybe cached by IOTLB and ATC −IOTLB/Context Cache Synchronization −Track guest invalidation of IOTLB − Invalidate physical IOTLB, and may invalidate ATC as well if the device has ATC −Track guest invalidation of Context Cache • Qemu device −DMA (IOVA->PA remapping) is emulated by Qemu −2 Options: Caching the remapping table, or No-Caching −Starting from simple solution: No caching −Qemu device is already slow 20
- 21. Performance Evaluation ofvirtual VT-d 1100 Bandwidth of Nested Guest Ideal Bandwidth 1000 900 800 Mb/s 700 600 500 400 300 200 100 0 TCP Send TCP Receive UDP Send UDP Receive Iperf testing with the assigned NIC to nested Guest Bandwidth is good enough! 21
- 22. Latency Evaluation ofvirtual VT-d ms Latency 120 Latency 100 80 60 40 20 0 Native Nested Guest Still have room to tune Latency 22
- 23. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 23
- 24. Preliminary Performance Based on Xen #25467 1.2 1 0.8 L1 Guest 0.6 0.4 L2 Guest 0.2 0 Platform: SNB-EP OS: RHEL5.4 Guest MEM:2GB Memory CPU: 2 VCPU 24
- 25. Agenda • Motivation andGoals • History −Nested VMX Architecture −Previous status • Latest status and new features −Stability Enhancements −Virtual EPT −Virtual VT-d • Preliminary Performance • Call to Action 25
- 26. Call to Action •Support more L1 VMMs − McAfee Deep Defender − VMware VMM − Hyper-V − Virtual Box • Virtual APIC-V − New Features for Interrupt/APIC Virtualization are coming − For more information, please come to Nakajima Jun’s talk “Intel Update” this afternoon. − Improve interrupt virtualization efficiency for both L1 and L2 • Performance Tuning 26
- 27. Reference • Nested Virtualizationon Xen − Qing He: − Xen Summit 2009: http://xen.org/xensummit/xensummit_fall_2009.html • Virtual APIC-V − Jun Nakajima: Intel Update − Xen Summit 2012: http://www.xen.org/xensummit/xs12na_talks/T10.html 27
- 28. Questions? • Or contactxiantao.zhang@intel.com 28