XPDS13: HVM Dom0 - Any unmodified OS as Dom0 - Will Auld, Intel


Published on

It should be great if we can use an unmodified guest for dom0 or the driver domain. We found a way to achieve that. Since Xen's inception, the first guest on Xen is always a para-virtualized domain, and it can be modified Linux, NetBSD, and Solaris etc. In this way, dom0 can achieve near-native performance, so it is commonly used in the server market. However, modifications to guest kernels also implies limitations. For example, it can't support Windows OS as the dom0 or the driver domain. With the rapid evolution of hardware-assisted virtualization (e.g. VMX, VT-d technologies), HVM domains also can achieve comparable performance with para-virtualization. And, it's high time for Xen to such an unmodified guest as the dom0. In the presentation, we discuss its architectural changes and its benefits compared with the traditional PV or HVM dom0, and we also introduce what we have done.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

XPDS13: HVM Dom0 - Any unmodified OS as Dom0 - Will Auld, Intel

  1. 1. HVM Dom0: Any Unmodified OS as dom0 Xiantao Zhang, Nakajima Jun, Dongxiao Xu Speaker: Auld Will Intel Corporation
  2. 2. Legal Disclaimer  INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.  Intel may make changes to specifications and product descriptions at any time, without notice.  All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.  Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.  Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.  *Other names and brands may be claimed as the property of others.  Copyright © 2013 Intel Corporation.
  3. 3. Outline • Para-virtualized Dom0 – history & problems • Why HVM dom0 ? • Technologies in HVM dom0 • Call to Action • Takeaways 3
  4. 4. Today’s Xen Architecture (Since Xen3.0) VM3 Dom0 Device Manager & Control s/w 32/64bit VM2 Unmodified User Software GuestOS AGP ACPI PCI VM1 Unmodified User Software GuestOS GuestOS (XenLinux) (XenLinux) (XenLinux) Back-End Back-End SMP Unmodified GuestOS (Windows OS Linux)) Front-End Device Drivers Front-End Device Drivers Native Device Driver Control IF Native Device Driver Safe HW IF Event Channel Virtual CPU Unmodified User Software Virtual MMU Xen Hypervisor Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) VT-x VT-d
  5. 5. History of PV Dom0 PV Dom0 evolution: • … • Xen Linux 2.6.18 • Xen Linux 2.6.27 • Linux 2.6.32 + PVOPS patchset • PVOPS Linux pushed to upstream 3.0 Challenges: • Tremendous effort spent on pushing patches to Linux upstream. • Maintaining effort. • Hard to push certain features/fixes into Linux upstream. − Xen PAT − Xen ACPI. − Xen RAS 5
  6. 6. Why was PV dom0 only? • Problems − Old x86 architecture (Pre-VT) −Virtualization is not considered by design. −Many virtualizations holes exist − X86 architecture with 1st Gen VT −Lack of performance optimizations −No hardware features to support memory/IO virtualization • Solution – PV dom0 − Modify dom0’s kernel source code to address virtualization holes. − Adopt PV interfaces to enhance system performance. −Network, storage, MMU, etc. − But Linux only 6
  7. 7. Limitations of PV Dom0 • Dom0 kernel is modified Linux (XenLinux) − Depends on Kernel changes − Hard to push some changes(RAS, PAT, ACPI) to upstream Linux − Can’t support unmodified OS (Windows, Mac OS, etc) − Can’t leverage VT for performance enhancement • Performance limitations of dom0 − 64bit dom0 has to suffer from poor performance − Super page can’t be supported well − Fast system call can’t be supported as well − Un-avoidable various traps to hypervisor −Thread switch − FPU, TLS, stack switch −MMU update operations −Guest page faults … 7
  8. 8. New Hardware Virtualization Technologies • CPU virtualization − Virtualization holes are finally closed by architecture − CR access acceleration − TPR shadow/APIC-v • Memory virtualization − EPT VPID – memory virtualization is done by hardware − EPT super page supported − Unrestricted guest • I/O virtualization − VT-d supports direct IO for guest − SR-IOV • Interrupt virtualization − APIC-V − Posted interrupts HVM domain comparable performance with PV domain 8
  9. 9. Good chance for improving dom0 • Goal − Remove PV dom0’s limitations − Leverage new VT technologies to enhance dom0’s performance • Options − PVH dom0: Running PV kernel in HVM container, leveraging some VT technologies (e.g. EPT) to enhance dom0’s performance. Only limited to Linux OS. − HVM dom0: Allows unmodified OS (may with PV drivers) running in HVM container with full VT technologies. Ideally, it can support any unmodified OS. Our choice: HVM dom0 9
  10. 10. Xen Architecture with HVM dom0 HVM Dom0 HVM Domain Device Manager & Control s/w Qemu (virtio backend) Ring3 Unmodified User Software VMX Non-Root Unmodified GuestOS (Windows OS Linux)) Linux/Window/etc. Back-End Device Drivers VM Entry/Exit VMX Root Control IF Native Device Driver Front-End Device Drivers (e.g. virtio) Interrupts Safe HW IF VM Entry/Exit Event Channel Xen Hypervisor Hardware 10 Virtual CPU Virtual MMU Ring0
  11. 11. Benefits of HVM dom0 • More choices for Dom0 (Windows, Mac OS, etc.). • Better performance compared with 64-bit PV Dom0. • Reduce the Xen Linux kernel complexity and maintenance effort. • Xen hypervisor becomes more flexible to support more usage cases. − Desktop virtualization to benefit Windows/Mac OS users. − New Xen client hypervisor usage. − Mobile virtualization. • Covers more virtualization model. − First windows/Mac OS based type-1 open source hypervisor. 11
  12. 12. How to make HVM dom0 work ? • CPU virtualization − Same as HVM domU. • Memory virtualization − Adopt EPT/NPT for memory virtualization − Super page is used for performance enhancement • IO virtualization − With VT-d, all physical devices are assigned to dom0 by default − IO access & mmio access doesn’t trigger vmexits. • Interrupt virtualization − Dom0 controls physical IOAPIC − Dom0’s local APIC is virtualized − Hypervisor owns physical local APIC 12
  13. 13. HVM dom0 Boot Sequence • EFI-based system as HVM Dom0 (e.g. Windows*) − Dynamically de-privilege EFI shell to a HVM guest environment − Boot flow: System power on  Boot EFI shell  execute startup.nsh  xen_loader.efi  Xen entry point  start_xen()  construct_dom0()  prepare_hvm_context()  VMLAUNCH back to original EFI shell  Load OS as usual startup.nsh: xen_loader.efi xen.gz /boot/efi/ia32.efi xen_loader.efi: − Used dynamically to de-privilege EFI shell − One EFI binary for loading Xen and setup return point from hypervisor − After back to EFI shell, EFI environment is in a HVM guest EFI is dynamically de-privileged to HVM container 13
  14. 14. HVM dom0 Boot Sequence (Cond) • Linux as HVM dom0 − System loaded by grub − System Power on  Power on to grub  Xen entry point  start_xen()  construct_dom0()  prepare_hvm_context()  VMLAUNCH to kernel entry. Similar with today’s PV dom0 14
  15. 15. Multi-domain Support • Qemu is a must − Both Linux & Windows supports Qemu • PV Driver support − Xen bus/event channel mechanism is needed − One virtual PCI device (PCI platform device) is virtualized − Port PCI platform device’s logic from Qemu to hypervisor Dom0 OS Back-end Driver Event Channel/ Xen Bus Driver* User-land tools/libs Qemu Linux Ready Ready Ready Ready Windows Virtio Ready Not ready Partially Ready* 15
  16. 16. Call for Action • Port (or simplify) Xen’s userland tools/libraries − For Windows • Enable PV drivers for DomU guests − For performance enhancement of DomU • Port Xen-Qemu logic to Windows 16
  17. 17. Takeaways • Unmodified OS as HVM dom0 • Only an add-on feature for Xen project − Doesn’t break existing Xen’s usage models − Only used for new x86 platforms • Can resolve PV dom0’s limitations • Can cover more usage models − New type of Xen Client (with Windows HVM Dom0) − Create Trusted Execution Environment for single Windows/Mac OS − Xen can be used in PC market like today’s type-2 VMMs 17
  18. 18. Questions? • Or contact xiantao.zhang@intel.com 18
  19. 19. 19