Nakajima hvm-be final

  • 1,469 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,469
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
57
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Moving Backend Drivers from Dom0 to HVM Domain 29 April 2010 Jun Nakajima, Eddie Dong, and Xiantao Zhang
  • 2. Legal Disclaimer  INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.  Intel may make changes to specifications and product descriptions at any time, without notice.  All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.  Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.  Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.  *Other names and brands may be claimed as the property of others.  Copyright © 2010 Intel Corporation. Xen Summit NA 2010 2
  • 3. HVM Driver Domain with Backends HVM Driver Domain Dom 0 (w/ HAP and Direct I/O) HVM Guest X86-64 (xm/xend) Models Models Device Device Control Linux Panel Unmodified OS PV Linux Backend FE Drivers driver Native FE Drivers Drivers FE Virtual Drivers Device Native Drivers Guest BIOS Guest BIOS Device Virtual Platform PV Virtual Platform PV Xen API Xen API VMExit VMExit Event channel Control Interface Scheduler Event Channel Hypercalls Processor Memory I/O: PIT, APIC, PIC, IOAPIC, RTC Device models Xen Hypervisor Direct I/O
  • 4. Benefits of HVM Driver Domain • HVM driver domain is kind of IDD (Isolated driver domain) − Reuse existing Linux • Same binaries for (certified) device drivers • Backend drivers with minor modifications • Provide more scalable and efficient I/O than dom0 − Use HVM with HAP − Use direct I/O feature to access physical devices • Allow to have restartable I/O capability − Restart HVM Driver domain if fails − Need additional work to restore BE-FE connections • Multiple instances of HVM driver domains − Avoid single point of faulure Xen Summit NA 2010 4
  • 5. Data from Proof of Concept • Added network backend driver to HVM Linux − Xenbus driver and minor modifications (some are for Xen as well) − Frontend in PV Linux talks to BE in HVM Linux • Better performance compared with PV domain (about 28%) with lower CPU utilization − PV-FE <-> HVM BE (CPU total utilization: 131%), 7.62 Gb/sec − PV-FE <-> PV BE (CPU total utilization: 156%), 5.95 Gb/sec • Comparing performance of EPT page flipping and grant- copying − Page flipping in EPT sounds much simpler than in PV Xen Summit NA 2010 5
  • 6. EPT Page Flipping vs. Page-Copying • Page-flipping: − Exchange pages of data between FE drivers and BE driver without copying data • For PV we replaced page flipping with grant-copying − Total cost of mapping/unmapping/TLB-shootdown/refcount-updates overheads was not cheaper • However, things are much simpler with HAP − Update HAP page tables entries − Invalidate HAP (e.g. INVEPT) like TLB shoot-down − Potentially need to do the same thing for VT-d/IOMMU Xen Summit NA 2010 6
  • 7. EPT Page-Flipping vs. Page-Copying (cont.) Mode hypercalls Cycles per Throughput CPU (per sec.) hypercall (Gb/s) Utiliziation (%) HVM (BE) 35208 52982 7.35 129 Page-flipping HVM (BE) 35531 52417 7.62 131 Page-copying PV BE 5.95 156 • Page-copying is faster with (a bit) higher CPU utilization Xen Summit NA 2010 7
  • 8. Dom0 w/o Physical Device Drivers HVM Driver Domain Dom 0 (w/ HAP and Direct I/O) HVM Guest X86-64 (xm/xend) Models Models Device Device Control Linux Panel Unmodified OS PV Linux Backend FE Drivers driver Native FE Drivers FE Drivers Device FE Virtual Drivers Drivers Semi-physical BIOS Guest BIOS Virtual Platform PV Virtual Platform PV Xen API Xen API VMExit VMExit Event channel Control Interface Scheduler Event Channel Hypercalls Processor Memory I/O: PIT, APIC, PIC, IOAPIC, RTC Device models Xen Hypervisor Direct I/O
  • 9. Booting w/o drivers in dom0 1. Grub loads Xen, dom0, and RAM filesystem image − RAM filesystem image contains HVM driver domain 2. Xen boots, and starts initialization 3. Dom0 is created, and mounts RAM filesystem 4. Dom0 creates HVM driver domain 5. HVM driver domain boots, and it runs device drivers and backend drivers 6. System is ready Xen Summit NA 2010 9
  • 10. Issues with Dom0 w/o Drivers • No NIC drivers on dom0 − We can use FE on dom0 − Network is not available until HVM driver domain is ready • We don’t want dom0 to depend on guest… • No storage drivers on dom0 − No persistent storage. This may be desirable. − Workaround: Move HBA between dom0 and HVM driver domain • A bit complex Xen Summit NA 2010 10
  • 11. Summary • Network backend driver in HVM provides better performance with EPT and Direct I/O enabled, compared with BE in PV domain (about 28%) • Next Steps: − More data analysis on “EPT page-flipping and page-copying” • Understand why HAP page-flipping is not faster • Opportunities for optimizations − Send out patches Xen Summit NA 2010 11