XS Boston 2008 SR-IOV


Published on

Eddie Dong & Yunhong Jiang: Support SR-IOV Devices in Xen -

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

XS Boston 2008 SR-IOV

  1. 1. SR-IOV support in Xen Yaozu (Eddie) Dong (Eddie.Dong@intel.com) Yunhong Jiang Kun (Kevin) Tian
  2. 2. Agenda • SR-IOV specification overview • Xen/SR-IOV architecture • Discussions 2
  3. 3. SR-IOV specification overview Start with a single function device VM1 IOVM VM0 • HW under the control of privileged SW • Includes an SR-IOV Extended Capability DD DD DD • Physical Function (PF) Replicate the resources needed by a VM PCIM • MMIO for direct communication • RID to tag DMA traffic Physical PF VF VF • Minimal configuration space Function Shared • Virtual Function (VF) Resource Virtual Introduces PCI Manager (PCIM) Function • Conceptual SW entity DMA, PIO, and Interrupts • Completes the configuration model Initialization and Configuration • Translates VF into a full function • Configures SR-IOV resources 3
  4. 4. Xen/SRIOV architecture SR-IOV generic VF CFGS emulation Dom0 (Service VM) user Guest App DM CP SR-IOV SR-IOV Normal PCI driver with hot VF management add/removal support Network configuration PF driver VF driver Alt1: Inter-VM communication Dom0 kernel Guest Kernel CFGS access Alt2: PIO/MMIO based virtual mailbox/doorbell Xen Alt3: HW mailbox/doorbell HW 4
  5. 5. Control Panel VF management: A generic tool with notification to PF for state change • VFnums setting / Show • Enable/Disable/Migrate VFs Network configuration • Can Ethtool service for this? • Or a totally new tool? 5
  6. 6. SR-IOV VF/PF Communication Channel Inter-VM APIs VMM cooperative VF driver • VF driver will depends on VMM, Guest OS, and even release from VMM vendor – Primitive hypercalls are not sufficient – There is no Windows based Xen Inter-VM high level APIs in upstream – Xen vendors may release their own abstract APIs with separately Guest hardware VMM independent VF driver • Real Hardware mailbox/doorbell – No SR-PCIM involvement – Good Performance, but IHVs may not implement • Virtual mailbox/doorbell – Need SR-PCIM support – Need SR-IOV standard VF/PF driver pair’s decision to use whatever mechanism, but suggest using guest hardware 6
  7. 7. Virtual hardware for PF/VF communication Dom0 (Service VM) user VF->PF doorbell can use I/O PF->VF doorbell uses virtual IRQ DM SRIOV 4: Emulation 5: Notification 3: Back to DM 6: Ack Guest Kernel PF driver VF driver Dom0 kernel 1: Guest write I/O 8: Done 7: Done Xen 2: VM Exit to hypervisor 7
  8. 8. PCI Device Instance of VF in Host? No create VF instance • Need access path for Qemu to R/W VF CFGS • Modifications are mostly in Qemu side • Start from this for TTM Created VF instance • Pros: Easy for assignment • Cons: Confuse to other pci modules, invasive change, need community decision on how to change • Long term solution 8
  9. 9. Summary IOMMU based SR-IOV support is an ideal IO virtualization • Each guest can get a portion of hardware • VMM doesn’t need to intercept at runtime • High throughput, Low CPU utilization, Perfect scalability Early VMM support for SR-IOV is critical to IHVs to implement PF/VF drivers Dom0 Linux version also matters in terms of PF driver development 9
  10. 10. Legal Information INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice. Intel is a trademark of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2007, Intel Corporation. All rights are protected. 10
  11. 11. 11