Multiple Device Emulators for HVM Guests, Paul Durrant, Citrix


Published on

Currently Xen only allows a single device emulator to be attached to each HVM guest in a system and, to date,this has been QEMU generally running as a process in the same domain as the toolstack, or in a stub domain.
To enable the deployment of virtual GPUs to HVM guests in XenServer, patches were created to allow multiple device emulators to be attached to each HVM guest. QEMU continues to be used to emulate the majority of the devices, but a second process is spawned to handle the virtual GPU. This opens up the possibility of the GPU vendors supplying 'appliance' driver domains in future.
I'd like to give an overview of the changes that we've made to Xen and QEMU to enable the use of multiple emulators, the potential benefits to driver domains, plus the knock on effect of emulator disaggregation on the 'unplug' protocol and what we could do about this.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Multiple Device Emulators for HVM Guests, Paul Durrant, Citrix

  1. 1. Multiple Device Emulators for HVM Guests Paul Durrant Principal Software Engineer, Citrix
  2. 2. Agenda “What we’ve done with device emulation in XenServer and what we’re planning to do” • Background • How device emulation is done for an HVM guest • nVidia GRID vGPU • Current implementation of vGPU • Future implementation of vGPU • Potential spin-off work
  3. 3. How are devices emulated for an HVM guest? Guest DOM0 QEMU PIO MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ RTC, HPET, etc. Problem: • All device emulation has to be done directly in Xen or in a single emulator i.e. QEMU
  4. 4. nVidia GRID Virtual GPU Windows Guest OpenGL Device Model DirectX Standard Hardware Driver Kernel Driver Xen Need an emulator for this bit GRID GPU
  5. 5. nVidia GRID Virtual GPU • Build into Xen? • Closed source codebase • GPL problems • Needs hooks into VGA • Build into QEMU? • Large patch • Maintenance issue • Binary plugin interface • Unlikely to gain traction upstream “Neither of those sounds good. Is there another way?”
  6. 6. IOREQ Server Abstraction • Based on patch created by Julien Grall from XenClient HVM_PARAM_DM_DOMAIN HVM_PARAM_IOREQ_PFN vcpu[X].arch.hvm_vcpu.xen_port HVM_PARAM_BUFIOREQ_PFN HVM_PARAM_BUFIOREQ_EVTCHN IOREQ Server
  7. 7. Multiple Emulators Guest EMU0 PIO EMU1 MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ IOREQ IOREQ Server Table IOREQ Server Table: • Demux on port address, memory address or PCI BDF
  8. 8. Where XenServer vGPU is now • Emulator process for each vGPU • Wrapper for nVidia plugin • Runs in dom0 • Patched QEMU • Stripped out graphics device model • Uses IOREQ Server API “That doesn’t sound ideal”
  9. 9. Where XenServer vGPU is going • ‘Appliance’ VM for vGPU • Pass GPU through • vGPU emulators • Use IOREQ Server API to register • libVNC based console • Unpatched QEMU • No graphics emulation required • Uses old HVM_PARAM interface • Instantiates ‘catch all’ IOREQ server “Multiple emulators sound cool. What else could we do with them?”
  10. 10. Service Domains Storage PV Guest EMU Network PV EMU Xen Co-locate emulators and PV backends: • No need to build complex emulated-to-PV data paths “Does this mean we’re not going to use QEMU any more?”
  11. 11. QEMU Disaggregation • We still want to use QEMU for most devices, but we want to use several instances • May be issues separating device emulation from machine emulation • Work already done by Julien • ‘Unplug’ becomes a problem
  12. 12. Idea • New HVMOPs to request unplug • Usual IOREQs issued to ‘catch-all’ emulator for compatibility • New emulators receive new UNPLUG IOREQs • If no ‘catch-all’ is present then we may be able to unplug individual devices • Xen Platform ‘fixed IO port’ emulation handled in Xen • Maps onto new HVMOP code • Needs prototyping
  13. 13. Q&A