Multiple Device Emulators for HVM Guests, Paul Durrant, Citrix

  • 4,599 views
Uploaded on

Currently Xen only allows a single device emulator to be attached to each HVM guest in a system and, to date,this has been QEMU generally running as a process in the same domain as the toolstack, or …

Currently Xen only allows a single device emulator to be attached to each HVM guest in a system and, to date,this has been QEMU generally running as a process in the same domain as the toolstack, or in a stub domain.
To enable the deployment of virtual GPUs to HVM guests in XenServer, patches were created to allow multiple device emulators to be attached to each HVM guest. QEMU continues to be used to emulate the majority of the devices, but a second process is spawned to handle the virtual GPU. This opens up the possibility of the GPU vendors supplying 'appliance' driver domains in future.
I'd like to give an overview of the changes that we've made to Xen and QEMU to enable the use of multiple emulators, the potential benefits to driver domains, plus the knock on effect of emulator disaggregation on the 'unplug' protocol and what we could do about this.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
4,599
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
53
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Multiple Device Emulators for HVM Guests Paul Durrant Principal Software Engineer, Citrix
  • 2. Agenda “What we’ve done with device emulation in XenServer and what we’re planning to do” • Background • How device emulation is done for an HVM guest • nVidia GRID vGPU • Current implementation of vGPU • Future implementation of vGPU • Potential spin-off work
  • 3. How are devices emulated for an HVM guest? Guest DOM0 QEMU PIO MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ RTC, HPET, etc. Problem: • All device emulation has to be done directly in Xen or in a single emulator i.e. QEMU
  • 4. nVidia GRID Virtual GPU Windows Guest OpenGL Device Model DirectX Standard Hardware Driver Kernel Driver Xen Need an emulator for this bit GRID GPU
  • 5. nVidia GRID Virtual GPU • Build into Xen? • Closed source codebase • GPL problems • Needs hooks into VGA • Build into QEMU? • Large patch • Maintenance issue • Binary plugin interface • Unlikely to gain traction upstream “Neither of those sounds good. Is there another way?”
  • 6. IOREQ Server Abstraction • Based on patch created by Julien Grall from XenClient HVM_PARAM_DM_DOMAIN HVM_PARAM_IOREQ_PFN vcpu[X].arch.hvm_vcpu.xen_port HVM_PARAM_BUFIOREQ_PFN HVM_PARAM_BUFIOREQ_EVTCHN IOREQ Server
  • 7. Multiple Emulators Guest EMU0 PIO EMU1 MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ IOREQ IOREQ Server Table IOREQ Server Table: • Demux on port address, memory address or PCI BDF
  • 8. Where XenServer vGPU is now • Emulator process for each vGPU • Wrapper for nVidia plugin • Runs in dom0 • Patched QEMU • Stripped out graphics device model • Uses IOREQ Server API “That doesn’t sound ideal”
  • 9. Where XenServer vGPU is going • ‘Appliance’ VM for vGPU • Pass GPU through • vGPU emulators • Use IOREQ Server API to register • libVNC based console • Unpatched QEMU • No graphics emulation required • Uses old HVM_PARAM interface • Instantiates ‘catch all’ IOREQ server “Multiple emulators sound cool. What else could we do with them?”
  • 10. Service Domains Storage PV Guest EMU Network PV EMU Xen Co-locate emulators and PV backends: • No need to build complex emulated-to-PV data paths “Does this mean we’re not going to use QEMU any more?”
  • 11. QEMU Disaggregation • We still want to use QEMU for most devices, but we want to use several instances • May be issues separating device emulation from machine emulation • Work already done by Julien • ‘Unplug’ becomes a problem
  • 12. Idea • New HVMOPs to request unplug • Usual IOREQs issued to ‘catch-all’ emulator for compatibility • New emulators receive new UNPLUG IOREQs • If no ‘catch-all’ is present then we may be able to unplug individual devices • Xen Platform ‘fixed IO port’ emulation handled in Xen • Maps onto new HVMOP code • Needs prototyping
  • 13. Q&A