Currently Xen only allows a single device emulator to be attached to each HVM guest in a system and, to date,this has been QEMU generally running as a process in the same domain as the toolstack, or in a stub domain. To enable the deployment of virtual GPUs to HVM guests in XenServer, patches were created to allow multiple device emulators to be attached to each HVM guest. QEMU continues to be used to emulate the majority of the devices, but a second process is spawned to handle the virtual GPU. This opens up the possibility of the GPU vendors supplying 'appliance' driver domains in future. I'd like to give an overview of the changes that we've made to Xen and QEMU to enable the use of multiple emulators, the potential benefits to driver domains, plus the knock on effect of emulator disaggregation on the 'unplug' protocol and what we could do about this.

1. Multiple Device
Emulators for HVM
Guests
Paul Durrant
Principal Software Engineer, Citrix

2. Agenda
“What we’ve done with device emulation in XenServer and what
we’re planning to do”
• Background
• How device emulation is done for an HVM guest
• nVidia GRID vGPU
• Current implementation of vGPU
• Future implementation of vGPU
• Potential spin-off work

3. How are devices emulated for an HVM guest?
Guest
DOM0
QEMU
PIO
MMIO
Instruction
Trap
Page Fault
VMEXIT
Xen
IOREQ
RTC,
HPET,
etc.
Problem:
• All device emulation has to be done directly in Xen or in a single
emulator i.e. QEMU

4. nVidia GRID Virtual GPU
Windows
Guest
OpenGL
Device Model
DirectX
Standard
Hardware
Driver
Kernel Driver
Xen
Need an emulator
for this bit
GRID GPU

5. nVidia GRID Virtual GPU
• Build into Xen?
• Closed source codebase
• GPL problems
• Needs hooks into VGA
• Build into QEMU?
• Large patch
• Maintenance issue
• Binary plugin interface
• Unlikely to gain traction upstream
“Neither of those sounds good. Is there another way?”

6. IOREQ Server Abstraction
• Based on patch created by Julien Grall from XenClient
HVM_PARAM_DM_DOMAIN
HVM_PARAM_IOREQ_PFN
vcpu[X].arch.hvm_vcpu.xen_port
HVM_PARAM_BUFIOREQ_PFN
HVM_PARAM_BUFIOREQ_EVTCHN
IOREQ
Server

7. Multiple Emulators
Guest
EMU0
PIO
EMU1
MMIO
Instruction
Trap
Page Fault
VMEXIT
Xen
IOREQ
IOREQ
IOREQ
Server
Table
IOREQ Server Table:
• Demux on port address, memory address or PCI BDF

8. Where XenServer vGPU is now
• Emulator process for each vGPU
• Wrapper for nVidia plugin
• Runs in dom0
• Patched QEMU
• Stripped out graphics device model
• Uses IOREQ Server API
“That doesn’t sound ideal”

9. Where XenServer vGPU is going
• ‘Appliance’ VM for vGPU
• Pass GPU through
• vGPU emulators
• Use IOREQ Server API to register
• libVNC based console
• Unpatched QEMU
• No graphics emulation required
• Uses old HVM_PARAM interface
• Instantiates ‘catch all’ IOREQ server
“Multiple emulators sound cool. What else could we do with them?”

10. Service Domains
Storage
PV
Guest
EMU
Network
PV
EMU
Xen
Co-locate emulators and PV backends:
• No need to build complex emulated-to-PV data paths
“Does this mean we’re not going to use QEMU any more?”

11. QEMU Disaggregation
• We still want to use QEMU for most devices, but we want to use
several instances
• May be issues separating device emulation from machine
emulation
• Work already done by Julien
• ‘Unplug’ becomes a problem

12. Idea
• New HVMOPs to request unplug
• Usual IOREQs issued to ‘catch-all’ emulator for compatibility
• New emulators receive new UNPLUG IOREQs
• If no ‘catch-all’ is present then we may be able to unplug
individual devices
• Xen Platform ‘fixed IO port’ emulation handled in Xen
• Maps onto new HVMOP code
• Needs prototyping

13. Q&A