This document provides an overview of device passthrough in ACRN. It discusses how device passthrough provides near-native performance by allowing direct access to devices but limits access to single VMs. It describes how ACRN utilizes DMA remapping and interrupt remapping to virtualize device access while maintaining isolation. Code handling is outlined for assigning devices, handling interrupts, and virtualizing ACPI. Limitations discussed include driver dependencies, reset handling, and interrupt programming. New device support requires testing functionality and dependencies in the hypervisor.

1. Device Passthrough Overall Introduction
Binbin Wu
Wu, Binbin <binbin.wu@intel.com>
04/15/2020

2. Agenda
● Device Passthrough Overview
● DMA Remapping
● Interrupt Remapping
● Passthrough related code overview
● Passthrough limitations

3. I/O virtualization
• Software Emulation
• :) Device sharing; Native driver
• :( Low performance, complex to full emulation modern hardware device
• Para-virtualization (VirtIO)
• :) Better performance, device sharing
• :( Specific driver in VM
• Direct I/O (Pass-through)
• :) Near-to-native performance; Good isolation; Native driver
• :( Exclusive device access

4. Device Passthrough Overview
Device
(PCIe)
CPU
Shared
Memory
Interrupt
Register Access
DMA
• CPU access to device
o ACRN traps PCI Configuration space access
o BARs (MMIO / PIO)
▪ No support for PIO BAR in UOS?
▪ Traps MSI-X Table access
▪ Passthrough other MMIO resource vie EPT mapping
• Device DMA to system memory directly with VT-d DMA Remapping
• Device notifies CPU / driver through interrupt
o When receiving a physical interrupt from device, hypervisor will convert the physical
interrupt to virtual interrupt and inject the virtual interrupt to VM

5. Passthrough device
• Passthrough object should be
• PCI/PCIe device Physical Function (PF)
• Virtual Function (VF, SR-IOV)
• Uniquely Identifiable Source-Id
• For isolation, the platform must be capable of uniquely identifying the requestor
(Source-Id) for each DMA request.
• PCI devices may use the same source-id
o Devices Behind PCI Express to PCI/PCI-X Bridges
o Devices Behind Conventional PCI Bridges
• Devices that may use a same source-id can only be passed-through collectively

6. DMA Remapping
• (Native) Driver in guest uses guest physical address as DMA target
• Device issue DMA access using address setup by guest (which is GPA)
• All DMA transactions are captured by chipset
• IOMMU do the translation from GPA to HPA according to the
o Device Assignment Structures
▪ Map a pass-through device to a domain, also the translation structures of the domain
o Address Translation Structures
▪ ACRN reuse EPT tables of a VM as the address translation tables

7. Interrupt Remapping VS Interrupt virtualization
MSI address MSI data
0xFEExxxxx vector 30
MSI address MSI data
0xFEExxxxx
Interrupt_format is 1 -> remappable
Interrupt_index
Interrupt Remapping Eanbled
IRT address
Interrupt Remapping Table
...
B:D.F, Vector 60, Dest LAPIC ID
...
Interrupt index
Remappable
interrupt request
Interrupt 60 VMCS configured to
cause vmexit on
external interrupt
VMEXIT
Hypervisor
Hypervisor
Injects
interrupt 30
Process interrupt 30
Guest VM
VMENTRY
IOMMUPhyscial vector 60
• Interrupt Remapping
o Control and censor external interrupt requests generated by passthrough device
o X2APIC support
• Interrupt Virtualization without Interrupt Posting
o physical interrupt to virtual interrupt
o Inject virtual interrupt to VM

8. Code for device passthrough
• Passthrough driver in Device Model
(devicemodel/hw/pci/passthrough.c)
o Allocate virtual BAR address spaces
o For device only support INTx
▪ Allocate virtual Interrupt Line/Pin
▪ Call API (passthru_bind_irq) to bind pin information
o Call API (vm_assign_pcidev) to assign a pci device to a VM
o ACPI Virtualization
• Passthrough device interrupt handling
(hypervisor/arch/x86/guest/assign.c, hypervisor/common/ptdev.c)
• vPCI related (hypervisor/dm/vpci/)
• VT-d support (hypervisor/arch/x86/vtd.c)

9. How to passthrough a device
• Unbind device driver in SOS
• Bind device to pci_stub driver
• Add the device as a option to acrn-dm
-s <slot>,passthru,B/D/F
Refer to laucn_xxx.sh in devicemodel/samples/<board>/

10. Device level reset
• Passthrough device should provide device level reset capability:
• FLR or PCI PM reset.
• Without device level reset, may have issue
o VM restart
o Used in SOS first and then passthrough to UOS

11. PCI Interrupt Pin Programming
• In general, ACRN doesn’t recommend to passthrough a device that only
support legacy interrupt
• Especially, ACRN doesn’t support to passthrough devices only support legacy
interrupt in pre-launched VM / RTVM.
• ACRN relies on BIOS / bootloader to program PCI interrupt pin
• ACRN device model relies on the value programmed in “Interrupt Line” (offset 0x3c)
by BIOS / bootloader.
o Although such information can be parsed and extracted from ACPI table, it would be too
complicated to do it in ACRN.
o Although Linux kernel will re-program the field when probe the driver, the driver for the
device may not be present in Service VM, so if BIOS / bootloader doesn’t program it, the
legacy interrupt will not work.

12. Avoid Global System Interrupt (GSI) sharing
• Devices physically sharing a same GSI pin make trouble when passed-
through to different VMs
• How to avoid
• Disable virtual IOAPIC link for MSI capable device
o Guest driver should use MSI instead of legacy interrupt.
o Some native driver still use legacy interrupt even when the device supports MSI !!
• GSI sharing violation check in device model
• Pre-launch VM doesn’t support devices using legacy interrupt
• Platform specific, need extra effort for a new platform, not scalable.
• BIOS / bootloader should avoid GSI pin sharing for GSI only device

13. Driver should no IOMMU dependency
• Driver should not depend on IOMMU for functionality.
• ACRN hypervisor take use of IOMMU for DMA remapping and Interrupt Remapping.
• ACRN hypervisor doesn’t support virtual IOMMU.
• Otherwise, need extra efforts to remove the dependency on IOMMU in guest
kernel framework or device driver
• Eg. Device using multiple MSI Vectors
o Device doesn’t support MSI-X but requires multiple vectors.
o Device driver use IOMMU to mitigate the issue, but it will not work on ACRN

14. Virtual PCI slot allocation requirements
• Device driver should not hardcode BDF for a device
• Some device drivers hardcoded BDF in code
o Need extra effort to debug
o Virtual slot should be the same as physical slot as a workaround

15. ACPI virtualization
• Blackbox
• Huge efforts when expose host ACPI to guest
• Additional resource access needs to be virtualized
• Refer to write_dsdt_xxx in devicemodel/hw/pci/passthrough.c

16. When enable a new passthrough device
• Figure out the BIOS / driver dependency
• Test functionality on native platform
• Test functionality in SOS on ACRN
• Test functionality is UOS on ACRN
• APIC virtualization needed?
• Other device dependency (e.g. GPIO?)
• Check if interrupt injected in guest properly
• Driver issue?
• ...

17. Q&A