PCI Passthrough and ITS Support in Xen / ARM :Xen Dev Summit 2015 Presentation

PCI Passthrough
and GICv3-ITS in Xen ARM
Manish Jaggi
Vijaya Kumar Kilari
Cavium, Inc. Proprietary
+ Demo on Dual Socket 48x2 Core ARMv8 Board

©2015 Cavium Inc. All rights reserved. Confidential and Proprietary.
Agenda
 Status of Xen Support from Cavium
 Top Level Architecture
 Additions in xen for pci-passthrough
 ITS architecture
– ARM specification
– Virtual ITS driver in Xen
 Xen NUMA Demo on Cavium ThunderX platform
 Questions

Status of Xen Support from Cavium
 Xen 4.5+ (Current)
– Demoed in Linaro Connect
– Initial Support NUMA
 Xen 4.6
– Basic ThunderX platform support
– Gicv3 Support.
 Xen 4.7
– vITS support
– PCI Passthrough patches in Xen and Linux
– NUMA Patches

Linaro Connect – Demo
Xen running on single socket 48 core -
ThunderX

ThunderX System Dual Socket Reference Platform
x2
Standard Industry Form Factor:
½ SSI Motherboard
2U 19” Rack Mount Chassis
Volume Server I/O:
PCIe Gen3
10Gb or 40Gb Ethernet
Integrated SATA
Up to 128GB Memory
Full Systems Management w/ BMC and IPMI
http://cavium.com/pdfFiles/ThunderX_CRB_2S_Rev1.pdf

Xen NUMA
running on dual socket 48x2 cores
vCPU
dom0 domU
vITS
vCPU
vITS
domU
vCPU
vITS
Xen Hypervisor
DDR DDR
Node 0 (48 Cores) Node 1 (48 Cores)

Top Level Architecture
R/WMSI/X
vCPU
IO Virtualizaton with System MMU
dom0 domU
vITS
vCPU
vITS
PCIe HostBridge
GICv3 ITS
DDR
Controller
DDR
Xen Virtual ITS Driver
PCIe-EP1 PCIe-EP2
(DeviceID,MSI_Index)=>LPI
Interrupt
Translation Table StreamID => ContextBank
ContextBank =
{…, Domain PageTable, … }
vLPIvLPI
}

additions in xen-arm… (proposed / implemented)
 PCIe HostController Support in Xen.
– pci_conf_read/write calls handled by host controller driver
– device_tree based
 vITS Emulation Support
 Hypercall to map Linux SegmentID to appropriate PCI HostController
 xl-toolstack additions
– Mapping of GITS_ITRANSLATER space in domain
– assign_device hypercall enhanced to support vDeviceID
 Frontend-Backend Changes
– no communication for MSI.
– Front-end PCI bus msi-parent => its node in guest device tree
 SMMU additions

PCIe Host Controller support in Xen
The init function in the pci host driver calls to register hostbridge callbacks:
int pci_hostbridge_register(pci_hostbridge_t *pcihb);
struct pci_hostbridge_ops {
     u32 (*pci_conf_read)(struct pci_hostbridge*, u32 bus, u32 devfn,u32 reg, u32 bytes);
     void (*pci_conf_write)(struct pci_hostbridge*, u32 bus, u32 devfn,
u32 reg, u32 bytes, u32 val);
};
struct pci_hostbridge{
     u32 segno;
     paddr_t cfg_base;
     paddr_t cfg_size;
     struct dt_device_node *dt_node;
     struct pci_hostbridge_ops ops;
     struct list_head list;
};

PHYSDEVOP_pci_host_bridge_add
#define PHYSDEVOP_pci_host_bridge_add 44
struct physdev_pci_host_bridge_add {
/* IN */
uint16_t seg;
uint64_t cfg_base;
uint64_t cfg_size;
};
This hypercall is invoked before dom0 invokes the PHYSDEVOP_pci_device_add
hypercall. The handler code invokes … to update segment number in pci_hostbridge:
int pci_hostbridge_setup(uint32_t segno, uint64_t cfg_base, uint64_t cfg_size);

xl toolstack additions - DOMCTL
For domU, while creating the domain, the toolstack reads the IPA from the
macro GITS_ITRANSLATER_SPACE from xen/include/public/arch-arm.h. The PA is
read from a new hypercall which returns the PA of the GITS_ITRANSLATER_SPACE.
Subsequently the toolstack sends a hypercall to create a stage 2 mapping.
Hypercall Details: XEN_DOMCTL_get_itranslater_space
/* XEN_DOMCTL_get_itranslater_space */
struct xen_domctl_get_itranslater_space {
/* OUT variables. */
uint64_aligned_t start_addr;
uint64_aligned_t size;
};
typedef struct xen_domctl_get_itranslater_space xen_domctl_get_itranslater_space;
DEFINE_XEN_GUEST_HANDLE(xen_domctl_get_itranslater_space;

xl toolstack additions – device assignmentReserved Areas in guest memory space
Parts of the guest address space is reserved for mapping assigned pci device’s BAR
regions. Toolstack is responsible for allocating ranges from this area and creating stage
2 mapping for the domain.This area is defined in public/arch-arm.h
/* For 32bit BARs*/
#define GUEST_BAR_BASE_32 <<>>
#define GUEST_BAR_SIZE_32 <<>>
/* For 64bit BARs*/
#define GUEST_BAR_BASE_64 <<>>
#define GUEST_BAR_SIZE_64 <<>>
New entries in xenstore for device BARs
/local/domain/0/backend/pci/1/0
vdev-N
BDF = ""
BAR-0-IPA = ""
BAR-0-PA = ""
BAR-0-SIZE = ""
...
BAR-M-IPA = ""
BAR-M-PA = ""
BAR-M-SIZE = "”

Hypercall Modification (XEN_DOMCTL_assign_device)
 struct xen_domctl_assign_device {
uint32_t dev; /* XEN_DOMCTL_DEV_* */
union {
struct {
uint32_t machine_sbdf; /* machine PCI ID of assigned device */
uint32_t guest_sbdf; /* guest PCI ID of assigned device */
} pci;
struct {
uint32_t size; /* Length of the path */
XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
} dt;
} u;
};

SMMU Code additions
iommu_ops functions
PHYSDEVOP_pci_add_device
.add_device = arm_smmu_add_dom0_dev,
PHYSDEVOP_pci_remove_device
.remove_device = arm_smmu_remove_device

Mapping between streamID - deviceID - pci sbdf - requesterID
 For a simpler case all should be equal to BDF.
 But there are some devices that use the different
requester ID for DMA transactions
 Suggestions How to handle this

pci-frontend bus gicv3-its node binding for domU
 It is assumed that toolstack would generate a gicv3-its node in domU device
tree.
 As of now the ARM PCI passthrough design supports device assignment to
the guests which have gicv3-its support.
 All the devices assigned to domU are enumerated on a PCI frontend bus.
On this bus interrupt parent is set as gicv3-its for ARM systems.
 As the gicv3-its is emulated in xen, all the access by domU driver is trapped.
This helps configuration & direct injection of MSI(LPI) into the guest. Frontend-backend
communication for MSI is no longer required.
 Frontend-backend communication is required only for reading PCI configuration
space by dom0 on behalf of domU.

ITS
 Interrupt Translation Service(ITS) is the specification from ARM to
support PCI MSI(x).
 MSI(x) are handled as Locality-specific Peripheral Interrupts (LPI)
starting from IRQ number 8192.
 LPIs are directly targeted to CPU.
 SW sends ITS command like MAPD, MAPVI, MOVI, INT, SYNC, INV to ITS
HW to prepare MSI(x) Translation.
 Command Completion notification using
– Polling
– Interrupt notification by placing INT command

ITS HW-SW Interaction
ITT Table
ITT Table
ITT Table
CWRITER
CREADER
BASER
Command
Queue
ITS Commands
Device Table
ITS HW
LPI
Configuration
Table
LPI Pending Table
(per CPU)
CPUS
CPUS
SOFTWARE
Allocated by
SW used by
HW
Allocated by SW
used by both
SW Write ITS
CMDs to Queue
HW reads ITS
CMDs and
configures
CPUS
CPUS
Legend:

Major challenges in virtualizing ITS
 ITS Commands should be processed in with minimal latency
without blocking VCPU for long duration
 All guests should get fair amount of time in processing guest ITS
commands
 Guest cannot put Xen in DoS by sending commands continuously
– Solution: Do not send Guest ITS commands to HW. Just emulate them.
 Processing global ITS commands like SYNC, INVALL etc., on
platforms with Multi-node ITS
– Solution: One Virtual ITS per domain and Ignore guests SYNC,
INVALL, DISCARD commands

Major challenges in virtualizing ITS
 Handling guest ITS emulation that uses INT
command for completion notification
– Solution: Xen injects back virtual LPI to guest
when INT command is emulated.

ITS virtualization in XEN
 ITS Virtualization
– Command Queue virtualization
– LPI configuration table virtualization
– GITS registers virtualization

XEN ITS Initialization
XEN
Guest
SOFTWARE
ITT Table
CWRITER
CREADER
BASER
Physical ITS
command Queue
ITS Commands
Device Table
Memory Allocated
by
XEN for ITS HW
(1) Guest sends
PCI_DEVICE_ADD
_PHYSDEVOPS
hyper call
(2) Allocates ITT
table for the
device and sends
MAPD command
to ITS HW
(3) Allocates LPIs
(physical LPI) to
device and sends
MAPVI commands

ITS command Virtualization
ITT Table
ITT Table
ITT Table
CWRITER
CREADER
BASER
Virtual
Command
Queue
ITS
Commands
Device Table
XEN
Guest
SOFTWARE
Memory Allocated
by
GUEST
(2) Xen uses Guest’s
Device and ITT table
memory to note-down
Guest ITS command
information
(1) Traps to Xen
on Guest update
of command in
Virtual Queue

MAPD/MAPVI ITS command Virtualization
ITT Table
CWRITER
CREADER
BASER
MAPD Devi ID,
ITT IPA, Size
Device Table
XEN
Guest
SOFTWARE
Memory Allocated
by
GUEST
(1) XEN reads
MAPD finds out
IPA of ITT table
and Size for the
devid
ITT IPA
(8 bytes)
Size
(8 bytes)
Virtual
Command
Queue
MAPVI Dev ID,
vID, Collection
Collection ID
vLPI (vID)
(3) XEN reads
MAPVI
Command
Device and ITT table
memory to note-down
ITT Table IPA and Size
Device Table to find
Address of ITT for the
device and updates ITT
indexed by ID with vLPI
and Collection ID

LPI Routing to Guest
ITT TableDevice Table
XEN
Guest
SOFTWARE
Memory
Allocated by
Guest
(2) Xen queries
Device Table
and gets ITT
table
(1) Xen
receives
pLPI
ITT IPA
(8 bytes)
Size
(8 bytes)
vLPI (vID)
Collection ID
HW
(3) From ITT
table, Xen get
Virtual LPI
(vLPI)
(4) Xen Injects
vLPI to Guest

References:
 vITS Design doc
– http://xenbits.xen.org/people/ianc/vits/draftG.pdf
 Patches ( 22 )
– http://osdir.com/ml/general/2015-07/msg35182.html
 PCI Pass through Design doc
– http://www.gossamer-
threads.com/lists/xen/devel/394962

Xen Dual(Socket / Node) NUMA Demo
vCPU
dom0 domU
vITS
vCPU
vITS
domU
vCPU
vITS
Xen Hypervisor
DDR DDRNode 0 (48 Cores) Node 1 (48 Cores)

 #xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 2048 8 r----- 128.9
domu-node0 1 2048 4 -b---- 1.4
domu-node1 2 2048 4 -b---- 0.6
 #xl cpupool-list
Name CPUs Sched Active Domain count
Pool-node0 48 credit y 2
Pool-node1 48 credit y 1
 xl cpupool-list -c
Name CPU list
Pool-node0
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,4
3,44,45,46,47
Pool-node1
48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,
88,89,90,91,92,93,94,95

Questions

MSI-x Routing (back up)

PCI Passthrough and ITS Support in Xen / ARM :Xen Dev Summit 2015 Presentation

More Related Content

What's hot

Similar to PCI Passthrough and ITS Support in Xen / ARM :Xen Dev Summit 2015 Presentation

Recently uploaded

PCI Passthrough and ITS Support in Xen / ARM :Xen Dev Summit 2015 Presentation