Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Software	&	Services	Group
vSVM	on	Xen
Author: Yi Sun/Yi Liu/Chao Peng
Speaker: Jun Nakajima
July 12, 2017
Software	&	Services	Group
Agenda
• Motivation
• SVM Extensions in Hardware
• SVM Virtualization
• vSVM Design On Xen
• Xen...
Software	&	Services	Group
High-Level View of SVM
Intel	Confidential	3
CPU Device
X86 page tables x86 page tables
EPT table...
Software	&	Services	Group
Example
• OpenCL 2.0 supports sharing virtual address between CPU and GPU.
• Reduce communicatio...
Software	&	Services	Group
SVM Extensions in
Hardware
5
• PASID
• Process Address Space ID
• New extension to PCI-e Spec.
•...
Software	&	Services	Group
SVM Virtualization
6
Caching-
Mode
Guest
Host
shadow ex-context
RTAR
Ext
Root
Ext
Context
RTAR
E...
Software	&	Services	Group
vSVM Design: Architecture on Xen
• vSVM depends on Virtual IOMMU
framework developed by Tianyu L...
Software	&	Services	Group
vSVM Design: PCI-e Extended Capabilities Exposure
• Expose below PCI capabilities to Guest:
– PA...
Software	&	Services	Group
vSVM Design: Architecture in details
9
Hardware
Memory
pIOMMU
Device1
Device0
Shared	VA
Request	...
Software	&	Services	Group
Xen IOMMU Status
• Root-table->Context-table->Second-level-table are
supported: GPA->HPA.
• Xen ...
Software	&	Services	Group
Q&A
11
Software	&	Services	Group
Terms
SVM: Shared Virtual Memory
vSVM: virtual SVM, SVM in guest
CSR: Mean the IOMMU registers i...
Software	&	Services	Group
13
Upcoming SlideShare
Loading in …5
×

XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu, Intel

1,068 views

Published on

Shared Virtual Memory (SVM) is a VT-d feature that allows sharing application address space with the I/O device. The feature works with the PCI sig Process Address Space ID (PASID). With SVM, programmer gets a consistent view of memory across host application and device, avoids pining or copying overheads. We have been working on supporting SVM in Xen to enable SVM usage in guest if a SVM capable device is assigned. e.g. assign IGD to a guest, applications like OpenCL would benefit if SVM is supported in guest. SVM virtualization requires exposing a virtual VT-d to guest. In this discussion, Yi would update the latest SVM virtualization implementation and foresee the future work about supporting SVM and IOVA a single virtual VT-d.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu, Intel

  1. 1. Software & Services Group vSVM on Xen Author: Yi Sun/Yi Liu/Chao Peng Speaker: Jun Nakajima July 12, 2017
  2. 2. Software & Services Group Agenda • Motivation • SVM Extensions in Hardware • SVM Virtualization • vSVM Design On Xen • Xen IOMMU Status 2
  3. 3. Software & Services Group High-Level View of SVM Intel Confidential 3 CPU Device X86 page tables x86 page tables EPT tables VT-d tables Host/Physical Memory GVA GVA GPA GPA HPA HPA OS Managed VMM Managed
  4. 4. Software & Services Group Example • OpenCL 2.0 supports sharing virtual address between CPU and GPU. • Reduce communication latency between CPU and GPU. • Simplified GPU programming. 4 SVM
  5. 5. Software & Services Group SVM Extensions in Hardware 5 • PASID • Process Address Space ID • New extension to PCI-e Spec. • Identify the targeted virtual address space • DMA Requests without PASID • Normal memory requests from endpoint devices • DMA Requests with PASID • request to application’s virtual address, SVM • Extended Root Entry • Upper 64-bit is used • LCTP: Lower Context Table Pointer • UCTP: Upper Context Table Pointer • Extended Context Entry • Extended to 256 bits • PASIDPTR: PASID table pointer • SLPTPTR: Second-Level Page Translation Pointer • PASID table • FLPTPTR: First-Level Page Translation Pointer Ex-Root Table PASID Table First Level Page Table Second Level Page Table Bus # devfn # RTA Ex-Context Table pasid # For request w/ PASID VT-d DMA Remapping Hierarchy
  6. 6. Software & Services Group SVM Virtualization 6 Caching- Mode Guest Host shadow ex-context RTAR Ext Root Ext Context RTAR Ext ContextExt Root PASID Table 1st level translation GVA à GPA GPA à HPA (i.e. ia32-e page tables) 2nd level translation (i.e. VT-d page tables) VT-d w/ SVM Ext. Virtual VT-d w/ SVM Ext Link to guest PASID Table
  7. 7. Software & Services Group vSVM Design: Architecture on Xen • vSVM depends on Virtual IOMMU framework developed by Tianyu Lan and Chao Gao from Intel. • Expose capabilities • Queued invalidation: – Shadow context entry: GPA of PASID table pointer used in nested mode – Forward 1st-level translation cache invalidation – PRQ response • Fault reporting – Unrecoverable fault – Recoverable fault reporting/servicing (PRQ) 7 IOMMU Driver vIOMMUIOMMU Driver IOMMU Hardware Xen Hypervisor Qemu Guest IOMMU Fault PRQ Response Translation Cache Invalidation vIOMMU Fault Shadow Ex-Context Entry Cache Invalidation for 1st Translation vIOMMU Capabilities IOMMU Capabilities PCIe Support PCIe Configuration
  8. 8. Software & Services Group vSVM Design: PCI-e Extended Capabilities Exposure • Expose below PCI capabilities to Guest: – PASID - Process Address Space ID Extended Capability Structure – ATS - Address Translation Services Extended Capability Structure – PRQ - Page Request Extended Capability Structure 8 • More work is required: – All these capabilities are PCI-e Extended Capability which is in PCI configuration space offset 256-4095 – Current PCI device emulation on Xen supports offset 0-255 only – Offset 256-4095 can only be accessed with MMCFG – A PCI-e device emulation is required – In QEMU only Q35 supports PCI-e, while the PIIX440 Xen uses does not – Finally we need to enable Q35 for Xen
  9. 9. Software & Services Group vSVM Design: Architecture in details 9 Hardware Memory pIOMMU Device1 Device0 Shared VA Request w/ PASID Request w/o PASID Assigned Device Xen IOMMU Driver Trans faults PRQ service PRQ queue QI interface Invalidate queue Ext Root Table Ext Ctx Table PASID Table SLP Table vIOMMU Guest gVAS Guest process1 gVAS Guest process2 Guest Kernel gIOMMU Driver gTrans faults Guest PRQ thread Guest QI interface gPRQ queue gInvalidate queue Qemu-xen Newly added Modify Trans faults Inject msi RTA/IQT/etc regs handling MMIO write (including PRQ response) Ctx-cache invalidation to update PASID ptr PCIe support Trans faults QI interface Guest User Space Shared VA RTA: Root Table Address IQT: Invalidation Queue Tail
  10. 10. Software & Services Group Xen IOMMU Status • Root-table->Context-table->Second-level-table are supported: GPA->HPA. • Xen does not support IOVA/SVM now. Only device assignment can work. • Xen IOMMU protection domain is managed per- VM. 10
  11. 11. Software & Services Group Q&A 11
  12. 12. Software & Services Group Terms SVM: Shared Virtual Memory vSVM: virtual SVM, SVM in guest CSR: Mean the IOMMU registers in this slide IOVA: IO Virtual Address PRQ: Page Request vIOMMU: Virtual IOMMU FLPT/FLT: First Level Page Table SLPT/SLT: Second Level Page Table QI: Queued Invalidation IQ: Invalidation Queue DMAR: DMA Remapping Reporting Structure 12
  13. 13. Software & Services Group 13

×