Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

XPDDS17: Reworking the ARM GIC Emulation & Xen Challenges in the ARM ITS Emulation


Published on

Part 1: Reworking the ARM GIC Emulation

The ARM Generic Interrupt Controller (GIC) provides some level of virtualization support in hardware. This still requires emulation of the distributor part, which has to integrate with the virtualization feature. Doing this in a performing and readable way is not trivial, especially the locking strategy tends to be complicated.

While extending the existing virtual GIC support in Xen to cover support for MSIs, some issues have been discovered which ask for some significant changes in the existing code.
The presentation will briefly describe the existing VGIC design and the issues we faced when trying to extend it. Based on this the changes will be presented and how they improve and ideally simplify the code.

Part 2: Xen Challenges in the ARM ITS Emulation

For being able to use MSIs on ARM systems in Xen domains we need to emulate the ARM GICv3 ITS controller. Its design is centered around a command queue located in normal system memory.

Emulating this in the Xen hypervisor brings some interesting challenges, ranging from safely accessing the guest memory and dealing with possible propagation of commands, to possible DOS attacks by domains keeping the emulation code busy.

The presentation outlines the main problems and how we hit Xen limits in emulating this correctly and efficiently. Also it presents our temporary workarounds and their drawbacks.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

XPDDS17: Reworking the ARM GIC Emulation & Xen Challenges in the ARM ITS Emulation

  1. 1. ARM® Interrupt Virtualization Andre Przywara <> ARM Ltd., Cambridge, UK Xen summit 12/07/2017 © ARM 2017
  2. 2. ARM interrrupt virtualization agenda GICv2 and virtualization overview Xen vGIC implementation Rework plans GICv3/ITS architecture changes ITS emulation challenges 2 © ARM 2017
  3. 3. GICv2 architecture Device CPU interface CPU interface CPU interface Device Device Core 0 Core 1 Core n.... AXI bus IRQ lines (SPIs) GICv2 .... SGIs Distributor PPIs IRQ/FIQ 3 © ARM 2017
  4. 4. ARM GICv2 at a glance ”distributor” is the central component supports up to 8 CPUs and 1020 interrupts (988 shared + 32 private) has an input pin for each wired interrupt (SPIs) connects to separate CPU interfaces (one per core) has per-core input pins for private interrupts (PPIs) handles inter-processor interrupts internally (SGIs) Programmed via MMIO accesses Configuration affects always a group of interrupts (32-bit registers) Some registers are banked per CPU (at the same memory address) 4 © ARM 2017
  5. 5. Virtualization support in GICv2 Virtual CPU interface allows IRQ ACKs and EOIs without exiting the guest Hypervisor sets up virtual IRQs in List Registers (LRs) In the guest the (virtual) CPU interface relates to these Allows connecting a physical interrupt to a virtual one Physical IRQs gets EOIed at the same time No need to trap or monitor EOI in this case anymore Distributor IRQ lines (SPIs) GICCGICV LRs GICH CPU interface 5 © ARM 2017
  6. 6. Xen vGIC implementation Lives in xen/arch/arm (shared between arm(32) and arm64) Distributor is emulated (mostly in vgic-v2.c) Pending interrupts are written into the list registers (LRs) Interrupt acknowledgment and EOI is handled without exiting the guest (by the GIC hardware) After guest run distributor emulation syncs back from the LRs Virtual IRQ configuration is stored in ranks (groups of IRQs) GICV LRs CPU interface Distributor (emulated) GICCGICH virtual IRQs traps 6 © ARM 2017
  7. 7. vGIC implementation design Code design is centered around MMIO IRQ grouping (ranks) One lock per VCPU and one lock per rank Sophisticated lockless tricks to avoid excessive rank locking IRQ 32 IRQ 33 ISENABLER: . . . . IRQ 63 IRQ 62 IRQ 61 031 IRQ 34 ITARGETSR: IRQ 32IRQ 33IRQ 34IRQ 35 0781516232431 BUT: 7 © ARM 2017
  8. 8. Implementation issues and limitations Distributor accesses are rare (during IRQ configuration only) VCPU entry/exit path is much more frequent and sensitive State is held in bitmaps and bytemaps covering contigious IRQ numbers proves problematic with sparse LPI allocation (later) Handling of level vs. edge triggered interrupts 8 © ARM 2017
  9. 9. vGIC rework To bring ITS/LPI support closer to the existing code base: Store all virtual IRQ properties in struct pending_irq Protect with one lock per IRQ Get rid of the ranks and their locks at all Simplify IRQ list management (remove lr_queue list) Introduce reference counting for dynamic pending_irq’s Model level-triggered interrupts correctly 9 © ARM 2017
  10. 10. GICv3 architecture Device Device Device Device Device .... AXI bus IRQ lines (SPIs) .... SGIs Distributor PPIs GICv3 Redistributor Redistributor Redistributor Core 0 Core 1 Core n CPU interface CPU interface CPU interface ITS Memory IRQ/FIQ MSI MSI 10 © ARM 2017
  11. 11. GICv3 changes System register access to CPU interface (drops banked MMIO) IRQ routing allows millions of cores Lifts the 8-CPU limit of GICv2 Uses MPIDR based values to specify one target core per IRQ Splits distributor to separate private and shared IRQs New class of interrupts (LPI) via an Interrupt Translation Service (ITS) Allows MSI/MSI-X support Supports indirections for target cores (via collections) Introduces device ID sampled from the bus New IRQ class with possibly thousands of LPIs and probably sparse allocation Tables are held in physical memory 11 © ARM 2017
  12. 12. GICv3/ITS Xen implications Potentially large, sparsely allocated LPIs spoil vGICv2 ranks Requires dynamic allocation (and de-allocation → ref-counting) Motivates storing all IRQ property in struct pending_irq Brings up locking issues (solution: per-IRQ lock) ITS data structures are held in guest physical memory Need to reach into domain memory (fortunately cheap on ARM64) Creates security question (we can’t trust this memory) Provides natural resource limitation (more LPIs → more memory) Caching is architected and common in hardware too 12 © ARM 2017
  13. 13. ITS tables (simplified) device command prop pend coll 13 © ARM 2017
  14. 14. ITS challenges Security and DOS challenges: Untrusted guest memory holds internal information Need to sanitise information on every access (sometimes not feasible) Accessing guest memory is not for free Looking at getting rid of storing information there Guest could fill (potentially large) emulated command queue Architecture allows defered execution But we need to schedule this! Some guest actions may require host action Disabling or enabling interrupts is one case To avoid IRQ storms we want to disable IRQs on the hardware Requires queueing an ITS command to the host command queue May open DOS vector when a domain often disables/enables LPIs 14 © ARM 2017
  15. 15. Preliminary fixes ITS emulation only for Dom0 → trust the guest Untrusted guest memory holds internal information Sanitise as much as possible Trust the guest! Guest could fill (potentially large) emulated command queue Real World Dom0s only schedule two or three commands Trust the guest! Some guest actions may require host action Keep interrupts enabled on the host Detect guest disabled IRQs early Ignore IRQ storms for now (trust the guest!) 15 © ARM 2017
  16. 16. Xen questions Accessing shared host resources on behalf of guests Could introduce deferred host command queueing How to schedule commands? Suspend vCPUs until commands have been executed? More costly emulation on behalf of a guest Return early when timeslice expires! But when to return to continue the work? Deschedule guest while in an MMIO emulation? Schedule tasklet on account of a guest? 16 © ARM 2017
  17. 17. The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. Copyright © 2016 ARM Limited © ARM 2017