Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BUD17-301: Zeno's paradox: Optimizing KVM/ARM

891 views

Published on

"Session ID: BUD17-301
Session Name: KVM/ARM Nested Virtualization - BUD17-301
Speaker: Christoffer Dall
Track: Virtualization


★ Session Summary ★
Nested virtualization, the ability to run a virtual machine inside another virtual machine, is increasingly important because of the need to deploy virtual machines running software stacks on top of virtualized cloud infrastructure, as well as for prototyping and testing. As ARM servers make inroads in various deployment scenarios, being able to support nested virtualization on ARM is a key requirement, which has been met recently with the introduction of nested virtualization support in the latest ARMv8.3 revision of the architecture. I will present the initial effort to introduce ARM nested virtualization support to KVM/ARM, which involves adding significant logic to core KVM/ARM code, MMU support, timers, and the GIC emulation. I will also briefly discuss a paravirtualization approach we have used to prototype and evaluate the implementation on current ARMv8 hardware without hardware support for nested virtualization.
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/bud17/bud17-301/
Presentation: https://www.slideshare.net/linaroorg/bud17301-zenos-paradox-optimizing-kvmarm
Video: https://youtu.be/Mqh9J83xFxo
---------------------------------------------------

★ Event Details ★
Linaro Connect Budapest 2017 (BUD17)
6-10 March 2017
Corinthia Hotel, Budapest,
Erzsébet krt. 43-49,
1073 Hungary

---------------------------------------------------
Keyword: virtualization, KVM, ARM
http://www.linaro.org
http://connect.linaro.org
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://twitter.com/linaroorg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961"

Published in: Technology
  • Be the first to comment

BUD17-301: Zeno's paradox: Optimizing KVM/ARM

  1. 1. BUD17-301: KVM/ARM Nested Virtualization Christoffer Dall
  2. 2. Hardware Hypervisor VM VM Kernel App App Nested Virtualization Hypervisor VM Kernel App App VM Kernel App App
  3. 3. Hardware Host Hypervisor VM VM Kernel App App Terminology Guest Hypervisor Nested VM Kernel App App Nested VM Kernel App App L0 L1 L2 L1 L0
  4. 4. Use Cases 1. IaaS hosting private clouds 2. Test your hypervisor in a VM 3. Debug your hypervisor in a VM 4. Develop hypervisors using a cloud
  5. 5. Theorem 2 “A conventional third generation computer is recursively virtualizable if it is: (a) virtualizable, and (b) a VMM without any timing dependencies can be constructed for it.” Formal requirements for virtualizable third generation architectures
 [Popek and Goldberg ’74]
  6. 6. Recursively Virtualizable • Only applies to virtualizable architectures • ARM and x86 are not virtualizable • Hardware support for virtualization
  7. 7. ARM Virtualization Extensions Kernel UserEL0 EL1 HypervisorEL2
  8. 8. VM ARM Virtualization Extensions EL0 EL1 EL2 Hypervisor Kernel User Space VM Kernel User Space
  9. 9. ARM Nested Virtualization EL0 EL1 EL2 Host Hypervisor Kernel User Space Kernel User Space Virtual
 EL2 Guest Hypervisor Guest Hypervisor
  10. 10. ARM Nested Virtualization EL0 EL1 EL2 Host Hypervisor Kernel User Space Kernel User Space EL ?? Guest Hypervisor Guest Hypervisor
  11. 11. ARM Nested Virtualization EL0 EL1 EL2 Host Hypervisor Kernel User Space Kernel User Space EL0 Guest Hypervisor Guest Hypervisor Trap-and-emulate
  12. 12. ARM Nested Virtualization EL0 EL1 EL2 Host Hypervisor Kernel User Space Kernel User Space EL1 Guest Hypervisor Guest Hypervisor ?? -and-emulate
  13. 13. ARMv8.3 • Supports running the guest hypervisor in EL1 • HCR_EL2.NV: • Traps EL2 operations executed in EL1 to EL2 • Traps eret to EL2
  14. 14. • CPU Virtualization • Memory Virtualization • Timer Virtualization • Interrupt Virtualization KVM/ARM Nested Virtualization
  15. 15. struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; } struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; } Nested CPU Virtualization
  16. 16. Host Linux AppApp VM Kernel AppApp KVM EL0 EL1 EL2 Restore EL1 sys_regs Save EL1 sys_regs Hypervisor-VM Switch
  17. 17. Host Linux AppApp VM Kernel KVM EL0 EL1 EL2 Save/restore EL1 sys_regs Guest Hypervisor Save/restore el2_regs Hypervisor-Hypervisor Switch
  18. 18. • Define mapping of EL2 registers to EL1 registers • Example: TTBR0_EL2 to TTBR0_EL1 • Example: SCTLR_EL2 adapted to SCTLR_EL1 • Shadow EL1 registers Emulating EL2 in EL1
  19. 19. &sys_regs u64 *vcpu->ctxtx.hw_regs &shadow_sys_regs PSTATE.mode == EL2PSTATE.mode == EL0/1 Shadow Registers
  20. 20. • Trap to virtual EL2 • “Forward” exceptions • Emulate virtual exceptions VM EL0 EL1 EL2 Host KVM Kernel User Space Guest KVMvEL2 Virtual Exceptions
  21. 21. • Returning from virtual EL2 • Trap eret to EL2 (ARMv8.3) • Emulate virtual exception return VM EL0 EL1 EL2 Host KVM Kernel User Space Guest KVMvEL2 Virtual Exceptions
  22. 22. • CPU Virtualization • Memory Virtualization • Timer Virtualization • Interrupt Virtualization KVM/ARM Nested Virtualization
  23. 23. Virtual Address (VA) Physical Address (PA) Memory Virtualization
  24. 24. Virtual Address (VA) Physical Address (PA) Intermediate Physical Address (IPA) Memory Virtualization Stage 1: VM kernel Stage 2: Hypervisor
  25. 25. Virtual Address (VA) Physical Address (PA) Intermediate Physical Address (IPA) Stage 1: Nested VM kernel Nested Intermediate Physical Address Stage 2: Host hypervisor Stage ?: Guest hypervisor Nested Memory Virtualization
  26. 26. Virtual Address (VA) Physical Address (PA) Intermediate Physical Address (IPA) Nested Memory Virtualization Stage 1: Nested VM kernel Stage 2: Host hypervisor Shadow
 Stage 2
 Page Table
  27. 27. Shadow Stage 2
 Page Tables • Translate IPA to PA • Entries are created by host KVM by walking guest hypervisor stage 2 page tables in software VM EL0 EL1 EL2 Host KVM Kernel User Space vEL2 Guest KVM IPA -> VM PA VA -> IPA VM PA -> PA
  28. 28. • CPU Virtualization • Memory Virtualization • Timer Virtualization • Interrupt Virtualization KVM/ARM Nested Virtualization
  29. 29. • ARM provides a virtual and physical timer in EL1 • EL2 provides a separate EL2 “hyp” timer • KVM must emulate a VM with EL2 and the hyp timer Nested Timer Virtualization
  30. 30. • CPU Virtualization • Memory Virtualization • Timer Virtualization • Interrupt Virtualization KVM/ARM Nested Virtualization
  31. 31. ARM Generic Interrupt Controller (GIC) GIC CPU 0 CPU 1 CPU Interface CPU Interface Dist. IRQ ACK Device
 Interrupt
 Lines
  32. 32. ARM Generic Interrupt Controller (GIC) GIC CPU 0 CPU 1 CPU Interface CPU Interface Dist. IRQ ACK Virtual CPU Interface Virtual CPU Interface VIRQ ACK List Registers (LRs) List Registers (LRs)
  33. 33. VM Nested VM • Deliver both virtual and nested virtual interrupts using the GIC • Multi-level virtualization using single-level virtualization hardware [Turtles - OSDI ‘10] Nested Interrupt Virtualization HostVMM Kernel User Space Guest VMM Virtual CPU Interface LRs
  34. 34. • Shadow LRs • Guest hypervisor traps when attempting to program virtual LRs • Host hypervisor handles traps by writing to shadow LRs • Hardware uses shadow LRs when running the nested VM Nested Interrupt Virtualization
  35. 35. Implementation Status • RFC v1 on @kvmarm by Jintack Lim
 (Columbia University) • CONFIG_KVM_ARM_NESTED_HYP • vcpu->arch.features & KVM_ARM_VCPU_NESTED_VIRT
  36. 36. To Do • Must expose EL2 registers to user space • Mostly scattered out over existing files. Should we try to isolate more? • Hard-coded addresses and interrupt numbers • Reverse map for shadow stage 2 page tables • More efficient emulation of TLBI instructions • Get rid of config option and use command line parameter instead • Hypercalls from the VM vs. virtual self-hypercalls and PSCI
  37. 37. Questions? and please review the patches…
  38. 38. Backup Slides
  39. 39. KVM/ARM Nested Virtualization • VHE is fun with nested virtualization • We don’t set the E2H bit • The VM thinks it runs in EL2 using VHE so uses EL1 register accesses to access EL2 registers • But really does run in EL1 and doesn’t need to trap except on a few registers with different bit configuration CPU Virtualization - VHE
  40. 40. KVM/ARM Nested Virtualization • EL2 is separate translation regime • No ASIDs - cannot alias with EL1 translations • VMID not used • Emulating virtual EL2 in EL1 • Separate VMID for virtual EL2 • Always use ASID 0 Memory Virtualization - VMIDs and ASIDs

×