Linaro connect : Introduction to Xen on ARM


Published on

With the introduction of virtualization extensions on ARM processors, the Xen community has taken steps to add ARM support for ARM CPUs to the Xen Hypervisor. This port is executed as part of of the Hypervisor project, with no separate codebase.

Xen engineers will describe the key challenges they have overcome, current technical status and discuss about the next steps.

Published in: Technology

Linaro connect : Introduction to Xen on ARM

  1. 1. Xen on ARMStefano Stabellini and Ian Campbell
  2. 2. Virtualization:why it matters
  3. 3. Xen: the gears of the cloud● large user base more than 10 million individuals users● power the largest clouds in production● not just for servers
  4. 4. Xen: Open SourceGPLv2 with DCO (like Linux)Diverse contributor community
  5. 5. Xen: Open Sourcesource:Mike Day
  6. 6. Xen Architecture Dom0 DomU DomU DomU PV backends PV Frontends PV Frontends PV Frontends HW drivers Xen Hardware
  7. 7. PV Protocols Request Producer Backend Frontend Request Consumer Response Consumer Response Producer
  8. 8. Xen Architecture:driver domains Disk Driver Network Dom0 DomU Domain Driver Domain BlockBack NetBack BlockFront Toolstack Disk Driver Network Driver NetFront Xen Hardware
  9. 9. Xen: advantages● small surface of attack● isolation● resilience● specialized algorithms (scheduler)
  10. 10. Xen Architecture:HVM guests Dom0 HVM DomU stubdom HVM DomU IO emulation IO emulation QEMU PV backends PV Frontends HW drivers Xen Hardware
  11. 11. Xen upstream status● Xen (Dom0 and DomU support, PV frontends and backends) fully upstream in Linux since v3.0 A single 3.0.0 Linux kernel image boots on native, on Xen as domU, as dom0 and PV on HVM guest● Xen upstream in QEMU since v1.3● Xen supported by SuSE, Debian, Ubuntu, Fedora, CentOS, NetBSD and more
  12. 12. ARM Servers coming to market4GB RAM, 4 cores per node3 x 6 x 4 x 4 = 288 cores single node virtualization - manageability -
  13. 13. Design goals● exploit the hardware as much as possible● one type of guest ○ no PVOPs ○ use PV interfaces for IO● Rearchitected for the modern age: ○ no QEMU ○ no compat code ○ no shadow pagetables
  14. 14. Xen on ARM architecture
  15. 15. Xen on ARM architecture
  16. 16. Exploit the hardwareExploit the hardware virtualization extensionssupport as much as possible:● hypervisor mode● MMU: second stage translation ○ no PV MMU calls: no need for PVOPs ○ no shadow pagetables: -10721 lines of code!!● hypercall: HVC● generic timers
  17. 17. General Interrupt Controlleran interrupt controller with virtualization support● use the GIC to inject hardware interrupts into dom0● use the GIC to inject event notifications into any guest domains with Xen support ○ use PPI 31 ○ advertise the IRQ via Device Tree
  18. 18. One type of guest to rulethem all
  19. 19. One type of guestLike PV guests do it:● support booting from a supplied kernel● no emulated devices● use PV interfaces for IO no need for QEMU
  20. 20. One type of guestLike HVM guests do it:● no PV MMU calls: exploit HW nested paging● same entry point on native and on Xen● use Device Tree to discover Xen presence● no unnecessary devices in the Device Tree● simple device emulation can be done in Xen no need for QEMU
  21. 21. Device TreeUse Device Tree to describe the virtual platformhypervisor { compatible = "xen,xen", "xen,xen-4.2"; reg = <0xb0000000 0x20000>; interrupts = <1 15 0xf08>;};
  22. 22. Device TreeUse Device Tree to describe the virtual platform version of the Xen ABIhypervisor { compatible = "xen,xen", "xen,xen-4.2"; reg = <0xb0000000 0x20000>; interrupts = <1 15 0xf08>;}; Grant table memory area event notifications IRQ
  23. 23. The hypercall callingconventionthe hypercall interface:● hvc instruction● hypervisor specific imm 0xEA1● hypercall arguments passed in registers
  24. 24. a 64 bit "ready" ABI● a single hypercall ABI for 32 bit guests and 64 bit guests no compat code in Xen ○ 2600 lines of code lighter
  25. 25. ARMv8● Builds on foundations laid by ARMv7 ○ xen/arch/arm mostly common code● Initially 32 bit dom0+domU on 64 ○ Kernels already ready ○ 64-bit guest support in progress● Running on AEMv8 model ○ Foundation Model soon
  26. 26. Code sizesometimes smaller is better Common ARMv7 ARMv8 Total xen/arch/arm 5,122 1,969 821 7,912 C 5,023 406 344 5,773 ASM 99 1,563 477 2,139 xen/include/asm-arm 2,315 563 666 3,544 TOTAL 7,437 2,532 1,487 11,456● Entire hypervisor ~200,000LOC ○ X86 (64-bit only) ~100,000LOC (~4,000 ASM) ■ ~22,000: HVM. ~14,000 MMU
  27. 27. Common Code mrc p15, 0, r0, c0, c0, 5 => WTF?xen/include/asm-arm/cpregs.h: #define __CP32(r, coproc, opc1, crn, crm, opc2) coproc, opc1, r, crn, crm, opc2 #define CP32(r, name...) __CP32(r, name) #define MPIDR p15,0,c0,c0,5 mrc CP32(r0, MPIDR)
  28. 28. ChallengesFrom the emulator to real hardware:
  29. 29. War Stories ChallengesFrom the emulator to real hardware:● barriers and flushes● cache coherency● GIC and race conditions● virt_timer documentation bugs
  30. 30. Status of the Project:ARMv7● Xen and Dom0 booting on Versatile Express Cortex A15 and Arndale● XL (Xen toolstack) ported to ARM● PV console, disk and network working● basic VM lifecycle operations functional● Xen and Linux ARM patches fully upstream
  31. 31. Status of the Project:ARMv8● Xen booting 64 bit● Dom0 32 bit boots on Xen 64 bit● 32 bit guest creation and destruction● Shared code means most features developed on ARMv7 Just Work
  32. 32. RoadmapXen 4.3 ○ full ARMv7 support ○ ARMv8 64-bit port of the hypervisorXen 4.4 ○ increase HCL ○ automated testing ○ ARMv8 64-bit virtual machines and tools ○ PCI passthroughLinux 3.10/3.11 ○ full ARMv8 64-bit Xen guest support
  33. 33. Open Questions● Boot Protocols ○ Multiboot?● Bootloaders ○ U-boot ○ UEFI (runtime services?)● ACPI● SysMMU
  34. 34. Demo
  35. 35. Xen and Linaro Opportunities: ○ testing ○ platform support ○ boot protocols ○ hardware features
  36. 36. More Information● Talk to us, we are here all week: ○ Keynote: Tuesday @ 08:30, Grand Ballroom ○ Xen at Demo Friday●● Xen on ARM●