Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

XPDS16: Porting Xen on ARM to a new SOC - Julien Grall, ARM


Published on

Adding support for you new shiny board in Xen on ARM is a simple task once you get a kernel running on bare metal.

This session will cover the different steps to port Xen on ARM from the firmware to the shell prompt in DOM0.

We will give you tips on the common pitfalls when you have your hypervisor, or your DOM0 kernel crashing. We will also provide suggestion on how to debug when the console is not working.

Published in: Technology
  • Login to see the comments

XPDS16: Porting Xen on ARM to a new SOC - Julien Grall, ARM

  1. 1. Porting Xen on ARM to a new SOC Julien Grall <> Xen Developper Summit 2016 © ARM 2016
  2. 2. Xen Architecture 2 © ARM 2016
  3. 3. Virtualization ARMv7 and ARMv8 Provides virtualization for Timer Interrupt Controller Page Table 3 © ARM 2016
  4. 4. Virtualization - 2 4 © ARM 2016
  5. 5. Xen on ARM 5 © ARM 2016
  6. 6. Dom0 First guest to start Known as the hardware domain Nearly all devices are assigned to DOM0 Serial, IOMMU, Timer and GIC are used by Xen Some devices can be blacklisted by Xen DOM0 kernel should discover devices via ACPI or Device Tree 6 © ARM 2016
  7. 7. Groundwork 7 © ARM 2016
  8. 8. Preparation before porting Before starting to port Xen, some groundwork needs to be done: Check the hardware support Having the firmware/bootloader to boot the image at EL2 Having an OS supporting the targeted platform 8 © ARM 2016
  9. 9. Hardware ARMv7 and ARMv8 processor with virtualization extension General Interrupt Controller (GIC) v2 or later 9 © ARM 2016
  10. 10. Firmware and bootloader The firmware or bootloader must drop into EL2 (hypervisor) before starting Xen. Some vendors locked down the firmware/bootloader to drop into Non-secure EL1 (kernel mode). Hypervisor Call instruction (HVC) must be enabled. It can be done by setting SCR EL3.HCE (AArch64) or SCR.HCE (AArch32) to 1. 10 © ARM 2016
  11. 11. Firmware tables Xen supports the below firmware tables out-of-box: Device Tree ACPI 6.0 and onwards UEFI only Technical preview in Xen 4.7 DOM0 with ACPI support has been merged for Linux 4.8 11 © ARM 2016
  12. 12. DOM0 kernel Before adding Xen in the equation, it is highly recommended to get the kernel booting natively DOM0 support is upstreamed in Linux Adding support to any other kernel is easy See 12 © ARM 2016
  13. 13. DOM0 kernel - Linux DOM0 support has been added in Linux 3.8 It is recommended to use the latest release when possible Minimal list of options to enable: CONFIG_XEN_DOM0=y CONFIG_XEN=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_HVC_XEN=y CONFIG_XEN_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y 13 © ARM 2016
  14. 14. Porting Xen 14 © ARM 2016
  15. 15. A single binary to rule them all A single Xen binary can be loaded via different methods (e.g multiboot, UEFI). boot on multiple hardware. 15 © ARM 2016
  16. 16. It is recommended to use the latest version of Xen when porting to a new SOC. 16 © ARM 2016
  17. 17. Early debugging with Xen Xen provides early printk to debug crash before the UART driver is initialized. Only available when CONFIG DEBUG=y Xen will not be portable, intented only for development UART selected on the build command line with CONFIG EARL PRINTK=mach CONFIG EARL PRINTK= INC>, BASE ADDRESS>, OTHER OPTIONS> More details on 17 © ARM 2016
  18. 18. Early debugging with Xen - 2 Major UARTs supported: pl011, 8250,... 8250: CONFIG EARL PRINTK=8250, BASE ADDRESS>, REG SHIFT> REG SHIFT> is the left-shift to apply to register offsets within the uart (optional). pl011: CONFIG EARL PRINTK=pl011, BASE ADDRESS>, BAUD RATE> BAUD RATE is optional. We recommend to let the bootloader setting the baud rate. 18 © ARM 2016
  19. 19. Getting the firmware to load Xen and DOM0 The firmware needs to load in memory Xen, DOM0 kernel and potentially others modules (e.g initramfs, XSM...). There are 3 methods to do it: Multiboot UEFI GRUB via UEFI (work in progress) 19 © ARM 2016
  20. 20. Loading Xen and DOM0 using multiboot Multiboot is a protocol based on Device Tree. It is used to describe where the kernel, initramfs... reside in memory. An example to generate multiboot nodes with U-Boot can be found on the wiki. Boot_Modules 20 © ARM 2016
  21. 21. Loading Xen and DOM0 via UEFI On AArch64, Xen is built as an EFI application. A configuration file is used to describe: The command line The binaries to load (device tree, kernel, initramfs...) in memory. The configure file could be passed to the EFI application using the parameter -cfg=myxen.cfg 21 © ARM 2016
  22. 22. Loading Xen and DOM0 via UEFI - Example Configuration file example for UEFI: [global] default=model [model] options=console=dtuart conswitch=x dom0_max_vcpus=2 dtuart=serial0 kernel=vmlinuz console=hvc0 earlycon=pl011,0xf2a00000 root=/dev/ram1 rootwait ramdisk=initrd.img dtb=model.dtb 22 © ARM 2016
  23. 23. Troubleshooting - Xen is not entering in EL2 Xen will panic when it is not entered in EL2 with the following message: For AArch32: - Xen must be entered in Hyp mode - - Please update the bootloader - For AArch64: - Xen must be entered in NS EL2 mode - - Please update the bootloader - 23 © ARM 2016
  24. 24. What to do if Xen is not entered in EL2? Even if the hardware supports virtualization extensions, the firmware/bootloader may be configured to enter the kernel/hypervisor in EL1. Find a version which dropped in EL2. Find the source code and modify it to enter the hypervisor in EL2. 24 © ARM 2016
  25. 25. Platform specific code In most of the case, platform specific code is not necessary. Hooks in the core code is provided specific initialization is required. Platform code resides in xen/arch/arm/platforms 25 © ARM 2016
  26. 26. Platform specific code - 2 List of callbacks available: /* Platform initialization */ int (*init)(void); int (*init_time)(void); int (*smp_init)(void); int (*cpu_up)(int cpu); /* Specific mapping for dom0 */ int (*specific_mapping)(struct domain *d); /* Platform reset */ void (*reset)(void); /* Platform power-off */ void (*poweroff)(void); /* * Platform blacklist devices * List of devices which must not pass-through to a guest */ const struct dt_device_match *blacklist_dev; 26 © ARM 2016
  27. 27. UART support Xen has multiple UART drivers (pl011, 8250,...). They can be found in xen/drivers/char. The UART used by Xen will not be available for DOM0. A virtual UART will shadow the real one. Useful if the kernel use early printk Very basic: Only write is supported The UART configuration can be read from: the parameter dtuart=cfg stdout-path in the device tree The SCPR table in ACPI 27 © ARM 2016
  28. 28. Debugging DOM0 kernel Xen console Switch from DOM0 console to Xen console via CTLR-a three times Useful key 0 Dump Dom0 vCPUs q Domains information e Event channel information R Reboot the machine 28 © ARM 2016
  29. 29. Using Xen debugging facilities in the kernel Use of hvc 0xFFXX Supported when Xen is compiled with debug=y Requires to modify the kernel 0xFFEX 0xFFFD 0xFFFE 0xFFFF Print the register rX/xX Print the program counter Print the character stored in r0/x0 Dump the state of the vCPU 29 © ARM 2016
  30. 30. SMP support Xen is able to bring up secondary processors via different protocols: Power State Coordination Interface (PSCI) This is the recommended protocol to bring up CPU. PSCI 0.1, 0.2 and 1.0 supported It can be used for rebooting the platform (PSCI >= 0.2). Spin table (AArch64 only) Platform specific bringup (AArch32 only) This protocol should be avoided in favor of PSCI. It can be implemented with the callcack smp init and cpu up. 30 © ARM 2016
  31. 31. Upstreaming Even if your platform does not require platform specific code, it is recommended to Document the step to boot Xen on the wiki. Testing new release of Xen. 31 © ARM 2016
  32. 32. Where to ask questions? devel ML: #xenarm or #xendevel on freenode 32 © ARM 2016
  33. 33. Questions? 33 © ARM 2016
  34. 34. The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. Copyright © 2016 ARM Limited © ARM 2016