Successfully reported this slideshow.
Sep 26, 2016
Adding support for you new shiny board in Xen on ARM is a simple task once you get a kernel running on bare metal.
This session will cover the different steps to port Xen on ARM from the firmware to the shell prompt in DOM0.
We will give you tips on the common pitfalls when you have your hypervisor, or your DOM0 kernel crashing. We will also provide suggestion on how to debug when the console is not working.
Porting Xen on ARM to a new SOC
Julien Grall <firstname.lastname@example.org>
Xen Developper Summit 2016
© ARM 2016
2 © ARM 2016
ARMv7 and ARMv8
Provides virtualization for
3 © ARM 2016
Virtualization - 2
4 © ARM 2016
Xen on ARM
5 © ARM 2016
First guest to start
Known as the hardware domain
Nearly all devices are assigned to DOM0
Serial, IOMMU, Timer and GIC are used by Xen
Some devices can be blacklisted by Xen
DOM0 kernel should discover devices via ACPI or Device Tree
6 © ARM 2016
7 © ARM 2016
Preparation before porting
Before starting to port Xen, some groundwork needs to be done:
Check the hardware support
Having the firmware/bootloader to boot the image at EL2
Having an OS supporting the targeted platform
8 © ARM 2016
ARMv7 and ARMv8 processor with virtualization extension
General Interrupt Controller (GIC) v2 or later
9 © ARM 2016
Firmware and bootloader
The firmware or bootloader must drop into EL2 (hypervisor) before starting Xen.
Some vendors locked down the firmware/bootloader to drop into Non-secure EL1 (kernel mode).
Hypervisor Call instruction (HVC) must be enabled.
It can be done by setting SCR EL3.HCE (AArch64) or SCR.HCE (AArch32) to 1.
10 © ARM 2016
Xen supports the below firmware tables out-of-box:
ACPI 6.0 and onwards
Technical preview in Xen 4.7
DOM0 with ACPI support has been merged for Linux 4.8
11 © ARM 2016
Before adding Xen in the equation, it is highly recommended to get the kernel booting natively
DOM0 support is upstreamed in Linux
Adding support to any other kernel is easy
12 © ARM 2016
DOM0 kernel - Linux
DOM0 support has been added in Linux 3.8
It is recommended to use the latest release when possible
Minimal list of options to enable:
13 © ARM 2016
14 © ARM 2016
A single binary to rule them all
A single Xen binary can
be loaded via different methods (e.g multiboot, UEFI).
boot on multiple hardware.
15 © ARM 2016
It is recommended to use the latest
version of Xen when porting to a new
16 © ARM 2016
Early debugging with Xen
Xen provides early printk to debug crash before the UART driver is initialized.
Only available when CONFIG DEBUG=y
Xen will not be portable, intented only for development
UART selected on the build command line with
CONFIG EARL PRINTK=mach
CONFIG EARL PRINTK= INC>, BASE ADDRESS>, OTHER OPTIONS>
More details on
17 © ARM 2016
Early debugging with Xen - 2
Major UARTs supported: pl011, 8250,...
8250: CONFIG EARL PRINTK=8250, BASE ADDRESS>, REG SHIFT>
REG SHIFT> is the left-shift to apply to register offsets within the uart (optional).
pl011: CONFIG EARL PRINTK=pl011, BASE ADDRESS>, BAUD RATE>
BAUD RATE is optional. We recommend to let the bootloader setting the baud rate.
18 © ARM 2016
Getting the firmware to load Xen and DOM0
The firmware needs to load in memory Xen, DOM0 kernel and potentially others modules (e.g
initramfs, XSM...). There are 3 methods to do it:
GRUB via UEFI (work in progress)
19 © ARM 2016
Loading Xen and DOM0 using multiboot
Multiboot is a protocol based on Device Tree.
It is used to describe where the kernel, initramfs... reside in memory.
An example to generate multiboot nodes with U-Boot can be found on the wiki.
20 © ARM 2016
Loading Xen and DOM0 via UEFI
On AArch64, Xen is built as an EFI application.
A configuration file is used to describe:
The command line
The binaries to load (device tree, kernel, initramfs...) in memory.
The configure file could be passed to the EFI application using the parameter -cfg=myxen.cfg
21 © ARM 2016
Loading Xen and DOM0 via UEFI - Example
Configuration file example for UEFI:
options=console=dtuart conswitch=x dom0_max_vcpus=2 dtuart=serial0
kernel=vmlinuz console=hvc0 earlycon=pl011,0xf2a00000 root=/dev/ram1 rootwait
22 © ARM 2016
Troubleshooting - Xen is not entering in EL2
Xen will panic when it is not entered in EL2 with the following message:
- Xen must be entered in Hyp mode -
- Please update the bootloader -
- Xen must be entered in NS EL2 mode -
- Please update the bootloader -
23 © ARM 2016
What to do if Xen is not entered in EL2?
Even if the hardware supports virtualization extensions, the firmware/bootloader may be configured
to enter the kernel/hypervisor in EL1.
Find a version which dropped in EL2.
Find the source code and modify it to enter the hypervisor in EL2.
24 © ARM 2016
Platform specific code
In most of the case, platform specific code is not necessary.
Hooks in the core code is provided specific initialization is required.
Platform code resides in xen/arch/arm/platforms
25 © ARM 2016
Platform specific code - 2
List of callbacks available:
/* Platform initialization */
int (*cpu_up)(int cpu);
/* Specific mapping for dom0 */
int (*specific_mapping)(struct domain *d);
/* Platform reset */
/* Platform power-off */
* Platform blacklist devices
* List of devices which must not pass-through to a guest
const struct dt_device_match *blacklist_dev;
26 © ARM 2016
Xen has multiple UART drivers (pl011, 8250,...).
They can be found in xen/drivers/char.
The UART used by Xen will not be available for DOM0.
A virtual UART will shadow the real one.
Useful if the kernel use early printk
Very basic: Only write is supported
The UART configuration can be read from:
the parameter dtuart=cfg
stdout-path in the device tree
The SCPR table in ACPI
27 © ARM 2016
Debugging DOM0 kernel
Switch from DOM0 console to Xen console via CTLR-a three times
0 Dump Dom0 vCPUs
q Domains information
e Event channel information
R Reboot the machine
28 © ARM 2016
Using Xen debugging facilities in the kernel
Use of hvc 0xFFXX
Supported when Xen is compiled with debug=y
Requires to modify the kernel
Print the register rX/xX
Print the program counter
Print the character stored in r0/x0
Dump the state of the vCPU
29 © ARM 2016
Xen is able to bring up secondary processors via different protocols:
Power State Coordination Interface (PSCI)
This is the recommended protocol to bring up CPU.
PSCI 0.1, 0.2 and 1.0 supported
It can be used for rebooting the platform (PSCI >= 0.2).
Spin table (AArch64 only)
Platform specific bringup (AArch32 only)
This protocol should be avoided in favor of PSCI.
It can be implemented with the callcack smp init and cpu up.
30 © ARM 2016
Even if your platform does not require platform specific code, it is recommended to
Document the step to boot Xen on the wiki.
Testing new release of Xen.
31 © ARM 2016
Where to ask questions?
devel ML: email@example.com
#xenarm or #xendevel on freenode
32 © ARM 2016
33 © ARM 2016
The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2016 ARM Limited
© ARM 2016