Julien Grall discusses how you can leverage Xen on ARM for BSD guests

1. How to port your BSD to run as Xen on ARM
guest
Julien Grall
julien.grall@citrix.com
BSDCan 2015

2. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Xen
Type-I hypervisor
Support for ARM v7 and ARM v8 with virtualization extension
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 2 / 27

3. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Example of hardware supported
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 3 / 27

4. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Xen architecture
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 4 / 27

5. Intro DOM0 Requirements Debugging FreeBSD Conclusion
ARM architecture
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 5 / 27

6. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Xen on ARM architecture
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 6 / 27

7. Intro DOM0 Requirements Debugging FreeBSD Conclusion
DOM0
First guest to start
Nearly every devices are assigned to DOM0
Serial, IOMMU, Timer and GIC are used by Xen
Some devices can be blacklisted by Xen
DOM0 kernel should use the Device Tree to discover the
hardware
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 7 / 27

8. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Requirements
Guest boot ABI
Device tree support
Specific memory attribute
Xen PV drivers
Copy of xen/include/public
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 8 / 27

9. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Guest boot ABI
Interface of the virtual machine:
Loaders:
UEFI image
Linux zImage/Image
See xc dom loader structure in libxc to add a new loader
Use of PSCI to bring up secondary CPUs
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 9 / 27

10. Intro DOM0 Requirements Debugging FreeBSD Conclusion
UEFI Image
Only supported for AArch64 guest
Require to build UEFI firmware for DOMU
tianocore edk II
https://wiki.linaro.org/LEG/UEFIforXEN
No difference with baremetal after the firmware is booted
Specification http://www.uefi.org/specifications
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 10 / 27

11. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Linux zImage/Image
Kernel loaded at a 4K aligned-address
Specific values on some registers
ARM 32 bits
r0 = 0
r1 = 0xffffffff
r2 = Device Tree physical address
ARM 64 bits
x0 = Device Tree physical address
x1 ... x4 = 0
MMU disabled
Data cache disabled
Instruction cache in an unknown state
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 11 / 27

12. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Device Tree: What is it?
Tree data structure with nodes that describes the physical
devices in a system
Working group to standardize device core bindings
devicetree@vger.kernel.org
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 12 / 27

13. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Device Tree provided by Xen
Basic Device Tree generated by the toolstack which contains:
CPUs
Memory
Timer
GIC
Hypervisor
PSCI
Guest must support Device Tree in order to run on every Xen
version
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 13 / 27

14. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Memory
Memory layout not fixed
Differ between Xen 4.4 and Xen 4.5
Documented in xen/include/public/arch-arm.h
At least one RAM region will be located under 4GB
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 14 / 27

15. Intro DOM0 Requirements Debugging FreeBSD Conclusion
MMU and Memory
MMU-less and data cache disabled guests are supported
except for hypercall-based IO
RAM restriction when MMU is enabled:
3GB when short-descriptor is used
1TB when LPAE is used
RAM attribute should be Write-Through or Write-Back
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 15 / 27

16. Intro DOM0 Requirements Debugging FreeBSD Conclusion
PV protocol
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 16 / 27

17. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Hypercalls
Use of HVC instruction
Hypercall tag XEN HYPERCALL TAG = 0xEA1
xen/public/arch-arm.h provides hypercalls convention
ARM 32 bits
r0 ...r4 used to store arguments
hypercall number is passed in r12
return value is in r0
ARM 64 bits
x0 ... x4 used to store arguments
hypercall number is passed in x16
return value is in x0
hypercall assembly wrappers provides in Linux
arch/arm{,64}/xen/hypercall.S
Dual-license GPLv2 and BSD
memory hypercall based on 4K page granularity
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 17 / 27

18. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Xen PV drivers
Xen core architecture
Xenstore
Grant-Table
Event-channel
Xen device drivers
Console
Block
Network
Framebuffer
Drivers already available under:
BSD license in FreeBSD/Mini-OS
Dual-license GPLv2 and BSD in Linux
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 18 / 27

19. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Debugging the guest kernel
xenctx domid vcpuid
Get the state of a VCPU
To use in DOM0
Xen console
Switch from DOM0 console to Xen console via CTLR-a three
times
Useful key
q Domains information
e Event channel informations
R Reboot the machine (useful for DOM0 debugging)
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 19 / 27

20. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Using Xen debugging facilities in the kernel
Use of HVC 0xFFxx
Supported when Xen is compiled with debug=y
Requires to modify the guest
0xFFEX
0xFFFD
0xFFFE
0xFFFF
Print the register rX/xX
Print the program counter
Print the character stored in r0/x0
Dump the state of the VCPU
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 20 / 27

21. Intro DOM0 Requirements Debugging FreeBSD Conclusion
FreeBSD on Xen
Support for x86 PVHVM and PVH
Support for PVH DOM0 on x86
Support of ARM 32 and ARM 64 architecture
Device Tree bindings compatible with the Linux ones
On going work to support Xen for ARM
Less than 500 lines of Xen arch specific code
PV drivers shared with x86
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 21 / 27

22. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Xen PV drivers
Update interface headers to Xen 4.4
FreeBSD is based on Xen 4.2 headers
ARM interface was not set in stone
Drivers common with x86
Use the right xen type (xen pfn t, xen ulong t,...)
Support for HVM in console drivers
Rework event channel handling
was x86 specific
still missing features
suspend/resume
pirq
common interrupt framework?
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 22 / 27

23. Intro DOM0 Requirements Debugging FreeBSD Conclusion
ARM 32
New kernel config XENHVM created
Only support for guest with 1 VCPU
Loader:
Missing support using Device Tree with Linux boot ABI
zImage support?
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 23 / 27

24. Intro DOM0 Requirements Debugging FreeBSD Conclusion
ARM 64
Xen enabled in the GENERIC kernel config
Driver to parse Xen DT bindings added
No modification necessary in the loader
UEFI based image
Kernel loaded from the Filesystem
Device Tree provided by UEFI
Missing SMP bring up based on PSCI
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 24 / 27

25. Intro DOM0 Requirements Debugging FreeBSD Conclusion
OS supported by Xen
Out-of-box
Future support
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 25 / 27

26. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Questions?
Xen devel ML: xen-devel@lists.xenproject.org
Xen user ML: xen-user@lists.xenproject.org
#xenarm on freenode
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 26 / 27

27. Intro DOM0 Requirements Debugging FreeBSD Conclusion
Fin
BSDCan 2015 How to port your BSD to run as Xen on ARM guest 27 / 27