SlideShare a Scribd company logo
Static Partitioning with Xen, LinuxRT, and Zephyr:
a concrete end-to-end example
Stefano Stabellini
Embedded Linux Conference 2022
2 |
Mixed-Criticality and Static Partitioning
• It is common to have a mix of critical and non-critical components
• Real-time vs. non real-time
• Function-critical vs. user interface or cloud interface
• Running them in the same environment put the critical function at risk
• A failure in the non-critical component causes a critical system crash
• The non-critical component using too much CPU time causes a missed deadline
• Don’t put all your eggs in one basket
• Static partitioning allows you to run them separately in isolation
• Many baskets!
• Each domain has direct hardware access (IOMMU protected)
• Real-time (< 5us latency)
• Strong isolation between the domains
• Real-time isolation
• Failure isolation
• By default, each component has only enough privilege to do what it needs to do
2
Xen
Critical
Non-
Critical
Non-
Critical
3 |
Static Partitioning
Linux Dom0
Dom1 Zephyr
LinuxRT
Programmable
Logic
GEM0 MMC0
GEM1 MMC1
TTC
SATA
4 |
Xen Features for Static Partitioning
• Xen Dom0less: static domains configuration for Xen
• Statically-defined domains
• Sub-second boot times
• No need to create new VMs at runtime (possible but not required)
• No need for Dom0 (possible but not required)
• Each domain has direct hardware access
• Real-time and Cache Coloring: isolation from cache interference
• Deterministic interrupt latency at 3us
• VM-to-VM communication
• Static shared memory and event channels
• PV drivers for device sharing
• Safety-certifiability improvements underway
• MISRA C
• Deterministic interrupt handling and memory allocations
5 |
Step1: a simple Xen dom0less static configuration
• Board: Xilinx ZCU102
• Xen and 3 VMs
• Regular Linux Dom0 from Xilinx BSP
• LinuxRT DomU with a minimal busybox rootfs
• Zephyr DomU
• Configuration parameters
• Memory allocation
• CPUs allocation
• Device assignment added later
• Do we need Dom0?
• Useful for system monitoring and domains restart
• If dom0 is required, the Xen userspace tools (xl) need to be built and added to the dom0 rootfs
Xen
LinuxRT
Linux
Dom0
Zephyr
6 |
Xen Dom0less boot
U-Boot
Xen
Linux Dom0 LinuxRT DomU Zephyr DomU
CPU 0 CPU 1-2 CPU 3
boots
boots
7 |
ImageBuilder
• A tool to automatically generate a bootable configuration from a simple text config file
• https://gitlab.com/xen-project/imagebuilder
MEMORY_START="0x0"
MEMORY_END="0x80000000"
DEVICE_TREE="elc22demo1/mpsoc.dtb"
XEN="elc22demo1/xen"
DOM0_KERNEL="elc22demo1/xilinx-linux"
DOM0_RAMDISK="elc22demo1/xen-rootfs.cpio.gz"
DOM0_MEM=1024
NUM_DOMUS=2
DOMU_KERNEL[0]="elc22demo1/linuxrt"
DOMU_RAMDISK[0]="elc22demo1/initrd.cpio"
DOMU_MEM[0]=2048
DOMU_VCPUS[0]=2
DOMU_KERNEL[1]="elc22demo1/zephyr.bin"
DOMU_MEM[1]=128
# bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c elc22demo/config
8 |
ImageBuilder boot.source
• ImageBuilder generates boot.source and boot.scr to be loaded by u-boot
• Editing boot.source is possible
• Then call mkimage by hand
# mkimage -A arm64 -T script -C none -a 0xc00000 -e 0xc00000 -d boot.source boot.scr
9 |
Building Xen with or without cache coloring support
• Building the hypervisor is very simple (it cross-compiles like Linux)
• Cache coloring is available as a menuconfig option
• Upstreaming in progress
• Currently available at https://github.com/Xilinx/xen
• Nothing else is needed unless you plan to use the Xen tools (xl) in dom0
# export CROSS_COMPILE=/path/to/cross-compiler
# export XEN_TARGET_ARCH=arm64
# cd xen.git/xen/
# make menuconfig
Architecture Features à L2 cache coloring
# make
10 |
Building a Linux/LinuxRT kernel for Xen
• Vanilla Linux releases and binaries work
• Using Linux v5.17 for the demo
• The following Kconfig options are recommended:
• CONFIG_XEN
• CONFIG_XEN_BLKDEV_BACKEND
• CONFIG_XEN_NETDEV_BACKEND
• CONFIG_XEN_GNTDEV
• CONFIG_XEN_GRANT_DEV_ALLOC
• CONFIG_XEN_GRANT_DMA_ALLOC
• CONFIG_SERIAL_AMBA_PL011
• CONFIG_SERIAL_AMBA_PL011_CONSOLE
• CONFIG_BRIDGE
• Either CONFIG_XEN or CONFIG_SERIAL_AMBA_PL011 is needed for console output
• 2 patches for PV drivers support for dom0less
• Patches already accepted and soon to be upstream
11 |
Building a Dom0 rootfs
• Dom0 requires the Xen userspace tools in Dom0
• Build a Linux rootfs for Dom0 with Xen support using Yocto:
DISTRO_FEATURES:append = " xen"
DISTRO_FEATURES:append = " virtualization"
IMAGE_INSTALL:append = " xen-base zlib-dev"
ASSUME_PROVIDED += ” iasl-native"
12 |
Building Zephyr with Xen support
• Initial Zephyr build environment setup required
• See https://docs.zephyrproject.org/latest/develop/getting_started/index.html
• Use the existing xenvm machine
• Build time Device Tree: boards/arm64/xenvm/xenvm.dts
• Memory mappings: soc/arm64/xenvm/mmu_regions.c
• To run Zephyr as Dom0less guest:
• change xen_hvc with xen_consoleio_hvc in xenvm.dts
• build Zephyr with Dom0 support
• build Xen with DEBUG enabled
# west build -t menuconfig
-> SoC/CPU/Configuration Selection (Xen virtual machine on aarch64)
-> Xen virtual machine on aarch64
-> Zephyr as Xen Domain 0
-> Device Drivers
-> Serial Drivers
-> Xen hypervisor Dom0 console UART driver
# west build -b xenvm samples/hello_world/
13 |
Step2: Add Device Assignment
• Device Assigment:
• Direct hardware access
• Protected by the IOMMU
• Example configuration:
• Assigning the Network Device (GEM) to LinuxRT
• Assigning the TTC timer to Zephyr
Xen
LinuxRT
Linux
Dom0
Zephyr
GEM0 TTC0
14 |
Xen Device Assignment
• Based on “partial device trees”
• They describe the device for the domU
• They configure device assignment
• Everything under passthrough is copied
to the guest device tree
• Special device assignment
configuration properties:
• xen,reg
• xen,path
• xen,force-assign-without-iommu
• interrupt is used for interrupt remapping
• interrupt-parent must be 0xfde8
/dts-v1/;
/ {
#address-cells = <0x2>;
#size-cells = <0x2>;
passthrough {
compatible = "simple-bus";
ranges;
#address-cells = <0x2>;
#size-cells = <0x2>;
ethernet@ff0e0000 {
compatible = "cdns,zynqmp-gem";
[…]
reg = <0x0 0xff0e0000 0x0 0x1000>;
interrupt-parent = <0xfde8>;
interrupts = <0x0 0x3f 0x4 0x0 0x3f 0x4>;
xen,reg = <0x0 0xff0e0000 0x0 0x1000 0x0 0xff0e0000>;
xen,path = "/amba/ethernet@ff0e0000";
};
};
};
15 |
Xen Device Assignment and ImageBuilder
• Add xen,passthrough; under the corresponding node on the host device tree
• Add the partial device tree binaries to the ImageBuilder config file
• ImageBuilder will load them automatically for you at boot
[…]
NUM_DOMUS=2
DOMU_KERNEL[0]="elc22demo2/linuxrt"
DOMU_RAMDISK[0]="elc22demo2/initrd.cpio"
DOMU_MEM[0]=2048
DOMU_VCPUS[0]=2
DOMU_PASSTHROUGH_DTB[0]="elc22demo2/ethernet@ff0e0000.dtb"
DOMU_KERNEL[1]="elc22demo2/zephyr.bin"
DOMU_PASSTHROUGH_DTB[1]="elc22demo2/timer@ff110000.dtb"
DOMU_MEM[1]=128
16 |
Xilinx Xen Device Tree examples repository
• Repository with device tree passthrough examples: https://github.com/Xilinx/xen-passthrough-device-trees
• Simplified ImageBuilder configuration with PASSTHROUGH_DTS_REPO, DOMU_PASSTHROUGH_PATHS
• Based on file names and example files, no automatic device tree generation yet
• Careful about “axi” vs “amba”, the name needs to match everywhere!
• host DTB, passthrough DTB, xen,path property, ImageBuider config file
[…]
PASSTHROUGH_DTS_REPO="git@github.com:Xilinx/xen-passthrough-device-trees.git device-trees-2021.2"
NUM_DOMUS=2
DOMU_KERNEL[0]="elc22demo2/linuxrt"
DOMU_RAMDISK[0]="elc22demo2/initrd.cpio"
DOMU_MEM[0]=2048
DOMU_VCPUS[0]=2
DOMU_PASSTHROUGH_PATHS[0]="/axi/ethernet@ff0e0000”
DOMU_KERNEL[1]="elc22demo2/zephyr.bin"
DOMU_MEM[1]=128
DOMU_PASSTHROUGH_PATHS[1]="/axi/timer@ff110000"
17 |
Xen Partial Device Trees Automatic Generation
• System Device Tree: Device Tree based description of a full heterogeneous system
• Lopper
• a Device Tree manipulation tool
• takes a System Device Tree as input and generates multiple Device Trees as output
• Use Lopper to automatically generate Xen passthrough DTBs
• Just need to provide the host DTB and the device path, e.g. /amba/ethernet@ff0e0000
• Attend the presentation “System Device Tree and Lopper: Concrete Examples”, Thursday June 23 at 12PM!
18 |
Make use of a new device in Zephyr
• Add it to the board device tree boards/arm64/xenvm/xenvm.dts
• Map the device memory in soc/arm64/xenvm/mmu_regions.c
MMU_REGION_FLAT_ENTRY("TTC",
DT_REG_ADDR_BY_IDX(DT_INST(0, xlnx_ttcps), 0),
DT_REG_SIZE_BY_IDX(DT_INST(0, xlnx_ttcps), 0),
MT_DEVICE_nGnRnE | MT_P_RW_U_RW | MT_NS),
ttc0: timer@ff110000 {
compatible = "xlnx,ttcps";
status = "okay";
interrupts = <GIC_SPI 36 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>,
<GIC_SPI 37 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>,
<GIC_SPI 38 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>;
interrupt-names = "irq_0", "irq_1", "irq_2";
reg = <0x0 0xff110000 0x0 0x1000>;
label = "ttc0";
interrupt-parent = <&gic>;
clock-frequency = <8320000>;
};
© Copyright 2020 Xilinx
Step3: Xen Cache Coloring
L2
Core 1 Core 2 Core 3 Core 4
DDR
L1 L1 L1 L1
© Copyright 2020 Xilinx
Xen Cache Coloring
L2
Core 1 Core 2 Core 3 Core 4
DDR
L1 L1 L1 L1
© Copyright 2020 Xilinx
4CPUs clusters often share L2 cache
4Interference via L2 cache affects performance
¬ App0 running on CPU0 can cause cache entries
evictions, which affect App1 running on CPU1
¬ App1 running on CPU1 could miss a deadline due to
App0’s behavior
¬ It can happen between Apps running on the same OS &
between VMs on the same hypervisor
4Hypervisor Solution: Cache Partitioning,
AKA Cache Coloring
¬ Each VM gets its own allocation of cache entries
¬ No shared cache entries between VMs
¬ Allows real-time apps to run with deterministic IRQ
latency
D
D
R
Xen Cache Coloring
Core
1
Core
2
Core
3
Core
4
L2
22 |
Xen Cache Coloring and ImageBuilder
• Cache Coloring on Xen ZCU102
• 16 colors, each color corresponds to 256MB
• Color bitmask is 0xF000
• Configuration:
• Xen à color 0
• Dom0 à color 1-5
• LinuxRT à color 6-14
• Zephyr à color 15
# Xen and Dom0, use XEN_CMD in the ImageBuilder config:
XEN_CMD=“…xen_colors=0-0 dom0_colors=1-5 way_size=65536”
DOMU_COLORS[0]="6-14”
DOMU_COLORS[1]="15-15"
Dom0
LinuxRT Zephyr
CPU 0 CPU 1-2 CPU 3
Linux Dom0
ring
buffer
Step4: Shared Memory and Event Channels
24 |
Shared Memory Configuration
• A static shared memory region chosen at build time
• E.g. 0x7fe0_1000
• The address matter for cache coloring: 0x7fe0_1000 means color “1”
• Xen event channels can be used for notifications
• A BSD-licensed xen.h file with Xen hypercall definitions is provided availble
• Can be embedded in Zephyr and other RTOSes
• Makes it easy to add event channel support anywhere
• Static event channel definitions based on device tree available in the next Xen release
• Today event channels are created dynamically
• From the next release it is possible to configure them statically as part of the Xen domains (i.e. ImageBuilder)
• Static shared memory support also coming in the next Xen release
• Possible to define shared memory without editing the memory node
• Work in progress by ARM
25 |
Shared Memory Configuration
• Carve out an address range from the memory node
• 0xf7fe00000 - 0xf7ff00000
• Add it as a separate sram node
memory {
device_type = "memory";
reg = <0x00 0x00 0x00 0x7fe00000 0x08 0x00 0x00 0x80000000>;
};
sram@7fe00000 {
compatible = "mmio-sram";
reg = <0x0 0x7fe00000 0x0 0x100000>;
xen,passthrough;
};
26 |
Shared Memory Configuration
• Add shared memory to the domU passthrough device tree
passthrough {
compatible = "simple-bus";
ranges;
#address-cells = <0x2>;
#size-cells = <0x2>;
/* timer has been tested with baremetal app only */
timer@ff110000 {
[…]
sram@7fe01000 {
compatible = "mmio-sram";
reg = <0x0 0x7fe01000 0x0 0x1000>;
xen,reg = <0x0 0x7fe01000 0x0 0x1000 0x0 0x7fe01000>;
xen,force-assign-without-iommu = <0x1>;
};
27 |
Make use of the shared memory and event channels in Zephyr
• Map the shared memory
• Setup event channels
MMU_REGION_FLAT_ENTRY("RAM_SHARED",
0x7fe01000,
0x1000,
MT_NORMAL_NC | MT_P_RW_U_RW | MT_NS),
#include "xen.h”
[…]
bind.remote_dom = remote_domid;
bind.remote_port = remote_port;
bind.local_port = 0;
ret = xen_hypercall(EVTCHNOP_bind_interdomain, (unsigned long)&bind, 0, 0, HYPERVISOR_event_channel_op);
28 |
Make use of the shared memory and event channels in Linux
• Write a new driver
static int __init shared_mem_init(void)
{
char *str = "go";
struct resource r;
struct device_node *np = of_find_compatible_node(NULL, NULL, "xen,shared-memory-v1");
struct evtchn_alloc_unbound alloc_unbound;
of_address_to_resource(np, 0, &r);
shared_mem = phys_to_virt(r.start);
alloc_unbound.dom = DOMID_SELF;
alloc_unbound.remote_dom = 1; /* domid 1 */
HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &alloc_unbound);
irq = bind_evtchn_to_irqhandler_lateeoi(alloc_unbound.port, evtchn_handler, 0, “shared_intf", NULL);
irq = rc;
memcpy(shared_mem, str, 3);
mb();
/* send port number to other domain */
memcpy(shared_mem + 4, &alloc_unbound.port, sizeof(alloc_unbound.port));
return 0;
}
29 |
Step5: Dom0less with PV Drivers
Dom0
U-Boot
Xen
LinuxRT
CPU CPU
NetBack NetFront
init-dom0less Zephyr
CPU 3
30 |
Dom0less and PV Drivers
Linux Dom0 LinuxRT DomU
Zephyr DomU
CPU 0 CPU 1-2 CPU 3
Xen
31 |
Dom0less and PV Drivers
• Xen PV Drivers are used to share a device across multiple domains
• Share the network across multiple domains
• Share a disk across multiple domains, a partition to each domain
• PV network can also be used for VM-to-VM communication
• With dom0less, PV Drivers are instantiated after the boot is finished from Dom0
• Fast boot times: the critical application can start immediately, no need to wait
• PV drivers connect later after Dom0 completed booting
• A Dom0 environment with Xen userspace tools is required to setup the connection
• Use init-dom0less in Dom0 to initialize PV drivers, then hotplug a PV network connection
/usr/lib/xen/bin/init-dom0less
brctl addbr xenbr0; ifconfig xenbr0 192.168.0.1 up
xl network-attach 1
Questions?

More Related Content

What's hot

Fosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using VirtualizationFosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using Virtualization
The Linux Foundation
 
Yocto Project introduction
Yocto Project introductionYocto Project introduction
Yocto Project introduction
Yi-Hsiu Hsu
 
Rootlinux17: An introduction to Xen Project Virtualisation
Rootlinux17:  An introduction to Xen Project VirtualisationRootlinux17:  An introduction to Xen Project Virtualisation
Rootlinux17: An introduction to Xen Project Virtualisation
The Linux Foundation
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training
The Linux Foundation
 
Anatomy of the loadable kernel module (lkm)
Anatomy of the loadable kernel module (lkm)Anatomy of the loadable kernel module (lkm)
Anatomy of the loadable kernel module (lkm)
Adrian Huang
 
Arm device tree and linux device drivers
Arm device tree and linux device driversArm device tree and linux device drivers
Arm device tree and linux device drivers
Houcheng Lin
 
Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
Stefano Stabellini
 
Audio Drivers
Audio DriversAudio Drivers
Audio Drivers
Anil Kumar Pugalia
 
Booting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot imagesBooting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot images
Chris Simmonds
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
The Linux Foundation
 
ACPI Debugging from Linux Kernel
ACPI Debugging from Linux KernelACPI Debugging from Linux Kernel
ACPI Debugging from Linux Kernel
SUSE Labs Taipei
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
The Linux Foundation
 
Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020
Stefano Stabellini
 
Scheduling in Android
Scheduling in AndroidScheduling in Android
Scheduling in Android
Opersys inc.
 
Embedded Android : System Development - Part IV
Embedded Android : System Development - Part IVEmbedded Android : System Development - Part IV
Embedded Android : System Development - Part IV
Emertxe Information Technologies Pvt Ltd
 
Embedded Android : System Development - Part II (Linux device drivers)
Embedded Android : System Development - Part II (Linux device drivers)Embedded Android : System Development - Part II (Linux device drivers)
Embedded Android : System Development - Part II (Linux device drivers)
Emertxe Information Technologies Pvt Ltd
 
ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)
The Linux Foundation
 
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
The Linux Foundation
 
Linux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBLinux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 

What's hot (20)

Fosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using VirtualizationFosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using Virtualization
 
Yocto Project introduction
Yocto Project introductionYocto Project introduction
Yocto Project introduction
 
Rootlinux17: An introduction to Xen Project Virtualisation
Rootlinux17:  An introduction to Xen Project VirtualisationRootlinux17:  An introduction to Xen Project Virtualisation
Rootlinux17: An introduction to Xen Project Virtualisation
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training
 
Anatomy of the loadable kernel module (lkm)
Anatomy of the loadable kernel module (lkm)Anatomy of the loadable kernel module (lkm)
Anatomy of the loadable kernel module (lkm)
 
Arm device tree and linux device drivers
Arm device tree and linux device driversArm device tree and linux device drivers
Arm device tree and linux device drivers
 
Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
 
Audio Drivers
Audio DriversAudio Drivers
Audio Drivers
 
Booting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot imagesBooting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot images
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
 
ACPI Debugging from Linux Kernel
ACPI Debugging from Linux KernelACPI Debugging from Linux Kernel
ACPI Debugging from Linux Kernel
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
 
Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020
 
Scheduling in Android
Scheduling in AndroidScheduling in Android
Scheduling in Android
 
Embedded Android : System Development - Part IV
Embedded Android : System Development - Part IVEmbedded Android : System Development - Part IV
Embedded Android : System Development - Part IV
 
Embedded Android : System Development - Part II (Linux device drivers)
Embedded Android : System Development - Part II (Linux device drivers)Embedded Android : System Development - Part II (Linux device drivers)
Embedded Android : System Development - Part II (Linux device drivers)
 
ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)
 
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
 
Linux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBLinux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKB
 

Similar to Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Example - ELC NA 2022

Qemu device prototyping
Qemu device prototypingQemu device prototyping
Qemu device prototyping
Yan Vugenfirer
 
RHEL5 XEN HandOnTraining_v0.4.pdf
RHEL5 XEN HandOnTraining_v0.4.pdfRHEL5 XEN HandOnTraining_v0.4.pdf
RHEL5 XEN HandOnTraining_v0.4.pdf
Paul Yang
 
RunX ELCE 2020
RunX ELCE 2020RunX ELCE 2020
RunX ELCE 2020
Stefano Stabellini
 
Stateless Hypervisors at Scale
Stateless Hypervisors at ScaleStateless Hypervisors at Scale
Stateless Hypervisors at Scale
Antony Messerl
 
From printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debuggingFrom printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debugging
The Linux Foundation
 
Genode Compositions
Genode CompositionsGenode Compositions
Genode Compositions
Vasily Sartakov
 
Platform Security Summit 18: Xen Security Weather Report 2018
Platform Security Summit 18: Xen Security Weather Report 2018Platform Security Summit 18: Xen Security Weather Report 2018
Platform Security Summit 18: Xen Security Weather Report 2018
The Linux Foundation
 
Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0
guest72e8c1
 
RMLL / LSM 2009
RMLL / LSM 2009RMLL / LSM 2009
RMLL / LSM 2009
Franck_Villaume
 
the NML project
the NML projectthe NML project
the NML project
Lei Yang
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with Xen
Tamas K Lengyel
 
XPDDS17: NoXS: Death to the XenStore - Filipe Manco, NEC
XPDDS17:  NoXS: Death to the XenStore - Filipe Manco, NECXPDDS17:  NoXS: Death to the XenStore - Filipe Manco, NEC
XPDDS17: NoXS: Death to the XenStore - Filipe Manco, NEC
The Linux Foundation
 
Fuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdfFuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdf
distortdistort
 
Release notes 3_d_v61
Release notes 3_d_v61Release notes 3_d_v61
Release notes 3_d_v61
sundar sivam
 
Device virtualization and management in xen
Device virtualization and management in xenDevice virtualization and management in xen
Device virtualization and management in xen
Lingfei Kong
 
My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)
Gustavo Rene Antunez
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
The Linux Foundation
 
MemVerge: The Software Stack for CXL Environments
MemVerge: The Software Stack for CXL EnvironmentsMemVerge: The Software Stack for CXL Environments
MemVerge: The Software Stack for CXL Environments
Memory Fabric Forum
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogic
Aleem Shariff
 
SR-IOV Introduce
SR-IOV IntroduceSR-IOV Introduce
SR-IOV Introduce
Lingfei Kong
 

Similar to Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Example - ELC NA 2022 (20)

Qemu device prototyping
Qemu device prototypingQemu device prototyping
Qemu device prototyping
 
RHEL5 XEN HandOnTraining_v0.4.pdf
RHEL5 XEN HandOnTraining_v0.4.pdfRHEL5 XEN HandOnTraining_v0.4.pdf
RHEL5 XEN HandOnTraining_v0.4.pdf
 
RunX ELCE 2020
RunX ELCE 2020RunX ELCE 2020
RunX ELCE 2020
 
Stateless Hypervisors at Scale
Stateless Hypervisors at ScaleStateless Hypervisors at Scale
Stateless Hypervisors at Scale
 
From printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debuggingFrom printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debugging
 
Genode Compositions
Genode CompositionsGenode Compositions
Genode Compositions
 
Platform Security Summit 18: Xen Security Weather Report 2018
Platform Security Summit 18: Xen Security Weather Report 2018Platform Security Summit 18: Xen Security Weather Report 2018
Platform Security Summit 18: Xen Security Weather Report 2018
 
Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0
 
RMLL / LSM 2009
RMLL / LSM 2009RMLL / LSM 2009
RMLL / LSM 2009
 
the NML project
the NML projectthe NML project
the NML project
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with Xen
 
XPDDS17: NoXS: Death to the XenStore - Filipe Manco, NEC
XPDDS17:  NoXS: Death to the XenStore - Filipe Manco, NECXPDDS17:  NoXS: Death to the XenStore - Filipe Manco, NEC
XPDDS17: NoXS: Death to the XenStore - Filipe Manco, NEC
 
Fuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdfFuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdf
 
Release notes 3_d_v61
Release notes 3_d_v61Release notes 3_d_v61
Release notes 3_d_v61
 
Device virtualization and management in xen
Device virtualization and management in xenDevice virtualization and management in xen
Device virtualization and management in xen
 
My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
 
MemVerge: The Software Stack for CXL Environments
MemVerge: The Software Stack for CXL EnvironmentsMemVerge: The Software Stack for CXL Environments
MemVerge: The Software Stack for CXL Environments
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogic
 
SR-IOV Introduce
SR-IOV IntroduceSR-IOV Introduce
SR-IOV Introduce
 

More from Stefano Stabellini

Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Stefano Stabellini
 
RunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edgeRunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edge
Stefano Stabellini
 
System Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and LopperSystem Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and Lopper
Stefano Stabellini
 
Xen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time SystemXen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time System
Stefano Stabellini
 
Xen Project for ARM Servers
Xen Project for ARM ServersXen Project for ARM Servers
Xen Project for ARM Servers
Stefano Stabellini
 
Xen and OpenStack
Xen and OpenStackXen and OpenStack
Xen and OpenStack
Stefano Stabellini
 
XDS15: Project Raisin
XDS15: Project RaisinXDS15: Project Raisin
XDS15: Project Raisin
Stefano Stabellini
 
OpenStack and Xen
OpenStack and XenOpenStack and Xen
OpenStack and Xen
Stefano Stabellini
 

More from Stefano Stabellini (8)

Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
 
RunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edgeRunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edge
 
System Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and LopperSystem Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and Lopper
 
Xen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time SystemXen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time System
 
Xen Project for ARM Servers
Xen Project for ARM ServersXen Project for ARM Servers
Xen Project for ARM Servers
 
Xen and OpenStack
Xen and OpenStackXen and OpenStack
Xen and OpenStack
 
XDS15: Project Raisin
XDS15: Project RaisinXDS15: Project Raisin
XDS15: Project Raisin
 
OpenStack and Xen
OpenStack and XenOpenStack and Xen
OpenStack and Xen
 

Recently uploaded

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Green Software Development
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
safelyiotech
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
TaghreedAltamimi
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
YousufSait3
 

Recently uploaded (20)

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
 

Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Example - ELC NA 2022

  • 1. Static Partitioning with Xen, LinuxRT, and Zephyr: a concrete end-to-end example Stefano Stabellini Embedded Linux Conference 2022
  • 2. 2 | Mixed-Criticality and Static Partitioning • It is common to have a mix of critical and non-critical components • Real-time vs. non real-time • Function-critical vs. user interface or cloud interface • Running them in the same environment put the critical function at risk • A failure in the non-critical component causes a critical system crash • The non-critical component using too much CPU time causes a missed deadline • Don’t put all your eggs in one basket • Static partitioning allows you to run them separately in isolation • Many baskets! • Each domain has direct hardware access (IOMMU protected) • Real-time (< 5us latency) • Strong isolation between the domains • Real-time isolation • Failure isolation • By default, each component has only enough privilege to do what it needs to do 2 Xen Critical Non- Critical Non- Critical
  • 3. 3 | Static Partitioning Linux Dom0 Dom1 Zephyr LinuxRT Programmable Logic GEM0 MMC0 GEM1 MMC1 TTC SATA
  • 4. 4 | Xen Features for Static Partitioning • Xen Dom0less: static domains configuration for Xen • Statically-defined domains • Sub-second boot times • No need to create new VMs at runtime (possible but not required) • No need for Dom0 (possible but not required) • Each domain has direct hardware access • Real-time and Cache Coloring: isolation from cache interference • Deterministic interrupt latency at 3us • VM-to-VM communication • Static shared memory and event channels • PV drivers for device sharing • Safety-certifiability improvements underway • MISRA C • Deterministic interrupt handling and memory allocations
  • 5. 5 | Step1: a simple Xen dom0less static configuration • Board: Xilinx ZCU102 • Xen and 3 VMs • Regular Linux Dom0 from Xilinx BSP • LinuxRT DomU with a minimal busybox rootfs • Zephyr DomU • Configuration parameters • Memory allocation • CPUs allocation • Device assignment added later • Do we need Dom0? • Useful for system monitoring and domains restart • If dom0 is required, the Xen userspace tools (xl) need to be built and added to the dom0 rootfs Xen LinuxRT Linux Dom0 Zephyr
  • 6. 6 | Xen Dom0less boot U-Boot Xen Linux Dom0 LinuxRT DomU Zephyr DomU CPU 0 CPU 1-2 CPU 3 boots boots
  • 7. 7 | ImageBuilder • A tool to automatically generate a bootable configuration from a simple text config file • https://gitlab.com/xen-project/imagebuilder MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="elc22demo1/mpsoc.dtb" XEN="elc22demo1/xen" DOM0_KERNEL="elc22demo1/xilinx-linux" DOM0_RAMDISK="elc22demo1/xen-rootfs.cpio.gz" DOM0_MEM=1024 NUM_DOMUS=2 DOMU_KERNEL[0]="elc22demo1/linuxrt" DOMU_RAMDISK[0]="elc22demo1/initrd.cpio" DOMU_MEM[0]=2048 DOMU_VCPUS[0]=2 DOMU_KERNEL[1]="elc22demo1/zephyr.bin" DOMU_MEM[1]=128 # bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c elc22demo/config
  • 8. 8 | ImageBuilder boot.source • ImageBuilder generates boot.source and boot.scr to be loaded by u-boot • Editing boot.source is possible • Then call mkimage by hand # mkimage -A arm64 -T script -C none -a 0xc00000 -e 0xc00000 -d boot.source boot.scr
  • 9. 9 | Building Xen with or without cache coloring support • Building the hypervisor is very simple (it cross-compiles like Linux) • Cache coloring is available as a menuconfig option • Upstreaming in progress • Currently available at https://github.com/Xilinx/xen • Nothing else is needed unless you plan to use the Xen tools (xl) in dom0 # export CROSS_COMPILE=/path/to/cross-compiler # export XEN_TARGET_ARCH=arm64 # cd xen.git/xen/ # make menuconfig Architecture Features à L2 cache coloring # make
  • 10. 10 | Building a Linux/LinuxRT kernel for Xen • Vanilla Linux releases and binaries work • Using Linux v5.17 for the demo • The following Kconfig options are recommended: • CONFIG_XEN • CONFIG_XEN_BLKDEV_BACKEND • CONFIG_XEN_NETDEV_BACKEND • CONFIG_XEN_GNTDEV • CONFIG_XEN_GRANT_DEV_ALLOC • CONFIG_XEN_GRANT_DMA_ALLOC • CONFIG_SERIAL_AMBA_PL011 • CONFIG_SERIAL_AMBA_PL011_CONSOLE • CONFIG_BRIDGE • Either CONFIG_XEN or CONFIG_SERIAL_AMBA_PL011 is needed for console output • 2 patches for PV drivers support for dom0less • Patches already accepted and soon to be upstream
  • 11. 11 | Building a Dom0 rootfs • Dom0 requires the Xen userspace tools in Dom0 • Build a Linux rootfs for Dom0 with Xen support using Yocto: DISTRO_FEATURES:append = " xen" DISTRO_FEATURES:append = " virtualization" IMAGE_INSTALL:append = " xen-base zlib-dev" ASSUME_PROVIDED += ” iasl-native"
  • 12. 12 | Building Zephyr with Xen support • Initial Zephyr build environment setup required • See https://docs.zephyrproject.org/latest/develop/getting_started/index.html • Use the existing xenvm machine • Build time Device Tree: boards/arm64/xenvm/xenvm.dts • Memory mappings: soc/arm64/xenvm/mmu_regions.c • To run Zephyr as Dom0less guest: • change xen_hvc with xen_consoleio_hvc in xenvm.dts • build Zephyr with Dom0 support • build Xen with DEBUG enabled # west build -t menuconfig -> SoC/CPU/Configuration Selection (Xen virtual machine on aarch64) -> Xen virtual machine on aarch64 -> Zephyr as Xen Domain 0 -> Device Drivers -> Serial Drivers -> Xen hypervisor Dom0 console UART driver # west build -b xenvm samples/hello_world/
  • 13. 13 | Step2: Add Device Assignment • Device Assigment: • Direct hardware access • Protected by the IOMMU • Example configuration: • Assigning the Network Device (GEM) to LinuxRT • Assigning the TTC timer to Zephyr Xen LinuxRT Linux Dom0 Zephyr GEM0 TTC0
  • 14. 14 | Xen Device Assignment • Based on “partial device trees” • They describe the device for the domU • They configure device assignment • Everything under passthrough is copied to the guest device tree • Special device assignment configuration properties: • xen,reg • xen,path • xen,force-assign-without-iommu • interrupt is used for interrupt remapping • interrupt-parent must be 0xfde8 /dts-v1/; / { #address-cells = <0x2>; #size-cells = <0x2>; passthrough { compatible = "simple-bus"; ranges; #address-cells = <0x2>; #size-cells = <0x2>; ethernet@ff0e0000 { compatible = "cdns,zynqmp-gem"; […] reg = <0x0 0xff0e0000 0x0 0x1000>; interrupt-parent = <0xfde8>; interrupts = <0x0 0x3f 0x4 0x0 0x3f 0x4>; xen,reg = <0x0 0xff0e0000 0x0 0x1000 0x0 0xff0e0000>; xen,path = "/amba/ethernet@ff0e0000"; }; }; };
  • 15. 15 | Xen Device Assignment and ImageBuilder • Add xen,passthrough; under the corresponding node on the host device tree • Add the partial device tree binaries to the ImageBuilder config file • ImageBuilder will load them automatically for you at boot […] NUM_DOMUS=2 DOMU_KERNEL[0]="elc22demo2/linuxrt" DOMU_RAMDISK[0]="elc22demo2/initrd.cpio" DOMU_MEM[0]=2048 DOMU_VCPUS[0]=2 DOMU_PASSTHROUGH_DTB[0]="elc22demo2/ethernet@ff0e0000.dtb" DOMU_KERNEL[1]="elc22demo2/zephyr.bin" DOMU_PASSTHROUGH_DTB[1]="elc22demo2/timer@ff110000.dtb" DOMU_MEM[1]=128
  • 16. 16 | Xilinx Xen Device Tree examples repository • Repository with device tree passthrough examples: https://github.com/Xilinx/xen-passthrough-device-trees • Simplified ImageBuilder configuration with PASSTHROUGH_DTS_REPO, DOMU_PASSTHROUGH_PATHS • Based on file names and example files, no automatic device tree generation yet • Careful about “axi” vs “amba”, the name needs to match everywhere! • host DTB, passthrough DTB, xen,path property, ImageBuider config file […] PASSTHROUGH_DTS_REPO="git@github.com:Xilinx/xen-passthrough-device-trees.git device-trees-2021.2" NUM_DOMUS=2 DOMU_KERNEL[0]="elc22demo2/linuxrt" DOMU_RAMDISK[0]="elc22demo2/initrd.cpio" DOMU_MEM[0]=2048 DOMU_VCPUS[0]=2 DOMU_PASSTHROUGH_PATHS[0]="/axi/ethernet@ff0e0000” DOMU_KERNEL[1]="elc22demo2/zephyr.bin" DOMU_MEM[1]=128 DOMU_PASSTHROUGH_PATHS[1]="/axi/timer@ff110000"
  • 17. 17 | Xen Partial Device Trees Automatic Generation • System Device Tree: Device Tree based description of a full heterogeneous system • Lopper • a Device Tree manipulation tool • takes a System Device Tree as input and generates multiple Device Trees as output • Use Lopper to automatically generate Xen passthrough DTBs • Just need to provide the host DTB and the device path, e.g. /amba/ethernet@ff0e0000 • Attend the presentation “System Device Tree and Lopper: Concrete Examples”, Thursday June 23 at 12PM!
  • 18. 18 | Make use of a new device in Zephyr • Add it to the board device tree boards/arm64/xenvm/xenvm.dts • Map the device memory in soc/arm64/xenvm/mmu_regions.c MMU_REGION_FLAT_ENTRY("TTC", DT_REG_ADDR_BY_IDX(DT_INST(0, xlnx_ttcps), 0), DT_REG_SIZE_BY_IDX(DT_INST(0, xlnx_ttcps), 0), MT_DEVICE_nGnRnE | MT_P_RW_U_RW | MT_NS), ttc0: timer@ff110000 { compatible = "xlnx,ttcps"; status = "okay"; interrupts = <GIC_SPI 36 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>, <GIC_SPI 37 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>, <GIC_SPI 38 IRQ_TYPE_LEVEL IRQ_DEFAULT_PRIORITY>; interrupt-names = "irq_0", "irq_1", "irq_2"; reg = <0x0 0xff110000 0x0 0x1000>; label = "ttc0"; interrupt-parent = <&gic>; clock-frequency = <8320000>; };
  • 19. © Copyright 2020 Xilinx Step3: Xen Cache Coloring L2 Core 1 Core 2 Core 3 Core 4 DDR L1 L1 L1 L1
  • 20. © Copyright 2020 Xilinx Xen Cache Coloring L2 Core 1 Core 2 Core 3 Core 4 DDR L1 L1 L1 L1
  • 21. © Copyright 2020 Xilinx 4CPUs clusters often share L2 cache 4Interference via L2 cache affects performance ¬ App0 running on CPU0 can cause cache entries evictions, which affect App1 running on CPU1 ¬ App1 running on CPU1 could miss a deadline due to App0’s behavior ¬ It can happen between Apps running on the same OS & between VMs on the same hypervisor 4Hypervisor Solution: Cache Partitioning, AKA Cache Coloring ¬ Each VM gets its own allocation of cache entries ¬ No shared cache entries between VMs ¬ Allows real-time apps to run with deterministic IRQ latency D D R Xen Cache Coloring Core 1 Core 2 Core 3 Core 4 L2
  • 22. 22 | Xen Cache Coloring and ImageBuilder • Cache Coloring on Xen ZCU102 • 16 colors, each color corresponds to 256MB • Color bitmask is 0xF000 • Configuration: • Xen à color 0 • Dom0 à color 1-5 • LinuxRT à color 6-14 • Zephyr à color 15 # Xen and Dom0, use XEN_CMD in the ImageBuilder config: XEN_CMD=“…xen_colors=0-0 dom0_colors=1-5 way_size=65536” DOMU_COLORS[0]="6-14” DOMU_COLORS[1]="15-15"
  • 23. Dom0 LinuxRT Zephyr CPU 0 CPU 1-2 CPU 3 Linux Dom0 ring buffer Step4: Shared Memory and Event Channels
  • 24. 24 | Shared Memory Configuration • A static shared memory region chosen at build time • E.g. 0x7fe0_1000 • The address matter for cache coloring: 0x7fe0_1000 means color “1” • Xen event channels can be used for notifications • A BSD-licensed xen.h file with Xen hypercall definitions is provided availble • Can be embedded in Zephyr and other RTOSes • Makes it easy to add event channel support anywhere • Static event channel definitions based on device tree available in the next Xen release • Today event channels are created dynamically • From the next release it is possible to configure them statically as part of the Xen domains (i.e. ImageBuilder) • Static shared memory support also coming in the next Xen release • Possible to define shared memory without editing the memory node • Work in progress by ARM
  • 25. 25 | Shared Memory Configuration • Carve out an address range from the memory node • 0xf7fe00000 - 0xf7ff00000 • Add it as a separate sram node memory { device_type = "memory"; reg = <0x00 0x00 0x00 0x7fe00000 0x08 0x00 0x00 0x80000000>; }; sram@7fe00000 { compatible = "mmio-sram"; reg = <0x0 0x7fe00000 0x0 0x100000>; xen,passthrough; };
  • 26. 26 | Shared Memory Configuration • Add shared memory to the domU passthrough device tree passthrough { compatible = "simple-bus"; ranges; #address-cells = <0x2>; #size-cells = <0x2>; /* timer has been tested with baremetal app only */ timer@ff110000 { […] sram@7fe01000 { compatible = "mmio-sram"; reg = <0x0 0x7fe01000 0x0 0x1000>; xen,reg = <0x0 0x7fe01000 0x0 0x1000 0x0 0x7fe01000>; xen,force-assign-without-iommu = <0x1>; };
  • 27. 27 | Make use of the shared memory and event channels in Zephyr • Map the shared memory • Setup event channels MMU_REGION_FLAT_ENTRY("RAM_SHARED", 0x7fe01000, 0x1000, MT_NORMAL_NC | MT_P_RW_U_RW | MT_NS), #include "xen.h” […] bind.remote_dom = remote_domid; bind.remote_port = remote_port; bind.local_port = 0; ret = xen_hypercall(EVTCHNOP_bind_interdomain, (unsigned long)&bind, 0, 0, HYPERVISOR_event_channel_op);
  • 28. 28 | Make use of the shared memory and event channels in Linux • Write a new driver static int __init shared_mem_init(void) { char *str = "go"; struct resource r; struct device_node *np = of_find_compatible_node(NULL, NULL, "xen,shared-memory-v1"); struct evtchn_alloc_unbound alloc_unbound; of_address_to_resource(np, 0, &r); shared_mem = phys_to_virt(r.start); alloc_unbound.dom = DOMID_SELF; alloc_unbound.remote_dom = 1; /* domid 1 */ HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &alloc_unbound); irq = bind_evtchn_to_irqhandler_lateeoi(alloc_unbound.port, evtchn_handler, 0, “shared_intf", NULL); irq = rc; memcpy(shared_mem, str, 3); mb(); /* send port number to other domain */ memcpy(shared_mem + 4, &alloc_unbound.port, sizeof(alloc_unbound.port)); return 0; }
  • 29. 29 | Step5: Dom0less with PV Drivers Dom0 U-Boot Xen LinuxRT CPU CPU NetBack NetFront init-dom0less Zephyr CPU 3
  • 30. 30 | Dom0less and PV Drivers Linux Dom0 LinuxRT DomU Zephyr DomU CPU 0 CPU 1-2 CPU 3 Xen
  • 31. 31 | Dom0less and PV Drivers • Xen PV Drivers are used to share a device across multiple domains • Share the network across multiple domains • Share a disk across multiple domains, a partition to each domain • PV network can also be used for VM-to-VM communication • With dom0less, PV Drivers are instantiated after the boot is finished from Dom0 • Fast boot times: the critical application can start immediately, no need to wait • PV drivers connect later after Dom0 completed booting • A Dom0 environment with Xen userspace tools is required to setup the connection • Use init-dom0less in Dom0 to initialize PV drivers, then hotplug a PV network connection /usr/lib/xen/bin/init-dom0less brctl addbr xenbr0; ifconfig xenbr0 192.168.0.1 up xl network-attach 1