This talk illustrates how to use the Jailhouse hypervisor for running Linux alongside an RTOS on modern ARM multi-core SoCs, aiming at building smarter devices for the automotive market. Recently, the industry has shown a growing interest for executing activities with different levels of criticality on the same multi-core SoC. These could consist, for example, of non-critical activities (e.g., monitoring, logging, human-machine intefaces) together with safety-critical tasks. The rationale behind this interest is the continuous need for reducing the time-to-market as well as the design and hardware costs. This is particularly suitable for the automotive market, where new infotainment functionalities might be coupled with traditional safety-critical tasks (e.g. engine/brake control). In this talk, we will present our experience (grown through the HERCULES EU project) in using the Jailhouse hypervisor for executing the Linux general-purpose OS alongside an automotive RTOS on modern ARM multi-core platforms. Besides providing useful instructions for using Jailhouse, we will illustrate a library designed for easing the communication between the two OSs as well as some mechanism for limiting the interference on shared hardware resources. Finally, a short video of a simple demo will show the effectiveness of the proposed approach.

1. 1
HANDLING MIXED CRITICALITY
ON EMBEDDED
MULTI-CORE SYSTEMS
Claudio Scordino
Evidence Srl
claudio@evidence.eu.com

2. 22
Summary
• Mixed criticality
• The HERCULES project
• ERIKA Enterprise
• Jailhouse hypervisor
• Communication
• Interference

3. 33
About Evidence Srl...
• Company specialized in firmware and software
for small electronic devices
Founded in 2002 in Pisa, Italy
~22 qualified people
15+ years of experience in industrial projects
• We look for:
... plus:
... excellent knowledge of C programming,
operating systems,
and computer architectures

4. 44
Mixed criticality
Non-critical tasks:
Critical tasks:Multimedia
HMI
Networking
Logging
Data backup
Autonomous driving
Industrial automation
Robotics
Engine control
Different levels of criticality
co-existing on the same
platform

5. 55
Rationale
Cost-reduction: reducing recurrent costs (i.e. hardware)
Flexibility: e.g. move tasks from one domain to another

6. 66
Use-case: automotive
• Mixed criticality is of particular interest for the
automotive market
• Non-critical tasks:
– Infotainment / multimedia
– Human-machine interface
– Navigation / dashboard
• Safety-critical tasks:
– Engine/brake control
– ADAS
– Autonomous driving

7. 77
1st approach
• Make Linux real-time!
• Dual kernel:
– RT-Linux (dead)
– RTAI (only x86)
– Xenomai (alive ?)
• PREEMPT_RT (soft real-time)
• Common issue: certification/standards for specific
domains

8. 88
2nd approach: resource partitioning
• Modern hardware provides support for
virtualization/partitioning
• Examples of existing technologies:
– Intel VT-x, VT-d
– AMD-V, AMD-Vi
– ARM TrustZone
– ARM virtualization extensions, SMMU
Idea: leverage hardware support for
running multiple operating systems

9. 99
The HERCULES project
Linux RT
RTOS
ARM
Cortex-A
• High-Performance Real-Time Architectures for Low-Power
Embedded Systems (HERCULES)
• http://hercules2020.eu
• Funded by the European Commission (H2020)
• Jan. 2016 – Dec. 2018
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
Jailhouse hypervisor
Infrastructure:
• UNIMORE
• Evidence
• CTU
• ETH Zurich
Use-cases:
• Magneti Marelli
• Airbus
• Pitom

10. 1010
ERIKA Enterprise
• Real-time operating system (RTOS)
• Developed by Evidence for automotive ECUs
• Minimal footprint (few KB) and multi-core support
• Certifications: OSEK/VDX, ISO26262 (ASIL B in-progress)
• Reference standards: MISRA-C, AUTOSAR OS
• Dual licensing: GPL (optional linking exception with a fee)
• Used by several companies and research projects:
http://www.erika-enterprise.com
https://github.com/evidence/erika3

11. 1111
Supported hardware
ERIKA Enterprise v3 supports:
CPUs/SoCs:
ARM Cortex-A, Cortex-M, Cortex-R
Infineon Tricore AURIX TC2xx, TC3xx*
Intel x86-64
Kalray MPPA Bostan
Microchip AVR, dsPIC33, PIC24
Renesas RH850
Hypervisors:
KVM*
Jailhouse
Vibrante (NVIDIA)
Xen* * Coming soon

12. 1212
Jailhouse
• Small, lightweight hypervisor
• Young project (2013) by Siemens
• License: GPLv2
• Code hosted on GitHub
https://github.com/siemens/jailhouse
• Goals: safety-critical & certification
(goal: 10.000 lines of code per architecture)

13. 1313
Jailhouse (2)
• A tool to run
... real-time and/or safety-critical tasks
... on multi-core platforms
... aside Linux
• It provides
... strong & clean isolation
... bare-metal like
performance & latencies

14. 1414
Jailhouse (3)
• Partitioning hypervisor
– More focused on isolation and resource
assignment than on virtualization
• Linux is required
– "Root cell"
– Similar to Xen's dom0 but w/out full control of hw
• Can't run unmodified OSs (e.g. Windows)

15. 1515
Jailhouse (4)
• Static design:
– 1:1 resource assignment
– Guests can't share a core (no scheduling)
– It doesn't support overcommitment of resources
(like CPUs, RAM or devices).
– No hw emulation
• Real-time guarantees:
– Must be provided by the guest OS
– Jailhouse just introduces the least
possible overhead

16. 1616
Jailhouse: naming conventions
The isolated compartments are
called cells:
• One root cell
• One or more non-root cells
The guest
software
is called inmate

17. 1717
Memory layout
• Typical RAM layout:
• Memory for hypervisor/inmates is reserved
at boot time
– x86-64: memmap= boot param
– ARM: device-tree or mem= boot param

18. 1818
Jailhouse architecture

19. 1919
Root cell configuration format
struct {
struct jailhouse_system header;
/*...*/
} __attribute__((packed)) config = {
.header = {
.signature = JAILHOUSE_SYSTEM_SIGNATURE,
.hypervisor_memory = {
.phys_start = 0xfc000000,
.size = 0x4000000,
},
.debug_console = {
.phys_start = 0x70006000,
.size = 0x1000,
.flags = JAILHOUSE_MEM_IO,
},
.platform_info.arm = {
.gicd_base = 0x50041000,
/* ... */
.maintenance_irq = 25,
},
.root_cell = {
.name = "Jetson-TK1",
/* ... */
Cell resources
• Configuration (e.g. addresses)
written through C files
• Generated on x86-64:
• Manually written on ARM
– Datasheet
– Device tree
– proc/
sudo jailhouse config create sysconfig.c

20. 2020
Non-root cell configuration format
Cell resources
struct {
struct jailhouse_cell_desc cell;
/*...*/
} __attribute__((packed)) config = {
.cell = {
.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
.name = "apic-demo",
.cpu_set_size = sizeof(config.cpus),
/*...*/
}
.cpus = {
0x8,
},

21. 2121
Cell resources
• Memory regions:
– Physical address, virtual address, size
– Flags (JAILHOUSE_MEM_*):
• PIO bitmap (x86 port-based I/O)
• IRQ chips
• Cache regions
• PCI devices and related capabilities
• Memory-mapped UARTs
All
configurations
written in C

22. 2222
Driver installation
• Build & install Jailhouse:
The build process also transforms all
configuration files from C to binary (i.e. *.cell files)
• Install the Jailhouse driver:
$ make
$ sudo make install
$ sudo modprobe jailhouse

23. 2323
Jailhouse enabling
• Enable the root cell passing the system
configuration:
$ sudo jailhouse enable configs/sysconfig.cell

24. 2424
Jailhouse enabling (2)

25. 2525
Non-root cell creation
• Create a non-root cell:
• Load a binary into the created cell:
• Start the cell:
• (optional) Get statistics about the cell:
$ sudo jailhouse cell create configs/file.cell
$ sudo jailhouse cell load cell_name code.bin
$ sudo jailhouse cell start cell_name
$ sudo jailhouse cell stats cell_name

26. 2626
Non-root cell creation (2)

27. 2727
Good practices
• Give the hypervisor access to a serial console
• Put #define CONFIG_TRACE_ERROR in file
include/jailhouse/config.h before compiling
• Example of misconfiguration:

28. 2828
NVIDIA support
• In HERCULES we have supported the
following platforms:
– NVIDIA TX1/TX2 (Cortex-A57 + Denver)
– Xilinx ZCU102 (Cortex-A53)
• NVIDIA's vendor kernel is not supported by
mainline Jailhouse:
– We created a specific Jailhouse tree:
https://github.com/evidence/linux-jailhouse-jetson
– Virtual machine containing Jailhouse + ERIKA:
http://www.erika-enterprise.com/index.php/download/virtual-
machines.html

29. 2929
Communication
• A sophisticated mechanism allows cells to
communicate through virtual PCI devices
– Model similar to ivshmem device from Qemu
– No multicast communications possible
• See Documentation/inter-cell-
communication.txt

30. 3030
Linux
kernel
AUTOSAR COM
Jailhouse hypervisor
SWC
AUTOSAR Classic
App
Linux OS
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
• Library on top of Jailhouse's
mechanism
• Blocking and non-blocking calls
• Dynamic-size messages
• Similar to AUTOSAR COM API:
Com_StatusType Com_GetStatus();
uint8 Com_SendSignal(Com_SignalIdType
SignalId, const void *SignalDataPtr);
uint8 Com_ReceiveSignal(Com_SignalIdType
SignalId, void* SignalDataPtr);
• Soon available (GPL)

31. 3131
Video (2)

32. 3232
Interference
Core
Linux RTOS
Hypervisor
Core
General-
Purpose
apps
Real-time
apps
L1 L1
L2 cache
SDRAM
Interference
• Even with partitioning, there is still
some interference on shared hardware
E.g. caches, memory bus, etc.
• One core can affect the real-time
responsiveness of other cores
• Software solutions:
1. Cache coloring to avoid data eviction (handle
virtual memory so that pages with different
"colors" have different positions in cache)
2. "Memguard" (force memory bandwidth by
monitoring performance counters)
3. Co-scheduling algorithms (orchestrating
memory accesses)
• Developed in HERCULES
• Soon released as open-source

33. 3333
Conclusions
• Software stack:
– Handling mixed criticality
– Targeting automotive (AUTOSAR compliant)
– Open-source (GPL)
– Working on COTS ARM hardware