1
HANDLING MIXED CRITICALITY
ON EMBEDDED
MULTI-CORE SYSTEMS
Claudio Scordino
Evidence Srl
claudio@evidence.eu.com
22
Summary
• Mixed criticality
• The HERCULES project
• ERIKA Enterprise
• Jailhouse hypervisor
• Communication
• Interference
33
About Evidence Srl...
• Company specialized in firmware and software
for small electronic devices
Founded in 2002 in Pisa, Italy
~22 qualified people
15+ years of experience in industrial projects
• We look for:
... plus:
... excellent knowledge of C programming,
operating systems,
and computer architectures
44
Mixed criticality
Non-critical tasks:
Critical tasks:Multimedia
HMI
Networking
Logging
Data backup
Autonomous driving
Industrial automation
Robotics
Engine control
Different levels of criticality
co-existing on the same
platform
55
Rationale
Cost-reduction: reducing recurrent costs (i.e. hardware)
Flexibility: e.g. move tasks from one domain to another
66
Use-case: automotive
• Mixed criticality is of particular interest for the
automotive market
• Non-critical tasks:
– Infotainment / multimedia
– Human-machine interface
– Navigation / dashboard
• Safety-critical tasks:
– Engine/brake control
– ADAS
– Autonomous driving
77
1st approach
• Make Linux real-time!
• Dual kernel:
– RT-Linux (dead)
– RTAI (only x86)
– Xenomai (alive ?)
• PREEMPT_RT (soft real-time)
• Common issue: certification/standards for specific
domains
88
2nd approach: resource partitioning
• Modern hardware provides support for
virtualization/partitioning
• Examples of existing technologies:
– Intel VT-x, VT-d
– AMD-V, AMD-Vi
– ARM TrustZone
– ARM virtualization extensions, SMMU
Idea: leverage hardware support for
running multiple operating systems
99
The HERCULES project
Linux RT
RTOS
ARM
Cortex-A
• High-Performance Real-Time Architectures for Low-Power
Embedded Systems (HERCULES)
• http://hercules2020.eu
• Funded by the European Commission (H2020)
• Jan. 2016 – Dec. 2018
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
Jailhouse hypervisor
Infrastructure:
• UNIMORE
• Evidence
• CTU
• ETH Zurich
Use-cases:
• Magneti Marelli
• Airbus
• Pitom
1010
ERIKA Enterprise
• Real-time operating system (RTOS)
• Developed by Evidence for automotive ECUs
• Minimal footprint (few KB) and multi-core support
• Certifications: OSEK/VDX, ISO26262 (ASIL B in-progress)
• Reference standards: MISRA-C, AUTOSAR OS
• Dual licensing: GPL (optional linking exception with a fee)
• Used by several companies and research projects:
http://www.erika-enterprise.com
https://github.com/evidence/erika3
1111
Supported hardware
ERIKA Enterprise v3 supports:
CPUs/SoCs:
ARM Cortex-A, Cortex-M, Cortex-R
Infineon Tricore AURIX TC2xx, TC3xx*
Intel x86-64
Kalray MPPA Bostan
Microchip AVR, dsPIC33, PIC24
Renesas RH850
Hypervisors:
KVM*
Jailhouse
Vibrante (NVIDIA)
Xen* * Coming soon
1212
Jailhouse
• Small, lightweight hypervisor
• Young project (2013) by Siemens
• License: GPLv2
• Code hosted on GitHub
https://github.com/siemens/jailhouse
• Goals: safety-critical & certification
(goal: 10.000 lines of code per architecture)
1313
Jailhouse (2)
• A tool to run
... real-time and/or safety-critical tasks
... on multi-core platforms
... aside Linux
• It provides
... strong & clean isolation
... bare-metal like
performance & latencies
1414
Jailhouse (3)
• Partitioning hypervisor
– More focused on isolation and resource
assignment than on virtualization
• Linux is required
– "Root cell"
– Similar to Xen's dom0 but w/out full control of hw
• Can't run unmodified OSs (e.g. Windows)
1515
Jailhouse (4)
• Static design:
– 1:1 resource assignment
– Guests can't share a core (no scheduling)
– It doesn't support overcommitment of resources
(like CPUs, RAM or devices).
– No hw emulation
• Real-time guarantees:
– Must be provided by the guest OS
– Jailhouse just introduces the least
possible overhead
1616
Jailhouse: naming conventions
The isolated compartments are
called cells:
• One root cell
• One or more non-root cells
The guest
software
is called inmate
1717
Memory layout
• Typical RAM layout:
• Memory for hypervisor/inmates is reserved
at boot time
– x86-64: memmap= boot param
– ARM: device-tree or mem= boot param
1818
Jailhouse architecture
1919
Root cell configuration format
struct {
struct jailhouse_system header;
/*...*/
} __attribute__((packed)) config = {
.header = {
.signature = JAILHOUSE_SYSTEM_SIGNATURE,
.hypervisor_memory = {
.phys_start = 0xfc000000,
.size = 0x4000000,
},
.debug_console = {
.phys_start = 0x70006000,
.size = 0x1000,
.flags = JAILHOUSE_MEM_IO,
},
.platform_info.arm = {
.gicd_base = 0x50041000,
/* ... */
.maintenance_irq = 25,
},
.root_cell = {
.name = "Jetson-TK1",
/* ... */
Cell resources
• Configuration (e.g. addresses)
written through C files
• Generated on x86-64:
• Manually written on ARM
– Datasheet
– Device tree
– proc/
sudo jailhouse config create sysconfig.c
2020
Non-root cell configuration format
Cell resources
struct {
struct jailhouse_cell_desc cell;
/*...*/
} __attribute__((packed)) config = {
.cell = {
.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
.name = "apic-demo",
.cpu_set_size = sizeof(config.cpus),
/*...*/
}
.cpus = {
0x8,
},
2121
Cell resources
• Memory regions:
– Physical address, virtual address, size
– Flags (JAILHOUSE_MEM_*):
• PIO bitmap (x86 port-based I/O)
• IRQ chips
• Cache regions
• PCI devices and related capabilities
• Memory-mapped UARTs
All
configurations
written in C
2222
Driver installation
• Build & install Jailhouse:
The build process also transforms all
configuration files from C to binary (i.e. *.cell files)
• Install the Jailhouse driver:
$ make
$ sudo make install
$ sudo modprobe jailhouse
2323
Jailhouse enabling
• Enable the root cell passing the system
configuration:
$ sudo jailhouse enable configs/sysconfig.cell
2424
Jailhouse enabling (2)
2525
Non-root cell creation
• Create a non-root cell:
• Load a binary into the created cell:
• Start the cell:
• (optional) Get statistics about the cell:
$ sudo jailhouse cell create configs/file.cell
$ sudo jailhouse cell load cell_name code.bin
$ sudo jailhouse cell start cell_name
$ sudo jailhouse cell stats cell_name
2626
Non-root cell creation (2)
2727
Good practices
• Give the hypervisor access to a serial console
• Put #define CONFIG_TRACE_ERROR in file
include/jailhouse/config.h before compiling
• Example of misconfiguration:
2828
NVIDIA support
• In HERCULES we have supported the
following platforms:
– NVIDIA TX1/TX2 (Cortex-A57 + Denver)
– Xilinx ZCU102 (Cortex-A53)
• NVIDIA's vendor kernel is not supported by
mainline Jailhouse:
– We created a specific Jailhouse tree:
https://github.com/evidence/linux-jailhouse-jetson
– Virtual machine containing Jailhouse + ERIKA:
http://www.erika-enterprise.com/index.php/download/virtual-
machines.html
2929
Communication
• A sophisticated mechanism allows cells to
communicate through virtual PCI devices
– Model similar to ivshmem device from Qemu
– No multicast communications possible
• See Documentation/inter-cell-
communication.txt
3030
Linux
kernel
AUTOSAR COM
Jailhouse hypervisor
SWC
AUTOSAR Classic
App
Linux OS
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
ARM
Cortex-A
• Library on top of Jailhouse's
mechanism
• Blocking and non-blocking calls
• Dynamic-size messages
• Similar to AUTOSAR COM API:
Com_StatusType Com_GetStatus();
uint8 Com_SendSignal(Com_SignalIdType
SignalId, const void *SignalDataPtr);
uint8 Com_ReceiveSignal(Com_SignalIdType
SignalId, void* SignalDataPtr);
• Soon available (GPL)
3131
Video (2)
3232
Interference
Core
Linux RTOS
Hypervisor
Core
General-
Purpose
apps
Real-time
apps
L1 L1
L2 cache
SDRAM
Interference
• Even with partitioning, there is still
some interference on shared hardware
E.g. caches, memory bus, etc.
• One core can affect the real-time
responsiveness of other cores
• Software solutions:
1. Cache coloring to avoid data eviction (handle
virtual memory so that pages with different
"colors" have different positions in cache)
2. "Memguard" (force memory bandwidth by
monitoring performance counters)
3. Co-scheduling algorithms (orchestrating
memory accesses)
• Developed in HERCULES
• Soon released as open-source
3333
Conclusions
• Software stack:
– Handling mixed criticality
– Targeting automotive (AUTOSAR compliant)
– Open-source (GPL)
– Working on COTS ARM hardware

Claudio Scordino - Handling mixed criticality on embedded multi-core systems

  • 1.
    1 HANDLING MIXED CRITICALITY ONEMBEDDED MULTI-CORE SYSTEMS Claudio Scordino Evidence Srl claudio@evidence.eu.com
  • 2.
    22 Summary • Mixed criticality •The HERCULES project • ERIKA Enterprise • Jailhouse hypervisor • Communication • Interference
  • 3.
    33 About Evidence Srl... •Company specialized in firmware and software for small electronic devices Founded in 2002 in Pisa, Italy ~22 qualified people 15+ years of experience in industrial projects • We look for: ... plus: ... excellent knowledge of C programming, operating systems, and computer architectures
  • 4.
    44 Mixed criticality Non-critical tasks: Criticaltasks:Multimedia HMI Networking Logging Data backup Autonomous driving Industrial automation Robotics Engine control Different levels of criticality co-existing on the same platform
  • 5.
    55 Rationale Cost-reduction: reducing recurrentcosts (i.e. hardware) Flexibility: e.g. move tasks from one domain to another
  • 6.
    66 Use-case: automotive • Mixedcriticality is of particular interest for the automotive market • Non-critical tasks: – Infotainment / multimedia – Human-machine interface – Navigation / dashboard • Safety-critical tasks: – Engine/brake control – ADAS – Autonomous driving
  • 7.
    77 1st approach • MakeLinux real-time! • Dual kernel: – RT-Linux (dead) – RTAI (only x86) – Xenomai (alive ?) • PREEMPT_RT (soft real-time) • Common issue: certification/standards for specific domains
  • 8.
    88 2nd approach: resourcepartitioning • Modern hardware provides support for virtualization/partitioning • Examples of existing technologies: – Intel VT-x, VT-d – AMD-V, AMD-Vi – ARM TrustZone – ARM virtualization extensions, SMMU Idea: leverage hardware support for running multiple operating systems
  • 9.
    99 The HERCULES project LinuxRT RTOS ARM Cortex-A • High-Performance Real-Time Architectures for Low-Power Embedded Systems (HERCULES) • http://hercules2020.eu • Funded by the European Commission (H2020) • Jan. 2016 – Dec. 2018 ARM Cortex-A ARM Cortex-A ARM Cortex-A Jailhouse hypervisor Infrastructure: • UNIMORE • Evidence • CTU • ETH Zurich Use-cases: • Magneti Marelli • Airbus • Pitom
  • 10.
    1010 ERIKA Enterprise • Real-timeoperating system (RTOS) • Developed by Evidence for automotive ECUs • Minimal footprint (few KB) and multi-core support • Certifications: OSEK/VDX, ISO26262 (ASIL B in-progress) • Reference standards: MISRA-C, AUTOSAR OS • Dual licensing: GPL (optional linking exception with a fee) • Used by several companies and research projects: http://www.erika-enterprise.com https://github.com/evidence/erika3
  • 11.
    1111 Supported hardware ERIKA Enterprisev3 supports: CPUs/SoCs: ARM Cortex-A, Cortex-M, Cortex-R Infineon Tricore AURIX TC2xx, TC3xx* Intel x86-64 Kalray MPPA Bostan Microchip AVR, dsPIC33, PIC24 Renesas RH850 Hypervisors: KVM* Jailhouse Vibrante (NVIDIA) Xen* * Coming soon
  • 12.
    1212 Jailhouse • Small, lightweighthypervisor • Young project (2013) by Siemens • License: GPLv2 • Code hosted on GitHub https://github.com/siemens/jailhouse • Goals: safety-critical & certification (goal: 10.000 lines of code per architecture)
  • 13.
    1313 Jailhouse (2) • Atool to run ... real-time and/or safety-critical tasks ... on multi-core platforms ... aside Linux • It provides ... strong & clean isolation ... bare-metal like performance & latencies
  • 14.
    1414 Jailhouse (3) • Partitioninghypervisor – More focused on isolation and resource assignment than on virtualization • Linux is required – "Root cell" – Similar to Xen's dom0 but w/out full control of hw • Can't run unmodified OSs (e.g. Windows)
  • 15.
    1515 Jailhouse (4) • Staticdesign: – 1:1 resource assignment – Guests can't share a core (no scheduling) – It doesn't support overcommitment of resources (like CPUs, RAM or devices). – No hw emulation • Real-time guarantees: – Must be provided by the guest OS – Jailhouse just introduces the least possible overhead
  • 16.
    1616 Jailhouse: naming conventions Theisolated compartments are called cells: • One root cell • One or more non-root cells The guest software is called inmate
  • 17.
    1717 Memory layout • TypicalRAM layout: • Memory for hypervisor/inmates is reserved at boot time – x86-64: memmap= boot param – ARM: device-tree or mem= boot param
  • 18.
  • 19.
    1919 Root cell configurationformat struct { struct jailhouse_system header; /*...*/ } __attribute__((packed)) config = { .header = { .signature = JAILHOUSE_SYSTEM_SIGNATURE, .hypervisor_memory = { .phys_start = 0xfc000000, .size = 0x4000000, }, .debug_console = { .phys_start = 0x70006000, .size = 0x1000, .flags = JAILHOUSE_MEM_IO, }, .platform_info.arm = { .gicd_base = 0x50041000, /* ... */ .maintenance_irq = 25, }, .root_cell = { .name = "Jetson-TK1", /* ... */ Cell resources • Configuration (e.g. addresses) written through C files • Generated on x86-64: • Manually written on ARM – Datasheet – Device tree – proc/ sudo jailhouse config create sysconfig.c
  • 20.
    2020 Non-root cell configurationformat Cell resources struct { struct jailhouse_cell_desc cell; /*...*/ } __attribute__((packed)) config = { .cell = { .signature = JAILHOUSE_CELL_DESC_SIGNATURE, .name = "apic-demo", .cpu_set_size = sizeof(config.cpus), /*...*/ } .cpus = { 0x8, },
  • 21.
    2121 Cell resources • Memoryregions: – Physical address, virtual address, size – Flags (JAILHOUSE_MEM_*): • PIO bitmap (x86 port-based I/O) • IRQ chips • Cache regions • PCI devices and related capabilities • Memory-mapped UARTs All configurations written in C
  • 22.
    2222 Driver installation • Build& install Jailhouse: The build process also transforms all configuration files from C to binary (i.e. *.cell files) • Install the Jailhouse driver: $ make $ sudo make install $ sudo modprobe jailhouse
  • 23.
    2323 Jailhouse enabling • Enablethe root cell passing the system configuration: $ sudo jailhouse enable configs/sysconfig.cell
  • 24.
  • 25.
    2525 Non-root cell creation •Create a non-root cell: • Load a binary into the created cell: • Start the cell: • (optional) Get statistics about the cell: $ sudo jailhouse cell create configs/file.cell $ sudo jailhouse cell load cell_name code.bin $ sudo jailhouse cell start cell_name $ sudo jailhouse cell stats cell_name
  • 26.
  • 27.
    2727 Good practices • Givethe hypervisor access to a serial console • Put #define CONFIG_TRACE_ERROR in file include/jailhouse/config.h before compiling • Example of misconfiguration:
  • 28.
    2828 NVIDIA support • InHERCULES we have supported the following platforms: – NVIDIA TX1/TX2 (Cortex-A57 + Denver) – Xilinx ZCU102 (Cortex-A53) • NVIDIA's vendor kernel is not supported by mainline Jailhouse: – We created a specific Jailhouse tree: https://github.com/evidence/linux-jailhouse-jetson – Virtual machine containing Jailhouse + ERIKA: http://www.erika-enterprise.com/index.php/download/virtual- machines.html
  • 29.
    2929 Communication • A sophisticatedmechanism allows cells to communicate through virtual PCI devices – Model similar to ivshmem device from Qemu – No multicast communications possible • See Documentation/inter-cell- communication.txt
  • 30.
    3030 Linux kernel AUTOSAR COM Jailhouse hypervisor SWC AUTOSARClassic App Linux OS ARM Cortex-A ARM Cortex-A ARM Cortex-A ARM Cortex-A • Library on top of Jailhouse's mechanism • Blocking and non-blocking calls • Dynamic-size messages • Similar to AUTOSAR COM API: Com_StatusType Com_GetStatus(); uint8 Com_SendSignal(Com_SignalIdType SignalId, const void *SignalDataPtr); uint8 Com_ReceiveSignal(Com_SignalIdType SignalId, void* SignalDataPtr); • Soon available (GPL)
  • 31.
  • 32.
    3232 Interference Core Linux RTOS Hypervisor Core General- Purpose apps Real-time apps L1 L1 L2cache SDRAM Interference • Even with partitioning, there is still some interference on shared hardware E.g. caches, memory bus, etc. • One core can affect the real-time responsiveness of other cores • Software solutions: 1. Cache coloring to avoid data eviction (handle virtual memory so that pages with different "colors" have different positions in cache) 2. "Memguard" (force memory bandwidth by monitoring performance counters) 3. Co-scheduling algorithms (orchestrating memory accesses) • Developed in HERCULES • Soon released as open-source
  • 33.
    3333 Conclusions • Software stack: –Handling mixed criticality – Targeting automotive (AUTOSAR compliant) – Open-source (GPL) – Working on COTS ARM hardware