Unrestricted © Siemens AG 2018
Jan Kiszka | Linaro Connect, March 19, 2018
Partitioning ARM Systems
With the Jailhouse Hypervisor
Unrestricted © Siemens AG 2017
Page 2 Corporate Technology
About /me, about this project
• Jan Kiszka <jan.kiszka@siemens.com>
• Member of embedded Linux team at Siemens Corporate Technology
• (In-house) consultant, architect, developer for OSS
• Focus on kernel, real-time, virtualization, embedded build systems
• Upstream contributor
• https://github.com/siemens/jailhouse
• Not a product of Siemens, rather an infrastructure component
• Started as open source project by Siemens
• Published for broader industrial usage and contributions
Unrestricted © Siemens AG 2017
Page 3 Corporate Technology
Agenda
Introduction to Jailhouse hypervisor
Current status on ARM
Architectural insights
Future directions
Summary
Discussion
Unrestricted © Siemens AG 2017
Page 4 Corporate Technology
Jailhouse: Static Partitioning for Multicore Systems
• Focus on maintaining static partitions
• No scheduling
• 1:1 resource assignment
• (Almost) no device emulation
• Keep runtime code base minimal
• Hard RT properties with minimal overhead
• Enable / simplify safety certification
Design Goals
RTOS /
Bare-
Metal
Hardware
Linux
Core 4Core 3Core 1 Core 2
Jailhouse Hypervisor
Device A Device B Device C Device D
Stahlkocher,
CC BY-SA 3.0
2nd
Linux
Unrestricted © Siemens AG 2017
Page 5 Corporate Technology
Boot Process of Jailhouse
Power-On
Boot
Loader
Typical
Hypervisor
Partition 1 OS
Partition n OS
Jailhouse
Boot
Loader
Partition 2 OS
Partition n OS
Partition 2 OS
Power-On Root LinuxLinux
(Yet Another
Boot Loader)
...
...
Unrestricted © Siemens AG 2017
Page 6 Corporate Technology
Management Interface via Linux
linux # jailhouse enable system.cell
linux # jailhouse cell create realtime.cell
linux # jailhouse cell load my-cell rtos.bin
linux # jailhouse cell start my-cell
linux # jailhouse cell destroy my-cell
linux # jailhouse cell linux linux.cell kernel -i initrd -d dtb
linux # jailhouse disable
Unrestricted © Siemens AG 2017
Page 7 Corporate Technology
Modes of Operation – Trusting Linux?
Linux
Jailhouse
Cell 1
Cell 2 Cell 3
Linux
Jailhouse
Cell 1
Cell 3
Cell 2
Open Model Safety Model
• Linux (root cell) is in control
• Cells not involved in management
decisions
• Sufficient if root cell is trusted
• Linux controls, but...
• Cells can be configured to vote
over management decisions
• Building block for safe operation
Unrestricted © Siemens AG 2017
Page 8 Corporate Technology
Jailhouse Status on ARM
ARMv7
• Support for Banana-Pi, Orange-Pi, NVIDIA Jetson TK1,
VExpress, emtrion emCON-RZ/G1x
• Non-upstream: TI Sitara AM572x-EVM
• GICv2 and v3
• SMMU on to-do list
ARMv8
• Support for AMD Seattle, LeMaker HiKey, Xilinx ZynqMP,
NVIDIA Jetson TX1, ESPRESSObin, NXP i.MX8MQ
• Works inside QEMU (via virt machine and GICv3)
It's small
• Currently ~7k lines of code (ARMv8)
Unrestricted © Siemens AG 2017
Page 9 Corporate Technology
Architectural Overview
Hypervisor
Hardware
Page MappingPage Allocator
Virtual CPU
IOMMU HW Access Filters
IRQ Controller
Arch. Specifics: Mapping, PCI, Life Cycle, ...
Inter-Cell Communication
PCI Access
Life Cycle Management
MMIO Access
Debug Output
VM, IRQ, Exception Entry UART Output
Minimal libc
Jailhouse Management Tool
/sys/devices/jailhouse /dev/jailhouse
Cell Image Cell
Config
Jailhouse Image
Cell
ConfigCell Image System
Config
Linux Kernel Jailhouse Driver Module
Unrestricted © Siemens AG 2017
Page 10 Corporate Technology
Sharing Devices under Jailhouse
Jailhouse
Guest B
Hardware
Core 1 Core 2
Storage
Core 3 Core 4
LAN
Guest A
Shared Memory Device
IRQ
vETH
ivshmem-net
vETH
ivshmem-net
NFS etc.
Open issue: ivshmem (v2.0) vs. vhost-pci (virtio)
Unrestricted © Siemens AG 2017
Page 11 Corporate Technology
Secure Boot with Jailhouse – Static Chain
Boot
Loader
Partition n OS
Partition 2 OS
Power-On
Full-featured
Linux
Minimal Linux
(kernel + initrd
with Jailhouse)
...
Jailhouse
• Simple model, feasible with all architectures
• Prevents undesired hardware access of full-featured Linux
• To-do: cell image validation by Jailhouse (if not part of initrd)
Unrestricted © Siemens AG 2017
Page 12 Corporate Technology
Ongoing Developments
Generated demo & testing images
• WiP at https://github.com/siemens/jailhouse-images
• Currently generates Debian x86 image for QEMU/KVM
• Allows easy exploration of Jailhouse “look & feel”
• Planned next: ARM64 QEMU image
• Then: reference board images
Speculation barriers
• Already well isolated in static setups
• Further isolate cells inside the hypervisor
→ CPU-local memory views
• Prototype exists for x86, to be extended to ARM now
Unrestricted © Siemens AG 2017
Page 13 Corporate Technology
Future Developments
Configuration format
• Binary format optimized for runtime usage → should remain
• Source format currently C structure → should be improved
• Device Tree? Also on x86?
• Custom YAML description?
Non-Linux root cells
• Straightforward with many RTOSes
• Catch: we need stable & versioned hypervisor boot interface
Early partitioning
• Create cells via boot loader or EFI helper
• Cell reload / restart during runtime without root cell?
Clock partitioning
• Provide infrastructure to help with moderating clock access
• Avoid clock driver reimplementations in hypervisor → firmware service?
Unrestricted © Siemens AG 2017
Page 14 Corporate Technology
Why Jailhouse?
• Designed for real-time
• Full CPU isolation
• Minimal I/O latencies
• Designed for safety & security
• No emulation, no scheduling, minimal interfaces
• Target code size: <10k LOC/arch (runtime even smaller)
• Safety certification under preparation (waiting for safe hardware)
• Designed as true Open Source
• GPLv2, public for 4.5 years
• Active community, including CPU vendors
• Could eventually make into the kernel
Page 15
Thank you!
Jan Kiszka <jan.kiszka@siemens.com>

HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor

  • 1.
    Unrestricted © SiemensAG 2018 Jan Kiszka | Linaro Connect, March 19, 2018 Partitioning ARM Systems With the Jailhouse Hypervisor
  • 2.
    Unrestricted © SiemensAG 2017 Page 2 Corporate Technology About /me, about this project • Jan Kiszka <jan.kiszka@siemens.com> • Member of embedded Linux team at Siemens Corporate Technology • (In-house) consultant, architect, developer for OSS • Focus on kernel, real-time, virtualization, embedded build systems • Upstream contributor • https://github.com/siemens/jailhouse • Not a product of Siemens, rather an infrastructure component • Started as open source project by Siemens • Published for broader industrial usage and contributions
  • 3.
    Unrestricted © SiemensAG 2017 Page 3 Corporate Technology Agenda Introduction to Jailhouse hypervisor Current status on ARM Architectural insights Future directions Summary Discussion
  • 4.
    Unrestricted © SiemensAG 2017 Page 4 Corporate Technology Jailhouse: Static Partitioning for Multicore Systems • Focus on maintaining static partitions • No scheduling • 1:1 resource assignment • (Almost) no device emulation • Keep runtime code base minimal • Hard RT properties with minimal overhead • Enable / simplify safety certification Design Goals RTOS / Bare- Metal Hardware Linux Core 4Core 3Core 1 Core 2 Jailhouse Hypervisor Device A Device B Device C Device D Stahlkocher, CC BY-SA 3.0 2nd Linux
  • 5.
    Unrestricted © SiemensAG 2017 Page 5 Corporate Technology Boot Process of Jailhouse Power-On Boot Loader Typical Hypervisor Partition 1 OS Partition n OS Jailhouse Boot Loader Partition 2 OS Partition n OS Partition 2 OS Power-On Root LinuxLinux (Yet Another Boot Loader) ... ...
  • 6.
    Unrestricted © SiemensAG 2017 Page 6 Corporate Technology Management Interface via Linux linux # jailhouse enable system.cell linux # jailhouse cell create realtime.cell linux # jailhouse cell load my-cell rtos.bin linux # jailhouse cell start my-cell linux # jailhouse cell destroy my-cell linux # jailhouse cell linux linux.cell kernel -i initrd -d dtb linux # jailhouse disable
  • 7.
    Unrestricted © SiemensAG 2017 Page 7 Corporate Technology Modes of Operation – Trusting Linux? Linux Jailhouse Cell 1 Cell 2 Cell 3 Linux Jailhouse Cell 1 Cell 3 Cell 2 Open Model Safety Model • Linux (root cell) is in control • Cells not involved in management decisions • Sufficient if root cell is trusted • Linux controls, but... • Cells can be configured to vote over management decisions • Building block for safe operation
  • 8.
    Unrestricted © SiemensAG 2017 Page 8 Corporate Technology Jailhouse Status on ARM ARMv7 • Support for Banana-Pi, Orange-Pi, NVIDIA Jetson TK1, VExpress, emtrion emCON-RZ/G1x • Non-upstream: TI Sitara AM572x-EVM • GICv2 and v3 • SMMU on to-do list ARMv8 • Support for AMD Seattle, LeMaker HiKey, Xilinx ZynqMP, NVIDIA Jetson TX1, ESPRESSObin, NXP i.MX8MQ • Works inside QEMU (via virt machine and GICv3) It's small • Currently ~7k lines of code (ARMv8)
  • 9.
    Unrestricted © SiemensAG 2017 Page 9 Corporate Technology Architectural Overview Hypervisor Hardware Page MappingPage Allocator Virtual CPU IOMMU HW Access Filters IRQ Controller Arch. Specifics: Mapping, PCI, Life Cycle, ... Inter-Cell Communication PCI Access Life Cycle Management MMIO Access Debug Output VM, IRQ, Exception Entry UART Output Minimal libc Jailhouse Management Tool /sys/devices/jailhouse /dev/jailhouse Cell Image Cell Config Jailhouse Image Cell ConfigCell Image System Config Linux Kernel Jailhouse Driver Module
  • 10.
    Unrestricted © SiemensAG 2017 Page 10 Corporate Technology Sharing Devices under Jailhouse Jailhouse Guest B Hardware Core 1 Core 2 Storage Core 3 Core 4 LAN Guest A Shared Memory Device IRQ vETH ivshmem-net vETH ivshmem-net NFS etc. Open issue: ivshmem (v2.0) vs. vhost-pci (virtio)
  • 11.
    Unrestricted © SiemensAG 2017 Page 11 Corporate Technology Secure Boot with Jailhouse – Static Chain Boot Loader Partition n OS Partition 2 OS Power-On Full-featured Linux Minimal Linux (kernel + initrd with Jailhouse) ... Jailhouse • Simple model, feasible with all architectures • Prevents undesired hardware access of full-featured Linux • To-do: cell image validation by Jailhouse (if not part of initrd)
  • 12.
    Unrestricted © SiemensAG 2017 Page 12 Corporate Technology Ongoing Developments Generated demo & testing images • WiP at https://github.com/siemens/jailhouse-images • Currently generates Debian x86 image for QEMU/KVM • Allows easy exploration of Jailhouse “look & feel” • Planned next: ARM64 QEMU image • Then: reference board images Speculation barriers • Already well isolated in static setups • Further isolate cells inside the hypervisor → CPU-local memory views • Prototype exists for x86, to be extended to ARM now
  • 13.
    Unrestricted © SiemensAG 2017 Page 13 Corporate Technology Future Developments Configuration format • Binary format optimized for runtime usage → should remain • Source format currently C structure → should be improved • Device Tree? Also on x86? • Custom YAML description? Non-Linux root cells • Straightforward with many RTOSes • Catch: we need stable & versioned hypervisor boot interface Early partitioning • Create cells via boot loader or EFI helper • Cell reload / restart during runtime without root cell? Clock partitioning • Provide infrastructure to help with moderating clock access • Avoid clock driver reimplementations in hypervisor → firmware service?
  • 14.
    Unrestricted © SiemensAG 2017 Page 14 Corporate Technology Why Jailhouse? • Designed for real-time • Full CPU isolation • Minimal I/O latencies • Designed for safety & security • No emulation, no scheduling, minimal interfaces • Target code size: <10k LOC/arch (runtime even smaller) • Safety certification under preparation (waiting for safe hardware) • Designed as true Open Source • GPLv2, public for 4.5 years • Active community, including CPU vendors • Could eventually make into the kernel
  • 15.
    Page 15 Thank you! JanKiszka <jan.kiszka@siemens.com>