SlideShare a Scribd company logo
Stefano Stabellini
VM-to-VM Communication
Mechanisms for Embedded
Existing VM-to-VM Communication Mechanisms
˃ Existing protocols: Xen PV Drivers, VirtIO
discoverable and dynamic
create new connections at runtime
made for IO virtualization
˃ A frontend in the guest connects to the
backend in Dom0 / hypervisor
˃ Created to virtualizedevices
˃ Typically based on memory sharing
˃ VirtIO expects privileged backends
>> 2
Xen
Dom0
Dom0
DomU
Frontend Backend
Hardware
Static Partitioning
Static Partitioning
˃ Static Partitioning is similar to virtualization with some key differences:
No dynamic VMs, a limited number of "partitions" instead
Focus on direct assignment of hardware resources
Configuration defined at "Build Time"
Real-Time, Safety, and Short Boot Times are often key requirements
˃ Example: Xen Dom0less
Xen Dom0less
U-Boot
Xen
Dom0 DomU 1 DomU 2
CPU CPU CPU
loads into memory
loads into memory
Xen Dom0less
U-Boot
Xen
Dom0 DomU 1 DomU 2
CPU CPU CPU
boots
boots
Xen Dom0less Current Status
˃ Static Partitions defined at build time
˃ Fast boot times, real-time support, easier to safety-certify
˃ Dom0 is not required
˃ No "out of the box" communication mechanisms available
Static Partitioning and Communication
˃ Often only VM-to-VM communication is required, not device virtualization
There are enough physical devices to directly assign them to VMs as needed
Device virtualization can be interesting for sharing an SD card among multiple VMs
˃ VM-to-VM communication is differentfrom device virtualization
A simple VM-to-VM channel to send and receive raw data
It doesn't need "frontends" and "backends"
It requires a smaller code base
It is faster for exchanging data but it is unwieldy for virtualizing devices
˃ Static definition of VM-to-VM communication channels
Define connections at "build time"
Required for safety
˃ No privileged backends
Required for safety
˃ Support Linux and non-Linux guests (Zephyr, FreeRTOS, WindRiver, QNX, etc.)
Static Partitioning VM-to-VM communication
˃ No privileged backends
Dom0 DomU 1 DomU 2
Hardware Device
Backend Frontend
Xen PV Drivers
˃ Solid and hardened in production for years (AWS)
˃ Made for device virtualization, can be used for communication:
Network
Block (disks)
Console
2D graphics, mouse and keyboard
Sound
Etc.
˃ Pros:
Very Fast Unprivileged Backends
Available for Linux, BSDs and Windows, less common among RTOSes
˃ Cons:
Might not be available in certain embedded RTOSes (but BSD versions exist for all PV drivers)
Dom0less support is work-in-progress
Xen PV Drivers and Dom0less
>> 11
U-Boot
Xen
Dom0
DomU 1 DomU 2
CPU CPU CPU
Dom0
Backend Frontend
xl
Xen PV Drivers and Dom0less
˃ Domains booted in parallel
˃ PV Drivers connections created after Dom0 is up and running
˃ Advantages compared to regular non-Dom0lessdeployments:
Domains are still started very quickly
Domains can immediately begin to perform critical tasks
Overall time to get PV Drivers up and running is shorter (no domain creation needed in Dom0)
˃ To become available by the end of the year (work by Hipert/Lab @ Unimore)
VirtIO
˃ Frontend Drivers are available in most Operating Systems
˃ "VMM" provides the backends (e.g. QEMU, kvmtools, etc.)
˃ Pros:
Many virtual device classes
˃ Cons:
Backends are currently required to be privileged – backends must be in Dom0
‒ Security implications
‒ Safety implications
Support for Xen is available, but it requires non-upstream toolstack patches
No Dom0less support
VirtIO
>> 14
Dom0
DomU 1 DomU 2
CPU CPU CPU
Dom0
Backend
"VMM"
Frontend
Xen
1. trap
2. ioreq
3. map and emulate
VirtIO
˃ IOREQ infrastructure upstream in Xen
It enables VirtIO backends and any other emulators to run in Dom0 (e.g. QEMU)
No support in the Xen tools for creating VirtIO frontends/backends yet (patch available)
PoC with virtio-block by EPAM
Requires Backends with full privileges, they have to be in Dom0
Good performance with full privileges
˃ Support for Unprivileged Backend is work-in-progress by Linaro Project Stratos
Based on memory copies to/from a pre-shared memory region
Performance to be determined (never done before, underlying protocols designed for sharing)
˃ How to enable VirtIO for Dom0less?
Could VirtIO device hotplug be used to avoid synchronous waiting during boot?
Argo
˃ Hypervisor-Mediated Data Transfers
Argo
˃ Pros:
Great performance and very strong security properties
‒ Hypervisor checks against malicious data senders
‒ Designed and optimized for memory copies
More lightweight than Xen PV Drivers and VirtIO
‒ No Xenstore, no PV backends, no VMM needed
‒ Requires Event Channels and Argo drivers (BSD drivers available here and here)
Straightforward Dom0less enablement: no need for any kind of "wait"
‒ No need to wait for Dom0 to complete booting to communicate with other VMs
˃ Current Status:
It requires one Linux patch to work with Dom0less
‒ Thanks Alec Kwapis from DornerWorks!
˃ Cons:
Requires Argo driver and Xen event channels for notifications
Static Shared Memory and Interrupts
U-Boot
Xen
Dom0
DomU 1 DomU 2
CPU CPU CPU
Dom0
ring
buffer
Static Shared Memory and Interrupts
˃ Plain shared memoryregion
Configured at "build time"
Guests setups ring buffers over shared memory
Can use OpenAMP RPMesg or any other communication libraries based on shared memory
˃ Interrupt-based notifications, work with any OSes
New hypercall to inject SGIs (patch by Xilinx)
˃ Pros:
Very simple
Works with any OS
Great performance if used correctly
˃ Cons:
One non-upstream patch to enable interrupt notifications
Require your own communication library
No dynamic connections
PL-based communication mechanisms
DomU 1 DomU 2
PL
Network
Device
Network
Device
Switch
PL-based communication mechanisms
DomU 1 DomU 2
PL
Data Movers
PL-based communication mechanisms
˃ Create Data Movers in Programmable Logic
From simple Network Devices to optimized Data Movers
˃ Assign PL resources to VMs
˃ VMs use PL to send and receive data to/from other VMs
˃ Pros:
Fastest for larger data sizes
Userspace drivers only
Easy to enable in any OS
˃ Cons:
Requires PL
Summary
Solution Upstream
Status for
regularXen
Upstream
Status for
Dom0less
VM-to-VM
Communication
vs. Device
Virtualization
Compatibility Performance Unprivileged
Backends
Plain shared
memory &
interrupts
Patch available
for interrupts
Patch
available for
interrupts
Static
VM-to-VM
Communication
Can run
anywhere
High if
implemented
correctly
Yes
Argo Upstream One patch
for Linux
available
Dynamic
VM-to-VM
Communication
Linux,
Windows with
a small effort
High Yes
Xen PV
Drivers
Upstream Patches
available
soon
Unprivileged
Device
Virtualization
Most traditional
OSes (Linux,
Windows,
BSDs)
High Yes
VirtIO Hypervisor:
upstream
No Privileged
Device
Virtualization
Most traditional
OSes (Linux,
Windows,
BSDs)
High with full
privileged
Otherwise: ?
No
(work in
progress)
Toolstack:
patches
available
Conclusions
˃ Several solutions are already available, but nothing works out of the box yet
˃ No one-size fits all:
Shared memory and notifications: best for OS compatibility
Argo: best for VM-to-VM communication
Xen PV Drivers: best for virtual devices with unprivileged backends
VirtIO: best for virtual device classes available
Demo
By Luca Miccio and Marco Solieri
Demo: Dom0less + PV Drivers
U-Boot
Xen
Dom0 DomU 1
CPU CPU
boots
boots
Demo: Dom0less + PV Drivers
U-Boot
Xen
CPU CPU
NetBack NetFront
xl
Adaptable.
Intelligent.

More Related Content

What's hot

Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
Stefano Stabellini
 
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
The Linux Foundation
 
OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training
The Linux Foundation
 
Rootlinux17: An introduction to Xen Project Virtualisation
Rootlinux17:  An introduction to Xen Project VirtualisationRootlinux17:  An introduction to Xen Project Virtualisation
Rootlinux17: An introduction to Xen Project Virtualisation
The Linux Foundation
 
Linux PV on HVM
Linux PV on HVMLinux PV on HVM
Linux PV on HVM
The Linux Foundation
 
OSSNA18: Xen Beginners Training (exercise script)
OSSNA18: Xen Beginners Training (exercise script)OSSNA18: Xen Beginners Training (exercise script)
OSSNA18: Xen Beginners Training (exercise script)
The Linux Foundation
 
ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)
The Linux Foundation
 
Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM
ShapeBlue
 
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
The Linux Foundation
 
Safety-Certifying Open Source Software: The Case of the Xen Hypervisor
Safety-Certifying Open Source Software: The Case of the Xen HypervisorSafety-Certifying Open Source Software: The Case of the Xen Hypervisor
Safety-Certifying Open Source Software: The Case of the Xen Hypervisor
Stefano Stabellini
 
Xen Project 15 Years down the Line
Xen Project 15 Years down the LineXen Project 15 Years down the Line
Xen Project 15 Years down the Line
The Linux Foundation
 
Fosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using VirtualizationFosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using Virtualization
The Linux Foundation
 
vSAN Beyond The Basics
vSAN Beyond The BasicsvSAN Beyond The Basics
vSAN Beyond The Basics
Sumit Lahiri
 
LFCollab14: Xen vs Xen Automotive
LFCollab14: Xen vs Xen AutomotiveLFCollab14: Xen vs Xen Automotive
LFCollab14: Xen vs Xen Automotive
The Linux Foundation
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
The Linux Foundation
 
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Shinya Takamaeda-Y
 
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime RipardKernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Anne Nicolas
 
Qemu Pcie
Qemu PcieQemu Pcie
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File System
Adrian Huang
 
Introduction to yocto
Introduction to yoctoIntroduction to yocto
Introduction to yocto
Alex Gonzalez
 

What's hot (20)

Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
 
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
 
OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training OSSNA18: Xen Beginners Training
OSSNA18: Xen Beginners Training
 
Rootlinux17: An introduction to Xen Project Virtualisation
Rootlinux17:  An introduction to Xen Project VirtualisationRootlinux17:  An introduction to Xen Project Virtualisation
Rootlinux17: An introduction to Xen Project Virtualisation
 
Linux PV on HVM
Linux PV on HVMLinux PV on HVM
Linux PV on HVM
 
OSSNA18: Xen Beginners Training (exercise script)
OSSNA18: Xen Beginners Training (exercise script)OSSNA18: Xen Beginners Training (exercise script)
OSSNA18: Xen Beginners Training (exercise script)
 
ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)ALSS14: Xen Project Automotive Hypervisor (Demo)
ALSS14: Xen Project Automotive Hypervisor (Demo)
 
Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM
 
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
XPDS13: Xen in OSS based In–Vehicle Infotainment Systems - Artem Mygaiev, Glo...
 
Safety-Certifying Open Source Software: The Case of the Xen Hypervisor
Safety-Certifying Open Source Software: The Case of the Xen HypervisorSafety-Certifying Open Source Software: The Case of the Xen Hypervisor
Safety-Certifying Open Source Software: The Case of the Xen Hypervisor
 
Xen Project 15 Years down the Line
Xen Project 15 Years down the LineXen Project 15 Years down the Line
Xen Project 15 Years down the Line
 
Fosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using VirtualizationFosdem 18: Securing embedded Systems using Virtualization
Fosdem 18: Securing embedded Systems using Virtualization
 
vSAN Beyond The Basics
vSAN Beyond The BasicsvSAN Beyond The Basics
vSAN Beyond The Basics
 
LFCollab14: Xen vs Xen Automotive
LFCollab14: Xen vs Xen AutomotiveLFCollab14: Xen vs Xen Automotive
LFCollab14: Xen vs Xen Automotive
 
Xen Debugging
Xen DebuggingXen Debugging
Xen Debugging
 
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)
 
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime RipardKernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
 
Qemu Pcie
Qemu PcieQemu Pcie
Qemu Pcie
 
Linux Kernel - Virtual File System
Linux Kernel - Virtual File SystemLinux Kernel - Virtual File System
Linux Kernel - Virtual File System
 
Introduction to yocto
Introduction to yoctoIntroduction to yocto
Introduction to yocto
 

Similar to ELC21: VM-to-VM Communication Mechanisms for Embedded

LOAD BALANCING OF APPLICATIONS USING XEN HYPERVISOR
LOAD BALANCING OF APPLICATIONS  USING XEN HYPERVISORLOAD BALANCING OF APPLICATIONS  USING XEN HYPERVISOR
LOAD BALANCING OF APPLICATIONS USING XEN HYPERVISOR
Vanika Kapoor
 
Xen revisited
Xen revisitedXen revisited
Xen revisited
Shahbaz Sidhu
 
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
The Linux Foundation
 
LFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and BeyondLFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and Beyond
The Linux Foundation
 
Aplura virtualization slides
Aplura virtualization slidesAplura virtualization slides
Aplura virtualization slides
The Linux Foundation
 
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
ShapeBlue
 
Medusa Project
Medusa ProjectMedusa Project
Medusa Project
Sameer Verma
 
Xen and Client Virtualization: the case of XenClient XT
Xen and Client Virtualization: the case of XenClient XTXen and Client Virtualization: the case of XenClient XT
Xen and Client Virtualization: the case of XenClient XT
The Linux Foundation
 
Slide final
Slide finalSlide final
Slide final
Nikita Gupta
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with Xen
Tamas K Lengyel
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
The Linux Foundation
 
Fuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdfFuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdf
distortdistort
 
.ppt
.ppt.ppt
Virtualization, The future of computing (archived)
Virtualization, The future of computing (archived)Virtualization, The future of computing (archived)
Virtualization, The future of computing (archived)
Bud Siddhisena
 
Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0
guest72e8c1
 
RMLL / LSM 2009
RMLL / LSM 2009RMLL / LSM 2009
RMLL / LSM 2009
Franck_Villaume
 
the NML project
the NML projectthe NML project
the NML project
Lei Yang
 
Handout2o
Handout2oHandout2o
Handout2o
Shahbaz Sidhu
 
Xen Euro Par07
Xen Euro Par07Xen Euro Par07
Xen Euro Par07
congvc
 
ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
The Linux Foundation
 

Similar to ELC21: VM-to-VM Communication Mechanisms for Embedded (20)

LOAD BALANCING OF APPLICATIONS USING XEN HYPERVISOR
LOAD BALANCING OF APPLICATIONS  USING XEN HYPERVISORLOAD BALANCING OF APPLICATIONS  USING XEN HYPERVISOR
LOAD BALANCING OF APPLICATIONS USING XEN HYPERVISOR
 
Xen revisited
Xen revisitedXen revisited
Xen revisited
 
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
CIF16/Scale14x: The latest from the Xen Project (Lars Kurth, Chairman of Xen ...
 
LFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and BeyondLFCOLLAB15: Xen 4.5 and Beyond
LFCOLLAB15: Xen 4.5 and Beyond
 
Aplura virtualization slides
Aplura virtualization slidesAplura virtualization slides
Aplura virtualization slides
 
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
 
Medusa Project
Medusa ProjectMedusa Project
Medusa Project
 
Xen and Client Virtualization: the case of XenClient XT
Xen and Client Virtualization: the case of XenClient XTXen and Client Virtualization: the case of XenClient XT
Xen and Client Virtualization: the case of XenClient XT
 
Slide final
Slide finalSlide final
Slide final
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with Xen
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
 
Fuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdfFuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdf
 
.ppt
.ppt.ppt
.ppt
 
Virtualization, The future of computing (archived)
Virtualization, The future of computing (archived)Virtualization, The future of computing (archived)
Virtualization, The future of computing (archived)
 
Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0Rmll Virtualization As Is Tool 20090707 V1.0
Rmll Virtualization As Is Tool 20090707 V1.0
 
RMLL / LSM 2009
RMLL / LSM 2009RMLL / LSM 2009
RMLL / LSM 2009
 
the NML project
the NML projectthe NML project
the NML project
 
Handout2o
Handout2oHandout2o
Handout2o
 
Xen Euro Par07
Xen Euro Par07Xen Euro Par07
Xen Euro Par07
 
ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
 

More from Stefano Stabellini

Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Stefano Stabellini
 
RunX ELCE 2020
RunX ELCE 2020RunX ELCE 2020
RunX ELCE 2020
Stefano Stabellini
 
RunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edgeRunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edge
Stefano Stabellini
 
System Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and LopperSystem Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and Lopper
Stefano Stabellini
 
Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020
Stefano Stabellini
 
Xen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time SystemXen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time System
Stefano Stabellini
 
Xen Project for ARM Servers
Xen Project for ARM ServersXen Project for ARM Servers
Xen Project for ARM Servers
Stefano Stabellini
 
Xen and OpenStack
Xen and OpenStackXen and OpenStack
Xen and OpenStack
Stefano Stabellini
 
XDS15: Project Raisin
XDS15: Project RaisinXDS15: Project Raisin
XDS15: Project Raisin
Stefano Stabellini
 
OpenStack and Xen
OpenStack and XenOpenStack and Xen
OpenStack and Xen
Stefano Stabellini
 

More from Stefano Stabellini (10)

Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
 
RunX ELCE 2020
RunX ELCE 2020RunX ELCE 2020
RunX ELCE 2020
 
RunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edgeRunX: deploy real-time OSes as containers at the edge
RunX: deploy real-time OSes as containers at the edge
 
System Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and LopperSystem Device Tree update: Bus Firewalls and Lopper
System Device Tree update: Bus Firewalls and Lopper
 
Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020Cache coloring Xen Summit 2020
Cache coloring Xen Summit 2020
 
Xen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time SystemXen Cache Coloring: Interference-Free Real-Time System
Xen Cache Coloring: Interference-Free Real-Time System
 
Xen Project for ARM Servers
Xen Project for ARM ServersXen Project for ARM Servers
Xen Project for ARM Servers
 
Xen and OpenStack
Xen and OpenStackXen and OpenStack
Xen and OpenStack
 
XDS15: Project Raisin
XDS15: Project RaisinXDS15: Project Raisin
XDS15: Project Raisin
 
OpenStack and Xen
OpenStack and XenOpenStack and Xen
OpenStack and Xen
 

Recently uploaded

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Green Software Development
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
Alberto Brandolini
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Marcin Chrost
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
AnkitaPandya11
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
YousufSait3
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 

Recently uploaded (20)

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 

ELC21: VM-to-VM Communication Mechanisms for Embedded

  • 2. Existing VM-to-VM Communication Mechanisms ˃ Existing protocols: Xen PV Drivers, VirtIO discoverable and dynamic create new connections at runtime made for IO virtualization ˃ A frontend in the guest connects to the backend in Dom0 / hypervisor ˃ Created to virtualizedevices ˃ Typically based on memory sharing ˃ VirtIO expects privileged backends >> 2 Xen Dom0 Dom0 DomU Frontend Backend Hardware
  • 4. Static Partitioning ˃ Static Partitioning is similar to virtualization with some key differences: No dynamic VMs, a limited number of "partitions" instead Focus on direct assignment of hardware resources Configuration defined at "Build Time" Real-Time, Safety, and Short Boot Times are often key requirements ˃ Example: Xen Dom0less
  • 5. Xen Dom0less U-Boot Xen Dom0 DomU 1 DomU 2 CPU CPU CPU loads into memory loads into memory
  • 6. Xen Dom0less U-Boot Xen Dom0 DomU 1 DomU 2 CPU CPU CPU boots boots
  • 7. Xen Dom0less Current Status ˃ Static Partitions defined at build time ˃ Fast boot times, real-time support, easier to safety-certify ˃ Dom0 is not required ˃ No "out of the box" communication mechanisms available
  • 8. Static Partitioning and Communication ˃ Often only VM-to-VM communication is required, not device virtualization There are enough physical devices to directly assign them to VMs as needed Device virtualization can be interesting for sharing an SD card among multiple VMs ˃ VM-to-VM communication is differentfrom device virtualization A simple VM-to-VM channel to send and receive raw data It doesn't need "frontends" and "backends" It requires a smaller code base It is faster for exchanging data but it is unwieldy for virtualizing devices ˃ Static definition of VM-to-VM communication channels Define connections at "build time" Required for safety ˃ No privileged backends Required for safety ˃ Support Linux and non-Linux guests (Zephyr, FreeRTOS, WindRiver, QNX, etc.)
  • 9. Static Partitioning VM-to-VM communication ˃ No privileged backends Dom0 DomU 1 DomU 2 Hardware Device Backend Frontend
  • 10. Xen PV Drivers ˃ Solid and hardened in production for years (AWS) ˃ Made for device virtualization, can be used for communication: Network Block (disks) Console 2D graphics, mouse and keyboard Sound Etc. ˃ Pros: Very Fast Unprivileged Backends Available for Linux, BSDs and Windows, less common among RTOSes ˃ Cons: Might not be available in certain embedded RTOSes (but BSD versions exist for all PV drivers) Dom0less support is work-in-progress
  • 11. Xen PV Drivers and Dom0less >> 11 U-Boot Xen Dom0 DomU 1 DomU 2 CPU CPU CPU Dom0 Backend Frontend xl
  • 12. Xen PV Drivers and Dom0less ˃ Domains booted in parallel ˃ PV Drivers connections created after Dom0 is up and running ˃ Advantages compared to regular non-Dom0lessdeployments: Domains are still started very quickly Domains can immediately begin to perform critical tasks Overall time to get PV Drivers up and running is shorter (no domain creation needed in Dom0) ˃ To become available by the end of the year (work by Hipert/Lab @ Unimore)
  • 13. VirtIO ˃ Frontend Drivers are available in most Operating Systems ˃ "VMM" provides the backends (e.g. QEMU, kvmtools, etc.) ˃ Pros: Many virtual device classes ˃ Cons: Backends are currently required to be privileged – backends must be in Dom0 ‒ Security implications ‒ Safety implications Support for Xen is available, but it requires non-upstream toolstack patches No Dom0less support
  • 14. VirtIO >> 14 Dom0 DomU 1 DomU 2 CPU CPU CPU Dom0 Backend "VMM" Frontend Xen 1. trap 2. ioreq 3. map and emulate
  • 15. VirtIO ˃ IOREQ infrastructure upstream in Xen It enables VirtIO backends and any other emulators to run in Dom0 (e.g. QEMU) No support in the Xen tools for creating VirtIO frontends/backends yet (patch available) PoC with virtio-block by EPAM Requires Backends with full privileges, they have to be in Dom0 Good performance with full privileges ˃ Support for Unprivileged Backend is work-in-progress by Linaro Project Stratos Based on memory copies to/from a pre-shared memory region Performance to be determined (never done before, underlying protocols designed for sharing) ˃ How to enable VirtIO for Dom0less? Could VirtIO device hotplug be used to avoid synchronous waiting during boot?
  • 17. Argo ˃ Pros: Great performance and very strong security properties ‒ Hypervisor checks against malicious data senders ‒ Designed and optimized for memory copies More lightweight than Xen PV Drivers and VirtIO ‒ No Xenstore, no PV backends, no VMM needed ‒ Requires Event Channels and Argo drivers (BSD drivers available here and here) Straightforward Dom0less enablement: no need for any kind of "wait" ‒ No need to wait for Dom0 to complete booting to communicate with other VMs ˃ Current Status: It requires one Linux patch to work with Dom0less ‒ Thanks Alec Kwapis from DornerWorks! ˃ Cons: Requires Argo driver and Xen event channels for notifications
  • 18. Static Shared Memory and Interrupts U-Boot Xen Dom0 DomU 1 DomU 2 CPU CPU CPU Dom0 ring buffer
  • 19. Static Shared Memory and Interrupts ˃ Plain shared memoryregion Configured at "build time" Guests setups ring buffers over shared memory Can use OpenAMP RPMesg or any other communication libraries based on shared memory ˃ Interrupt-based notifications, work with any OSes New hypercall to inject SGIs (patch by Xilinx) ˃ Pros: Very simple Works with any OS Great performance if used correctly ˃ Cons: One non-upstream patch to enable interrupt notifications Require your own communication library No dynamic connections
  • 20. PL-based communication mechanisms DomU 1 DomU 2 PL Network Device Network Device Switch
  • 21. PL-based communication mechanisms DomU 1 DomU 2 PL Data Movers
  • 22. PL-based communication mechanisms ˃ Create Data Movers in Programmable Logic From simple Network Devices to optimized Data Movers ˃ Assign PL resources to VMs ˃ VMs use PL to send and receive data to/from other VMs ˃ Pros: Fastest for larger data sizes Userspace drivers only Easy to enable in any OS ˃ Cons: Requires PL
  • 23. Summary Solution Upstream Status for regularXen Upstream Status for Dom0less VM-to-VM Communication vs. Device Virtualization Compatibility Performance Unprivileged Backends Plain shared memory & interrupts Patch available for interrupts Patch available for interrupts Static VM-to-VM Communication Can run anywhere High if implemented correctly Yes Argo Upstream One patch for Linux available Dynamic VM-to-VM Communication Linux, Windows with a small effort High Yes Xen PV Drivers Upstream Patches available soon Unprivileged Device Virtualization Most traditional OSes (Linux, Windows, BSDs) High Yes VirtIO Hypervisor: upstream No Privileged Device Virtualization Most traditional OSes (Linux, Windows, BSDs) High with full privileged Otherwise: ? No (work in progress) Toolstack: patches available
  • 24. Conclusions ˃ Several solutions are already available, but nothing works out of the box yet ˃ No one-size fits all: Shared memory and notifications: best for OS compatibility Argo: best for VM-to-VM communication Xen PV Drivers: best for virtual devices with unprivileged backends VirtIO: best for virtual device classes available
  • 25. Demo By Luca Miccio and Marco Solieri
  • 26. Demo: Dom0less + PV Drivers U-Boot Xen Dom0 DomU 1 CPU CPU boots boots
  • 27. Demo: Dom0less + PV Drivers U-Boot Xen CPU CPU NetBack NetFront xl