SlideShare a Scribd company logo
1 of 21
Download to read offline
ACRN: A Big Little Hypervisor for IoT Development
CHEN, Jason
Jason.cj.chen@intel.com
3/11/2020
Agenda
❑ Overview
- ACRN Introduction
- ACRN Scenario
- ACRN Architecture
❑ Modules
- ACRN Boot
- ACRN CPU
- ACRN Memory
- ACRN Interrupt
- ACRN PTDev
- ACRN DM – IO Request
Introduction
ACRN™ is a flexible, open-source, lightweight hypervisor -
intended to enable consolidation of heterogeneous
workloads, and to streamline IoT edge development.
- A Linux Foundation Project
- Launched in March 2018
- Version 1.0 released in May 2019
Ready for Production
• Released in May 2019
Key Features
• Safety and Security Isolation (Cluster + IVI)
• Extensive Sharing Capabilities
• Graphics, media, USB, audio, camera etc.
• Advanced DMA/graphics buffer sharing
• Multiple OS Support
• Clear Linux, Yocto, Ubuntu
• Android, AGL, AliOS
• MISRA-C Compliance
ACRN hypervisor
Device Model
Service VM User VMs
Trusty
ACRN 1.0 Scenario
Trusty
World
ACRN 1.0 Architecture
Subtext goes here
Service VM
(PIT, PCI, ACPI ..)
Hypercalls
VT-d EPT
ACRN Hypervisor
VMX
Trusty API vPIC/vLAPIC/
vIOAPIC/vMSI
ACRN Device
Model
(Mediators)
VM
Manager
Linux VM
virtio
FE Drivers
User
Kernel
User
Kernel
VM API
SOC Platform (Apollo Lake etc.)
Firmware (UEFI, SlimBoot etc.)
CSE
Virtual Firmware
IVI App
Normal
World
virtio
FE Drivers
User
Kernel
Virtual Firmware
VMX non-root
operation
VMX root
operation
Android VM
Encrypt/
signature
Native Device Driver
Native Device Driver
Kernel
Mediators
Hybrid-Mode
• Partition + Shared
VM Type
• Safety VM
• Real-time VM
Kata Container
More OS Support:
• Zephyr, VxWorks,
RT-Linux, Windows
FUSA Certification
ACRN hypervisor
Device Model
Safety App
Service VM User VMsSafety VM
(RTVM)
ACRN 2.0 Scenario
ACRN 2.0 Architecture
`
Service VM
ACRN Device
Model
VMX
VT-d EPT
ACRN
OSPM
vPIC/vLAPIC/ vIOAPIC/vMSI
ACRN DM
(Device Model)
Hypercalls
VM Manager
Native Device
Drivers
User VM1
(Post-Launched)
virtio
FE Drivers
Virtual Firmware
(OVMF)
Emulated Device
Drivers
User VM0
(Pre-Launched)
Passthru Device
Drivers
Passthru Device
Drivers
vPCI/vBridge/vUART/vRTC/vWDT…
RT VM
(Post-Launched)
Virtual Firmware
(OVMF)
HV Emulated
Device Drivers
ACRN Service
Driver
CPU Scheduler
virtio
FE Drivers(PMD)
Passthru Device
Drivers
ACRN Module Interaction Arch
Agenda
❑ Overview
- ACRN Introduction
- ACRN Scenario
- ACRN Architecture
❑ Modules
- ACRN Boot
- ACRN CPU
- ACRN Memory
- ACRN Interrupt
- ACRN PTDev
- ACRN DM – IO Request
ACRN Boot
• Support SBL & UEFI
Firmware boot
Firmware
(SBL or UEFI)
Multiboot
Interface
mmap info
acrn.out
sos image
module
ACRN
Hypervisor
• Multiboot1/2 interface
• Hypervisor boot from
BSP
• Hypervisor launch
SOS directly
• Hypervisor launch
UOS through SOS-DM
hypercall
ACRN CPU Virtualization
VM0
Hypervisor
CPU Scheduler
VCPU0 VCPU1
VCPU0-VM0
vCR
vMSR
vCPUID
VCPU1-VM0
vCR
vMSR
vCPUID
VCPU0-VM1
vCR
vMSR
vCPUID
VCPU1-VM1
vCR
vMSR
vCPUID
VM1
VCPU0 VCPU1
Hardware
PCPU0 PCPU1
• Dedicated VCPU instances
for each VM in hypervisor
• Emulation for
• CPUID
• Control Register
• MSR
• VM-Exit
• CPUID Instruction
• Control Register Access
• RDMSR/WRMSR Instruction
ACRN Memory (Concept)
• GVA to GPA
• MMU + Page Table in Guest
• GPA to HPA
• EPT + Page Table in
Hypervisor (per VM)
• HPA to HVA
• MMU + Page Table in
Hypervisor
• VM-Exit – EPT Violation
• no page mapping in EPT
page tables
• for MMIO emulation
Hypervisor
VM1
VCPU1 VCPU2
MMU
GVA
GPA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
VM2
VCPU1 VCPU2
MMU
GVA
GPA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
EPT EPT
Page Table Page Table
HPAHPA
HVA
MMU
Page Table
ACRN Memory Layout
• Identical mapping for HPA <-> HVA
• Identical mapping for SOS GPA <-> HPA
• SOS DM allocate UOS memory based on Hugetlb
ACRN Interrupt Controller
• In hypervisor(native) layer, ACRN configures
physical PIC/IOAPIC/LAPIC to support different
interrupt source from local timer/IPI to external
INTx/MSI.
• In guest layer, ACRN may emulates virtual PIC,
virtual IOAPIC and virtual LAPIC, and provides
full APIs which allow the virtual interrupt
injection from emulated or pass-thru devices.
• In guest layer, ACRN may choose pass-thru
LAPIC, which pass-thru almost all LAPIC
registers except ICR/XAPICID/LDR, and guest
will run under x2apic mode w/o vPIC &
vIOAPIC.
• VM-Exit
- PIO Access : VPIC (0x20/0x21/0xA0/0xA1…)
- EPT Violation: VIOAPIC/VLAPIC
- APIC Access/Write: APICV
Hypervisor
VM0
VCPU0
VLAPIC0
VCPU1
VLAPIC1
VIOAPIC
PCPU0
LAPIC0
PCPU1
LAPIC1
VM1
VCPU2
VLAPIC0
VCPU0
LAPIC-PT
PCPU2
LAPIC2
PCPU3
LAPIC3
IOAPIC
VPIC
PIC
Hardware
Local Timer INTx Devices MSI DevicesIPI
VPIC
VIOAPIC
Note:
under LAPIC PT mode, hypervisor will not handle native interrupt, and only MSI interrupt is supported for
this guest and such MSI interrupt will be injected into guest directly
ACRN Interrupt Injection
• Source
- Emulated device
- Pass-thru device
- Others (timer/IPI) – not show here
• Type
- INTx – through PIC/IOAPIC -> LAPIC
- MSI – through LAPIC
- IPI/LVT – through LAPIC
• Physical Interrupt Handler
- do irq for physical interrupt
- physical interrupt may happen in
hypervisor(vmx root mode) or guest (vmx
non-root mode -> vm-exit)
• Virtual Interrupt Handler
- virtual interrupt injection to guest
- request may come from DM(hypercall) or
PTDev(physical interrupt handler)
Hardware
Hypervisor
Service
VM
PCPU0
LAPIC0
PCPU1
LAPIC1
User
VM
PCPU2
LAPIC2
IOAPIC
Pass-thru Dev
w/ INTx
DM
Emulated Device
(INTx or MSI)
VCPU0
VLAPIC0
VIOAPIC
VCPU1
VLAPIC1
Pass-thru Dev
w/ MSI
Physical Interrupt Handler
Virtual Interrupt Handler
ACRN Physical Interrupt Handler
Enable Intr
VM Exit
Other Exit
Handler
Disable Intr
External Intr Exit
Handler
Enable Intr
Handle pending
req
VM Entry
Dispatch Interrupt
irq_desc[vector_2_irq(vec)]
Spurious
Handler
LAPIC ACK EOI
Action Handler
Edge
Handler
LAPIC ACK EOI
Action Handler
Level
Handler
LAPIC ACK EOI
Action Handler
IOAPIC Mask Intr
• Unregistered Vector • VCPU Notification IPI
• Local Timer
• PTDev MSI
• Non-PTDev INTx
1
1
2
1 Physical interrupt could come from VMX root mode after enabled interrupt
2 Physical interrupt could come from VMX non-root mode which trigger a vm exit
Dev Level
Handler
LAPIC ACK EOI
Action Handler
IOAPIC Mask Intr
• PTDev INTx (IOAPIC
Unmask Intr when
vIOAPIC/vPIC get
ACK EOI)
IOAPIC unMask
Intr
Do Softirq
Note:
action for PTDev leads to
virtual interrupt injection
ACRN Virtual Interrupt/Exception Injection
** it’s part of acrn_handle_pending_request
High priority
exception exist?
Inject hi exception
through
VMX_INT_INFO
Interrupt Window
Check
Interrupt
Pending from vPIC or
vLAPIC
(basic APICv)?
NMI exist?
Previous
Pending IDT Intr/
Excep?
Guest Irq
Enabled?
Inject NMI
through
VMX_INT_INFO
Inject VMX IDT
Pending Intr/Excep
through
VMX_INT_INFO
Low priority
Exception exist?
Y
Enable Interrupt
Window
Y
VM Exit
N
N
Y
N
Y
N
Y
N
Inject lo exception
through
VMX_INT_INFO
Y
N
Pending extint
from vPIC?
Inject extint
through
VMX_INT_INFO
Y
Pending
interrupt from
vLAPIC?
N
N
Inject lapic
pending interrupt
through
VMX_INT_INFO
Guest
irq enabled
&&
!irq_injected
Basic APICv
Y
N
Inject lapic
pending interrupt
through
VID PIR
Advanced
APICv
Inject virtual interrupt
from
vPIC/vIOAPIC/vLAPIC
-
acrn_inject_pending_intr
VM Entry
• Request for virtual interrupt injection come
from virtual interrupt controller - vPIC,
vIOAPIC, vLAPIC
• Request for virtual exception injection come
from different modules who need emulate
an exception – instruction emulation,
CR/MSR emulation …
• The injection be done based on priority
ACRN Pass-thru Device
• ACRN support PCI-based
pass-thru devices
• With the help of EPT, no VM-
Exit for MMIO access
• Necessary VM-Exit for PCI
config space access and
interrupt injection
• ACRN support PT IRQ
mapping mechanism for
pass-thru devices
Hardware
Hypervisor
VM
Pass-thru Dev
vPCI EPT
PCI Config Space MMIO Interrupt
PT IRQ
Pass-thru Dev
PCI Config Space MMIO Interrupt
Main Purpose – try best to avoid VM-Exit!!!
ACRN Pass-thru IRQ
• PTDev
- a PTDev may trigger interrupt from INTx
or MSI/MSIX
- exclusively assigned to different VMs
• PTIRQ Entry:
- maintain PTDev interrupt source (INTx &
MSI) mapping between native and virtual
- exclusively assigned to different VMs
associated with PTDev
• Physical Interrupt Management:
- allocate physical vector in hypervisor
when PTDev assign PTIRQ entry
- physical IOAPIC RTE & MSI fields can
be update according to virtual
configuration (except vector field)
- physical interrupt handler will look up
PTIRQ entries to find virtual dest then do
virtual interrupt injection after receive
external interrupt from PTDev
Hypervisor(Native)
VM0
IOAPIC
vPIC
PT PCI Dev
vBDF = (a, b, c)
PT PCI Dev
BDF = (A, B, C)
1 MSI, 1 INTx
VM1
vIOAPIC
PT PCI Dev
vBDF = (d, e, f)
PT PCI Dev
BDF = (D, E, F)
2 MSIX, 1 INTx
ptirq entry = {
type = intx,
vm = vm0,
int_src = pic,
virt_pin = a,
phy_pin = A,
}
ptirq entry = {
type = intx,
vm = vm1,
int_src = iopic,
virt_pin = c,
phy_pin = C,
}
ptirq entry = {
type = msix,
vm = vm1,
virt_bdf = def,
virt_idx = x,
phy_bdf = DEF,
phy_idx = X,}
INTx pins (PIC/IOAPIC)
MSI/MSIX entry
vecPhy RTE:
Vector is fixed after allocated
Others fields could be changed
according to vRTE changes
Physical Interrupt
Management
Phy MSI addr:
Vector is fixed after allocated
Others fields could be changed
according to vMSI changes
vecPhy MSI data:
Not exist for vPT-
LAPIC case
External interrupt
from PTDev
A C
a c x
X
SOS
Device
Model
ACRN HV
UOS
Virtual Dev
HV Device
Model
ACRN Device Model - IO Request
• IO Trap
- hypervisor trap out (VM-Exit) virtual
device’s PIO/MMIO access from a guest
(UOS)
• IO Request:
- hypervisor figure out the target address
and target size for the PIO/MMIO access
, to form a IO request
- instruction decode needed for a MMIO
access
- the IO request will dispatch to specific
device model based on target address
• IO Emulation
- a virtual device provide IO emulation for its
PIO or MMIO based on the IO request
- the virtual device may be emulated by
different device models from HV DM, SOS
kernel BE or SOS user space DM
Q & A

More Related Content

What's hot

The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
hugo lu
 

What's hot (20)

Physical Memory Models.pdf
Physical Memory Models.pdfPhysical Memory Models.pdf
Physical Memory Models.pdf
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
 
Kata Container - The Security of VM and The Speed of Container | Yuntong Jin
Kata Container - The Security of VM and The Speed of Container | Yuntong Jin	Kata Container - The Security of VM and The Speed of Container | Yuntong Jin
Kata Container - The Security of VM and The Speed of Container | Yuntong Jin
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
 
05.2 virtio introduction
05.2 virtio introduction05.2 virtio introduction
05.2 virtio introduction
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
 
Enhancing Network and Runtime Security with Cilium and Tetragon by Raymond De...
Enhancing Network and Runtime Security with Cilium and Tetragon by Raymond De...Enhancing Network and Runtime Security with Cilium and Tetragon by Raymond De...
Enhancing Network and Runtime Security with Cilium and Tetragon by Raymond De...
 
Qemu Introduction
Qemu IntroductionQemu Introduction
Qemu Introduction
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on Lab
 
Build your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto projectBuild your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto project
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoCapturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
 
Qemu JIT Code Generator and System Emulation
Qemu JIT Code Generator and System EmulationQemu JIT Code Generator and System Emulation
Qemu JIT Code Generator and System Emulation
 
Project ACRN expose and pass through platform hidden PCIe devices to SOS
Project ACRN expose and pass through platform hidden PCIe devices to SOSProject ACRN expose and pass through platform hidden PCIe devices to SOS
Project ACRN expose and pass through platform hidden PCIe devices to SOS
 
introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack
 
Dockerと外部ルータを連携させる仕組みを作ってみた
Dockerと外部ルータを連携させる仕組みを作ってみたDockerと外部ルータを連携させる仕組みを作ってみた
Dockerと外部ルータを連携させる仕組みを作ってみた
 
ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門
 
Linux Instrumentation
Linux InstrumentationLinux Instrumentation
Linux Instrumentation
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
GPU仮想化最前線 - KVMGTとvirtio-gpu -
GPU仮想化最前線 - KVMGTとvirtio-gpu -GPU仮想化最前線 - KVMGTとvirtio-gpu -
GPU仮想化最前線 - KVMGTとvirtio-gpu -
 
Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)
 

Similar to Project ACRN hypervisor introduction

Similar to Project ACRN hypervisor introduction (20)

Implementing SR-IOv failover for Windows guests during live migration
Implementing SR-IOv failover for Windows guests during live migrationImplementing SR-IOv failover for Windows guests during live migration
Implementing SR-IOv failover for Windows guests during live migration
 
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
 
ACRN vMeet-Up EU 2021 - Introduction and Architecture Look Forward
ACRN vMeet-Up EU 2021 - Introduction and Architecture Look ForwardACRN vMeet-Up EU 2021 - Introduction and Architecture Look Forward
ACRN vMeet-Up EU 2021 - Introduction and Architecture Look Forward
 
Static partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-VStatic partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-V
 
Embedded Systems Conference 2014 Presentation
Embedded Systems Conference 2014 PresentationEmbedded Systems Conference 2014 Presentation
Embedded Systems Conference 2014 Presentation
 
Sierraware ARM hypervisor
Sierraware ARM hypervisor Sierraware ARM hypervisor
Sierraware ARM hypervisor
 
ACRN vMeet-Up EU 2021 - Bridging Orchestrator and Hard Realtime Workload Cons...
ACRN vMeet-Up EU 2021 - Bridging Orchestrator and Hard Realtime Workload Cons...ACRN vMeet-Up EU 2021 - Bridging Orchestrator and Hard Realtime Workload Cons...
ACRN vMeet-Up EU 2021 - Bridging Orchestrator and Hard Realtime Workload Cons...
 
Zero footprint guest memory introspection from xen
Zero footprint guest memory introspection from xenZero footprint guest memory introspection from xen
Zero footprint guest memory introspection from xen
 
Project ACRN Device Passthrough Introduction
Project ACRN Device Passthrough IntroductionProject ACRN Device Passthrough Introduction
Project ACRN Device Passthrough Introduction
 
NFV Orchestration for Optimal Performance
NFV Orchestration for Optimal PerformanceNFV Orchestration for Optimal Performance
NFV Orchestration for Optimal Performance
 
Using VPP and SRIO-V with Clear Containers
Using VPP and SRIO-V with Clear ContainersUsing VPP and SRIO-V with Clear Containers
Using VPP and SRIO-V with Clear Containers
 
Contrail Enabler for agile cloud services
Contrail Enabler for agile cloud servicesContrail Enabler for agile cloud services
Contrail Enabler for agile cloud services
 
Realtime scheduling for virtual machines in SKT
Realtime scheduling for virtual machines in SKTRealtime scheduling for virtual machines in SKT
Realtime scheduling for virtual machines in SKT
 
Marriage of ESX and OpenStack - PayPal - VMWorld US 2013
Marriage of ESX and OpenStack - PayPal - VMWorld US 2013Marriage of ESX and OpenStack - PayPal - VMWorld US 2013
Marriage of ESX and OpenStack - PayPal - VMWorld US 2013
 
Intel update
Intel updateIntel update
Intel update
 
2011-11-03 Intelligence Community Cloud Users Group
2011-11-03 Intelligence Community Cloud Users Group2011-11-03 Intelligence Community Cloud Users Group
2011-11-03 Intelligence Community Cloud Users Group
 
See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...
 
Windows Server 2012 Hyper-V Networking Evolved
Windows Server 2012 Hyper-V Networking Evolved Windows Server 2012 Hyper-V Networking Evolved
Windows Server 2012 Hyper-V Networking Evolved
 
Project ACRN schedule framework introduction
Project ACRN schedule framework introductionProject ACRN schedule framework introduction
Project ACRN schedule framework introduction
 
High Performance Linux Virtual Machine on Microsoft Azure: SR-IOV Networking ...
High Performance Linux Virtual Machine on Microsoft Azure: SR-IOV Networking ...High Performance Linux Virtual Machine on Microsoft Azure: SR-IOV Networking ...
High Performance Linux Virtual Machine on Microsoft Azure: SR-IOV Networking ...
 

More from Project ACRN

More from Project ACRN (20)

ACRN vMeet-Up EU 2021 - installation and configuration introduction
ACRN vMeet-Up EU 2021 - installation and configuration introductionACRN vMeet-Up EU 2021 - installation and configuration introduction
ACRN vMeet-Up EU 2021 - installation and configuration introduction
 
ACRN vMeet-Up EU 2021 - Boot Process and Secure Boot
ACRN vMeet-Up EU 2021 - Boot Process and Secure BootACRN vMeet-Up EU 2021 - Boot Process and Secure Boot
ACRN vMeet-Up EU 2021 - Boot Process and Secure Boot
 
ACRN vMeet-Up EU 2021 - debug ACRN hypervisor
ACRN vMeet-Up EU 2021 - debug ACRN hypervisorACRN vMeet-Up EU 2021 - debug ACRN hypervisor
ACRN vMeet-Up EU 2021 - debug ACRN hypervisor
 
ACRN vMeet-Up EU 2021 - functional safety design and certification plan
ACRN vMeet-Up EU 2021 -  functional safety design and certification planACRN vMeet-Up EU 2021 -  functional safety design and certification plan
ACRN vMeet-Up EU 2021 - functional safety design and certification plan
 
ACRN vMeet-Up EU 2021 - community and development model
ACRN vMeet-Up EU 2021 - community and development modelACRN vMeet-Up EU 2021 - community and development model
ACRN vMeet-Up EU 2021 - community and development model
 
ACRN vMeet-Up EU 2021 - hypervisor new platform enabling
ACRN vMeet-Up EU 2021 - hypervisor new platform enablingACRN vMeet-Up EU 2021 - hypervisor new platform enabling
ACRN vMeet-Up EU 2021 - hypervisor new platform enabling
 
ACRN vMeet-Up EU 2021 - shared memory based inter-vm communication introduction
ACRN vMeet-Up EU 2021 - shared memory based inter-vm communication introductionACRN vMeet-Up EU 2021 - shared memory based inter-vm communication introduction
ACRN vMeet-Up EU 2021 - shared memory based inter-vm communication introduction
 
ACRN vMeet-Up EU 2021 - Real Time Management and Performance Optimization
ACRN vMeet-Up EU 2021 - Real Time Management and Performance OptimizationACRN vMeet-Up EU 2021 - Real Time Management and Performance Optimization
ACRN vMeet-Up EU 2021 - Real Time Management and Performance Optimization
 
ACRN Kata Container on ACRN
ACRN Kata Container on ACRNACRN Kata Container on ACRN
ACRN Kata Container on ACRN
 
Project ACRN Yocto Project meta-acrn layer introduction
Project ACRN Yocto Project meta-acrn layer introductionProject ACRN Yocto Project meta-acrn layer introduction
Project ACRN Yocto Project meta-acrn layer introduction
 
Project ACRN USB mediator introduction
Project ACRN USB mediator introductionProject ACRN USB mediator introduction
Project ACRN USB mediator introduction
 
Project ACRN I2C mediator introduction
Project ACRN I2C mediator introductionProject ACRN I2C mediator introduction
Project ACRN I2C mediator introduction
 
Project ACRN system debug
Project ACRN system debugProject ACRN system debug
Project ACRN system debug
 
Project ACRN SR-IOV on ACRN
Project ACRN SR-IOV on ACRNProject ACRN SR-IOV on ACRN
Project ACRN SR-IOV on ACRN
 
Project ACRN Device Model architecture introduction
Project ACRN Device Model architecture introductionProject ACRN Device Model architecture introduction
Project ACRN Device Model architecture introduction
 
Project ACRN configuration scenarios and config tool
Project ACRN configuration scenarios and config toolProject ACRN configuration scenarios and config tool
Project ACRN configuration scenarios and config tool
 
Project ACRN CPU sharing BVT scheduler in ACRN hypervisor
Project ACRN CPU sharing BVT scheduler in ACRN hypervisorProject ACRN CPU sharing BVT scheduler in ACRN hypervisor
Project ACRN CPU sharing BVT scheduler in ACRN hypervisor
 
Project ACRN how to build a Yocto Project-based SOS
Project ACRN how to build a Yocto Project-based SOSProject ACRN how to build a Yocto Project-based SOS
Project ACRN how to build a Yocto Project-based SOS
 
Project ACRN EtherCAT 101
Project ACRN EtherCAT 101Project ACRN EtherCAT 101
Project ACRN EtherCAT 101
 
Project ACRN GVT-d introduction and tutorial
Project ACRN GVT-d introduction and tutorialProject ACRN GVT-d introduction and tutorial
Project ACRN GVT-d introduction and tutorial
 

Recently uploaded

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
 
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million PeopleWSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
 
WSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration ToolingWSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration Tooling
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public AdministrationWSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMs
 

Project ACRN hypervisor introduction

  • 1. ACRN: A Big Little Hypervisor for IoT Development CHEN, Jason Jason.cj.chen@intel.com 3/11/2020
  • 2. Agenda ❑ Overview - ACRN Introduction - ACRN Scenario - ACRN Architecture ❑ Modules - ACRN Boot - ACRN CPU - ACRN Memory - ACRN Interrupt - ACRN PTDev - ACRN DM – IO Request
  • 3. Introduction ACRN™ is a flexible, open-source, lightweight hypervisor - intended to enable consolidation of heterogeneous workloads, and to streamline IoT edge development. - A Linux Foundation Project - Launched in March 2018 - Version 1.0 released in May 2019
  • 4. Ready for Production • Released in May 2019 Key Features • Safety and Security Isolation (Cluster + IVI) • Extensive Sharing Capabilities • Graphics, media, USB, audio, camera etc. • Advanced DMA/graphics buffer sharing • Multiple OS Support • Clear Linux, Yocto, Ubuntu • Android, AGL, AliOS • MISRA-C Compliance ACRN hypervisor Device Model Service VM User VMs Trusty ACRN 1.0 Scenario
  • 5. Trusty World ACRN 1.0 Architecture Subtext goes here Service VM (PIT, PCI, ACPI ..) Hypercalls VT-d EPT ACRN Hypervisor VMX Trusty API vPIC/vLAPIC/ vIOAPIC/vMSI ACRN Device Model (Mediators) VM Manager Linux VM virtio FE Drivers User Kernel User Kernel VM API SOC Platform (Apollo Lake etc.) Firmware (UEFI, SlimBoot etc.) CSE Virtual Firmware IVI App Normal World virtio FE Drivers User Kernel Virtual Firmware VMX non-root operation VMX root operation Android VM Encrypt/ signature Native Device Driver Native Device Driver Kernel Mediators
  • 6. Hybrid-Mode • Partition + Shared VM Type • Safety VM • Real-time VM Kata Container More OS Support: • Zephyr, VxWorks, RT-Linux, Windows FUSA Certification ACRN hypervisor Device Model Safety App Service VM User VMsSafety VM (RTVM) ACRN 2.0 Scenario
  • 7. ACRN 2.0 Architecture ` Service VM ACRN Device Model VMX VT-d EPT ACRN OSPM vPIC/vLAPIC/ vIOAPIC/vMSI ACRN DM (Device Model) Hypercalls VM Manager Native Device Drivers User VM1 (Post-Launched) virtio FE Drivers Virtual Firmware (OVMF) Emulated Device Drivers User VM0 (Pre-Launched) Passthru Device Drivers Passthru Device Drivers vPCI/vBridge/vUART/vRTC/vWDT… RT VM (Post-Launched) Virtual Firmware (OVMF) HV Emulated Device Drivers ACRN Service Driver CPU Scheduler virtio FE Drivers(PMD) Passthru Device Drivers
  • 9. Agenda ❑ Overview - ACRN Introduction - ACRN Scenario - ACRN Architecture ❑ Modules - ACRN Boot - ACRN CPU - ACRN Memory - ACRN Interrupt - ACRN PTDev - ACRN DM – IO Request
  • 10. ACRN Boot • Support SBL & UEFI Firmware boot Firmware (SBL or UEFI) Multiboot Interface mmap info acrn.out sos image module ACRN Hypervisor • Multiboot1/2 interface • Hypervisor boot from BSP • Hypervisor launch SOS directly • Hypervisor launch UOS through SOS-DM hypercall
  • 11. ACRN CPU Virtualization VM0 Hypervisor CPU Scheduler VCPU0 VCPU1 VCPU0-VM0 vCR vMSR vCPUID VCPU1-VM0 vCR vMSR vCPUID VCPU0-VM1 vCR vMSR vCPUID VCPU1-VM1 vCR vMSR vCPUID VM1 VCPU0 VCPU1 Hardware PCPU0 PCPU1 • Dedicated VCPU instances for each VM in hypervisor • Emulation for • CPUID • Control Register • MSR • VM-Exit • CPUID Instruction • Control Register Access • RDMSR/WRMSR Instruction
  • 12. ACRN Memory (Concept) • GVA to GPA • MMU + Page Table in Guest • GPA to HPA • EPT + Page Table in Hypervisor (per VM) • HPA to HVA • MMU + Page Table in Hypervisor • VM-Exit – EPT Violation • no page mapping in EPT page tables • for MMIO emulation Hypervisor VM1 VCPU1 VCPU2 MMU GVA GPA Page Table GVA Page Table GVA Page Table GVA Page Table GVA Page Table GVA Page Table VM2 VCPU1 VCPU2 MMU GVA GPA Page Table GVA Page Table GVA Page Table GVA Page Table GVA Page Table GVA Page Table EPT EPT Page Table Page Table HPAHPA HVA MMU Page Table
  • 13. ACRN Memory Layout • Identical mapping for HPA <-> HVA • Identical mapping for SOS GPA <-> HPA • SOS DM allocate UOS memory based on Hugetlb
  • 14. ACRN Interrupt Controller • In hypervisor(native) layer, ACRN configures physical PIC/IOAPIC/LAPIC to support different interrupt source from local timer/IPI to external INTx/MSI. • In guest layer, ACRN may emulates virtual PIC, virtual IOAPIC and virtual LAPIC, and provides full APIs which allow the virtual interrupt injection from emulated or pass-thru devices. • In guest layer, ACRN may choose pass-thru LAPIC, which pass-thru almost all LAPIC registers except ICR/XAPICID/LDR, and guest will run under x2apic mode w/o vPIC & vIOAPIC. • VM-Exit - PIO Access : VPIC (0x20/0x21/0xA0/0xA1…) - EPT Violation: VIOAPIC/VLAPIC - APIC Access/Write: APICV Hypervisor VM0 VCPU0 VLAPIC0 VCPU1 VLAPIC1 VIOAPIC PCPU0 LAPIC0 PCPU1 LAPIC1 VM1 VCPU2 VLAPIC0 VCPU0 LAPIC-PT PCPU2 LAPIC2 PCPU3 LAPIC3 IOAPIC VPIC PIC Hardware Local Timer INTx Devices MSI DevicesIPI VPIC VIOAPIC Note: under LAPIC PT mode, hypervisor will not handle native interrupt, and only MSI interrupt is supported for this guest and such MSI interrupt will be injected into guest directly
  • 15. ACRN Interrupt Injection • Source - Emulated device - Pass-thru device - Others (timer/IPI) – not show here • Type - INTx – through PIC/IOAPIC -> LAPIC - MSI – through LAPIC - IPI/LVT – through LAPIC • Physical Interrupt Handler - do irq for physical interrupt - physical interrupt may happen in hypervisor(vmx root mode) or guest (vmx non-root mode -> vm-exit) • Virtual Interrupt Handler - virtual interrupt injection to guest - request may come from DM(hypercall) or PTDev(physical interrupt handler) Hardware Hypervisor Service VM PCPU0 LAPIC0 PCPU1 LAPIC1 User VM PCPU2 LAPIC2 IOAPIC Pass-thru Dev w/ INTx DM Emulated Device (INTx or MSI) VCPU0 VLAPIC0 VIOAPIC VCPU1 VLAPIC1 Pass-thru Dev w/ MSI Physical Interrupt Handler Virtual Interrupt Handler
  • 16. ACRN Physical Interrupt Handler Enable Intr VM Exit Other Exit Handler Disable Intr External Intr Exit Handler Enable Intr Handle pending req VM Entry Dispatch Interrupt irq_desc[vector_2_irq(vec)] Spurious Handler LAPIC ACK EOI Action Handler Edge Handler LAPIC ACK EOI Action Handler Level Handler LAPIC ACK EOI Action Handler IOAPIC Mask Intr • Unregistered Vector • VCPU Notification IPI • Local Timer • PTDev MSI • Non-PTDev INTx 1 1 2 1 Physical interrupt could come from VMX root mode after enabled interrupt 2 Physical interrupt could come from VMX non-root mode which trigger a vm exit Dev Level Handler LAPIC ACK EOI Action Handler IOAPIC Mask Intr • PTDev INTx (IOAPIC Unmask Intr when vIOAPIC/vPIC get ACK EOI) IOAPIC unMask Intr Do Softirq Note: action for PTDev leads to virtual interrupt injection
  • 17. ACRN Virtual Interrupt/Exception Injection ** it’s part of acrn_handle_pending_request High priority exception exist? Inject hi exception through VMX_INT_INFO Interrupt Window Check Interrupt Pending from vPIC or vLAPIC (basic APICv)? NMI exist? Previous Pending IDT Intr/ Excep? Guest Irq Enabled? Inject NMI through VMX_INT_INFO Inject VMX IDT Pending Intr/Excep through VMX_INT_INFO Low priority Exception exist? Y Enable Interrupt Window Y VM Exit N N Y N Y N Y N Inject lo exception through VMX_INT_INFO Y N Pending extint from vPIC? Inject extint through VMX_INT_INFO Y Pending interrupt from vLAPIC? N N Inject lapic pending interrupt through VMX_INT_INFO Guest irq enabled && !irq_injected Basic APICv Y N Inject lapic pending interrupt through VID PIR Advanced APICv Inject virtual interrupt from vPIC/vIOAPIC/vLAPIC - acrn_inject_pending_intr VM Entry • Request for virtual interrupt injection come from virtual interrupt controller - vPIC, vIOAPIC, vLAPIC • Request for virtual exception injection come from different modules who need emulate an exception – instruction emulation, CR/MSR emulation … • The injection be done based on priority
  • 18. ACRN Pass-thru Device • ACRN support PCI-based pass-thru devices • With the help of EPT, no VM- Exit for MMIO access • Necessary VM-Exit for PCI config space access and interrupt injection • ACRN support PT IRQ mapping mechanism for pass-thru devices Hardware Hypervisor VM Pass-thru Dev vPCI EPT PCI Config Space MMIO Interrupt PT IRQ Pass-thru Dev PCI Config Space MMIO Interrupt Main Purpose – try best to avoid VM-Exit!!!
  • 19. ACRN Pass-thru IRQ • PTDev - a PTDev may trigger interrupt from INTx or MSI/MSIX - exclusively assigned to different VMs • PTIRQ Entry: - maintain PTDev interrupt source (INTx & MSI) mapping between native and virtual - exclusively assigned to different VMs associated with PTDev • Physical Interrupt Management: - allocate physical vector in hypervisor when PTDev assign PTIRQ entry - physical IOAPIC RTE & MSI fields can be update according to virtual configuration (except vector field) - physical interrupt handler will look up PTIRQ entries to find virtual dest then do virtual interrupt injection after receive external interrupt from PTDev Hypervisor(Native) VM0 IOAPIC vPIC PT PCI Dev vBDF = (a, b, c) PT PCI Dev BDF = (A, B, C) 1 MSI, 1 INTx VM1 vIOAPIC PT PCI Dev vBDF = (d, e, f) PT PCI Dev BDF = (D, E, F) 2 MSIX, 1 INTx ptirq entry = { type = intx, vm = vm0, int_src = pic, virt_pin = a, phy_pin = A, } ptirq entry = { type = intx, vm = vm1, int_src = iopic, virt_pin = c, phy_pin = C, } ptirq entry = { type = msix, vm = vm1, virt_bdf = def, virt_idx = x, phy_bdf = DEF, phy_idx = X,} INTx pins (PIC/IOAPIC) MSI/MSIX entry vecPhy RTE: Vector is fixed after allocated Others fields could be changed according to vRTE changes Physical Interrupt Management Phy MSI addr: Vector is fixed after allocated Others fields could be changed according to vMSI changes vecPhy MSI data: Not exist for vPT- LAPIC case External interrupt from PTDev A C a c x X
  • 20. SOS Device Model ACRN HV UOS Virtual Dev HV Device Model ACRN Device Model - IO Request • IO Trap - hypervisor trap out (VM-Exit) virtual device’s PIO/MMIO access from a guest (UOS) • IO Request: - hypervisor figure out the target address and target size for the PIO/MMIO access , to form a IO request - instruction decode needed for a MMIO access - the IO request will dispatch to specific device model based on target address • IO Emulation - a virtual device provide IO emulation for its PIO or MMIO based on the IO request - the virtual device may be emulated by different device models from HV DM, SOS kernel BE or SOS user space DM
  • 21. Q & A