ACRN: A Big Little Hypervisor for IoT Development
CHEN, Jason
Jason.cj.chen@intel.com
3/11/2020

Agenda
❑ Overview
- ACRN Introduction
- ACRN Scenario
- ACRN Architecture
❑ Modules
- ACRN Boot
- ACRN CPU
- ACRN Memory
- ACRN Interrupt
- ACRN PTDev
- ACRN DM – IO Request

Introduction
ACRN™ is a flexible, open-source, lightweight hypervisor -
intended to enable consolidation of heterogeneous
workloads, and to streamline IoT edge development.
- A Linux Foundation Project
- Launched in March 2018
- Version 1.0 released in May 2019

Ready for Production
• Released in May 2019
Key Features
• Safety and Security Isolation (Cluster + IVI)
• Extensive Sharing Capabilities
• Graphics, media, USB, audio, camera etc.
• Advanced DMA/graphics buffer sharing
• Multiple OS Support
• Clear Linux, Yocto, Ubuntu
• Android, AGL, AliOS
• MISRA-C Compliance
ACRN hypervisor
Device Model
Service VM User VMs
Trusty
ACRN 1.0 Scenario

Trusty
World
ACRN 1.0 Architecture
Subtext goes here
Service VM
(PIT, PCI, ACPI ..)
Hypercalls
VT-d EPT
ACRN Hypervisor
VMX
Trusty API vPIC/vLAPIC/
vIOAPIC/vMSI
ACRN Device
Model
(Mediators)
VM
Manager
Linux VM
virtio
FE Drivers
User
Kernel
User
Kernel
VM API
SOC Platform (Apollo Lake etc.)
Firmware (UEFI, SlimBoot etc.)
CSE
Virtual Firmware
IVI App
Normal
World
virtio
FE Drivers
User
Kernel
Virtual Firmware
VMX non-root
operation
VMX root
operation
Android VM
Encrypt/
signature
Native Device Driver
Native Device Driver
Kernel
Mediators

Hybrid-Mode
• Partition + Shared
VM Type
• Safety VM
• Real-time VM
Kata Container
More OS Support:
• Zephyr, VxWorks,
RT-Linux, Windows
FUSA Certification
ACRN hypervisor
Device Model
Safety App
Service VM User VMsSafety VM
(RTVM)
ACRN 2.0 Scenario

ACRN 2.0 Architecture
`
Service VM
ACRN Device
Model
VMX
VT-d EPT
ACRN
OSPM
vPIC/vLAPIC/ vIOAPIC/vMSI
ACRN DM
(Device Model)
Hypercalls
VM Manager
Native Device
Drivers
User VM1
(Post-Launched)
virtio
FE Drivers
Virtual Firmware
(OVMF)
Emulated Device
Drivers
User VM0
(Pre-Launched)
Passthru Device
Drivers
Passthru Device
Drivers
vPCI/vBridge/vUART/vRTC/vWDT…
RT VM
(Post-Launched)
Virtual Firmware
(OVMF)
HV Emulated
Device Drivers
ACRN Service
Driver
CPU Scheduler
virtio
FE Drivers(PMD)
Passthru Device
Drivers

ACRN Module Interaction Arch

Agenda
❑ Overview
- ACRN Introduction
- ACRN Scenario
- ACRN Architecture
❑ Modules
- ACRN Boot
- ACRN CPU
- ACRN Memory
- ACRN Interrupt
- ACRN PTDev
- ACRN DM – IO Request

ACRN Boot
• Support SBL & UEFI
Firmware boot
Firmware
(SBL or UEFI)
Multiboot
Interface
mmap info
acrn.out
sos image
module
ACRN
Hypervisor
• Multiboot1/2 interface
• Hypervisor boot from
BSP
• Hypervisor launch
SOS directly
• Hypervisor launch
UOS through SOS-DM
hypercall

ACRN CPU Virtualization
VM0
Hypervisor
CPU Scheduler
VCPU0 VCPU1
VCPU0-VM0
vCR
vMSR
vCPUID
VCPU1-VM0
vCR
vMSR
vCPUID
VCPU0-VM1
vCR
vMSR
vCPUID
VCPU1-VM1
vCR
vMSR
vCPUID
VM1
VCPU0 VCPU1
Hardware
PCPU0 PCPU1
• Dedicated VCPU instances
for each VM in hypervisor
• Emulation for
• CPUID
• Control Register
• MSR
• VM-Exit
• CPUID Instruction
• Control Register Access
• RDMSR/WRMSR Instruction

ACRN Memory (Concept)
• GVA to GPA
• MMU + Page Table in Guest
• GPA to HPA
• EPT + Page Table in
Hypervisor (per VM)
• HPA to HVA
• MMU + Page Table in
Hypervisor
• VM-Exit – EPT Violation
• no page mapping in EPT
page tables
• for MMIO emulation
Hypervisor
VM1
VCPU1 VCPU2
MMU
GVA
GPA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
VM2
VCPU1 VCPU2
MMU
GVA
GPA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
GVA
Page
Table
EPT EPT
Page Table Page Table
HPAHPA
HVA
MMU
Page Table

ACRN Memory Layout
• Identical mapping for HPA <-> HVA
• Identical mapping for SOS GPA <-> HPA
• SOS DM allocate UOS memory based on Hugetlb

ACRN Interrupt Controller
• In hypervisor(native) layer, ACRN configures
physical PIC/IOAPIC/LAPIC to support different
interrupt source from local timer/IPI to external
INTx/MSI.
• In guest layer, ACRN may emulates virtual PIC,
virtual IOAPIC and virtual LAPIC, and provides
full APIs which allow the virtual interrupt
injection from emulated or pass-thru devices.
• In guest layer, ACRN may choose pass-thru
LAPIC, which pass-thru almost all LAPIC
registers except ICR/XAPICID/LDR, and guest
will run under x2apic mode w/o vPIC &
vIOAPIC.
• VM-Exit
- PIO Access : VPIC (0x20/0x21/0xA0/0xA1…)
- EPT Violation: VIOAPIC/VLAPIC
- APIC Access/Write: APICV
Hypervisor
VM0
VCPU0
VLAPIC0
VCPU1
VLAPIC1
VIOAPIC
PCPU0
LAPIC0
PCPU1
LAPIC1
VM1
VCPU2
VLAPIC0
VCPU0
LAPIC-PT
PCPU2
LAPIC2
PCPU3
LAPIC3
IOAPIC
VPIC
PIC
Hardware
Local Timer INTx Devices MSI DevicesIPI
VPIC
VIOAPIC
Note:
under LAPIC PT mode, hypervisor will not handle native interrupt, and only MSI interrupt is supported for
this guest and such MSI interrupt will be injected into guest directly

ACRN Interrupt Injection
• Source
- Emulated device
- Pass-thru device
- Others (timer/IPI) – not show here
• Type
- INTx – through PIC/IOAPIC -> LAPIC
- MSI – through LAPIC
- IPI/LVT – through LAPIC
• Physical Interrupt Handler
- do irq for physical interrupt
- physical interrupt may happen in
hypervisor(vmx root mode) or guest (vmx
non-root mode -> vm-exit)
• Virtual Interrupt Handler
- virtual interrupt injection to guest
- request may come from DM(hypercall) or
PTDev(physical interrupt handler)
Hardware
Hypervisor
Service
VM
PCPU0
LAPIC0
PCPU1
LAPIC1
User
VM
PCPU2
LAPIC2
IOAPIC
Pass-thru Dev
w/ INTx
DM
Emulated Device
(INTx or MSI)
VCPU0
VLAPIC0
VIOAPIC
VCPU1
VLAPIC1
Pass-thru Dev
w/ MSI
Physical Interrupt Handler
Virtual Interrupt Handler

ACRN Physical Interrupt Handler
Enable Intr
VM Exit
Other Exit
Handler
Disable Intr
External Intr Exit
Handler
Enable Intr
Handle pending
req
VM Entry
Dispatch Interrupt
irq_desc[vector_2_irq(vec)]
Spurious
Handler
LAPIC ACK EOI
Action Handler
Edge
Handler
LAPIC ACK EOI
Action Handler
Level
Handler
LAPIC ACK EOI
Action Handler
IOAPIC Mask Intr
• Unregistered Vector • VCPU Notification IPI
• Local Timer
• PTDev MSI
• Non-PTDev INTx
1
1
2
1 Physical interrupt could come from VMX root mode after enabled interrupt
2 Physical interrupt could come from VMX non-root mode which trigger a vm exit
Dev Level
Handler
LAPIC ACK EOI
Action Handler
IOAPIC Mask Intr
• PTDev INTx (IOAPIC
Unmask Intr when
vIOAPIC/vPIC get
ACK EOI)
IOAPIC unMask
Intr
Do Softirq
Note:
action for PTDev leads to
virtual interrupt injection

ACRN Virtual Interrupt/Exception Injection
** it’s part of acrn_handle_pending_request
High priority
exception exist?
Inject hi exception
through
VMX_INT_INFO
Interrupt Window
Check
Interrupt
Pending from vPIC or
vLAPIC
(basic APICv)?
NMI exist?
Previous
Pending IDT Intr/
Excep?
Guest Irq
Enabled?
Inject NMI
through
VMX_INT_INFO
Inject VMX IDT
Pending Intr/Excep
through
VMX_INT_INFO
Low priority
Exception exist?
Y
Enable Interrupt
Window
Y
VM Exit
N
N
Y
N
Y
N
Y
N
Inject lo exception
through
VMX_INT_INFO
Y
N
Pending extint
from vPIC?
Inject extint
through
VMX_INT_INFO
Y
Pending
interrupt from
vLAPIC?
N
N
Inject lapic
pending interrupt
through
VMX_INT_INFO
Guest
irq enabled
&&
!irq_injected
Basic APICv
Y
N
Inject lapic
pending interrupt
through
VID PIR
Advanced
APICv
Inject virtual interrupt
from
vPIC/vIOAPIC/vLAPIC
-
acrn_inject_pending_intr
VM Entry
• Request for virtual interrupt injection come
from virtual interrupt controller - vPIC,
vIOAPIC, vLAPIC
• Request for virtual exception injection come
from different modules who need emulate
an exception – instruction emulation,
CR/MSR emulation …
• The injection be done based on priority

ACRN Pass-thru Device
• ACRN support PCI-based
pass-thru devices
• With the help of EPT, no VM-
Exit for MMIO access
• Necessary VM-Exit for PCI
config space access and
interrupt injection
• ACRN support PT IRQ
mapping mechanism for
pass-thru devices
Hardware
Hypervisor
VM
Pass-thru Dev
vPCI EPT
PCI Config Space MMIO Interrupt
PT IRQ
Pass-thru Dev
PCI Config Space MMIO Interrupt
Main Purpose – try best to avoid VM-Exit!!!

ACRN Pass-thru IRQ
• PTDev
- a PTDev may trigger interrupt from INTx
or MSI/MSIX
- exclusively assigned to different VMs
• PTIRQ Entry:
- maintain PTDev interrupt source (INTx &
MSI) mapping between native and virtual
- exclusively assigned to different VMs
associated with PTDev
• Physical Interrupt Management:
- allocate physical vector in hypervisor
when PTDev assign PTIRQ entry
- physical IOAPIC RTE & MSI fields can
be update according to virtual
configuration (except vector field)
- physical interrupt handler will look up
PTIRQ entries to find virtual dest then do
virtual interrupt injection after receive
external interrupt from PTDev
Hypervisor(Native)
VM0
IOAPIC
vPIC
PT PCI Dev
vBDF = (a, b, c)
PT PCI Dev
BDF = (A, B, C)
1 MSI, 1 INTx
VM1
vIOAPIC
PT PCI Dev
vBDF = (d, e, f)
PT PCI Dev
BDF = (D, E, F)
2 MSIX, 1 INTx
ptirq entry = {
type = intx,
vm = vm0,
int_src = pic,
virt_pin = a,
phy_pin = A,
}
ptirq entry = {
type = intx,
vm = vm1,
int_src = iopic,
virt_pin = c,
phy_pin = C,
}
ptirq entry = {
type = msix,
vm = vm1,
virt_bdf = def,
virt_idx = x,
phy_bdf = DEF,
phy_idx = X,}
INTx pins (PIC/IOAPIC)
MSI/MSIX entry
vecPhy RTE:
Vector is fixed after allocated
Others fields could be changed
according to vRTE changes
Physical Interrupt
Management
Phy MSI addr:
Vector is fixed after allocated
Others fields could be changed
according to vMSI changes
vecPhy MSI data:
Not exist for vPT-
LAPIC case
External interrupt
from PTDev
A C
a c x
X

SOS
Device
Model
ACRN HV
UOS
Virtual Dev
HV Device
Model
ACRN Device Model - IO Request
• IO Trap
- hypervisor trap out (VM-Exit) virtual
device’s PIO/MMIO access from a guest
(UOS)
• IO Request:
- hypervisor figure out the target address
and target size for the PIO/MMIO access
, to form a IO request
- instruction decode needed for a MMIO
access
- the IO request will dispatch to specific
device model based on target address
• IO Emulation
- a virtual device provide IO emulation for its
PIO or MMIO based on the IO request
- the virtual device may be emulated by
different device models from HV DM, SOS
kernel BE or SOS user space DM

Q & A