This document discusses the challenges in designing a type 1 hypervisor for ARM v7 virtualization extensions. It covers topics like hypervisor layering in software stack, monolithic vs microkernel design approaches, ARM v7 virtualization features, bare minimal hypervisor design, hypervisor exception handling, guest interrupt handling using GIC, timer virtualization, managing virtual devices using virtio MMIO, guest debugging support, VM management functions like context save/restore and scheduling, and remote device management use cases.

1. Challenges
in
designing
a
type
1
hypervisor
for
ARM
v7
VE
Manish
Jaggi
manish.jaggi@gmail.com
http://in.linkedin.com/in/manishjaggi

2. Hypervisor
Layering
in
Software
Stack
• Type
1
HyperOne,
Xen,
PikeOS,
OKL4,
HyperV,
Vmware
ESX
!
• Type
2
KVM,
VirtualBox,
Virtual
PC,
Parallels,
BlueStacks
Hypervisor
Hardware
UserSpace
Kernel
Hardware
OS
Kernel
UserSpace
Kernel
Hypervisor
VM1 VM2
UserSpace
Kernel
UserSpace

3. Monolithic
vs
Microkernel
Design

4. HyperOne
Microkernel
Design..
VM2 Driver
Device
Drivers
Hypervisor
Hardware
VM
VM1
Root
InterVM
IPC
Virtual
Scheduling Memory
Interrupt
Routing
Virtualization
Stack
Device
Drivers Device
Drivers

5. ARM
v7
Virtualization
Extensions
• Hypervisor
Privilege
Mode
(PL2)
NS
• Different
set
of
CP15
registers
for
hypervisor
• Stage
2
MMU
and
SMMU
• GIC
Virtual
CPU
I/f
• Timer
Virtualization
(Virtual
Timer
Interrupts
for
guest
/
hypervisor)

6. Guest
Stage
2
MMU
Translation

7. GICv2

8. SMMU

9. Bare
Minimal
Hypervisor
• A
hypervisor
boots
up
like
a
normal
smp
os
– MMU
/
GIC
/
UART
/
Timer
drivers
– Board
specific
/
platform
specific
code
– Needs
a
device
tree
– Has
a
console
(shell)
• Hypervisor
functions
– Trap
guests
accesses
– Emulate
– Load/run/suspend/resume
guests

10. IPC
Channels
vm_t
Core_regs
Cp15_regs
interrupts
vcpu_t
VM
State
vGIC
State
virtual
devices
state
Memory
Assigned
Hypervisor
keeps
the
state
of
each
VM
and
its
associated
vcpus.
!
A
VM
has
a
machine
definition
and
physical
and
virtual
devices
connected
to
it
!
State
of
virtual
devices
is
kept
with
hypervisor
like
VGIC,
console,
network
and
other
virtual
devices.
!
VCPU
has
cpu_regs
context,
the
cp15
regs
and
the
descriptors
of
interrupts
injected
into
the
vcpu.
!
This
state
is
saved
and
restored
as
and
when
the
VM
or
vcpu
is
scheduled
out
/
in.
!
There
are
two
types
of
context
saves,
Shallow
and
deep.

11. Hypervisor
Exception
Handling
• Hypervisor
handles
exceptions
originated
from
hypervisor
mode
• Hypervisor
also
handles
the
exceptions
originated
from
Guest
Mode,
these
are
routed
to
hypervisor
as
hypervisor
trap
and
hypervisor
identifies
the
type
of
exception
using
a
syndrome
register.
• Virtual
Devices’
Memory
mapped
address
space
access
is
trapped
by
hypervisor
and
the
read
write
is
emulated
in
hypervisor
mode
• Guests
can
also
issue
hyper
calls
to
the
hypervisor.
These
are
mainly
used
by
para
virtualised
drivers.
• Privileged
instructions
such
as
WFI
and
WFE
trap
into
hypervisor.

12. Guest
Interrupt
Handling
• Hypervisor
emulated
GICD
and
injects
interrupts
in
guest
using
GICH
• Hypervisor
programs
the
Link
registers
• Guest
access
the
virtual
CPU
I/
f
in
the
GIC
itself,
no
emulation
is
required
• There
are
2
methods
for
hypervisor
to
b
notified
that
guest
has
deactivated
the
interrupt
• Maintenance
Interrupt
• UnderFlow
Interrupt
GICD
Driver
Guest
Hypervisor
VCPU
I/F CPU
I/F Hyp
Trap
I/f Distributor
Level
Interrupt
UART SoC
GIC
GICD
Emulation
GICC
Driver

13. • ARM
Timer
Virtualization
provides
a
Physical
timer
register
and
a
Virtual
Timer
register.
Virtual
Timer
is
an
offset
from
physical
timer
• Hypervisor
Programs
the
CNTVOFF
register
before
guest
is
resumed.
• ARM
provides
separate
timer
interrupt
for
hypervisor
and
guest.
• Timer
interrupt
is
injected
in
same
way
as
other
interrupts

14. Managing
Virtual
Device
-­‐
Virtio
MMIO
• virtio
is
an
abstraction
layer
over
devices
in
a
paravirtualized
hypervisor.
• virtio
mmio
does
not
require
a
PCI
device

15. 15

16. Guest
Debug
Support
• Debugging
a
guest
can
be
done
on
hypervisor
console
• for
dumping
guest
state
on
console
• for
setting
software
breakpoints
• or
gdb
interface
can
be
added
in
hypervisor
to
debug
a
guest

17. VM
Management
Summary
• Context
Save
and
Restore
• Booting
and
Loading
• Scheduling
• VCPU
Management
• Remote
Guest
Image
loading

18. Remote
Device
Management
A typical use case of Virtual Machines on a device is an
enterprise Virtual Machine to provide a secure and
separate workspace environment for employees.
!
Basic management operations include creating/
deleting, resuming/suspending and powering on/off of
a Virtual Machine
18

19. Questions