This presentation has been moved to this address: https://www.slideshare.net/linaroorg/optimizing-the-design-and-implementation-of-kvmarm-sfo17403-81026985

1. connect.linaro.org
SFO17-403: Optimizing the Design and Implementation of KVM/ARM
Christoffer Dall

2. ENGINEERS AND DEVICES
WORKING TOGETHER
–Popek and Golberg
[Formal requirements for virtualizable third generation architectures ’74]
““Efficient, isolated duplicate
of the real machine””

3. ENGINEERS AND DEVICES
WORKING TOGETHER
Hardware
OS Kernel
App AppApp
Hardware
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
Native Virtual Machines
Virtualization

4. ENGINEERS AND DEVICES
WORKING TOGETHER
Hypervisor Design
Hardware
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
Type 1 (Standalone)

5. ENGINEERS AND DEVICES
WORKING TOGETHER
Hypervisor Design
Hardware
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
Type 1 (Standalone)
Hardware
OS Kernel
VM
Kernel
App App
VM
Kernel
App App
Type 2 (Hosted)
Hypervisor
App

6. ENGINEERS AND DEVICES
WORKING TOGETHER
Hypervisor Design
Hardware
Xen
Dom0
Linux
App App
DomU
Linux
App App
Hardware
Linux
VM
Linux
App App
VM
Linux
App App
KVM
App

7. ENGINEERS AND DEVICES
WORKING TOGETHER
ARM Virtualization Extensions
Kernel
UserEL0
EL1
HypervisorEL2

8. ENGINEERS AND DEVICES
WORKING TOGETHER
ARM VE and Hypervisors
Xen
Dom0
Linux
App App
DomU
Linux
App AppEL0
EL1
EL2
?

9. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM
Host
Linux
AppApp
VM
Kernel
AppApp
KVM
KVM lowvisor
EL0
EL1
EL2
1. Hypercall
2. Return3. Hypercall
4. Return
switch
state

10. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM
Host
Linux
AppApp
VM
Kernel
AppApp
KVM
EL0
EL1
EL2
1. Hypercall 2. Return

11. ENGINEERS AND DEVICES
WORKING TOGETHER
ARMv8.1 VHE
• Virtualization Host Extensions
• Supports running unmodified
OSes in EL2 without using EL1
Linux
EL0
EL1
EL2
AppApp

12. ENGINEERS AND DEVICES
WORKING TOGETHER
ARMv8.1 VHE in Details
1. HCR_EL2.E2H complete enables and disables VHE
2. Addition registers in EL2 to have same support as EL1
3. TGE bit traps EL0 to EL2 without disabling EL0 Stage 1 MMU
4. EL2 translation regime works like EL1
5. Redirects system register accesses from EL1 to EL2

13. ENGINEERS AND DEVICES
WORKING TOGETHER
VHE Register Redirection
TCR_EL1
mrs x0, TCR_EL1
HCR_EL2.E2H==0
TCR_EL2
HCR_EL2.E2H==1

14. ENGINEERS AND DEVICES
WORKING TOGETHER
VHE Register Redirection
TCR_EL1mrs x0, TCR_EL12

15. ENGINEERS AND DEVICES
WORKING TOGETHER
More VHE Register Redirection
• Some registers change bit position and content
• Example: CNTHTCL_EL2 changes layout to match CNTKCTL_EL1 with extra
bits

16. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM with VHE
HypervisorLinux
EL2
EL1
KVM
Lowvisor
Trap
Run VM

17. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM with VHE
HypervisorLinux
EL2
KVM
Lowvisor
Function
Call
Run VM

18. ENGINEERS AND DEVICES
WORKING TOGETHER
Experimental Setup
• AMD Seattle B0
• 64-bit ARMv8-A
• 2.0 GHz AMD A1100 CPU
• 8-way SMP
• 16 GB RAM
• 10 GB Ethernet (passthrough)
*Measurements obtained using Linux in EL2. See BKK16 talk.

19. ENGINEERS AND DEVICES
WORKING TOGETHER
VHE Performance at First Glance
CPU Clock Cycles non-VHE VHE*
Hypercall 3.181 3.045
*Measurements obtained using Linux in EL2. See BKK16 talk.

20. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM Optimization #1
VM
Kernel
AppAppEL0
EL1
EL2
Host
AppApp
Linux KVM
• Avoid saving/restoring
EL1 register state

21. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM Optimization #2
VM
Kernel
AppAppEL0
EL1
EL2
Host
AppApp
Linux KVM
• Leave virtualization
features enabled
• No traps from host EL0
when E2H and TGE are
both set
• EL2 not affected by
traps

22. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM Optimization #3
• Don’t context switch
the timer on every exit
from the VM
• Completely reworks the
timer code to take
interrupts and load/put
state as necessary
• 20 patches on list

23. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM Optimization #4
• Defer as much work as possible to vcpu_load and vcpu_put
• Called when entering/exiting run-loop
• Called when preempted/scheduled

24. ENGINEERS AND DEVICES
WORKING TOGETHER
KVM/ARM Optimization #5
• Rewrite the world
switch code
• Avoids many
conditionals
• Adds has_vhe() calls in
pre_run_check and
post_run_check.
kvm_arch_vcpu_ioctl_run
{
...
while (1) {
pre_run_checks();
if (has_vhe() /* static key */
ret = kvm_vcpu_vhe_run(vcpu);
else
ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
post_run_checks();
}
...
}

25. ENGINEERS AND DEVICES
WORKING TOGETHER
Microbenchmark Results
CPU Clock Cycles non-VHE VHE OPT * x86
Hypercall 3,181 752 1.437
I/O Kernel 3,992 1.604 2.565
I/O User 6,665 7.630 6.732
Virtual IPI 14,155 2.526 3.102
*Measurements obtained using Linux in EL2. See BKK16 talk.

26. Application Workloads
Application Description
Kernbench Kernel compile
Hackbench Scheduler stress
Netperf Network performance
Apache Web server stress
Memcached Key-Value store

27. ENGINEERS AND DEVICES
WORKING TOGETHER
Application Workloads
0.00
0.50
1.00
1.50
2.00
Kernbench
Hackbench
TCP_STREAM
TCP_MAERTS
TCP_RR
Apache
Memcached
non-VHE VHE OPT*
*Measurements obtained using Linux in EL2. See BKK16 talk.
Normalized overhead
(lower is better)

28. ENGINEERS AND DEVICES
WORKING TOGETHER
Conclusions
• Optimize and redesign KVM/ARM for VHE
• Reduce hypercall overhead by more than 75% (better than x86!)
• Network benchmark overhead reduced by 50%
• Memcached overhead reduced by more than 80%

29. ENGINEERS AND DEVICES
WORKING TOGETHER
Upstream Status
• Timer patches on list
• Optimization patches in final stages
• Priority to upstream soon, hopefully ready for v4.16