RISC-V Summit 2020 presentation

1. Static Partitioning
Virtualization on RISC-V
José Martins and Sandro Pinto
RISC-V Summit @ Virtual
December 8th, 2020

2. Agenda
Introduction
Motivation and Background
01
RISC-V Virtualization
RISC-V virtualization in a nutshell
02
RISC-V H-Extension
Implementation & enhancements
03
Evaluation
Tests, experiments, and results
04
Conclusion
Final thoughts and Takeaways
05

3. Introduction
Motivation and Background

4. Mixed-Criticality
4
ESRGv3 RISC-V Summit 20
• High Complexity
• Multiple Subsystems
• Heterogeneous Software Stacks (RTOS, GPOS)
• Different Criticality Levels
• Size, Weight, Power and Cost (SWaP-C)
Automotive
Industrial
Automation

5. Embedded Virtualization
5
ESRGv3
ESRGv3 RISC-V Summit 20
Consolidation
Size
Weight
Power
Cost
Low Engineering Cost
Full-Virtualization allows
direct porting of guest OSs
Fault Containment
Sandboxed Enviroments
Performance
Low Virtualization Overhead
Close to Native Performance
Security
Small TCB
Side-channel
TEE support
Open Design
Real-time
Low latencies
Determinism/Predictability
Freedom-from-interference

6. Traditional Hypervisors
6
ESRGv3
▪ Not designed for embedded/MCS:
▪ Although retrofitted with success
▪ High-overhead IO:
▪ Emulated
▪ Para-virtualization/Backend-drivers
▪ Large Code Base :
▪ Hosted hypervisor
▪ Privileged VMs run large monolithic OSs
(typically Linux) part of TCB.
ESRGv3 RISC-V Summit 20

7. Static Partitioning Virtualization
7
ESRGv3
Jailhouse
 Static Partitioning:
 Thin configuration/partition layer
 1:1 vCPU-to-pCPU mapping
 Static memory assignment
 Device Pass-through
 Hardware interrupts
 Jailhouse:
 Needs “root cell” to boot and manage VMs
 Large boot time
 Xen Dom0-less*:
 DomUs may boot w/o Dom0
 Direct Device assignment
Dom0-less
ESRGv3 RISC-V Summit 20
* https://www.youtube.com/watch?v=OrtV6gyHW74

8. Bao Overview
8
ESRGv3
01
02 03
 Type-1 / Bare-metal
 Static Partitioning:
 1:1 vCPU-to-pCPU mapping
 Static memory assignment
 Device Pass-through
 Hardware-assisted:
 2nd-stage translation
 Interrupt virtualization support
 IOMMU
 Dependencies:
 No external libraries / privileged VMs
 Small TCB (~7K SLoC)
 Real-time & Security:
 Predictability / Freedom-from-interference
 Side-channels / TEE support
ESRGv3 RISC-V Summit 20
José Martins, Adriano Tavares, Marco Solieri, Marko Bertogna, and Sandro Pinto. "Bao: A Lightweight
Static Partitioning Hypervisor for Modern Multi-Core Embedded Systems". In NG-RES 2020
https://drops.dagstuhl.de/opus/volltexte/2020/11779/

9. Bao Hypervisor Support
9
ESRGv3
01
03
 Architectures:
 Armv8-A
 RISC-V (v0.6.1 Hyp Spec.)
 Platforms:
 Zynq US+ (ZCUx and Ultra96)
 HiSilicon96 (Hikey96)
 NXP i.MX8
 Nvidia Tegra TX2
 QEMU
 Rocket @ ZCU
 Firmware:
 Arm Trusted Firmware (PSCI) on Arm
 Supervisor Binary Interface (SBI) on RISC-V
 U-boot
 Guests:
 Bare-metal
 Linux / Android
 RTOSs (FreeRTOS, Erika)
ESRGv3 RISC-V Summit 20

10. RISC-V Virtualization
RISC-V virtualization in a nutshell

11. Hypervisor Extension
11
ESRGv3
 Designed for type-1 and type-2 hypervisors
 Additional execution modes
 HS-mode (hypervisor-extended supervisor)
 VS-mode & VU-mode
 Additional CSR registers:
 HS-mode CSRs for hypervisor capabilities (e.g., hstatus)
 HS-mode CSRs for accessing Guest/VM state (e.g., vsstatus)
 Two-Stage MMU:
 Stage 1: VS-mode page table (GVA -> GPA)
 Stage 2: HS-mode guest page table (GPA -> HPA)
ESRGv3 RISC-V Summit 20
https://github.com/riscv/riscv-isa-manual/releases/latest

12. Hypervisor Extension
12
ESRGv3
ESRGv3 RISC-V Summit 20
Hypervisor
Firmware (SBI)
Decreasing
Privilege level
M
Guest OS
Guest User Space Host User Space
Virtualised Environment Non-virtualised Environment
HS
VS
Adapted from: Alistair Francis (WD), “Developing the RISC-V Hypervisor Extensions in QEMU”, Embedded Linux Conference Europe, 2019
VU
M
HS
U

13. Spec Status
13
ESRGv3
ESRGv3 RISC-V Summit 20
 Currently version 0.6.1
 Feedback from open source projects
 Contributions from organizations and individuals
 H-Extension spec close to freeze state
 Hypervisor “group” (join RISC-V Hypervisor sync-up calls!)
 Function completeness (KVM & Xvisor)
 RTL implementations (we have contributed with one )
 KVM and Xvisor running on FPGA implementation (we have ran Xvisor on an FPGA )
 Open source projects support:
 QEMU (v0.6.1)
 KVM and Xvisor (v0.6.1)

14. RISC-V Hypervisors
14
ESRGv3
 KVM:
 Type-2 hypervisor, Linux-based architecture, open-source
 Enterprise virtualization setups (datacenters and private clouds)
 https://github.com/kvm-riscv/linux
 Xvisor:
 Type-1 hypervisor, monolithic architecture, open-source
 Embedded systems with soft real-time requirements
 https://github.com/avpatel/xvisor-next
 Bao:
 Type-1 hypervisor, static partitioning architecture, open-source
 Mixed-criticality systems with strong real-time and security requirements
 https://github.com/bao-project/bao-hypervisor
ESRGv3 RISC-V Summit 20

15. RISC-V H-Extension
Implementation & enhancements

16. Spec Checklist
16
ESRGv3
01
02 03
 H-Extension, Version 0.6.1
 RV64 and sv39 only
 Implemented:
 h- and vs- csrs and respective functionality
 m- registers extended accordingly
 hypervisor load/store instructions
 new virtual instruction exception
 hfence instructions (limited, flushes complete TLB)
 guest external interrupts (w/ PLIC virtualization extension)
 Not Implemented:
 ASID/VMID support
 Transformed Instruction or Pseudoinstruction for htinst/mtinst
ESRGv3 RISC-V Summit 20

17. H-Extension Implementation
17
ESRGv3
01
02 03
 Available at: https://github.com/josecm/rocket-chip/tree/hyp
 Synced with chipyard’s master branch commit of rocket-chip (@1872f5d, 6
months ago)
 ~1100 sLOC modifications to rocket-chip. Mainly on:
 CSR
 PTW
 TLB
ESRGv3 RISC-V Summit 20

18. H-Extension Implementation
18
ESRGv3
01
02 03
 Next Steps:
 Refactoring/Optimizations/Clean-up
 ASID/VMID support
 Increase hfence granularity
 Improve 2nd-stage translation data structures (e.g. dedicated guest physical address TLB)
 Software:
 Ad hoc testing
 Bao Hypervisor
 Xvisor
ESRGv3 RISC-V Summit 20

19. PLIC H-Extension
19
ESRGv3
01
02 03
 Main requirements:
 minimal design
 direct injection of physical interrupts for the active guest
 no traps on claim/complete
 mix of physical and virtual interrupts
 Extra GEILEN VS-contexts per physical hart
 Hypervisor must trap-and-emulate VS-context
priority and enable registers.
 Hypervisor gives guests direct access to VS-
context’s claim/complete registers.
 Virtual interrupts are injected by writing to virtual
interrupt injection registers (VIIR) which can be
grouped.
ESRGv3 RISC-V Summit 20
02
PLIC
RISC-V
Hart 0
RISC-V
Hart 1
RISC-V
Hart n
M-mode External Interrupt
S-mode External Interrupt
VS-mode External Interrupt 0
VS-mode GEILEN-1
* The PLIC itself might be deprecated as a new standard interrupt controller is
in the workings.
https://github.com/josecm/riscv-plic-spec/tree/virt

20. Cache and Bandwidth Partitioning
20
ESRGv3
 Software-based techniques:
 Increases TCB
 High Overheads
 Coarse-grained
 E.g.: page coloring, PMU event based throttling
 Cache Partitioning:
 Per bus-master way-locking
 Lightweight modification to eviction circuit
 Memory bandwidth throttling:
 Bandwidth regulation unit (BRU)
 Per partition memory bandwidth budgets
ESRGv3 RISC-V Summit 20
Adapted from: "BRU: Bandwidth Regulation Unit for Real-
Time Multicore Processors,". F. Farshchi, Q. Huang and H.
Yun. RTAS 2020

21. Evaluation
Tests, experiments, and results

22. Experimental Setup
22
ESRGv3
Hardware
▪ Hexa-core rocket
▪ 3.2GHz
▪ 16 KB iL1$ and dL1$
▪ 8-way, 512 KB unified L2$ w/
way-locking
▪ Plic Virt extensions
▪ BRU
System Configurations
▪ Bare-metal / no hypervisor (bare)
▪ Single guest (solo)
▪ Interference (interf)
▪ Partitioning (solo+part)
▪ Partitioning w/ Interference (interf+part)
Metrics
▪ Performance Overhead
▪ Interrupt Latency
RISC-V Summit 20
Software Stack
▪ OpenSBI
▪ Bao Hypervisor
▪ Linux OS (VM)
▪ Baremetal App (VMs)
Memory Resource
Partitioning
▪ 4 L2$ ways
▪ 1800 MiB/s

23. Performance Overhead
23
ESRGv3
ESRGv3 RISC-V Summit 20
▪ MiBench Benchmark
Suite
▪ Automotive subset
▪ Relative to bare
▪ Bare absolute values
▪ Higher is worse
▪ Negligible variance
Hosted execution brings non-negligible overheads (2 to 6%) . We believe this is mainly due to
high cost of tick timer handling and non-optimized two-stage translation
Memory subsystem interference has performance impacts from 5% to 70%
Cache and memory bandwidth partition mitigations significantly reduce interference (1 to
23%) but not completely

24. Interrupt Latency
24
ESRGv3
ESRGv3 RISC-V Summit 20
▪ Custom benchmark
▪ Auto-Reload timer at 100 Hz
▪ Absolute time in ns
▪ L1 Instruction cache
invalidated at each tick
Trap-and-emulate approach introduces an extra 650 ns (≈800%) in guest average interrupt latency
Interference increases latency but can be somewhat attenuated by memory resource
partitioning
Direct interrupt injection keeps native latency and is not very susceptible to the effects of interference

25. Conclusion
Final thoughts and Takeaways

26. We have presented the first public implementation
valuation of the hypervisor extensions in a RISC-V

27. 27
ESRGv3
 No "real" silicon with H-extension yet
 probably not for the foreseeable future
 Static Partitioning “Virtualization”
 M-mode = hypervisor
 PMP configuration per hart
 PLIC emulation
 IOPMP
 Bao porting
 WiP version @ PolarFire Icicle (Renode)
 Virtualization for RISC-V “as of today”
 Extended platform support
ESRGv3 RISC-V Summit 20
But… do we
really need it ?!

28. TAKEAWAYS
28
• Current state of the RISC-V
hypervisor extension
specification
• First implementation of the
hypervisor extensions in a
RISC-V core
• Static partitioning
virtualization and what
hardware supports it needs
The spec is close to freeze but we
encourage the community to
contribute
1
There is a need for additional
implementations in other RISC-V
cores
2
There is a need to define standard
extensions that specify memory
resource partition interfaces and
IOMMUs
3
ESRGv3 RISC-V Summit

29. THANK YOU!
jose.martins@dei.uminho.pt
LinkedIn - https://www.linkedin.com/in/josecmar/
Twitter - https://twitter.com/josecarmartins
Github - https://github.com/josecm
sandro.pinto@dei.uminho.pt
LinkedIn - https://www.linkedin.com/in/sandro2pinto
Twitter - https://twitter.com/sandro2pinto
Github - https://github.com/sandro2pinto/

30. Q&A
https://github.com/bao-project/bao-hypervisor
https://github.com/josecm/rocket-chip/tree/hyp