Ruslan Bukin has contributed ports of FreeBSD to the RISC-V architecture as well as various device drivers. He began work in June 2015, initially porting the kernel, then userland. Key challenges included implementing the page table with RISC-V's single base register and handling the thread pointer register. His work resulted in a fully functional FreeBSD/RISC-V port within 6 months. He continues working on improvements and additional hardware support.

1. Ruslan Bukin: FreeBSD/RISC-Vand Device Drivers
Personal Background
● FreeBSD arch ports (ARM, MIPS, RISC-V, x86)
● Drivers (Sound, DMA, Ethernet, MMC, …)
● DTrace support (ARMv8, RISC-V)
● HWPMC
Overview
● History
● porting process
● challenges
● status updates
● drivers support
Timeline
● start: jun 15
● printf: aug 15
● kernel works: oct 15
● userland works: dec 15
● Commit to FreeBSD svn: jan 16
● SMP: feb 16
● KDB: mar 16
● :Dtrace: apr 16
● Modules: may 16
● ISA v1.9: Jun 16
● FPU: nov 16
● ISA v1.10:
RISC-V privilege levels
● M-mode : bootloader (BBL)
● S-mode: FreeBSD kernel
● U-mode: /bin/init
● supported combination of modes
○ M
○ M, U embedded with protection
○ M, U, S
BBL
● firmware/bootloader
● M-mode
● console, timers, IPI
● All traps switch mode to M
○ can delegate traps to S-mode using mtdeleg instruction
Start of Work
● toolchain
○ GCC
○ LLVM (no support at the time)
Template
● cp -R sys/arm64 sys/riscv
● do {
● vi sys/riscv/…
● make TARGET_ARCH=riscv64
● } while ($?)
Prepare Hardware
● Emulators

2. ○ spike
○ QEMU
● FPGA
● root disk
● emulators speed up booting and can inspect regs, single step etc
Porting: early asm code
● _start in locore.S
● put machine in known state
● initial page tables
● enable MMU and branch to virtual addressing
● setup stack pointer
● jump to c-code
Porting: kernel (1)
● console
● atomics: atomic_add(), atomic_readandclear(), …
○ need to test them well, debugging later will be painful
● Initialize physmap table
○ can hard code at first
● PMAP implementation
PMAP
● 40 functions to implement
○ pmap_enter()
○ extract
○ remove
○ invalidate
○ protect
○ activate
○ unwire
○ …
● most difficult part in the kernel
Porting kernel (2)
● Exceptions
● cpu_switch(): only changing kernel context
● fork_trampoline(): newly created user thread, kernel -> user mode change
● Kernel VA <-> user VA copy functions
○ copyin
○ copyout
○ ...
Porting kernel (3)
● Timer driver
● Interrupt controller driver
● Disk driver
● Done
Challenges #1: page table base
RISC-V only has 1 page table base register, AArch64 has 2
kernel and user space can’t each have its down
● kernel page directory
● user page directory
● Solution: merge kernel user into one user page table directory
○ takes CPU time
○ RISC-V community rejected 2 base reg proposal
Challenge #2: single Thread Pointer register
● PCPU pointer
● solution: store GP (global pointer) to supervisor stack

3. ● Reload GP on return from user thread
Porting: userspace (1)
● jemalloc
● csu: crt1.S, crtN.S, crti.S
● libc
○ syscalls
○ setjmp, longjmp, _set_tp
● msun (libm)
Porting: userspace(2)
● Compile world
● statically linked /bin/sh
● rtld
● signals (required by e.g. csh job control)
Porting: finale
● challenging parts
● pmap in kernel
● rtld in userspace
● 6 months from scratch
● 25k line diff (200 new files)
● (thanks lots of people)
Status Update
● v1.10 of Privileged spec
● HiFive1 board release
● FreeBSD board is planned on Q1 2018
● GCC 7 target upstream
● NVIDIA wants to ship RISC-V uC
● Next workshop in San Jose
SiFive HiFive1
● TSMC 180nm
● 320+ MHZ
● 16k SRAM
VC707
● SiFive Xilinx prototype board
Privileged Arch: v1.10 changes
● Not compatible on S-mode with v1.9
○ next version “should” be compatible with v1.10 on S-mode
● Changes
○ built-in macros and compiler arguments changed
○ SBI interface, VM changes, BBL changes
○ Physical Memory Protection (PMP) Unit introduced
○ Support for FDT
Compiler Arg changes
● -mno-float, -msoft-float removed
● -march=rvimafdc -mabi=lp64
● a:; atomic
● …
Compiler Built-in defines
● __riscv ...
SBI interface changed
● old way: pre-defined function address in physical memory
● new way: ecall to upper (machine) privilege level
● required for
○ timer
○ console

4. ○ ipi
○ shutdown
○ Hart ID
PMP unit
● R/W/Xon 4-byte granularity, 16 regions
● composable with MMU
● can be locked in M-mode
● when enabled, modes below M hae no memory permissions
Virtual Memory changes
● VM turn on/off using satp register
● supervisor can’t access user pages by default
○ syscalls: set SUM bit in sstatus reg
● R, W, X controlled separately
○ X-only pages
○ W-only pages
“C” - Compressed Extension
● saves space
● compresses some instructions to 2 bytes (like Thumb)
● comparable to microMIPS
○ space savings: 16%(RISC-V 64-bit kernel) vs 20% (MIPS 32-bit kernel)
FDT Support
● GENERIC kernel
● timer/console moved from OFW bus to nexus bus
● #./spike -dump-dts
● #./spike -m2048 -p8 /path/to/bbl
Hardware support for v1.10 privilege spec
impl status
Spike full support
RocketChip unknown
lowRISC un
QEMU un
GEM5 un
real hw N/A
Future plans
● ASIC chips and boards
● LLVM
● ports and pkgs
○ QEMU user (syscall emulation) mode
● bhyve support?
Device driver
● How to attach
● Resources
○ how to access
○ where to request
Driver Frameworks
● kld
● newbus(9)
● rman(9)
● cdevsw
● ofw and FDT
● bus_space
● bus_dma
● sysctl
● SYSINIT

5. SYSINIT
test_handler(mod, what, arg) {
switch (what) {
case MOD_UNLOAD:
case MOD_UNLOAD:
}
}
DECLARE_MODULE(test, SI_SUB_VM,...)
newbus(9)
● rooted tree
● device is a bus if it has children
● (root -> nexus bus -> ofwbus -> simplebus -> spi, uart0
● -> RISC-V console)
newbus methods
● methods: probe, attach
● DRIVER_MODULE(test, simplebus, methods)
Resources
● Device tree sources (DTS)
● bus_alloc_resources
● bus_setup_intr(dev, intr_func, …)
cdevsw
● char device
● open, read, ioctl, mmap, poll
● fd = open(“/dev/test”, …)
Frameworks / subsystems
● SPI, I2C, xdma, ifnet, callout, pci, sound(9), wifi, vt(4), taskqueue(9)
Self Made FreeBSD/MIPS board
● (custom designed board)
● 27 capacitors
● 1 resistor
● 1 pin header
● 1 xtal
● 1 MIPS CPU (from PIC)
● 2 layer PCB
● 2MB flash (1.7 MB used for FreeBSD)
Q&A
● How do you test your atomics implementations? A: didn’t test. lost time
● Schematics for MIPS board? A: designed using Windows tools. Only took two hours
● Any driver challenges? A: lots, spent 6 months, worked hard.
● Did single page table cause rework of core kernel code? A: only machine dependent parts
○ PTE has bits for supervisor and userspace access
● Challenge of rtld? A: I don’t remember. I remember it was difficult.