Abstract The NetBSD rump kernel has been developed for some years now, allowing NetBSD kernel drivers to be used unmodified in many environments, for example as userspace code. However it is only since last year that it has become possible to easily run unmodified applications on the rump kernel, initially with the rump kernel on Xen port, and then with the rumprun tools to run them in userspace on Linux, FreeBSD and NetBSD. This talk will look at how this is achieved, and look at use cases, including kernel driver development, and lightweight process virtualization. Speaker bio Justin Cormack has been a Unix user, developer and sysadmin since the early 1990s. He is based in London and works on open source cloud applications, Lua, and the NetBSD rump kernel project. He has been a NetBSD developer since early 2014.

1. Running Applications on the
NetBSD Rump Kernel
Justin Cormack @justincormack

2. Rump kernels

3. Slides at http://eurobsdcon.myriabit.eu/
3

4. What is a rump kernel?
It is the (NetBSD) kernel without support for
• executing binaries
• scheduling threads
• managing hardware privilege levels
• most memory management eg virtual memory
4

5. Just drivers
• Drivers are the bit of the kernel that turns raw hardware into nice
abstractions like files and sockets
• Get something else (host) to do the other parts (host environment)
• Or do without it, just compile in single purpose application.
5

6. What use is that?
• You get the missing capabilities from a hypercall layer.
• Like a very simple virtual machine
• man 3 rumpuser
• Provides memory allocation, threads, mutexes, console output, random
numbers, clock
• This can run in userspace for example, but can be implemented in any
environment fairy easily
6

7. What use is that?
• Original use cases around testing drivers.
• You can test a kernel driver without booting the OS.
• You can test much of the OS without booting into it.
• Debugging is easy as it is just a userpsace program
• Also for running kernel drivers in userspace eg for file systems
7

8. How was it used?
• Typically either written for rump like the NetBSD tests
rump_init(); rump_sys_mkdir(path, 0777);
• or using LD_PRELOAD to change some calls using rumphijack
library
• The interface was essentially the syscall interface (minus parts not
supported by rump)
8

9. New developments

10. Xen port
• In August last year we got a Xen port of the rump kernel working. In
addition to porting the hypercall layer, we also got NetBSD libc to
compile against the rump kernel.
• We replaced the syscalls in libc with calls to the rump kernel syscall
implementations.
• This enabled running real applications, in particular we had LuaJIT
running, and a simple web server.
• Not just syscalls, more of a POSIX API.
10

11. Increased portability
The POSIX implementation of the rump kernel now runs on
• NetBSD
• FreeBSD, OpenBSD, DragonflyBSD
• Solaris
• Linux, including Android
• on architectures including arm, x86, amd64, powerpc, sparc64 and mips
11

12. Increased testing
There is continuous integration using TravisCI and buildbot
• Tests all commits to the buildrump.sh script on multiple operating
systems and architectures
• Tests NetBSD -current hourly on multiple operating systems and
architectures
• Good test of NetBSD portability issues
• Uses ljsyscall to test functionality - good syscall coverage, fast.
12

13. Continuous integration
13

14. Running
applications

15. rumprun-posix
• Late last year I started work on rumprun, which allowed running
unmodified NetBSD binaries with the rump kernel.
• The main issue with this is that you have two libcs involved, so there are
symbol conflicts. However with judicious symbol renaming it turned out
to be possible to fix this.
• In particular you can compile userspace binaries for the core NetBSD
commands like ifconfig, etc.
• Can test userspace without booting into OS
15

16. How does it work?
• Compile NetBSD object code and libraries on host machine
• Link code and libraries to new object with -nostdinc
• Rename read symbols to rump___sysimpl_read ...
• Fix up main and other things which will be called from a stub
• Localise symbols to avoid conflicts
• Link with host libc and rump libraries
• See script for more details, it is messy...
16

17. # ./rumpdyn/bin/rump_allserver unix://sock
# export RUMP_SERVER=unix://sock
# ./bin/ifconfig
lo0: flags=8049 mtu 33648
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
# ./bin/ping -o 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 64 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=0.000000 ms
----127.0.0.1 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.000000/0.000000/0.000000/0.000000 ms
17

18. Supported commands
arp cp dumpfs fsck_msdos ls mount_ext2fs ndp pax
reboot sysctl wpa_supplicant cat dd ed halt mkdir
mount_ffs newfs pcictl rm umount cgdconfig df fsck
ifconfig mknod mount_msdos newfs_ext2fs ping rmdir
vnconfig chmod disklabel fsck_ext2fs ktrace modstat
mount_tmpfs newfs_msdos ping6 rndctl wlanctl chown
dump fsck_ffs ln mount mv npfctl raidctl route
wpa_passphrase
18

19. What runs?
• Almost anything in the NetBSD core can probably be compiled
• There is now a cross compiler for rumprun, that should work with most
code
• There is experimental pthreads support, not yet well tested
• Programs that fork or use other things not supported by rump may be a
problem
• Some of the missing rump features are emulated or partially emulated,
eg anon mmap
19

20. rumprun - TODOs
• Clean up and upstream libc build modifications
• Fix other architectures, currently only working on x86, amd64
• Build more of userspace tools and libraries
• Use for NetBSD tests - ability to run tests on a kernel without booting it
very useful when developing.
• Improve build process
• Continuous integration and testing on NetBSD current.
20

21. More new
developments

22. rump on green threads
• A few months back I added a "green threads" userspace implementation
to the userspace rump implementation.
• The default implementation uses pthreads.
• Uniprocessor, single thread
• Useful for embedded implementations
• Also useful if you want to make a deterministic implementation
22

23. PCI support
• Some PCI support has been added
• Linux userspace support.
• Linux has uio and vfio frameworks for userspace pci drivers.
• uio supported now, only works for some cards
• vfio may be later; requires an iommu.
• Can be used for developing NetBSD drivers even so - used for wireless
development
• Really want to add BSD support
23

24. rump on bare metal
• New project to run a rump kernel on (currently x86) bare metal
• https://github.com/rumpkernel/rumpuser-baremetal
• Includes PCI support, with virtio drivers for KVM, bhyve
• Only a few hundred lines of code to deal with interrupts etc, written in a
week
• Planning ARM port to mbed
24

25. rump on microkernels
• Genode, a microkernel OS framework, has started using the rump kernel
to support NetBSD file system drivers on microkernels.
• Less than 3,000 lines of (untrusted) glue code.
• See genode.org for more
• Minix3 is another potential user, it already uses a lot of NetBSD code.
25

26. Logo
26

27. Architecture

28. Four different environments
• hosted, e.g. userspace
• paravirtualized, e.g. Xen
• "bare metal", e.g. hardware or hypervisor with virtio
• microkernels (as servers)
28

29. Architecture
29

30. Documentation
• Documentation is much improved.
• All at http://wiki.rumpkernel.org/
• Best places to start are Getting Started and Kernel development
tutorials.
• A longer introduction is the rump kernel book
30

31. Use cases
1. Driver development
2. Tests
3. Drivers for other environments
4. Applications with userspace drivers, eg networking
5. Running code securely eg file system code
6. Very lightweight "containers" with their own OS library
7. ...
31

32. What needs doing?
• Upstream and improve rumprun
• Improve documentation
• Portability: native Windows, OSX
• Userspace IP stacks: need good performance on 10Gb
• dogfooding
• ...
32

33. Get involved
• http://rumpkernel.org/
• Freenode #rumpkernel
• Mailing list rumpkernel-users
• twitter @rumpkernel
• 25 November operatingsystems.io conference in London
• 26 November hackday in London
• Fosdem 2015
33