1. Simon Kuenzer <simon.kuenzer@neclab.eu>
Senior Researcher, NEC Laboratories Europe GmbH
Xen Summit 2019, Chicago
Weather Report
This work has received funding from the European Union’s Horizon 2020 research and
innovation program under grant agreements no. 675806 (“5G CITY”) and no. 825377
(“UNICORE”). This work reflects only the author’s views and the European
Commission is not responsible for any use that may be made of the information it
contains.

3. An SDK for highly specialized Unikernels

4. 4 © NEC Corporation 2019
Unikernels as VMs
Unikernels
Kernel
Hardware
Kernel
App B
Libs B
Traditional VMs
Hypervisor
App A
Libs A
Hardware
Hypervisor

5. 5 © NEC Corporation 2019
Unikernels as VMs
App A
Libs A
App B
Libs B
Unikernels
Kernel
Hardware
Kernel
App B
Libs B
Traditional VMs
Hypervisor
App A
Libs A
Hardware
Hypervisor

6. 6 © NEC Corporation 2019
Unikernels as VMs
▌Unikernels are purpose-built
 Thin kernel layer, only what application needs
 Single monolithic binary that contains OS and application
App A
Libs A
App B
Libs B
Unikernels
Kernel
Hardware
Kernel
App B
Libs B
Traditional VMs
Hypervisor
App A
Libs A
Hardware
Hypervisor

7. 7 © NEC Corporation 2019
Unikernels as VMs
▌Unikernels are purpose-built
 Thin kernel layer, only what application needs
 Single monolithic binary that contains OS and application
▌No isolation within Unikernel, done with hypervisor
 One application  Flat and single address space
App A
Libs A
App B
Libs B
Unikernels
Kernel
Hardware
Kernel
App B
Libs B
Traditional VMs
Hypervisor
App A
Libs A
Hardware
Hypervisor

8. 8 © NEC Corporation 2019
Unikernels as VMs
▌Unikernels are purpose-built
 Thin kernel layer, only what application needs
 Single monolithic binary that contains OS and application
▌No isolation within Unikernel, done with hypervisor
 One application  Flat and single address space
▌Further advantages from specialization
 Performance and efficiency; reduced attack vector; small memory footprint
App A
Libs A
App B
Libs B
Unikernels
Kernel
Hardware
Kernel
App B
Libs B
Traditional VMs
Hypervisor
App A
Libs A
Hardware
Hypervisor

9. 9 © NEC Corporation 2019
Unikernel Framework
Motivation
▌Support wide range of use cases
▌Simplify building and optimizing
▌Simplify porting of existing applications
▌Common and shared code base for Unikernel creators
▌Support different hypervisors and CPU architectures

10. 10 © NEC Corporation 2019
Unikernel Framework
Motivation
▌Support wide range of use cases
▌Simplify building and optimizing
▌Simplify porting of existing applications
▌Common and shared code base for Unikernel creators
▌Support different hypervisors and CPU architectures
▌Concept: “Everything is a library”
Decomposed OS functionality
▌Two components:
1. Library Pool
2. Build Tool

11. 11 © NEC Corporation 2019
architecture
libs
main
libs
platform
libs
libarm32arch.olibx86_64arch.o
libxenplat.o
drivers
libconsole.o
libixgbe.o
libvirtio.o
network stack
liblwip.o
libtcpip.o
libhttp.o
memory allocators
libbuddy.o
libheap.o
libmempool.o
filesystems
libvfs.o
libfat.o
libext3.o
runtimes
libocaml.o
libpython.o
liberlang.o
schedulers
libcoop.o
libpreempt.o
librt.o
and more…
standard libs
musl.o
libnewlibc.o
libopenssl.o
1) Library Pool
libkvmplat.o liblinuxuplat.o
libmipsarch.o
libxenplat.o

12. 12 © NEC Corporation 2019
architecture
libs
main
libs
platform
libs
libarm32arch.olibx86_64arch.o
libxenplat.o
drivers
libconsole.o
libixgbe.o
libvirtio.o
network stack
liblwip.o
libtcpip.o
libhttp.o
memory allocators
libbuddy.o
libheap.o
libmempool.o
filesystems
libvfs.o
libfat.o
libext3.o
runtimes
libocaml.o
libpython.o
liberlang.o
schedulers
libcoop.o
libpreempt.o
librt.o
MyApplication
and more…
standard libs
musl.o
libnewlibc.o
libopenssl.o
1) Library Pool
libkvmplat.o liblinuxuplat.o
libmipsarch.o
libxenplat.o
Select/create
application
1 MyLib_A

13. 13 © NEC Corporation 2019
architecture
libs
main
libs
platform
libs
libarm32arch.olibx86_64arch.o
libxenplat.o
drivers
libconsole.o
libixgbe.o
libvirtio.o
network stack
liblwip.o
libtcpip.o
libhttp.o
memory allocators
libbuddy.o
libheap.o
libmempool.o
filesystems
libvfs.o
libfat.o
libext3.o
runtimes
libocaml.o
libpython.o
liberlang.o
schedulers
libcoop.o
libpreempt.o
librt.o
MyApplication
and more…
standard libs
musl.o
libnewlibc.o
libopenssl.o
1) Library Pool
libkvmplat.o liblinuxuplat.o
libmipsarch.o
libxenplat.o
Select/create
application
1
Selectand
configurelibraries
2
MyLib_A

14. 14 © NEC Corporation 2019
Unikernels
architecture
libs
main
libs
platform
libs
libarm32arch.olibx86_64arch.o
libxenplat.o
drivers
libconsole.o
libixgbe.o
libvirtio.o
network stack
liblwip.o
libtcpip.o
libhttp.o
memory allocators
libbuddy.o
libheap.o
libmempool.o
filesystems
libvfs.o
libfat.o
libext3.o
runtimes
libocaml.o
libpython.o
liberlang.o
schedulers
libcoop.o
libpreempt.o
librt.o
MyApplication
and more…
standard libs
musl.o
libnewlibc.o
libopenssl.o
1) Library Pool
libkvmplat.o liblinuxuplat.o
libmipsarch.o
libxenplat.o
Select/create
application
1
Selectand
configurelibraries
2
Build3Run4
myapp_xen_x86-64 myapp_kvm_x86-64 myapp_bare_arm64 etc.
MyLib_A

15. 15 © NEC Corporation 2019
2) Build Tool

16. 16 © NEC Corporation 2019
2) Build Tool
▌KConfig based and Makefile “Magic”

17. 17 © NEC Corporation 2019
2) Build Tool
▌KConfig based and Makefile “Magic”
▌Type “make menuconfig”
Choose options in the menu that you want for your application
Choose your target platform(s), e.g., Xen, KVM, bare-metal, Linux

18. 18 © NEC Corporation 2019
2) Build Tool
▌KConfig based and Makefile “Magic”
▌Type “make menuconfig”
Choose options in the menu that you want for your application
Choose your target platform(s), e.g., Xen, KVM, bare-metal, Linux
▌Save your config and type “make”

19. Community Status and Achievements

20. 20 © NEC Corporation 2019
Timeline
▌Early 2017: NEC-Internal project launch; 0.1
Build system
Initial port from Mini-OS and Solo5/KVM

21. 21 © NEC Corporation 2019
Timeline
▌Early 2017: NEC-Internal project launch; 0.1
Build system
Initial port from Mini-OS and Solo5/KVM
▌Dec/2017: Public Launch; RELEASE-0.2 Titan
As Xen Incubator project
Arm32 Xen, x86 Xen, x86 KVM, x86 Linux
Binary buddy allocator (heap)
Cooperative scheduling

22. 22 © NEC Corporation 2019
Timeline
▌Early 2017: NEC-Internal project launch; 0.1
Build system
Initial port from Mini-OS and Solo5/KVM
▌Dec/2017: Public Launch; RELEASE-0.2 Titan
As Xen Incubator project
Arm32 Xen, x86 Xen, x86 KVM, x86 Linux
Binary buddy allocator (heap)
Cooperative scheduling
▌Feb/2019: RELEASE-0.3 Iapetus
Arm64 support for KVM
Networking (uknetdev, lwip, virtio-net)
Initial VFS with in-RAM filesystem
newlib

23. 23 © NEC Corporation 2019
Timeline
▌Early 2017: NEC-Internal project launch; 0.1
Build system
Initial port from Mini-OS and Solo5/KVM
▌Dec/2017: Public Launch; RELEASE-0.2 Titan
As Xen Incubator project
Arm32 Xen, x86 Xen, x86 KVM, x86 Linux
Binary buddy allocator (heap)
Cooperative scheduling
▌Feb/2019: RELEASE-0.3 Iapetus
Arm64 support for KVM
Networking (uknetdev, lwip, virtio-net)
Initial VFS with in-RAM filesystem
newlib
▌Upcoming release: RELEASE-0.4 Rhea
Language support: C++, Python, etc.
Block & console devices
9pfs filesystem (Xen, KVM)
Libraries: intel-intrinsics, libunwind, libuuid, pthread-embedded,
compiler-rt, eigen, fp16, fxdiv, pthreadpool

24. 24 © NEC Corporation 2019
Changed lines of code by affiliation
▌3 contributors
▌Project Launch
86020
14903
27720
11736
32149
86020
31369
60401
2017 2018 2019
NEC Politehnica University of Bucharest Arm Other
▌~15 contributors
▌*Many patches
still under review
▌~20 contributors
▌Start of student
projects

25. Used Tools

26. 26 © NEC Corporation 2019
Documentation
▌http://unikraft.org
▌Generated by Sphinx
▌Part of base repository: /doc/

27. 27 © NEC Corporation 2019
Patch Reviews
▌Patchwork: http://patchwork.unikraft.org
▌Coordination of reviewers
▌Review status and reviewer/maintainer in charge visible

28. 28 © NEC Corporation 2019
Automated Testing (coming soon)
▌Today: Two-branch model: master & staging
Staging: Latest accepted patches
Master: Manual follow-up: Releases and tested code

29. 29 © NEC Corporation 2019
Automated Testing (coming soon)
▌Today: Two-branch model: master & staging
Staging: Latest accepted patches
Master: Manual follow-up: Releases and tested code
▌Jenkins for automate testing
Common combinations of configurations and libraries
Execution test on common hypervisors and architectures
(Xen, KVM, Linux, x86, Arm)

30. 30 © NEC Corporation 2019
Automated Testing (coming soon)
▌Today: Two-branch model: master & staging
Staging: Latest accepted patches
Master: Manual follow-up: Releases and tested code
▌Jenkins for automate testing
Common combinations of configurations and libraries
Execution test on common hypervisors and architectures
(Xen, KVM, Linux, x86, Arm)
▌Goal
Integrate test results into review processes

31. Ongoing and upcoming Projects

32. 32 © NEC Corporation 2019
Binary Compatible Unikernels
▌Even with complete library pool,
manual porting is non-trivial
Existing build system need to be ported or
instrumented (e.g., cross-compilation)
Pre-compiled binaries cannot be executed
(e.g., proprietary executables)

33. 33 © NEC Corporation 2019
Binary Compatible Unikernels
▌Even with complete library pool,
manual porting is non-trivial
Existing build system need to be ported or
instrumented (e.g., cross-compilation)
Pre-compiled binaries cannot be executed
(e.g., proprietary executables)
▌ELF binary compatibility, Linux ABI
Same executable for Linux should run on
Unikraft without recompilation
ELF loader
System call emulation
[1] Pierre et. al, A Binary-Compatible Unikernel, VEE’19
[2] Kiviti et. al, OSv—Optimizing the Operating System for Virtual Machines, USENIX ATC ’14
Image: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format

34. 34 © NEC Corporation 2019
Language Runtimes

35. 35 © NEC Corporation 2019
Language Runtimes
▌Support applications written in higher-level
language as Unikernel
Primarily Serverless and IoT use cases

36. 36 © NEC Corporation 2019
Language Runtimes
▌Support applications written in higher-level
language as Unikernel
Primarily Serverless and IoT use cases
▌Upcoming:
C++
Rust
Go
Ruby
Javascript (v8)
Python

37. 37 © NEC Corporation 2019
Language Runtimes
▌Support applications written in higher-level
language as Unikernel
Primarily Serverless and IoT use cases
▌Upcoming:
C++
Rust
Go
Ruby
Javascript (v8)
Python
Unikernel Image sizes
(uncompressed, Xen)
Micropython 828 KB
Python 2 3,9 MB
Go runtime 552 KB

38. 38 © NEC Corporation 2019
NFV
▌Virtualized Network Functions
Package vNF directly as VM with Unikraft
Remove maintenance effort of hosting OS
Minimal OS overhead
Minimal OS noise
High networking performance & throughput

39. 39 © NEC Corporation 2019
NFV
▌Virtualized Network Functions
Package vNF directly as VM with Unikraft
Remove maintenance effort of hosting OS
Minimal OS overhead
Minimal OS noise
High networking performance & throughput
▌Click
Programmable vNF

40. 40 © NEC Corporation 2019
NFV
▌Virtualized Network Functions
Package vNF directly as VM with Unikraft
Remove maintenance effort of hosting OS
Minimal OS overhead
Minimal OS noise
High networking performance & throughput
▌Click
Programmable vNF
▌Intel DPDK
Dataplane development Kit
SDK for building high-performance VNFs
Directly build Unikernel instead of kernel-bypassing application

41. 41 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft

42. 42 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft
▌Webservers
Nginx, lighttpd

43. 43 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft
▌Webservers
Nginx, lighttpd
▌Databases
Memcached, Redis, sqlite

44. 44 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft
▌Webservers
Nginx, lighttpd
▌Databases
Memcached, Redis, sqlite
▌Machine Learning
Pytorch

45. 45 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft
▌Webservers
Nginx, lighttpd
▌Databases
Memcached, Redis, sqlite
▌Machine Learning
Pytorch
▌Goal: Only minimal effort to integrate to build system
Makefile.uk
Config.uk

46. 46 © NEC Corporation 2019
Application Porting
▌Initial set of ported application
Typical for cloud deployments
Learn about missing components in Unikraft
▌Webservers
Nginx, lighttpd
▌Databases
Memcached, Redis, sqlite
▌Machine Learning
Pytorch
▌Goal: Only minimal effort to integrate to build system
Makefile.uk
Config.uk
SQLite Unikernel
Image size (uncompressed, Xen) 844 KB

47. 47 © NEC Corporation 2019
Xenstore
Xen Dom0-Disaggregation
▌Disaggregate functionality from Dom0
Xen
Linux
Dom0
xl
toolstack

48. 48 © NEC Corporation 2019
Xenstore
Xen Dom0-Disaggregation
▌Disaggregate functionality from Dom0
XenStore stub domain (e.g., cxenstore)
Xen
Linux
Dom0
xl
toolstack

49. 49 © NEC Corporation 2019
Xenstore
Xen Dom0-Disaggregation
▌Disaggregate functionality from Dom0
XenStore stub domain (e.g., cxenstore)
Xen
Linux
Qemu
HVM DomUDom0
xl
toolstack

50. 50 © NEC Corporation 2019
Xenstore
Xen Dom0-Disaggregation
▌Disaggregate functionality from Dom0
XenStore stub domain (e.g., cxenstore)
QEMU stub domain
Xen
Linux
Qemu
HVM DomUDom0
xl
toolstack

51. 51 © NEC Corporation 2019
Hardening
▌Already small attack vector due to specialization

52. 52 © NEC Corporation 2019
Hardening
▌Already small attack vector due to specialization
▌Common attack prevention features need to be
implemented1, for instance:
ASLR (via boot loader or toolstack)
Stack canaries
Page protection bits
Heap integrity checks
[1] NCC Group, Assessing Unikernel Security,
https://www.nccgroup.trust/us/our-research/assessing-unikernel-security/

53. 53 © NEC Corporation 2019
Hardening
▌Already small attack vector due to specialization
▌Common attack prevention features need to be
implemented1, for instance:
ASLR (via boot loader or toolstack)
Stack canaries
Page protection bits
Heap integrity checks
▌Potential to enable enhanced preventions with little
performance costs due to direct access to privileged
functionality
e.g., malloc() maps pages, free() unmaps pages
[1] NCC Group, Assessing Unikernel Security,
https://www.nccgroup.trust/us/our-research/assessing-unikernel-security/

54. More talks at XenSummit 2019

55. 55 © NEC Corporation 2019
Talks

56. 56 © NEC Corporation 2019
Talks
▌Check-out related talks:
Wednesday, 11:55am
Secure Unikraft Applications with Solo5
Haibo Xu, Arm
Wednesday, 1:50pm
When Unikraft Meets Arm64
Jia He, Arm
Wednesday, 3:45pm
A Journey to Mirage OS as Xen PVH
Marek Marczykowski-Górecki, Invisible Things Lab

57. 57 © NEC Corporation 2019
Talks
▌Check-out related talks:
Wednesday, 11:55am
Secure Unikraft Applications with Solo5
Haibo Xu, Arm
Wednesday, 1:50pm
When Unikraft Meets Arm64
Jia He, Arm
Wednesday, 3:45pm
A Journey to Mirage OS as Xen PVH
Marek Marczykowski-Górecki, Invisible Things Lab
▌Design Sessions
Tuesday, 2:40pm
Building Stub Domains with Unikraft
Wednesday, 4:35pm
A Journey through Unikraft's Build System