An evaluation of microservices in Unikernels (presented at Microservices and Cloud Native Apps meetup in San Jose on 9/22/16).

1. Microservices in Unikernels
Rean Griffith, Madhuri Yechuri
1

2. Agenda
Introduction - bios
Unikernel Background
Developer/DevOps care about
Metric Set 1: Application lifecycle overhead
CIO cares about
Metric Set 2: Application datacenter footprint
2

3. What is a Unikernel?
A single purpose (virtual) appliance (Madhavapeddy et al.)
Specialized at compile-time into a standalone kernel
A single-process, single address-space runtime environment
No fork()
No shared memory
No IPC
Smaller attack surface (potentially) 3
fork() Shared
memory
Application
IPC
networking
sched Application
networking
thread
sched
services
vmm
vmm

4. Unikernel Background
4
Unmodified Legacy App
support
Multi-threaded App support
OSv Partial Yes (1: glibc subset, no
fork/exec)
Yes* (pthread subset)
Rumprun Yes* (no
fork/execve/sigaction/mmap)
Yes (pthread)
MirageOS No* (until non-OCAML language
bindings are available, no
fork/execve)
Green threads (event loop) only
IncludeOS No Green threads (event loop) only

5. Developer/DevOps care about
Enterprise Application Lifecycle management
Developer: Time to build app from source code, preferably unmodified
DevOps: Time to configure runtime parameters (ex: TCP port, log file
location)
DevOps: Time to deploy application
DevOps: Qualitative ease of managing+debugging long-running (weeks /
months) application 5

6. App Lifecycle Experiment Environment
Machine
CPU: Intel(R) Core(TM) i3 CPU M 380 @ 2.53GHz
Memory: 4GB RAM
OS: Ubuntu 16.04 LTS
Applications
Web tier: Nginx
Application tier: Tomcat
Deployment Options (local image)
6

7. Metric Set 1: Application Lifecycle
Convert Code to Image (Hours)
VM 8 (Nginx, Issues: 1 , 2, 3)
8 (Tomcat, Issues with Alpine glibc availability)
Container 0 (Nginx)
0 (Tomcat)
Unikernel 40 (Nginx, Issues: 1, 2)
4 (Tomcat)
7

8. Unikernel conversion - ukonvrt
Demo
8

9. Metric Set 1: Application Lifecycle
Convert Code to
Image (Hours)
Start Time
(Seconds)
VM 8 (Nginx)
8 (Tomcat)
66.557 (Nginx)
68.964 (Tomcat)
Container 0 (Nginx)
0 (Tomcat)
1.113 (Nginx)
4.1 (Tomcat)
Unikernel 40 (Nginx)
0 (Tomcat)
0.483 (Nginx)
10 (Tomcat)
9

10. Metric Set 1: Application Lifecycle
Convert
Code to
Image
(Hours)
Start Time
(Seconds)
Stop Time
(Seconds)
VM 8 (Nginx)
8 (Tomcat)
66.557 (Nginx)
68.964 (Tomcat)
7.478 (Nginx)
5.418 (Tomcat)
Container 0 (Nginx)
0 (Tomcat)
1.113 (Nginx)
4.1 (Tomcat)
0.685 (Nginx)
0.016 (Tomcat)
Unikernel 40 (Nginx)
4 (Tomcat)
0.483 (Nginx)
10 (Tomcat)
0.019 (Nginx)
0.006 (Tomcat)
10

11. Metric Set 1: Application Lifecycle
Code to
Image
(Hours)
Start Time
(Seconds)
Stop Time
(Seconds)
Debuggability
VM 8 (Nginx)
8 (Tomcat)
66.557 (Nginx)
68.964 (Tomcat)
7.478 (Nginx)
5.418 (Tomcat)
Container 0 (Nginx)
0 (Tomcat)
1.113 (Nginx)
4.1 (Tomcat)
0.685 (Nginx)
0.016 (Tomcat)
Unikernel 40 (Nginx)
4 (Tomcat)
0.483 (Nginx)
10 (Tomcat)
0.019 (Nginx)
0.006 (Tomcat)

12. CIO cares about
Consolidation of applications on finite hardware resources
Multi-tenant security isolation amongst applications on a compute node
Multi-tenant Resource Management
Manageability, Accounting, Auditability
Infrastructure Power consumption
12

13. Metric Set 2: Data center footprint
Image Size
(MB)
VM 143 (Nginx)
447 (Tomcat, Issue 1 - Alpine musl
vs glibc)
Container 182.8 (Nginx)
357.5 (Tomcat)
Unikernel 27.8 (Nginx)
106 (Tomcat)

14. Metric Set 2: Data center footprint
Image Size
(MB)
Runtime Memory Overhead
(MB)
VM 143 (Nginx)
447 (Tomcat)
619 (Nginx)
878 (Tomcat)
(/proc/{vboxpid}/status/{VmSize} - Configured)
Container 182.8 (Nginx)
357.5 (Tomcat)
274.4 (Nginx)
210.5 (Tomcat)
(containerd-shim /proc/{pid}/status/{VmSize})
Unikernel 7.8 (Nginx)
106 (Tomcat)
1222 (Nginx)
2056 (Tomcat)
(/proc/{qemupid}/status/{VmSize} - Configured)

15. Metric Set 2: Data center footprint
Image Size
(MB)
Runtime Memory
Overhead (MB)
Security (Tenant
Isolation)
VM 143 (Nginx)
447 (Tomcat)
619 (Nginx)
878 (Tomcat)
Strong
Container 182.8 (Nginx)
357.5 (Tomcat)
274.4 (Nginx)
210.5 (Tomcat)
Weak
Unikernel 7.8 (Nginx)
106 (Tomcat)
1222 (Nginx)
2056 (Tomcat)
Strong

16. Metric Set 2: Data center footprint
Image Size
(MB)
Runtime Memory
Overhead (MB)
Security (Tenant
Isolation)
Resource Knobs
VM 143 (Nginx)
447 (Tomcat)
619 (Nginx)
878 (Tomcat)
Strong Strong
(Reservation,
Limits)
Container 182.8 (Nginx)
357.5 (Tomcat)
274.4 (Nginx)
210.5 (Tomcat)
Weak Moderate (Limits)
Unikernel 7.8 (Nginx)
106 (Tomcat)
1222 (Nginx)
2056 (Tomcat)
Strong Moderate (knobs
available, not used
yet)

17. Customer cares about
Application Performance
Resource Isolation
Security
Application high-availability
17

18. Performance Experiment Environment
Machine
Lenovo W520, CPU: Intel i7-2760QM CPU 2.40 GHz x 8 logical cores, Memory: 19.5 GB RAM
OS: Ubuntu 16.04.1 LTS (64-bit)
Deployment Options
Linux (host machine) - Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP
VM: VirtualBox (v5.1.2) - Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP, 8GB RAM, 4 vCPUs
Container: Docker (v1.12.0) - Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP
Unikernel: OSv (based on git hash: f53c0c39) - v0.24-176-g2e19ba4 (Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic
#53-Ubuntu SMP), 4 vCPUs, 2GB RAM 18

19. Metric Set 3a: Application Performance
(nginx)
19

20. Metrics Set 3: Throughput Explanation
nginx-osv > nginx-linux > nginx-docker > nginx-vm
Baseline: 1 thread/client
Nginx-linux (bare metal) ~600 requests/sec
Nginx-vm slightly lower: expected because the client request needs to traverse two I/O
stacks - the hypervisor’s and the Guest OS’s
Nginx-docker is close to bare metal: expected since the only thing separating the container
from the workload generator is a network bridge
Nginx-osv slightly better than bare metal: client requests still have to go through the
unikernel’s I/O stack but the I/O stack for OSV was designed to be light/lower-overhead -
influenced by a design based on Van Jacobson’s net channels
10 threads
Results get slightly more than 10X better (this is mostly because of reductions in average
latency - next graph) but the ordering remains the same 20

21. Metrics Set 3: Response Time Explanation
nginx-osv > nginx-linux > nginx-docker > nginx-vm
Overall response times between 1ms and 2ms
Single thread case ~1.5ms, and 10 thread case < 1.5ms
Reduction in response time moving 1 to 10 threads is mostly a result of
caching and multiplexing.
With multiple threads, more work gets done per-unit time. While thread A is processing the
results of a response, thread B, which was waiting, can quickly be given a cached copy of
the static file being served.
21

22. Metric Set 3b Application Performance
(Apache Tomcat)
22

23. Summary
Many microservice tools can be deployed in a unikernel
Nginx, Tomcat, JVM, Nodejs, Redis, Memcached etc. (list is growing)
Performance is comparable
Smaller “attack” surface (no extraneous services)
Lean network-stack (e.g., OSv)
Lean OS (no kernel-userspace crossing, no context-switching, heavy mem mgt etc.,)
Opportunities in tooling to help flesh out the workflow for planning or
23

24. Acknowledgements
Thank you!
OSv: Nadav Har’El
Nirmata: Jim Bugwadia
Microservices and Cloud Native Apps Meetup
Mike Larkin, Carl Waldspurger, Anne Holler
24

25. Links
Tools: Ukonvrt, ukdctl
Rain Workload Toolkit
Nginx VirtualBox repo
Nginx OSv
OSv networking hack
Performance evaluation of OSv
25

26. Q & A
Madhuri
cosmokramer@gmail.com
GitHub: myechuri
Rean
rean@caa.columbia.edu
GitHub: rean
26