1. Quantifying Container Runtime
Performance
A Serverless Platform Case Study
Phil Estes
Senior Technical Staff, IBM Cloud
CTO Architecture Tech Team, Containers
@estesp

3. @estesp
● Virtualization
● IaaS
● PaaS
● Containers
● CaaS
● Serverless
(FaaS)

4. SERVER-
less?

5. Hint: There are still servers.
(It just so happens that you don’t have to run them.)
Step 3
Use triggers, actions, etc. as
supported by your FaaS
provider to handle function
input/output chaining
Step 1
Write your function, in a
language supported by the
FaaS runtime (Swift,
Node.js, etc.)
Step 2
Register your function with
your FaaS framework

6. Serverless Servers Matter.
(Because you expect your functions to run perfectly.)
Expectation 3
I only pay for the execution
runtime of my functions.
Expectation 1
(Near?) infinite* scaling of
your functions.
Expectation 2
Perfect uptime. My
functions always run when
triggered with no
perceptible delay.

7. FaaS pricing is based on “GB-s”
Function execution runtime (rounded ~ 100ms)
Memory allocated for the function (in GB)
Gigabyte Seconds
x
=

8. So, about those servers...

10. But...
Standard container lifecycle operations are not
sufficient for our performance guarantees!
Cannot “docker build”, “docker run”, “docker rm”
on each function invocation..
Containers make good sense
as the function invocation
vehicle.

11. /usr/bin/docker
libnetwork
VolumeAPI
AuthZcontainerd
ctr-shim
runc
{
/usr/bin/docker
/usr/bin/dockerd
DOCKER
CONTAINERD
RUNC
We Have Options!
Docker Engine architecture
Docker 1.11 and above; April 2016-current

12. docker
Complete container
engine with lifecycle
management,
orchestration, remote
API (daemon model),
plugin support, SDN
networking, image
building, image
registry/local cache
management.
containerd
High-performance,
standards-based
lightweight container
runtime with gRPC API,
daemon model.
Expanding to contain
complete lifecycle and
image management in
2017.
runc
Open Container
Initiative (OCI)
compliant
implementation of the
runtime specification.
Lightweight container
executor; no network,
image registry or
image creation
capability.

13. https://github.com/estesp/bucketbench
A Go-based framework for benchmarking container
lifecycle operations (under load) against docker,
containerd, and runc.
Usage:
bucketbench run [flags]
Flags:
-b, --bundle string Path of test runc image bundle (default ".")
-c, --containerd int Number of threads to execute against containerd
--ctr-binary string Name/path of containerd client (ctr) binary (default "ctr")
-d, --docker int Number of threads to execute against Docker
--docker-binary string Name/path of Docker binary (default "docker")
-i, --image string Name of test Docker image (default "busybox")
-r, --runc int Number of threads to execute against runc
--runc-binary string Name/path of runc binary (default "runc")
Global Flags:
--log-level string set the logging level (info,warn,err,debug) (default "warn")
H
O
W
CAN
W
E
CO
M
PARE
TH
E
RU
N
TIM
E
PERFO
RM
AN
CE
O
F
TH
ESE
O
PTIO
N
S?

14. Goals
- Assess runtime stability under significant load/parallelism
- Compare operational throughput of each container runtime
Table shows the rate of operation sequences per second. * indicates errors.
----------------------------------------------------------------------------------------------------------------------------
Iter/Thd 1 thrd 2 thrds 3 thrds 4 thrds 5 thrds 6 thrds 7 thrds 8 thrds 9 thrds 10 thrds 11 thrds 12 thrds 13 thrds
Limit 1000 651.3 829.4 834.5 809.6 827.6 848.6 774.8 843.2 800.3 839.2 804.2 806.7 813.0
DockerBasic 15 1.99 2.44 3.02* 3.24* 3.59* 3.90* 4.07*
DockerPause 15 10.22 13.53 15.67 17.69 19.18 19.11 18.56
DockerFull 15 1.66 2.18* 2.69* 3.05* 3.21* 3.36* 3.63*
ConBasic 50 2.90 4.95 6.54 7.49 8.10 8.33 8.65 9.02 9.25 9.17 9.43 9.22 9.25
RuncBasic 50 2.90 5.26 7.37 8.61 9.61 11.07 11.68 12.44 13.56 13.65 14.11 14.29 13.97
Caveats
- Flexibility of lower layer configurations has significant impact
- Stability & performance of runtimes release-dependant

15. Architecture
Two key interfaces:
● Driver
○ Drives the container runtime
● Bench
○ Defines the container operations
and provides results/statistics
type Driver interface {
Type() Type
Info() (string, error)
Create(name, image string, detached bool, trace bool) (Container, error)
Clean() error
Run(ctr Container) (string, int, error)
Stop(ctr Container) (string, int, error)
Remove(ctr Container) (string, int, error)
Pause(ctr Container) (string, int, error)
Unpause(ctr Container) (string, int, error)
}
type Bench interface {
Init(driverType driver.Type, binaryPath,
imageInfo string, trace bool) error
Validate() error
Run(threads, iterations int) error
Stats() []RunStatistics
Elapsed() time.Duration
State() State
Type() Type
}
Driver implementations support:
● Docker, containerd, and runc today
● Can easily be extended to support any
runtime which can implement the
interface shown above

16. Go tools: pprof, trace, block prof..
Also useful: strace, flame graphs..

17. @estesp
Discoveries
Network namespace creation/deletion has significant impact under load
▪ 300ms (and higher) delay in network spin lock under multi-threaded contention
▪ Known issue:
http://stackoverflow.com/questions/28818452/how-to-identify-performance-bottlen
eck-in-linux-system-call-unshareclone-newnet
API overhead, libnetwork
setup/teardown, & metadata
sync/update (locking) all add to
differential from runc “bare” container
start performance
Filesystem setup also measurable for
large # of layers, depending on
storage backend

18. @estesp
Bucketbench: What’s Left To Do
● Easier way to specify/define benchmark runs
○ Requiring Go code to create new benchmark type too high a bar
○ Should provide a way to define via JSON/YAML and input to `bucketbench`
● Structured Output option vs. human readable format
○ Selectable JSON out for displaying or parsing/post-processing with other tools
○ Provide itemized metrics per operation (not currently exposed) in structured output
● Update containerd driver implementation
○ Use gRPC API instead of `ctr` external binary client; use image/storage capabilities
● Other driver implementations?

19. So What?
▪ Want to learn more about OpenWhisk?
- Here at OSCON: Daniel Krook, IBM, Wed, 11:50am / Meeting Room 14
- https://openwhisk.org
- https://github.com/openwhisk/openwhisk
▪ Get involved in improvements to bucketbench:
- https://github.com/estesp/bucketbench
- See list of TODO items
▪ Use bucketbench to improve stability/performance of container
runtimes:
- Propose better integration with tracing/performance tooling
- Find and fix performance bottlenecks in any layer/runtimerunc

20. @estesp
Thank You!
1. Check out, critique, contribute to:
http://github.com/estesp/bucketbench
2. Connect with me to ask questions, or
provide your own perspective and findings
at @estesp on Twitter or
estesp@gmail.com
3. Have fun with containers, whether you
use Docker, containerd, runc, lxc/lxd, rkt,
Kubernetes, Swarm, Mesos, Rancher,
Nomad, OpenShift, ...