Quantifying Container Runtime Performance: OSCON 2017 Open Container Day

Phil Estes
Phil EstesSTSM, IBM Cloud Division, Open Technologies group at IBM
Quantifying Container Runtime
Performance
A Serverless Platform Case Study
Phil Estes
Senior Technical Staff, IBM Cloud
CTO Architecture Tech Team, Containers
@estesp
Quantifying Container Runtime Performance: OSCON 2017 Open Container Day
@estesp
● Virtualization
● IaaS
● PaaS
● Containers
● CaaS
● Serverless
(FaaS)
SERVER-
less?
Hint: There are still servers.
(It just so happens that you don’t have to run them.)
Step 3
Use triggers, actions, etc. as
supported by your FaaS
provider to handle function
input/output chaining
Step 1
Write your function, in a
language supported by the
FaaS runtime (Swift,
Node.js, etc.)
Step 2
Register your function with
your FaaS framework
Serverless Servers Matter.
(Because you expect your functions to run perfectly.)
Expectation 3
I only pay for the execution
runtime of my functions.
Expectation 1
(Near?) infinite* scaling of
your functions.
Expectation 2
Perfect uptime. My
functions always run when
triggered with no
perceptible delay.
FaaS pricing is based on “GB-s”
Function execution runtime (rounded ~ 100ms)
Memory allocated for the function (in GB)
Gigabyte Seconds
x
=
So, about those servers...
Quantifying Container Runtime Performance: OSCON 2017 Open Container Day
But...
Standard container lifecycle operations are not
sufficient for our performance guarantees!
Cannot “docker build”, “docker run”, “docker rm”
on each function invocation..
Containers make good sense
as the function invocation
vehicle.
/usr/bin/docker
libnetwork
VolumeAPI
AuthZcontainerd
ctr-shim
runc
{
/usr/bin/docker
/usr/bin/dockerd
DOCKER
CONTAINERD
RUNC
We Have Options!
Docker Engine architecture
Docker 1.11 and above; April 2016-current
docker
Complete container
engine with lifecycle
management,
orchestration, remote
API (daemon model),
plugin support, SDN
networking, image
building, image
registry/local cache
management.
containerd
High-performance,
standards-based
lightweight container
runtime with gRPC API,
daemon model.
Expanding to contain
complete lifecycle and
image management in
2017.
runc
Open Container
Initiative (OCI)
compliant
implementation of the
runtime specification.
Lightweight container
executor; no network,
image registry or
image creation
capability.
https://github.com/estesp/bucketbench
A Go-based framework for benchmarking container
lifecycle operations (under load) against docker,
containerd, and runc.
Usage:
bucketbench run [flags]
Flags:
-b, --bundle string Path of test runc image bundle (default ".")
-c, --containerd int Number of threads to execute against containerd
--ctr-binary string Name/path of containerd client (ctr) binary (default "ctr")
-d, --docker int Number of threads to execute against Docker
--docker-binary string Name/path of Docker binary (default "docker")
-i, --image string Name of test Docker image (default "busybox")
-r, --runc int Number of threads to execute against runc
--runc-binary string Name/path of runc binary (default "runc")
Global Flags:
--log-level string set the logging level (info,warn,err,debug) (default "warn")
H
O
W
CAN
W
E
CO
M
PARE
TH
E
RU
N
TIM
E
PERFO
RM
AN
CE
O
F
TH
ESE
O
PTIO
N
S?
Goals
- Assess runtime stability under significant load/parallelism
- Compare operational throughput of each container runtime
Table shows the rate of operation sequences per second. * indicates errors.
----------------------------------------------------------------------------------------------------------------------------
Iter/Thd 1 thrd 2 thrds 3 thrds 4 thrds 5 thrds 6 thrds 7 thrds 8 thrds 9 thrds 10 thrds 11 thrds 12 thrds 13 thrds
Limit 1000 651.3 829.4 834.5 809.6 827.6 848.6 774.8 843.2 800.3 839.2 804.2 806.7 813.0
DockerBasic 15 1.99 2.44 3.02* 3.24* 3.59* 3.90* 4.07*
DockerPause 15 10.22 13.53 15.67 17.69 19.18 19.11 18.56
DockerFull 15 1.66 2.18* 2.69* 3.05* 3.21* 3.36* 3.63*
ConBasic 50 2.90 4.95 6.54 7.49 8.10 8.33 8.65 9.02 9.25 9.17 9.43 9.22 9.25
RuncBasic 50 2.90 5.26 7.37 8.61 9.61 11.07 11.68 12.44 13.56 13.65 14.11 14.29 13.97
Caveats
- Flexibility of lower layer configurations has significant impact
- Stability & performance of runtimes release-dependant
Architecture
Two key interfaces:
● Driver
○ Drives the container runtime
● Bench
○ Defines the container operations
and provides results/statistics
type Driver interface {
Type() Type
Info() (string, error)
Create(name, image string, detached bool, trace bool) (Container, error)
Clean() error
Run(ctr Container) (string, int, error)
Stop(ctr Container) (string, int, error)
Remove(ctr Container) (string, int, error)
Pause(ctr Container) (string, int, error)
Unpause(ctr Container) (string, int, error)
}
type Bench interface {
Init(driverType driver.Type, binaryPath,
imageInfo string, trace bool) error
Validate() error
Run(threads, iterations int) error
Stats() []RunStatistics
Elapsed() time.Duration
State() State
Type() Type
}
Driver implementations support:
● Docker, containerd, and runc today
● Can easily be extended to support any
runtime which can implement the
interface shown above
Go tools: pprof, trace, block prof..
Also useful: strace, flame graphs..
@estesp
Discoveries
Network namespace creation/deletion has significant impact under load
▪ 300ms (and higher) delay in network spin lock under multi-threaded contention
▪ Known issue:
http://stackoverflow.com/questions/28818452/how-to-identify-performance-bottlen
eck-in-linux-system-call-unshareclone-newnet
API overhead, libnetwork
setup/teardown, & metadata
sync/update (locking) all add to
differential from runc “bare” container
start performance
Filesystem setup also measurable for
large # of layers, depending on
storage backend
@estesp
Bucketbench: What’s Left To Do
● Easier way to specify/define benchmark runs
○ Requiring Go code to create new benchmark type too high a bar
○ Should provide a way to define via JSON/YAML and input to `bucketbench`
● Structured Output option vs. human readable format
○ Selectable JSON out for displaying or parsing/post-processing with other tools
○ Provide itemized metrics per operation (not currently exposed) in structured output
● Update containerd driver implementation
○ Use gRPC API instead of `ctr` external binary client; use image/storage capabilities
● Other driver implementations?
So What?
▪ Want to learn more about OpenWhisk?
- Here at OSCON: Daniel Krook, IBM, Wed, 11:50am / Meeting Room 14
- https://openwhisk.org
- https://github.com/openwhisk/openwhisk
▪ Get involved in improvements to bucketbench:
- https://github.com/estesp/bucketbench
- See list of TODO items
▪ Use bucketbench to improve stability/performance of container
runtimes:
- Propose better integration with tracing/performance tooling
- Find and fix performance bottlenecks in any layer/runtimerunc
@estesp
Thank You!
1. Check out, critique, contribute to:
http://github.com/estesp/bucketbench
2. Connect with me to ask questions, or
provide your own perspective and findings
at @estesp on Twitter or
estesp@gmail.com
3. Have fun with containers, whether you
use Docker, containerd, runc, lxc/lxd, rkt,
Kubernetes, Swarm, Mesos, Rancher,
Nomad, OpenShift, ...
1 of 20

Recommended

Container Runtimes: Comparing and Contrasting Today's Engines by
Container Runtimes: Comparing and Contrasting Today's EnginesContainer Runtimes: Comparing and Contrasting Today's Engines
Container Runtimes: Comparing and Contrasting Today's EnginesPhil Estes
2.3K views22 slides
Containerd Internals: Building a Core Container Runtime by
Containerd Internals: Building a Core Container RuntimeContainerd Internals: Building a Core Container Runtime
Containerd Internals: Building a Core Container RuntimePhil Estes
2.1K views36 slides
Embedding Containerd For Fun and Profit by
Embedding Containerd For Fun and ProfitEmbedding Containerd For Fun and Profit
Embedding Containerd For Fun and ProfitPhil Estes
828 views8 slides
Docker Engine Evolution: From Monolith to Discrete Components by
Docker Engine Evolution: From Monolith to Discrete ComponentsDocker Engine Evolution: From Monolith to Discrete Components
Docker Engine Evolution: From Monolith to Discrete ComponentsPhil Estes
951 views21 slides
Intro- Docker Native for OSX and Windows by
Intro- Docker Native for OSX and WindowsIntro- Docker Native for OSX and Windows
Intro- Docker Native for OSX and WindowsThomas Chacko
670 views5 slides
containerd the universal container runtime by
containerd the universal container runtimecontainerd the universal container runtime
containerd the universal container runtimeDocker, Inc.
11.2K views35 slides

More Related Content

What's hot

Devoxx 2016: A Developer's Guide to OCI and runC by
Devoxx 2016: A Developer's Guide to OCI and runCDevoxx 2016: A Developer's Guide to OCI and runC
Devoxx 2016: A Developer's Guide to OCI and runCPhil Estes
1.1K views10 slides
Docker and the Linux Kernel by
Docker and the Linux KernelDocker and the Linux Kernel
Docker and the Linux KernelDocker, Inc.
16.7K views22 slides
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi by
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea LuzzardiWhat's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea LuzzardiMike Goelzer
20.5K views40 slides
Docker Distributed application bundle & Stack - Overview by
Docker Distributed application bundle & Stack - Overview Docker Distributed application bundle & Stack - Overview
Docker Distributed application bundle & Stack - Overview Thomas Chacko
683 views11 slides
Introduction to CRI and OCI by
Introduction to CRI and OCIIntroduction to CRI and OCI
Introduction to CRI and OCIHungWei Chiu
616 views34 slides
Kubernetes deep dive - - Huawei 2015-10 by
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Vishnu Kannan
1.2K views65 slides

What's hot(20)

Devoxx 2016: A Developer's Guide to OCI and runC by Phil Estes
Devoxx 2016: A Developer's Guide to OCI and runCDevoxx 2016: A Developer's Guide to OCI and runC
Devoxx 2016: A Developer's Guide to OCI and runC
Phil Estes1.1K views
Docker and the Linux Kernel by Docker, Inc.
Docker and the Linux KernelDocker and the Linux Kernel
Docker and the Linux Kernel
Docker, Inc.16.7K views
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi by Mike Goelzer
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea LuzzardiWhat's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
Mike Goelzer20.5K views
Docker Distributed application bundle & Stack - Overview by Thomas Chacko
Docker Distributed application bundle & Stack - Overview Docker Distributed application bundle & Stack - Overview
Docker Distributed application bundle & Stack - Overview
Thomas Chacko683 views
Introduction to CRI and OCI by HungWei Chiu
Introduction to CRI and OCIIntroduction to CRI and OCI
Introduction to CRI and OCI
HungWei Chiu616 views
Kubernetes deep dive - - Huawei 2015-10 by Vishnu Kannan
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10
Vishnu Kannan1.2K views
Scaling Docker Containers using Kubernetes and Azure Container Service by Ben Hall
Scaling Docker Containers using Kubernetes and Azure Container ServiceScaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
Ben Hall804 views
Driving containerd operations with gRPC by Docker, Inc.
Driving containerd operations with gRPCDriving containerd operations with gRPC
Driving containerd operations with gRPC
Docker, Inc.10.7K views
DockerCon US 2016 - Extending Docker With APIs, Drivers, and Plugins by Arnaud Porterie
DockerCon US 2016 - Extending Docker With APIs, Drivers, and PluginsDockerCon US 2016 - Extending Docker With APIs, Drivers, and Plugins
DockerCon US 2016 - Extending Docker With APIs, Drivers, and Plugins
Arnaud Porterie1.8K views
Wordcamp Bratislava 2017 - Docker! Why? by Adam Štipák
Wordcamp Bratislava 2017 - Docker! Why?Wordcamp Bratislava 2017 - Docker! Why?
Wordcamp Bratislava 2017 - Docker! Why?
Adam Štipák380 views
Kubernetes 101 - A Cluster Operating System by mikaelbarbero
Kubernetes 101 - A Cluster Operating SystemKubernetes 101 - A Cluster Operating System
Kubernetes 101 - A Cluster Operating System
mikaelbarbero8.3K views
The state of containerd by Docker, Inc.
The state of containerdThe state of containerd
The state of containerd
Docker, Inc.10K views
Docker for PHP Developers - Jetbrains by Chris Tankersley
Docker for PHP Developers - JetbrainsDocker for PHP Developers - Jetbrains
Docker for PHP Developers - Jetbrains
Chris Tankersley9.1K views
AtlasCamp 2015: The age of orchestration: From Docker basics to cluster manag... by Atlassian
AtlasCamp 2015: The age of orchestration: From Docker basics to cluster manag...AtlasCamp 2015: The age of orchestration: From Docker basics to cluster manag...
AtlasCamp 2015: The age of orchestration: From Docker basics to cluster manag...
Atlassian2.9K views
runC: The little engine that could (run Docker containers) by Docker Captain ... by Docker, Inc.
runC: The little engine that could (run Docker containers) by Docker Captain ...runC: The little engine that could (run Docker containers) by Docker Captain ...
runC: The little engine that could (run Docker containers) by Docker Captain ...
Docker, Inc.13.3K views
Platform Orchestration with Kubernetes and Docker by Julian Strobl
Platform Orchestration with Kubernetes and DockerPlatform Orchestration with Kubernetes and Docker
Platform Orchestration with Kubernetes and Docker
Julian Strobl1K views

Similar to Quantifying Container Runtime Performance: OSCON 2017 Open Container Day

Anton Moldovan "Building an efficient replication system for thousands of ter... by
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Fwdays
150 views114 slides
Porting a Streaming Pipeline from Scala to Rust by
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustEvan Chan
7 views38 slides
Introduction to LAVA Workload Scheduler by
Introduction to LAVA Workload SchedulerIntroduction to LAVA Workload Scheduler
Introduction to LAVA Workload SchedulerNopparat Nopkuat
2.1K views53 slides
Troubleshooting Complex Performance issues - Oracle SEG$ contention by
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTanel Poder
44.1K views43 slides
Road to sbt 1.0 paved with server by
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with serverEugene Yokota
4.9K views53 slides
Java Memory Model by
Java Memory ModelJava Memory Model
Java Memory ModelŁukasz Koniecki
575 views65 slides

Similar to Quantifying Container Runtime Performance: OSCON 2017 Open Container Day(20)

Anton Moldovan "Building an efficient replication system for thousands of ter... by Fwdays
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...
Fwdays150 views
Porting a Streaming Pipeline from Scala to Rust by Evan Chan
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to Rust
Evan Chan7 views
Introduction to LAVA Workload Scheduler by Nopparat Nopkuat
Introduction to LAVA Workload SchedulerIntroduction to LAVA Workload Scheduler
Introduction to LAVA Workload Scheduler
Nopparat Nopkuat2.1K views
Troubleshooting Complex Performance issues - Oracle SEG$ contention by Tanel Poder
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder44.1K views
Road to sbt 1.0 paved with server by Eugene Yokota
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with server
Eugene Yokota4.9K views
Experience sql server on l inux and docker by Bob Ward
Experience sql server on l inux and dockerExperience sql server on l inux and docker
Experience sql server on l inux and docker
Bob Ward690 views
Watch Re-runs on your SQL Server with RML Utilities by dpcobb
Watch Re-runs on your SQL Server with RML UtilitiesWatch Re-runs on your SQL Server with RML Utilities
Watch Re-runs on your SQL Server with RML Utilities
dpcobb1.3K views
Copper: A high performance workflow engine by dmoebius
Copper: A high performance workflow engineCopper: A high performance workflow engine
Copper: A high performance workflow engine
dmoebius6.8K views
Resume_CQ_Edward by caiqi wang
Resume_CQ_EdwardResume_CQ_Edward
Resume_CQ_Edward
caiqi wang467 views
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St... by Athens Big Data
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
Athens Big Data200 views
Developing Realtime Data Pipelines With Apache Kafka by Joe Stein
Developing Realtime Data Pipelines With Apache KafkaDeveloping Realtime Data Pipelines With Apache Kafka
Developing Realtime Data Pipelines With Apache Kafka
Joe Stein4.9K views
Cray XT Porting, Scaling, and Optimization Best Practices by Jeff Larkin
Cray XT Porting, Scaling, and Optimization Best PracticesCray XT Porting, Scaling, and Optimization Best Practices
Cray XT Porting, Scaling, and Optimization Best Practices
Jeff Larkin782 views
Ansible & Salt - Vincent Boon by MyNOG
Ansible & Salt - Vincent BoonAnsible & Salt - Vincent Boon
Ansible & Salt - Vincent Boon
MyNOG998 views
Fighting Against Chaotically Separated Values with Embulk by Sadayuki Furuhashi
Fighting Against Chaotically Separated Values with EmbulkFighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with Embulk
Sadayuki Furuhashi2.1K views

More from Phil Estes

Enabling Security via Container Runtimes by
Enabling Security via Container RuntimesEnabling Security via Container Runtimes
Enabling Security via Container RuntimesPhil Estes
613 views16 slides
Extended and embedding: containerd update & project use cases by
Extended and embedding: containerd update & project use casesExtended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use casesPhil Estes
522 views27 slides
Cloud Native TLV Meetup: Securing Containerized Applications Primer by
Cloud Native TLV Meetup: Securing Containerized Applications PrimerCloud Native TLV Meetup: Securing Containerized Applications Primer
Cloud Native TLV Meetup: Securing Containerized Applications PrimerPhil Estes
295 views18 slides
Securing Containerized Applications: A Primer by
Securing Containerized Applications: A PrimerSecuring Containerized Applications: A Primer
Securing Containerized Applications: A PrimerPhil Estes
639 views19 slides
Securing Containerized Applications: A Primer by
Securing Containerized Applications: A PrimerSecuring Containerized Applications: A Primer
Securing Containerized Applications: A PrimerPhil Estes
625 views21 slides
Let's Try Every CRI Runtime Available for Kubernetes by
Let's Try Every CRI Runtime Available for KubernetesLet's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for KubernetesPhil Estes
11.9K views28 slides

More from Phil Estes(20)

Enabling Security via Container Runtimes by Phil Estes
Enabling Security via Container RuntimesEnabling Security via Container Runtimes
Enabling Security via Container Runtimes
Phil Estes613 views
Extended and embedding: containerd update & project use cases by Phil Estes
Extended and embedding: containerd update & project use casesExtended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use cases
Phil Estes522 views
Cloud Native TLV Meetup: Securing Containerized Applications Primer by Phil Estes
Cloud Native TLV Meetup: Securing Containerized Applications PrimerCloud Native TLV Meetup: Securing Containerized Applications Primer
Cloud Native TLV Meetup: Securing Containerized Applications Primer
Phil Estes295 views
Securing Containerized Applications: A Primer by Phil Estes
Securing Containerized Applications: A PrimerSecuring Containerized Applications: A Primer
Securing Containerized Applications: A Primer
Phil Estes639 views
Securing Containerized Applications: A Primer by Phil Estes
Securing Containerized Applications: A PrimerSecuring Containerized Applications: A Primer
Securing Containerized Applications: A Primer
Phil Estes625 views
Let's Try Every CRI Runtime Available for Kubernetes by Phil Estes
Let's Try Every CRI Runtime Available for KubernetesLet's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for Kubernetes
Phil Estes11.9K views
CraftConf 2019: CRI Runtimes Deep Dive: Who Is Running My Pod? by Phil Estes
CraftConf 2019:  CRI Runtimes Deep Dive: Who Is Running My Pod?CraftConf 2019:  CRI Runtimes Deep Dive: Who Is Running My Pod?
CraftConf 2019: CRI Runtimes Deep Dive: Who Is Running My Pod?
Phil Estes613 views
JAX Con 2019: Containers. Microservices. Cloud. Open Source. Fantasy or Reali... by Phil Estes
JAX Con 2019: Containers. Microservices. Cloud. Open Source. Fantasy or Reali...JAX Con 2019: Containers. Microservices. Cloud. Open Source. Fantasy or Reali...
JAX Con 2019: Containers. Microservices. Cloud. Open Source. Fantasy or Reali...
Phil Estes8K views
Giving Back to Upstream | DockerCon 2019 by Phil Estes
Giving Back to Upstream | DockerCon 2019Giving Back to Upstream | DockerCon 2019
Giving Back to Upstream | DockerCon 2019
Phil Estes460 views
What's Running My Containers? A review of runtimes and standards. by Phil Estes
What's Running My Containers? A review of runtimes and standards.What's Running My Containers? A review of runtimes and standards.
What's Running My Containers? A review of runtimes and standards.
Phil Estes435 views
Docker London Meetup: Docker Engine Evolution by Phil Estes
Docker London Meetup: Docker Engine EvolutionDocker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine Evolution
Phil Estes378 views
FOSDEM 2019: A containerd Project Update by Phil Estes
FOSDEM 2019: A containerd Project UpdateFOSDEM 2019: A containerd Project Update
FOSDEM 2019: A containerd Project Update
Phil Estes649 views
CRI Runtimes Deep-Dive: Who's Running My Pod!? by Phil Estes
CRI Runtimes Deep-Dive: Who's Running My Pod!?CRI Runtimes Deep-Dive: Who's Running My Pod!?
CRI Runtimes Deep-Dive: Who's Running My Pod!?
Phil Estes635 views
Docker Athens: Docker Engine Evolution & Containerd Use Cases by Phil Estes
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Phil Estes495 views
It's 2018. Are My Containers Secure Yet!? by Phil Estes
It's 2018. Are My Containers Secure Yet!?It's 2018. Are My Containers Secure Yet!?
It's 2018. Are My Containers Secure Yet!?
Phil Estes2.9K views
An Open Source Story: Open Containers & Open Communities by Phil Estes
An Open Source Story: Open Containers & Open CommunitiesAn Open Source Story: Open Containers & Open Communities
An Open Source Story: Open Containers & Open Communities
Phil Estes617 views
Whose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes by Phil Estes
Whose Job Is It Anyway? Kubernetes, CRI, & Container RuntimesWhose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Whose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Phil Estes3.1K views
Containerd Project Update: FOSDEM 2018 by Phil Estes
Containerd Project Update: FOSDEM 2018Containerd Project Update: FOSDEM 2018
Containerd Project Update: FOSDEM 2018
Phil Estes1.7K views
Bucketbench: Benchmarking Container Runtime Performance by Phil Estes
Bucketbench: Benchmarking Container Runtime PerformanceBucketbench: Benchmarking Container Runtime Performance
Bucketbench: Benchmarking Container Runtime Performance
Phil Estes2.2K views
AtlanTEC 2017: Containers! Why Docker, Why NOW? by Phil Estes
AtlanTEC 2017: Containers! Why Docker, Why NOW?AtlanTEC 2017: Containers! Why Docker, Why NOW?
AtlanTEC 2017: Containers! Why Docker, Why NOW?
Phil Estes560 views

Recently uploaded

Generic or specific? Making sensible software design decisions by
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisionsBert Jan Schrijver
6 views60 slides
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...TomHalpin9
5 views29 slides
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... by
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Donato Onofri
795 views34 slides
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...Deltares
10 views32 slides
Agile 101 by
Agile 101Agile 101
Agile 101John Valentino
7 views20 slides
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDeltares
8 views17 slides

Recently uploaded(20)

Generic or specific? Making sensible software design decisions by Bert Jan Schrijver
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisions
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin95 views
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... by Donato Onofri
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Donato Onofri795 views
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares10 views
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by Deltares
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
Deltares8 views
FIMA 2023 Neo4j & FS - Entity Resolution.pptx by Neo4j
FIMA 2023 Neo4j & FS - Entity Resolution.pptxFIMA 2023 Neo4j & FS - Entity Resolution.pptx
FIMA 2023 Neo4j & FS - Entity Resolution.pptx
Neo4j6 views
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema by Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - GeertsemaDSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema
Deltares17 views
Software testing company in India.pptx by SakshiPatel82
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptx
SakshiPatel827 views
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by Marc Müller
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Marc Müller37 views
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J... by Deltares
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
Deltares9 views
A first look at MariaDB 11.x features and ideas on how to use them by Federico Razzoli
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use them
Federico Razzoli45 views
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko... by Deltares
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
Deltares14 views
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge... by Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
Deltares17 views
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by Lisi Hocke
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Lisi Hocke28 views
Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana8 views

Quantifying Container Runtime Performance: OSCON 2017 Open Container Day

  • 1. Quantifying Container Runtime Performance A Serverless Platform Case Study Phil Estes Senior Technical Staff, IBM Cloud CTO Architecture Tech Team, Containers @estesp
  • 3. @estesp ● Virtualization ● IaaS ● PaaS ● Containers ● CaaS ● Serverless (FaaS)
  • 5. Hint: There are still servers. (It just so happens that you don’t have to run them.) Step 3 Use triggers, actions, etc. as supported by your FaaS provider to handle function input/output chaining Step 1 Write your function, in a language supported by the FaaS runtime (Swift, Node.js, etc.) Step 2 Register your function with your FaaS framework
  • 6. Serverless Servers Matter. (Because you expect your functions to run perfectly.) Expectation 3 I only pay for the execution runtime of my functions. Expectation 1 (Near?) infinite* scaling of your functions. Expectation 2 Perfect uptime. My functions always run when triggered with no perceptible delay.
  • 7. FaaS pricing is based on “GB-s” Function execution runtime (rounded ~ 100ms) Memory allocated for the function (in GB) Gigabyte Seconds x =
  • 8. So, about those servers...
  • 10. But... Standard container lifecycle operations are not sufficient for our performance guarantees! Cannot “docker build”, “docker run”, “docker rm” on each function invocation.. Containers make good sense as the function invocation vehicle.
  • 12. docker Complete container engine with lifecycle management, orchestration, remote API (daemon model), plugin support, SDN networking, image building, image registry/local cache management. containerd High-performance, standards-based lightweight container runtime with gRPC API, daemon model. Expanding to contain complete lifecycle and image management in 2017. runc Open Container Initiative (OCI) compliant implementation of the runtime specification. Lightweight container executor; no network, image registry or image creation capability.
  • 13. https://github.com/estesp/bucketbench A Go-based framework for benchmarking container lifecycle operations (under load) against docker, containerd, and runc. Usage: bucketbench run [flags] Flags: -b, --bundle string Path of test runc image bundle (default ".") -c, --containerd int Number of threads to execute against containerd --ctr-binary string Name/path of containerd client (ctr) binary (default "ctr") -d, --docker int Number of threads to execute against Docker --docker-binary string Name/path of Docker binary (default "docker") -i, --image string Name of test Docker image (default "busybox") -r, --runc int Number of threads to execute against runc --runc-binary string Name/path of runc binary (default "runc") Global Flags: --log-level string set the logging level (info,warn,err,debug) (default "warn") H O W CAN W E CO M PARE TH E RU N TIM E PERFO RM AN CE O F TH ESE O PTIO N S?
  • 14. Goals - Assess runtime stability under significant load/parallelism - Compare operational throughput of each container runtime Table shows the rate of operation sequences per second. * indicates errors. ---------------------------------------------------------------------------------------------------------------------------- Iter/Thd 1 thrd 2 thrds 3 thrds 4 thrds 5 thrds 6 thrds 7 thrds 8 thrds 9 thrds 10 thrds 11 thrds 12 thrds 13 thrds Limit 1000 651.3 829.4 834.5 809.6 827.6 848.6 774.8 843.2 800.3 839.2 804.2 806.7 813.0 DockerBasic 15 1.99 2.44 3.02* 3.24* 3.59* 3.90* 4.07* DockerPause 15 10.22 13.53 15.67 17.69 19.18 19.11 18.56 DockerFull 15 1.66 2.18* 2.69* 3.05* 3.21* 3.36* 3.63* ConBasic 50 2.90 4.95 6.54 7.49 8.10 8.33 8.65 9.02 9.25 9.17 9.43 9.22 9.25 RuncBasic 50 2.90 5.26 7.37 8.61 9.61 11.07 11.68 12.44 13.56 13.65 14.11 14.29 13.97 Caveats - Flexibility of lower layer configurations has significant impact - Stability & performance of runtimes release-dependant
  • 15. Architecture Two key interfaces: ● Driver ○ Drives the container runtime ● Bench ○ Defines the container operations and provides results/statistics type Driver interface { Type() Type Info() (string, error) Create(name, image string, detached bool, trace bool) (Container, error) Clean() error Run(ctr Container) (string, int, error) Stop(ctr Container) (string, int, error) Remove(ctr Container) (string, int, error) Pause(ctr Container) (string, int, error) Unpause(ctr Container) (string, int, error) } type Bench interface { Init(driverType driver.Type, binaryPath, imageInfo string, trace bool) error Validate() error Run(threads, iterations int) error Stats() []RunStatistics Elapsed() time.Duration State() State Type() Type } Driver implementations support: ● Docker, containerd, and runc today ● Can easily be extended to support any runtime which can implement the interface shown above
  • 16. Go tools: pprof, trace, block prof.. Also useful: strace, flame graphs..
  • 17. @estesp Discoveries Network namespace creation/deletion has significant impact under load ▪ 300ms (and higher) delay in network spin lock under multi-threaded contention ▪ Known issue: http://stackoverflow.com/questions/28818452/how-to-identify-performance-bottlen eck-in-linux-system-call-unshareclone-newnet API overhead, libnetwork setup/teardown, & metadata sync/update (locking) all add to differential from runc “bare” container start performance Filesystem setup also measurable for large # of layers, depending on storage backend
  • 18. @estesp Bucketbench: What’s Left To Do ● Easier way to specify/define benchmark runs ○ Requiring Go code to create new benchmark type too high a bar ○ Should provide a way to define via JSON/YAML and input to `bucketbench` ● Structured Output option vs. human readable format ○ Selectable JSON out for displaying or parsing/post-processing with other tools ○ Provide itemized metrics per operation (not currently exposed) in structured output ● Update containerd driver implementation ○ Use gRPC API instead of `ctr` external binary client; use image/storage capabilities ● Other driver implementations?
  • 19. So What? ▪ Want to learn more about OpenWhisk? - Here at OSCON: Daniel Krook, IBM, Wed, 11:50am / Meeting Room 14 - https://openwhisk.org - https://github.com/openwhisk/openwhisk ▪ Get involved in improvements to bucketbench: - https://github.com/estesp/bucketbench - See list of TODO items ▪ Use bucketbench to improve stability/performance of container runtimes: - Propose better integration with tracing/performance tooling - Find and fix performance bottlenecks in any layer/runtimerunc
  • 20. @estesp Thank You! 1. Check out, critique, contribute to: http://github.com/estesp/bucketbench 2. Connect with me to ask questions, or provide your own perspective and findings at @estesp on Twitter or estesp@gmail.com 3. Have fun with containers, whether you use Docker, containerd, runc, lxc/lxd, rkt, Kubernetes, Swarm, Mesos, Rancher, Nomad, OpenShift, ...