Being a Moby maintainer

Copyright©2017 NTT Corp. All Rights Reserved.
Akihiro Suda ( @_AkihiroSuda_ )
NTT Software Innovation Center
Being a Moby maintainer
Docker Tokyo (Nov 2, 2017)
https://slideshare.net/AkihiroSuda

2
• Software Engineer at NTT
• github: @AkihiroSuda
• Twitter: @_AkihiroSuda_
• Docker Moby core maintainer (github.com/docker/docker moby/moby)
• In April 2017, Docker [ as a project ] transited into Moby.
• Now Docker [ as a product ] has been developed as one of downstream of Moby.
: ≒ :
RHEL Fedora
Who am I

3
• BuildKit initial maintainer (github.com/moby/buildkit)
• Next-generation `docker build`
• Executes DAG vertices of Dockerfile-equivalent concurrently
• Soon: cache-aware distributed mode
• containerd maintainer (github.com/containerd/containerd)
• Industry-standard container runtime
• Can be used as a Docker-replacement for Kubernetes
Who am I

4
• What is the Moby Project
• Recent updates in the Moby Project
• How I became a maintainer (and how you can!)
Agenda

5
What is the Moby Project?

6

7
• A collaborative project for the container ecosystem to assemble
container-based systems
runc
BuildKit
Moby registry
DataKit
VPNKit
HyperKit
Moby engine
(dockerd)Moby tool
libnetwork
libentitlement

8
Relationship between Moby and Docker (and Balena)
Docker Community Edition Docker Enterprise Edition
Balena: Moby-based container engine for IoT, by Resin.io
+ Support, GUI..
Add your own downstream here...
?
Third parties
Docker, Inc.'s products
Similarity

9
• `dockerd` command is part of Moby
• Will be renamed to `moby-engine` soon
• `docker` command is NOT part of Moby
• Because UX is basically out of scope of Moby
• Solely maintained by Docker, Inc. But still opensource.
• Desktop and cloud installers are NOT part of Moby
(Docker for Mac / Windows / AWS / Azure)
• Docker, Inc. 's proprietary software at the moment
• Dockerfile is likely to be removed from Moby... But no worry!
• Moby BuildKit provides a new low-level instruction set
Some Docker components are not included in Moby

10
• Announcement at DockerCon EU (October 17, 2017)
Recent updates in Moby and Docker
Kubernetes
Docker
containerd
Docker
Kubernetes
containerd
API translator (unreleased)
`docker` CLI
can be used for
managing k8s
Container Runtime Interface (CRI)
k8s no longer
depends on
Docker

11
• Announcement at DockerCon EU (October 17, 2017)
https://blog.docker.com/2017/10/kubernetes-docker-platform-and-moby-project/
Swarm is still kept
(User can choose either one)

12
• This was not a surprise, because Docker, Inc. and the Moby
community has been already working on Kubernetes-related stuff
for a year
https://blog.mobyproject.org/moby-and-kubernetes-bf888ab31e38

13
containerd: Industry-standard container runtime
• Simpler architecture than the "monolith" of Docker
• containerd is "a la carte" of well-decoupled subsystems (runtime, image, snapshot..)
• Can be used as a Docker-replacement for Kubernetes (and so on)
• CRI-containerd, the glue module for k8s is mainly maintained by Google employees
• More collaborative than the past Docker in the pre-Moby era
• Docker, Inc. donated containerd to Cloud Native Computing Foundation (CNCF)
runc
containerd v1.0
CRI-containerd
KubernetesKubernetes
Docker (Moby)
runc
containerd v0.2
Docker (Moby)
Docker 18.XX (unreleased)

14
• Assembles VM images for running certain service like Docker
• Now officially supports Kubernetes as well (with CRI-containerd)
• Usecases are not limited to Docker/Kubernetes
• e.g. "RedisOS" without Docker nor Kubernetes
• Everything is containerd container
• No systemd nor SysV init
• Everything is immutable by design
• When you need to update or customize your LinuxKit VM instances, you just
dispose the existing instances and create new ones (as in Docker containers!)
LinuxKit: Toolkit for custom Linux distribution
LinuxKit
namespace
Kubernetes
namespace
dhcpd
ntpd
kubelet
cri-containerd
e.g., nginx
containerd

15
• Deploys distributed system like Docker Swarm-mode to IaaS like EC2
• Self-healing
• Recreate instances if the actual state differs from the desired state
• Now supports deploying Kubernetes as well
(implemented by Yuji Oshima)
• InfraKit maintainer
• My colleague at NTT
InfraKit: Toolkit for infrastructure orchestration
Infrakit LinuxKit LinuxKit
EC2 EC2
LinuxKit
EC2
Docker Swarm-mode / Kubernetes

16
• Compiles Dockerfile to LLB DAG
• LLB: low-level build instruction set
• DAG: directed acyclic graph
• LLB DAG allows concurrent build
• Third party languages could be compiled to LLB DAG as well
BuildKit: next-generation `docker build`
Compile
Dockerfile
LLB DAG
Third party languages
docker-image://alpine
Image
git://foo/bar
docker-image://gcc
Run("apk add ..")Run("make")

17
• Distributed mode (on Kubernetes) is on plan
• Workers reports LLB DAG vertex cache info and performance stats to masters
• Master assigns a vertex job to the worker which seems the best (just heuristic)
• Designed to be stateless as much as possible for ease of deployment
Master
Master
Master
LBClient
Worker
Worker
Worker

18
Even useful for non-container
usecases, as a generic
concurrent / distributed
compiler toolkit

19
libentitlement: high-level permissions and security profile
https://www.slideshare.net/Docker/moby-and-kubernetes-entitlements

20
libentitlement: high-level permissions and security profile
https://www.slideshare.net/Docker/moby-and-kubernetes-entitlements

21
• Previously, Solomon Hykes (CTO, Docker, Inc.) was the BDFL
• Benevolent Dictator For Life
• Now "Technical Steering Committee" is being established (in this
November) as the replacement for the BDFL role
• A single company cannot hold more than 1/3 seats
Moby governance

22
• Some components under the Moby umbrella belong to other
organizations
• containerd, Notary: CNCF (Cloud Native Computing Foundation)
• InfraKit is also proposed to CNCF
• runc: OCI (Open Containers Initiative)
• Moby Technical Steering Committee will help cross-project and
cross-organization collaboration
Moby Project, CNCF, and OCI

23
• Maintainers (aka "committers" in other FLOSS communities)
• Can approve other contributors' pull requests (with 2 LGTMs usually)
• Can manage GitHub issues
• Can add and remove other maintainers (with 66% approval vote)
• Elected from active contributors who:
• Send pull requests
• Bug fix
• Enhancement
• New feature
• Review other contributors pull requests
• Triage GitHub issues
Who are maintainers and how they are elected

24
• Began contribution to Docker in December, 2015
• I was working on some fault injection tool (github.com/osrg/namazu), and got
stuck in "false ZooKeeper bug" due to AUFS hang-up: #18180
• This is not a bug of Docker but mainly tracked in Docker community
• Became a Docker maintainer in November, 2016
• Docker  Moby in April, 2017
• (I think) Mainly contributed to filesystem issues
• Both AUFS and overlayfs have some stability and compatibility issues
• Also proposed some new features
• `docker network prune` (merged): #27525
• introspection mount (procfs-like stuff for containers. unmerged yet): #24893, #26331
• TCP port forwarder (withdrawn): #26365
• ...
How I became a Docker/Moby maintainer

25
How I became a Docker/Moby maintainer
JVM processes in Docker were extremely unstable due to an
AUFS issue.
The issue was very critical for many users, but very hard to
debug because of non-determinism.

Actually, it was not me who firstly identified
the cause of the issue (Also, I didn't wrote
the patch to fix the issue in the fact)
But I made some demonstration
tools for confirming the cause of
the issue

I made some reports to AUFS community,
and the AUFS maintainer (not me) fixed the
issue
Created "cheat sheet" for Linux
distribution-specific workarounds
Encouraged Linux distributors to
apply the AUFS patch

Created comprehensive list of filesystem
issues
(github.com/AkihiroSuda/issues-docker)

Began code contribution
(Details are out of scope of this talk)
Discussed with maintainers in person
(Using mobile phones!)

http://www.publickey1.jp/blog/16/docker_enginenttdocker.html
"Suda at NTT became a maintainer of
Docker Engine"

32
• Initial maintainer from the beginning of the project (2017 summer)
• Initially I proposed DAG-based builder (but without idea of LLB)
• Coincidently, Tõnis Tiigi (Docker, Inc.) was planning similar but
even better idea, which turned into BuildKit
• Tõnis invited me to an initial maintainer of BuildKit 
How I became a BuildKit maintainer

35
• Began contribution in December, 2016
• Became a maintainer in September, 2017
• Mainly contributed to filesystem and image issues
• Though contribution to Docker / Moby, I found filesystem issues are hard to
maintain
• I needed to reform containerd interfaces and data formats for my experimental OCI
Image extension
• FILEgrain: transport-agnostic, fine-grained content-addressable container image layout
(github.com/AkihiroSuda/filegrain)
How I became a containerd maintainer

36
• Collaborativeness (the most important)
• Comprehensiveness
• Issue analysis, Bug-fix, enhancement, feature addition...
• But when you plan to add a big feature, please coordinate with maintainers in GitHub
issue or Slack before opening a PR!
• Continuity
• One-shot contribution is always welcomed, but maintainership requires continuous
activity
• Number of git commits and LOCs are not so important
And how you can become maintainers! (my personal view)

37
• Moby engine
• Recently integrated containerd v1.0 runtime. We need to make sure there is no
regression.
• The next step is to integrate containerd v1.0 snapshot subsystem. (Much harder for
compatibility)
• BuildKit
• Dockerfile2LLB compiler is not stable. Testing is highly welcomed.
• Design for distributed mode is still under discussion.
• containerd
• Adding tests and performance optimizations are welcomed.
• And more!
Good chances to contribution (my personal view)

38
• The Moby Project is getting more collaborative
• You can contribute and become a maintainer as well!
Recap
https://blog.docker.com/2017/04/introducing-the-moby-project/

Being a Moby maintainer

More Related Content

What's hot

Viewers also liked

Similar to Being a Moby maintainer

More from Akihiro Suda

Recently uploaded

Being a Moby maintainer

Editor's Notes