Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Being a Moby maintainer

893 views

Published on

Recent updates in the Moby Project and my experiences as a maintainer

Published in: Software
  • Be the first to comment

Being a Moby maintainer

  1. 1. Copyright©2017 NTT Corp. All Rights Reserved. Akihiro Suda ( @_AkihiroSuda_ ) NTT Software Innovation Center Being a Moby maintainer Docker Tokyo (Nov 2, 2017) https://slideshare.net/AkihiroSuda
  2. 2. 2 Copyright©2017 NTT Corp. All Rights Reserved. • Software Engineer at NTT • github: @AkihiroSuda • Twitter: @_AkihiroSuda_ • Docker Moby core maintainer (github.com/docker/docker moby/moby) • In April 2017, Docker [ as a project ] transited into Moby. • Now Docker [ as a product ] has been developed as one of downstream of Moby. : ≒ : RHEL Fedora Who am I
  3. 3. 3 Copyright©2017 NTT Corp. All Rights Reserved. • BuildKit initial maintainer (github.com/moby/buildkit) • Next-generation `docker build` • Executes DAG vertices of Dockerfile-equivalent concurrently • Soon: cache-aware distributed mode • containerd maintainer (github.com/containerd/containerd) • Industry-standard container runtime • Can be used as a Docker-replacement for Kubernetes Who am I
  4. 4. 4 Copyright©2017 NTT Corp. All Rights Reserved. • What is the Moby Project • Recent updates in the Moby Project • How I became a maintainer (and how you can!) Agenda
  5. 5. 5 Copyright©2017 NTT Corp. All Rights Reserved. What is the Moby Project?
  6. 6. 6 Copyright©2017 NTT Corp. All Rights Reserved. What is the Moby Project?
  7. 7. 7 Copyright©2017 NTT Corp. All Rights Reserved. • A collaborative project for the container ecosystem to assemble container-based systems What is the Moby Project? runc BuildKit Moby registry DataKit VPNKit HyperKit Moby engine (dockerd)Moby tool libnetwork libentitlement
  8. 8. 8 Copyright©2017 NTT Corp. All Rights Reserved. Relationship between Moby and Docker (and Balena) Docker Community Edition Docker Enterprise Edition Balena: Moby-based container engine for IoT, by Resin.io + Support, GUI.. Add your own downstream here... ? Third parties Docker, Inc.'s products Similarity
  9. 9. 9 Copyright©2017 NTT Corp. All Rights Reserved. • `dockerd` command is part of Moby • Will be renamed to `moby-engine` soon • `docker` command is NOT part of Moby • Because UX is basically out of scope of Moby • Solely maintained by Docker, Inc. But still opensource. • Desktop and cloud installers are NOT part of Moby (Docker for Mac / Windows / AWS / Azure) • Docker, Inc. 's proprietary software at the moment • Dockerfile is likely to be removed from Moby... But no worry! • Moby BuildKit provides a new low-level instruction set Some Docker components are not included in Moby
  10. 10. 10 Copyright©2017 NTT Corp. All Rights Reserved. • Announcement at DockerCon EU (October 17, 2017) Recent updates in Moby and Docker Kubernetes Docker containerd Docker Kubernetes containerd API translator (unreleased) `docker` CLI can be used for managing k8s Container Runtime Interface (CRI) k8s no longer depends on Docker
  11. 11. 11 Copyright©2017 NTT Corp. All Rights Reserved. • Announcement at DockerCon EU (October 17, 2017) Recent updates in Moby and Docker https://blog.docker.com/2017/10/kubernetes-docker-platform-and-moby-project/ Swarm is still kept (User can choose either one)
  12. 12. 12 Copyright©2017 NTT Corp. All Rights Reserved. • This was not a surprise, because Docker, Inc. and the Moby community has been already working on Kubernetes-related stuff for a year Recent updates in Moby and Docker https://blog.mobyproject.org/moby-and-kubernetes-bf888ab31e38
  13. 13. 13 Copyright©2017 NTT Corp. All Rights Reserved. containerd: Industry-standard container runtime • Simpler architecture than the "monolith" of Docker • containerd is "a la carte" of well-decoupled subsystems (runtime, image, snapshot..) • Can be used as a Docker-replacement for Kubernetes (and so on) • CRI-containerd, the glue module for k8s is mainly maintained by Google employees • More collaborative than the past Docker in the pre-Moby era • Docker, Inc. donated containerd to Cloud Native Computing Foundation (CNCF) runc containerd v1.0 CRI-containerd KubernetesKubernetes Docker (Moby) runc containerd v0.2 Docker (Moby) Docker 18.XX (unreleased)
  14. 14. 14 Copyright©2017 NTT Corp. All Rights Reserved. • Assembles VM images for running certain service like Docker • Now officially supports Kubernetes as well (with CRI-containerd) • Usecases are not limited to Docker/Kubernetes • e.g. "RedisOS" without Docker nor Kubernetes • Everything is containerd container • No systemd nor SysV init • Everything is immutable by design • When you need to update or customize your LinuxKit VM instances, you just dispose the existing instances and create new ones (as in Docker containers!) LinuxKit: Toolkit for custom Linux distribution LinuxKit namespace Kubernetes namespace dhcpd ntpd kubelet cri-containerd e.g., nginx containerd
  15. 15. 15 Copyright©2017 NTT Corp. All Rights Reserved. • Deploys distributed system like Docker Swarm-mode to IaaS like EC2 • Self-healing • Recreate instances if the actual state differs from the desired state • Now supports deploying Kubernetes as well (implemented by Yuji Oshima) • InfraKit maintainer • My colleague at NTT InfraKit: Toolkit for infrastructure orchestration Infrakit LinuxKit LinuxKit EC2 EC2 LinuxKit EC2 Docker Swarm-mode / Kubernetes
  16. 16. 16 Copyright©2017 NTT Corp. All Rights Reserved. • Compiles Dockerfile to LLB DAG • LLB: low-level build instruction set • DAG: directed acyclic graph • LLB DAG allows concurrent build • Third party languages could be compiled to LLB DAG as well BuildKit: next-generation `docker build` Compile Dockerfile LLB DAG Third party languages docker-image://alpine Image git://foo/bar docker-image://gcc Run("apk add ..")Run("make")
  17. 17. 17 Copyright©2017 NTT Corp. All Rights Reserved. • Distributed mode (on Kubernetes) is on plan • Workers reports LLB DAG vertex cache info and performance stats to masters • Master assigns a vertex job to the worker which seems the best (just heuristic) • Designed to be stateless as much as possible for ease of deployment BuildKit: next-generation `docker build` Master Master Master LBClient Worker Worker Worker
  18. 18. 18 Copyright©2017 NTT Corp. All Rights Reserved. Even useful for non-container usecases, as a generic concurrent / distributed compiler toolkit BuildKit: next-generation `docker build`
  19. 19. 19 Copyright©2017 NTT Corp. All Rights Reserved. libentitlement: high-level permissions and security profile https://www.slideshare.net/Docker/moby-and-kubernetes-entitlements
  20. 20. 20 Copyright©2017 NTT Corp. All Rights Reserved. libentitlement: high-level permissions and security profile https://www.slideshare.net/Docker/moby-and-kubernetes-entitlements
  21. 21. 21 Copyright©2017 NTT Corp. All Rights Reserved. • Previously, Solomon Hykes (CTO, Docker, Inc.) was the BDFL • Benevolent Dictator For Life • Now "Technical Steering Committee" is being established (in this November) as the replacement for the BDFL role • A single company cannot hold more than 1/3 seats Moby governance
  22. 22. 22 Copyright©2017 NTT Corp. All Rights Reserved. • Some components under the Moby umbrella belong to other organizations • containerd, Notary: CNCF (Cloud Native Computing Foundation) • InfraKit is also proposed to CNCF • runc: OCI (Open Containers Initiative) • Moby Technical Steering Committee will help cross-project and cross-organization collaboration Moby Project, CNCF, and OCI
  23. 23. 23 Copyright©2017 NTT Corp. All Rights Reserved. • Maintainers (aka "committers" in other FLOSS communities) • Can approve other contributors' pull requests (with 2 LGTMs usually) • Can manage GitHub issues • Can add and remove other maintainers (with 66% approval vote) • Elected from active contributors who: • Send pull requests • Bug fix • Enhancement • New feature • Review other contributors pull requests • Triage GitHub issues Who are maintainers and how they are elected
  24. 24. 24 Copyright©2017 NTT Corp. All Rights Reserved. • Began contribution to Docker in December, 2015 • I was working on some fault injection tool (github.com/osrg/namazu), and got stuck in "false ZooKeeper bug" due to AUFS hang-up: #18180 • This is not a bug of Docker but mainly tracked in Docker community • Became a Docker maintainer in November, 2016 • Docker  Moby in April, 2017 • (I think) Mainly contributed to filesystem issues • Both AUFS and overlayfs have some stability and compatibility issues • Also proposed some new features • `docker network prune` (merged): #27525 • introspection mount (procfs-like stuff for containers. unmerged yet): #24893, #26331 • TCP port forwarder (withdrawn): #26365 • ... How I became a Docker/Moby maintainer
  25. 25. 25 Copyright©2017 NTT Corp. All Rights Reserved. How I became a Docker/Moby maintainer JVM processes in Docker were extremely unstable due to an AUFS issue. The issue was very critical for many users, but very hard to debug because of non-determinism.
  26. 26. Actually, it was not me who firstly identified the cause of the issue (Also, I didn't wrote the patch to fix the issue in the fact) But I made some demonstration tools for confirming the cause of the issue
  27. 27. I made some reports to AUFS community, and the AUFS maintainer (not me) fixed the issue Created "cheat sheet" for Linux distribution-specific workarounds Encouraged Linux distributors to apply the AUFS patch
  28. 28. Created comprehensive list of filesystem issues (github.com/AkihiroSuda/issues-docker)
  29. 29. Began code contribution (Details are out of scope of this talk) Discussed with maintainers in person (Using mobile phones!)
  30. 30. http://www.publickey1.jp/blog/16/docker_enginenttdocker.html "Suda at NTT became a maintainer of Docker Engine"
  31. 31. 32 Copyright©2017 NTT Corp. All Rights Reserved. • Initial maintainer from the beginning of the project (2017 summer) • Initially I proposed DAG-based builder (but without idea of LLB) • Coincidently, Tõnis Tiigi (Docker, Inc.) was planning similar but even better idea, which turned into BuildKit • Tõnis invited me to an initial maintainer of BuildKit  How I became a BuildKit maintainer
  32. 32. 35 Copyright©2017 NTT Corp. All Rights Reserved. • Began contribution in December, 2016 • Became a maintainer in September, 2017 • Mainly contributed to filesystem and image issues • Though contribution to Docker / Moby, I found filesystem issues are hard to maintain • I needed to reform containerd interfaces and data formats for my experimental OCI Image extension • FILEgrain: transport-agnostic, fine-grained content-addressable container image layout (github.com/AkihiroSuda/filegrain) How I became a containerd maintainer
  33. 33. 36 Copyright©2017 NTT Corp. All Rights Reserved. • Collaborativeness (the most important) • Comprehensiveness • Issue analysis, Bug-fix, enhancement, feature addition... • But when you plan to add a big feature, please coordinate with maintainers in GitHub issue or Slack before opening a PR! • Continuity • One-shot contribution is always welcomed, but maintainership requires continuous activity • Number of git commits and LOCs are not so important And how you can become maintainers! (my personal view)
  34. 34. 37 Copyright©2017 NTT Corp. All Rights Reserved. • Moby engine • Recently integrated containerd v1.0 runtime. We need to make sure there is no regression. • The next step is to integrate containerd v1.0 snapshot subsystem. (Much harder for compatibility) • BuildKit • Dockerfile2LLB compiler is not stable. Testing is highly welcomed. • Design for distributed mode is still under discussion. • containerd • Adding tests and performance optimizations are welcomed. • And more! Good chances to contribution (my personal view)
  35. 35. 38 Copyright©2017 NTT Corp. All Rights Reserved. • The Moby Project is getting more collaborative • You can contribute and become a maintainer as well! Recap https://blog.docker.com/2017/04/introducing-the-moby-project/

×