Downloaded 42 times
Uploaded byDocker, Inc.
Docker Platform Internals: Taking runtimes and image creation to the next level with containerd and BuildKit
The document discusses the features and architecture of Docker's BuildKit and Containerd, emphasizing its smart client design and efficiency improvements for container building. Key highlights include its ability to manage multiple inputs/outputs, seamless integration with Kubernetes, and substantial performance gains through caching mechanisms. It also notes BuildKit's adaptability and the launch of experimental features, inviting contributions from the community.
More Related Content
Similar to Docker Platform Internals: Taking runtimes and image creation to the next level with containerd and BuildKit
![[KubeConEU] Building images efficiently and securely on Kubernetes with BuildKit](https://cdn.slidesharecdn.com/ss_thumbnails/buildingimagesefficientlyandsecurelyonkuberneteswithbuildkit-190521101727-thumbnail.jpg?width=640&height=640&fit=bounds)
Docker Platform Internals: Taking runtimes and image creation to the next level with containerd and BuildKit
- 1. Michael Crosby Engineer, DockerInc. Docker Platform Internals containerd and BuildKit Tonis Tiigi Engineer, Docker Inc.
- 2. ● Container runtimefor platform builders ● Scoped to a local node ● Stable GRPC API ● Native Kube support in 1.1 via CRI containerd
- 3.
- 4. ● Responsibilities splitbetween client and server ○ Image fetch vs image storage ○ i/o creation and redirection ○ Snapshot creation vs snapshot unpacking What is a smart client?
- 5. Why build asmart client?
- 6. ● Single daemonand API regardless of consumer ● Innovation kept at platform level ● Runtime kept boring and stable One daemon to rule them all...
- 7. What do youget?
- 8. ● Multi Tenancy ○Containerd namespaces What do you get?
- 9.
- 10. ● Direct I/Oaccess What do you get?
- 11.
- 12. ● Containers astypes What do you get?
- 13.
- 14. ● youtube-dl (mp4)| ffmpeg > audio.mp3 ● No I/O Copy https://github.com/crosbymichael/pipelines-dockercon Pipelines
- 15.
- 16.
- 17.
- 18. BuildKit New open source buildertoolkit Fast, safe, extensible
- 19. BuildKit Replacing builder backend inDocker daemon Experimental opt-in feature in v18.06
- 20. BuildKit Used by Moby /Docker CE - experimental support in v18.06 genuinetools/img (Jessie Frazelle) OpenFaas Cloud (Alex Ellis) containerbuilding/cbi (Akihiro Suda)
- 21. BuildKit Features No side-effects Quality,performance, extensibility Parallel requests support Concurrent solver Easy storage management (GC) Custom language support Custom Dockerfile features Incremental build context sending Persistent cache checksums Secure git cache roots HTTP dependency caching Multiple inputs/outputs Nested invocations Multi-format export OpenTracing support Signable image manifests Automatic build context detection Remote build cache Skip unused stages Per-host registry authentication
- 22. BuildKit Performance example Based ongithub.com/moby/moby Dockerfile, master branch. Smaller is better.
- 23. BuildKit Performance example Based ongithub.com/moby/moby Dockerfile, master branch. Smaller is better. Time for full build from empty state 2.0x faster Measured on DO 4vcpu droplet
- 24. BuildKit Performance example Based ongithub.com/moby/moby Dockerfile, master branch. Smaller is better. Repeated build with matching cache 7.2x faster Measured on DO 4vcpu droplet
- 25. BuildKit Performance example Based ongithub.com/moby/moby Dockerfile, master branch. Smaller is better. Repeated build with new source code 2.5x faster Measured on DO 4vcpu droplet
- 26. BuildKit Performance example Based ongithub.com/moby/moby Dockerfile, master branch. Smaller is better. Fresh build with --cache-from from remote source 9.1x faster Measured on DO 4vcpu droplet
- 27. BuildKit Embedding or usingdirectly Usable as a library, as a standalone daemon or with containerd
- 28.
- 29. BuildKit buildkitd & buildctl gRPCAPI Go client library rootless execution OpenTracing multi-worker model
- 30. BuildKit buildkitd & buildctl gRPCAPI Go client library rootless execution OpenTracing multi-worker model llb.Image("alpine:latest"). AddEnv("FOO", "BAR"). Dir("/dir"). Run(llb.Shlex("ls -l"), llb.AddEnv("key", "val")) r := Run(llb.Shlex("...")) r.AddMount("/alpine", llb.Image("alpine:latest"), llb.Reado
- 31. BuildKit buildkitd & buildctl gRPCAPI Go client library rootless execution OpenTracing multi-worker model
- 32. BuildKit buildkitd & buildctl gRPCAPI Go client library rootless execution OpenTracing multi-worker model
- 33. BuildKit buildkitd & buildctl gRPCAPI Go client library rootless execution OpenTracing multi-worker model containerd worker oci worker buildkitd
- 34.
- 35. Architecture Completely new design LLB:Low-level intermediate build format Side-effect free
- 36. Architecture LLB Format Describes areal dependency graph Intermediate machine generated format Content addressable protobuf Unlimited inputs and outputs per vertex
- 37. Architecture LLB Solver Find mostoptimal execution path Build up cache for repeated invocations - based on secure root node checksums - checksums by definition and content - smart remote registry cache sources Supports nested invocation
- 38. LLB Solver Built toscale https://github.com/tonistiigi/llb-gobuild
- 39. Architecture LLB Authoring Go clientlibrary https://godoc.org/github.com/moby/buildkit/client/llb llb.Image("alpine:latest"). AddEnv("FOO", "BAR"). Dir("/dir"). Run(llb.Shlex("ls -l"), llb.AddEnv("key", "val")) r := Run(llb.Shlex("...")) r.AddMount("/alpine", llb.Image("alpine:latest"), llb.Readonly)
- 40. BuildKit Frontends Convert any definitioninto LLB Distributable as container images Example: Dockerfile Anyone can create a new frontend Doesn’t need to be anything like Dockerfile
- 41. BuildKit Dockerfile frontend Automatically picksup all LLB concurrency/cache features Fully compatible with current Docker New: stages build in parallel New: unused stages skipped New: incremental build context updates New: detecting context paths used by ADD/COPY New: syntax directive
- 42. BuildKit Dockerfile experimental features Problem:Very hard to add new features All changes are forever Requires daemon upgrade Breaks users / CI that have not updated Solution: Load updates from images using frontends feature
- 43. BuildKit Dockerfile syntax directive -If present the remote frontend is used instead - User can choose to rely on experimental features … or use channel that is backwards compatible - Everyone automatically gets the correct version - Easy to develop because of limited scope # syntax = registry/user/repo:tag
- 44. Dockerfile syntax directive Example:RUN --mount Access files directly from build context or between the stages Persistent special cache mounts allow application specific caching.
- 45.
- 46. 33.1x 10.6x 8.1x Dockerfile syntax directive Example:RUN --mount moby/buildkit Dockerfile: time to binary rebuild after code change slower slower slower Measured on DO 4vcpu droplet
- 47. BuildKit Get involved Try DockerCE v18.06 and provide feedback - export DOCKER_BUILDKIT=1 - Release later this month https://github.com/moby/buildkit #buildkit Docker community slack
- 48.