SlideShare a Scribd company logo

[KubeConUS2019 Docker, Inc. Booth] Distributed Builds on Kubernetes with BuildKit and Docker Buildx

•
•
A
Akihiro Suda

1. Kubernetes can be used to distribute Docker builds across multiple nodes for improved performance and scalability using BuildKit and the new BuildX toolkit. 2. BuildX introduces a Kubernetes driver that allows builds to run on Kubernetes pods with no complex YAML or TLS configuration required, providing the same user experience as local Docker builds. 3. Builds can be executed without root privileges on Kubernetes for improved security, and the Kubernetes driver supports caching of build artifacts both locally on pods and remotely to a registry for faster rebuilds.

1 of 15
Downloaded 12 times
Tibor Vass, Docker Akihiro Suda, NTT Akihiro Suda, NTT Distributed Builds on Kubernetes with BuildKit and Docker BuildX 1
Why build images on Kube? 2 Two kinds of motivation 1. CI/CD 2. Developer Experience
About us 3 Why build images on Kube? 3 BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD
About us 4 Why build images on Kube? 4poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 4
About us 5 Why build images on Kube? 5poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 5 BuildX
Kubernetes driver for BuildX 6 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Set up BuildKit cluster on Kube automatically No YAML mess No TLS certificate mess
Kubernetes driver for BuildX 7 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Same user experience as legacy docker build But specify --load to load the result to local Docker (or --push to push)
Rootless mode 8 • You don’t want to run privileged pods? Don’t worry • BuildKit pods can be executed as a non-root user (UID 1000) without securityContext.privileged − Protect the host from potential BuildKit vulns − Friendly to PSP-restricted clusters − Fake root is emulated for Dockerfile RUNs docker buildx create … --driver-opt rootless=true
About us 9 Demo
Caching 10 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry Request Cache
Caching 11 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry CacheRequest
Caching 12 • But remote cache might be slow compared to the BuildKit daemons’ local cache (/var/lib/buildkit in each of the Pods) • Example: − No cache: 2m50s − Remote cache: 36s − Daemon-local cache: 0.5s
Caching 13 Sticky mode allows sticking a build request to a specific Pod based on the Dockerfile path • Always hit cache • But non-optimal LB buildkitd-1 buildkitd-0 buildkitd-2 foo/Dockerfile bar/Dockerfile baz/Dockerfile --driver-opt loadbalance=(sticky|random) Circular hashing space
Parallelism 14 • Multiple Dockerfiles: built in parallel using multiple nodes • Multiple stages in a single Dockerfile: built in parallel using a single node − Future work: scatter stages across multiple nodes in parallel
About us 15 github.com/docker/buildx

More Related Content

[KubeConUS2019 Docker, Inc. Booth] Distributed Builds on Kubernetes with BuildKit and Docker Buildx

  • 1. Tibor Vass, Docker Akihiro Suda, NTT Akihiro Suda, NTT Distributed Builds on Kubernetes with BuildKit and Docker BuildX 1
  • 2. Why build images on Kube? 2 Two kinds of motivation 1. CI/CD 2. Developer Experience
  • 3. About us 3 Why build images on Kube? 3 BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD
  • 4. About us 4 Why build images on Kube? 4poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 4
  • 5. About us 5 Why build images on Kube? 5poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 5 BuildX
  • 6. Kubernetes driver for BuildX 6 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Set up BuildKit cluster on Kube automatically No YAML mess No TLS certificate mess
  • 7. Kubernetes driver for BuildX 7 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Same user experience as legacy docker build But specify --load to load the result to local Docker (or --push to push)
  • 8. Rootless mode 8 • You don’t want to run privileged pods? Don’t worry • BuildKit pods can be executed as a non-root user (UID 1000) without securityContext.privileged − Protect the host from potential BuildKit vulns − Friendly to PSP-restricted clusters − Fake root is emulated for Dockerfile RUNs docker buildx create … --driver-opt rootless=true
  • 10. Caching 10 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry Request Cache
  • 11. Caching 11 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry CacheRequest
  • 12. Caching 12 • But remote cache might be slow compared to the BuildKit daemons’ local cache (/var/lib/buildkit in each of the Pods) • Example: − No cache: 2m50s − Remote cache: 36s − Daemon-local cache: 0.5s
  • 13. Caching 13 Sticky mode allows sticking a build request to a specific Pod based on the Dockerfile path • Always hit cache • But non-optimal LB buildkitd-1 buildkitd-0 buildkitd-2 foo/Dockerfile bar/Dockerfile baz/Dockerfile --driver-opt loadbalance=(sticky|random) Circular hashing space
  • 14. Parallelism 14 • Multiple Dockerfiles: built in parallel using multiple nodes • Multiple stages in a single Dockerfile: built in parallel using a single node − Future work: scatter stages across multiple nodes in parallel