Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
[KubeConUS2019 Docker, Inc. Booth] Distributed Builds on Kubernetes with BuildKit and Docker Buildx
1. Tibor Vass, Docker
Akihiro Suda, NTT
Akihiro Suda, NTT
Distributed Builds on Kubernetes
with BuildKit and Docker BuildX
1
2. Why build images on Kube?
2
Two kinds of motivation
1. CI/CD
2. Developer Experience
3. About us
3
Why build images on Kube?
3
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD
4. About us
4
Why build images on Kube?
4poor CPU, RAM, Wi-FI, battery
2. Developer Experience
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD
4
5. About us
5
Why build images on Kube?
5poor CPU, RAM, Wi-FI, battery
2. Developer Experience
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD
5
BuildX
6. Kubernetes driver for BuildX
6
$ docker buildx create
--driver kubernetes
--driver-opt replicas=3
--use
$ docker buildx build -t foo --load .
Set up BuildKit cluster
on Kube automatically
No YAML mess
No TLS certificate mess
7. Kubernetes driver for BuildX
7
$ docker buildx create
--driver kubernetes
--driver-opt replicas=3
--use
$ docker buildx build -t foo --load .
Same user experience
as legacy docker build
But specify --load to load
the result to local Docker
(or --push to push)
8. Rootless mode
8
• You don’t want to run privileged pods? Don’t
worry
• BuildKit pods can be executed as a non-root user
(UID 1000) without securityContext.privileged
− Protect the host from potential BuildKit vulns
− Friendly to PSP-restricted clusters
− Fake root is emulated for Dockerfile RUNs
docker buildx create … --driver-opt rootless=true
10. Caching
10
• Cache can be pushed to a registry as well as image
BK Pod
BK Pod
BK Pod
BuildX Registry
Request Cache
11. Caching
11
• Cache can be pushed to a registry as well as image
BK Pod
BK Pod
BK Pod
BuildX Registry
CacheRequest
12. Caching
12
• But remote cache might be slow compared to
the BuildKit daemons’ local cache
(/var/lib/buildkit in each of the Pods)
• Example:
− No cache: 2m50s
− Remote cache: 36s
− Daemon-local cache: 0.5s
13. Caching
13
Sticky mode allows
sticking a build request
to a specific Pod based
on the Dockerfile path
• Always hit cache
• But non-optimal LB
buildkitd-1
buildkitd-0
buildkitd-2
foo/Dockerfile
bar/Dockerfile
baz/Dockerfile
--driver-opt loadbalance=(sticky|random)
Circular hashing space
14. Parallelism
14
• Multiple Dockerfiles:
built in parallel using multiple nodes
• Multiple stages in a single Dockerfile:
built in parallel using a single node
− Future work: scatter stages across multiple
nodes in parallel
