BuildKit is a new toolkit for building Docker images that addresses performance and design shortcomings of the original Docker build system. It converts build processes into an efficient, cacheable, and parallelizable intermediate format called LLB. BuildKit can be used standalone with buildkitd and buildctl or embedded in Docker. It provides significant speed improvements over the original builder, especially for incremental builds using cached layers.

1. An introduction to BuildKit
Arnaud Porterie - @icecrime - 2019-04-04

2. Arnaud Porterie - @icecrime
Mandatory introduction
● Arnaud Porterie - @icecrime
● VP Engineering at Veepee
○ Well, of course we are hiring! Thank you for asking!
● Previously Senior Engineering Manager at Docker, Inc.
○ Maintainer of the Docker Engine and managing the Engine team
○ Running most of the open source community activities
● Member of Moby Project Technical Steering Committee

3. Arnaud Porterie - @icecrime
The importance of Docker build
● Massively relied on feature
○ Extremely important (and groundbreaking at the time) for developers
○ Heavily participated in Docker widespread adoption
● Many engineering lessons to learn from its story
○ How the MVP turned out to be a game-changer
○ How to redesign a feature used by millions and enable innovation
○ How to deliver significant improvements in a backward compatible way

4. Arnaud Porterie - @icecrime
● Purpose: build a Docker image from a Dockerfile in a repeatable way
Docker Build

5. Arnaud Porterie - @icecrime
Context
The client packages the
source directory together
with the Dockerfile (the
build context) and sends it
to the daemon.
Command: docker build
API endpoint: /build
Dockerfile
Daemon parses the
Dockerfile and executes
instructions in order.
Each instruction produces a
layer in the final image
which is kept as cache for
ulterior builds.
Image
The final image is
generated and tagged as
requested by the client.
Docker Build: original design

6. Arnaud Porterie - @icecrime
Docker Build: original design shortcomings
● Suboptimal performance
○ All the context is sent at every build invocation
○ Dockerfile evaluation is sequential in nature

7. Arnaud Porterie - @icecrime
Docker Build: original design shortcomings
● Suboptimal performance
○ All the context is sent at every build invocation
○ Dockerfile evaluation is sequential in nature
● Impractical cache management
○ Build cache implemented as untagged images

8. Arnaud Porterie - @icecrime
Docker Build: original design shortcomings
● Suboptimal performance
○ All the context is sent at every build invocation
○ Dockerfile evaluation is sequential in nature
● Impractical cache management
○ Build cache implemented as untagged images
● Difficult to evolve
○ Syntax is tied to a particular daemon version
○ Lots of opinions and feature requests (IF, INCLUDE, mounts, …)
○ Dockerfile syntax essentially is the API

9. Arnaud Porterie - @icecrime
Introducing BuildKit
BuildKit is a toolkit for converting source code to build artifacts
in an efficient, expressive, and repeatable manner.
https://github.com/moby/buildkit
● Main author: Tõnis Tiigi (Docker maintainer and employee)

10. Arnaud Porterie - @icecrime
Introducing BuildKit
● BuildKit builds use a binary intermediary format called LLB
○ “LLB is to Dockerfile what LLVM IR is to C”
Dockerfile.v0
Dockerfile.vX
Custom Syntax
...
LLB BuildKit

11. Arnaud Porterie - @icecrime
Frontend
A frontend converts
arbitrary input (typically a
human readable description
of a build operation) into a
dependency graph
expressed as LLB.
LLB
The LLB representation
captures all the necessary
steps to produce the
desired build artifact.
Because it is a content
addressable directed
acyclic graph, it allows for
efficient caching and
parallelisation.
BuildKit
The LLB is evaluated and
necessary context files are
lazily requested to the
client as required.
The build artifact is
outputted using the
specified exporter, for
example as Docker image.
Introducing BuildKit: design overview

12. Arnaud Porterie - @icecrime
Using BuildKit: standalone
● BuildKit is usable as a standalone daemon (buildkitd)
○ Dependent on runc for execution
○ Exposed over gRPC and through a user-friendly CLI (buildctl)
buildctl
Docker for Mac
buildkitdgRPC

13. Arnaud Porterie - @icecrime
Demo: buildkitd and buildctl

14. Arnaud Porterie - @icecrime
What have we seen?
● The user experience of buildkitd and buildctl
● The new concepts and specificities
○ Frontends (using the dockerfile.v0 frontend)
○ Outputs (using the built-in docker exporter)
○ Local context passing
● Using built-in support for OpenTracing

15. Arnaud Porterie - @icecrime
Using BuildKit: embedded
● Embeddable as a library, as it is in Docker
○ Since 18.06 as an experimental feature
○ Since 18.09 as an opt-in feature (DOCKER_BUILDKIT=1)

16. Arnaud Porterie - @icecrime
Demo: using BuildKit in Docker 18.09

17. Arnaud Porterie - @icecrime
What have we seen?
● Using BuildKit through Docker without any change
○ Simply export DOCKER_BUILDKIT=1
● Allowing additional frontends as docker images
○ Select a frontend with the #syntax=registry/user/repo:tag meta-directive
● Benefiting from all BuildKit improvements under the hood

18. Arnaud Porterie - @icecrime
Scenario: clean build
Benchmark repository: github.com/moby/moby
Numbers taken from Tõnis Tiigi’s presentation at DockerCon 2018
https://dockercon2018.hubs.vidyard.com/watch/nQSZHgPuoUDT1736dQvXxD
Performance numbers
x2

19. Arnaud Porterie - @icecrime
Scenario: build with up-to-date cache
Benchmark repository: github.com/moby/moby
Numbers taken from Tõnis Tiigi’s presentation at DockerCon 2018
https://dockercon2018.hubs.vidyard.com/watch/nQSZHgPuoUDT1736dQvXxD
Performance numbers
x7.2

20. Arnaud Porterie - @icecrime
Scenario: incremental build with code change
Benchmark repository: github.com/moby/moby
Numbers taken from Tõnis Tiigi’s presentation at DockerCon 2018
https://dockercon2018.hubs.vidyard.com/watch/nQSZHgPuoUDT1736dQvXxD
Performance numbers
x2.5

21. Arnaud Porterie - @icecrime
And all the rest...!
● Rootless execution
● Multi-platform support
● Sharable build cache
● Multi-format export
● Automatic storage management (GC)
● No side effects
● ...

22. Thanks! Questions?
About this presentation, about Veepee,
about Docker, about life, the universe, and everything, ...
Arnaud Porterie - @icecrime