I'd like to talk to everyone about someone we often neglect. That someone keeps our code flowing into production, runs all those automated tests, does things that we might not even remember to deploy to all our environments. That someone is our build servers. They tirelessly work to get our changes to our end users and we do neglect them. We don't treat our build servers like production servers, we hack them, we play with them, we mess up the environments. We may have several agents with different runtimes and gems and patch levels and who knows what. We can do better. In this talk I'd like to take you through something I've been working on over the last couple of months: Containerising your build. The tool I'll be using for this is Docker, but it's achievable with any VM or container technology. I'll take you through going from a working build on local to building local inside a container and then to moving that build to a build server, explaining what how and why as I go.

1. R A P I D
REPRODUCIBLE BUILDS
treating everything like production
Dewald Viljoen
@dewald_v

2. who am i?
2

3. HOW'S THIS TALK GOING TO WORK
and what are we actually talking
about?
3

4. "a server is a server is a server, there
really are no test servers, build
servers, dev servers and production
servers... and spoons of course"
- @dewald_v
4

5. DOCKER IN 5
MINUTES
a quick and dirty look into images, containers and all that lies below
5

6. DOCKER IN 5 MINUTES - IMAGES
6
NGINX IMAGE
UBUNTU 14.04
ADD NGINX
ADD NGINX CONFIGLayer 3
Layer 2
Base (Layer 1)

7. DOCKER IN 5 MINUTES - CONTAINERS
7
NGINX CONTAINER
UBUNTU 14.04
ADD NGINX
ADD NGINX CONFIGLayer 3
Layer 2
Base (Layer 1)
CONTAINER R/W FILESYSTEM
Layer 4
(Only in the
Container)

8. NGINX-1
CONTAINER R/W
NGINX-2
CONTAINER R/W
NGINX-3
CONTAINER R/W
NGINX IMAGE
DOCKER IN 5 MINUTES - CONTAINERS AND LAYERS
8

9. DOCKER IN 5 MINUTES - CONTAINER VS VM
9
Image credit: Docker Inc.

10. what can we use containers for?
10

11. WORKSTATIONS
AND BUILD AGENTS
the darkest corners of the mind
11

12. what's on a workstation?
12

13. A TYPICAL WORKSTATION
13
WORKSTATION

14. A TYPICAL WORKSTATION
14
WORKSTATION
Basically everything
and anything you would
need to work on your
stack

15. what's on a build agent?
15

16. A TYPICAL BUILD AGENT
16
(SUPER) BUILD AGENT

17. A TYPICAL BUILD AGENT
17
(SUPER) BUILD AGENT
Basically everything
and anything you would
need to build your app

18. what's the alternative?
18

19. DOCKER-POWERED
BUILDS
applying images and containers to build pipelines
19

20. NODEJS BUILD IMAGE
THE BUILD IMAGE - NODE + GRUNT
20
UBUNTU 14.04
ADD NODE 4.2.4
ADD GRUNT
ADD PACKAGE.JSON &
NPM INSTALL
Base Image
Build Runtime
Build Config &
Dependencies
Build Tool

21. MAVEN BUILD IMAGE
THE BUILD IMAGE - JAVA + MAVEN
21
UBUNTU 14.04
ADD JDK 1.8
ADD MAVEN 3.3.3
ADD MAVEN SETTINGS.XML
Base Image
Build Runtime
Build Config
Build Tool

22. THE BUILD CONTAINER
22
MAVEN BUILD IMAGE
+
=
SOURCE CODE
BUILD ARTIFACT

23. JAVA-APP-BUILD-1
CONTAINER R/W
JAVA-APP-BUILD-3
CONTAINER R/W
JAVA-APP-BUILD-2
CONTAINER R/W
THE BUILD CONTAINER - MULTIPLIED
23
MAVEN BUILD IMAGE

24. ENVIRONMENTS, ENVIRONMENTS, ENVIRONMENTS...
24
LOCAL MACHINE BUILD AGENT
DOCKER REGISTRY

25. THE BUILD CONTAINER - EVEN MORE MULTIPLIED
25
MAVEN 3 MAVEN 2 NODEJS RUBY
JAVA-APP-
BUILD
JAVA-APP-
BUILD
NODE-APP-
BUILD
RUBY-APP-
BUILD
BUILD AGENT
ARTIFACT ARTIFACT ARTIFACT ARTIFACT

26. OF PIPELINES AND
CONTAINERS
how do we use this then?
26

27. enterprises build all the things...
27

28. THE SUPER BUILD AGENT
28
BUILD AGENT
• Complex
• Can be containerised but needs a lot of config
• Holds all build tools and configuration for any type
of build it can service
• Results in high build-contention
• Difficult to scale
• Pushes artifacts to repositories
MAVEN 3
MAVEN 2
JDK 1.5
JDK 1.6
JDK 1.7
RUBY
PYTHON
GRADLE

29. THE MICRO BUILD AGENT
29
BUILD AGENT
• Very simple
• Can be easily containerised
• Holds build tools and configuration for only the
type of build it can service
• Easier to scale
• Can be managed with configuration as code
• Pushes artifacts to repositories
MAVEN 3
JDK 1.7

30. THE CONTAINER BUILD AGENT
30
BUILD AGENT
DOCKER
• Very simple
• Can be containerised it self (Docker in Docker)
• Doesn't have any build dependencies inside, only
docker
• Pulls build containers from a registry
• Simple to scale
• Pushes artifacts to repositories
• Basically a micro agent running only Docker

31. so what does a containerised build
pipeline look like?
31

32. PIPELINING CONTAINERS
32
Application build directly on agent
Failed due to environmental issues

33. PIPELINING CONTAINERS
33
Build Container
(Maven & JDK on Ubuntu)
Application build in container

34. PIPELINING CONTAINERS
34
Build Container
(Maven & JDK on Ubuntu)
Application Builds in containers

35. THE GO SCRIPT
not configuring the build on the server
35

36. what happens when the build
server gets killed?
36

37. BUILD CONFIGURATION
37
Backup your configuration
Configuration as code
Keep as little configuration on the server as possible
Keep build configuration next to the app it builds

38. THE GO SCRIPT
38
A Script for your build actions
Knows the commands to build your app
Keep it simple. Bash is a good start.
Everything that works on the build server should
work UNMODIFIED on the local environment.
Build server should just call this script
Use environment variables for environment
changes

39. PUTTING IT ALL
TOGETHER
demo time
39

40. live demo time...
(cross your fingers)
40

41. DEWALD VILJOEN
@dewald_v
dewaldv@thoughtworks.com
THANKS!
QUESTIONS?