This document discusses how Docker and Jenkins can be used together for continuous integration. It describes how Docker provides repeatable environments that reduce unintended variations, and how Jenkins pipelines allow for frequent integration that detects issues earlier. Specific examples show how Docker images can standardize build and test environments, while multi-branch Jenkins pipelines allow testing proposed changes before merging.

1. Docker & Jenkins Pipeline
Continuous Integration – Mark Waite
Mile High Agile 2018

2. The
Intersection
Accident
Problem
Accidents frequently happen
at intersections

3. The Software Integration Problem
Merge
Team 1
Team 2
Team 3
• Integration frequently fails
• Conflicting changes
• Misunderstood requirements
• Missed capabilities
• Larger integrations fail more

4. Integration surprises
• Frequently surprised
• That should have worked
• Works on my machine
• Why would anyone do that?
• That’s a feature, not a bug
• …

5. The
Intersection
Accident
Solution
Roundabouts have
fewer accidents and
less severe accidents

6. The Software Integration Solution
• Integrate frequently • Combine changes often
• Evaluate changes often
• Detect issues early
Merge
Team 1
Team 2
Team 3
Merge
Team 1
Team 2
Team 3
Merge
Team 1
Team 2
Team 3
Merge
Team 1
Team 2
Team 3
Merge
Team 1
Team 2
Team 3

7. Implementation: Meet the Red Team
• Goal: Deliver great software from a happy team
• Integrate
• Build
• Test
• Deliver
• Find failures sooner, faster

8. Red Team Uses Jenkins Pipeline
• Jenkins Continuous Integration
• Open source
• Full-featured
• Scalable
• Powerful
• Easy to configure

9. Red Team Demo – Jenkins Pipeline
• Red Team finds team level problems sooner
• Conflicts discovered earlier
• Missed requirements identified sooner
• Integration is less painful because it is more frequent

10. Meet Ms. Green and Mrs. Black
• Red team members
• Ms. Green
• Mrs. Black
• Find problems sooner
Merge
Ms. Green
Red Team
Mrs. Black
What if I could find
problems before merge?

11. Check Proposed Changes
• Detect conflicts before merge
• Run tests before code review
• Try Pipeline improvements
Merge
Ms. Green
Red Team
Mrs. Black
What if I could find
problems before merge?

12. Red Team Uses Jenkins Multi-branch Pipeline
• Multi-branch Pipeline
• Domain specific language (DSL)
• Developer defined and modified
• Stored with the source code
• Job per branch
• Jobs created automatically
• Jobs destroyed automatically
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'make package'
}
}
}
}

13. Multi-branch Jenkins Pipeline Demo
• Ms. Green and Mrs. Black finds problems sooner
• Private merge of proposed change
• Merge conflicts detected
• Tests run before code review

14. The
Unintended
Variation
Problem
Surprises aren’t always happy
This Photo by BlarSnow is licensed under CC BY-SA

15. Variation in Software Development
• Good: Intentional Variation
• Platform specific build
• Multi-platform tests
• Web browser tests
• Low bandwidth tests
This icon by Sean Martell is
licensed under CC BY-SA

16. Variation in Software Development
• Bad: Unintentional Variation
• Wrong build tools
• Outdated dependencies
• Contaminated tests
• Full file systems

17. Reducing Unintentional Variation with Docker
• Docker Containers
• Repeatable
• Isolated file system
• Isolated process - lightweight
• Open source

18. Building with Docker – Apache Maven
• Java 7, latest Maven 3
• docker run maven:3-jdk-7 mvn –v
• Java 8, latest Maven 3
• docker run maven:3-jdk-8 mvn –v
• Java 9, latest Maven 3
• docker run maven:3-jdk-9 mvn –v
• Java 10, latest Maven 3
• docker run maven:3-jdk-10 mvn -v
• Java 11, latest Maven 3
• docker run maven:3-jdk-11 mvn -v
See https://hub.docker.com/_/maven/ for more command line details

19. Building with Docker – Gradle
• Java 8, Gradle 4.7.0
• docker run gradle:4.7.0-jre8 gradle –-version
• Java 9, Gradle 4.7.0
• docker run gradle:4.7.0-jre9 gradle –-version
• Java 10, Gradle 4.7.0
• docker run gradle:4.7.0-jre10 gradle –-version
See https://hub.docker.com/_/gradle/ for more command line details

20. Define Your Own Toolchain
• Dockerfile specifies
• Operating system (CentOS, Debian, Red Hat, SUSE, Ubuntu, …)
• Packages
• File system contents
• Volume mount points (reuse, persistence across run, etc.)

21. Build with Docker
• Repeatable builds in precisely defined environment
• Specify exact tool versions in Dockerfile
• Clean file system is easy
• Examples
• I need gcc 6.1 and go 1.8, they need gcc 4.3 and java 1.8
• Both can be satisfied by each compiling from a purpose-built docker image
• I need specific library versions, they need different versions
• Both satisfied by their own Dockerfile

22. Docker for Unit and Integration Tests
Fewer environment surprises

23. Docker for Better Unit & Integration Testing
• Docker provides more consistent runtime
• Reduce differences between test environments
• Examples
• Unit tests run in Docker start from a clean state
• Test environment cleanup is simpler

24. Specific Example – Jenkins Test Harness
• Jenkins test harness uses a specific named temporary directory
• Multiple users on same machine cannot use same directory
• Docker file system isolation solves it

25. Docker for End to End Tests
Rapid configuration of complex scenarios

26. Docker for Better End to End Testing
• Docker automates complicated configuration
• Repeatable test environment creation
• Compose multi-component test configurations
• Spend less time configuring test environments
• Spend more time testing
• Examples:
• Jenkins LTS with plugins
• https://github.com/MarkEWaite/docker-lfs/tree/lts-with-plugins

27. Docker for Bug Reports
Repeatable environments in the bug report

28. Docker for Better Bug Reports
• Simplify long sequences of pre-conditions in Dockerfile
• Focus the report on specific bug details
• Simplify setup for bug fixer
• Some examples:
• JENKINS-41948 – noSuchMethodError in scanning
• JENKINS-42204 – NPE due to GitSCMSource change
• JENKINS-46724 – Illegal reflective access from remoting

29. Docker & Jenkins Together
• Jenkins for Pipeline as code
• Docker for infrastructure as code