This document summarizes a presentation about using Docker in continuous integration systems. It describes some initial challenges faced when first using Docker, such as images becoming large over time and dependencies on external resources. It then outlines improvements made such as creating common base images, reviewing images, restricting external access, and monitoring Docker usage and failures. The presentation emphasizes that Docker can help stabilize CI pipelines if used properly but also requires ongoing learning and challenges to address.

1. Alexander Akbashev
RootConf | June 06, 2017
Docker in Continuous
Integration

3. Agenda
01. Context
02. Very naive time
03. Start Project Docker
04. Something wend wrong
05. Chaos
06. Still not perfect
07. New day - new challenges
08. Morale

4. Context
What does CI System means
for us

5. Context
• Self-hosted Jenkins
• Cloud based
• Tons of configured project
• All changes are going through pre-commit validation
pipelines
• Different platform and different products
• Our users are our colleagues

6. Day 0
Very naive time
Everything was so simple… to break

7. Mutable host
- Yes, I really want to change /etc/
hosts for my integration test
- …

8. One agent - one package
You don’t want to mix some stuff on
one host
• one version of python
• one version of system library
• one version of everything

9. New package - new pain
- Oops, I didn’t know that
libXYZ-1.2 comes with new API
compare to libXYZ-1.1

10. Painful verification process
To test new package you need:
• new node
• new label
• cloned job (multiple jobs?)
• … but it’s used in 100+ projects…

11. Bad utilization
Some nodes are needed only in
rare cases
I want to test only on CentOS 5! It’s
my favourite production OS!

12. Download to build
Java, Python, Ruby, nodeJs tends
to download staff on-the-fly

13. External dependency
It’s not safe to query Internet in
pre-commit
> Could not resolve commons-io:commons-io:2.4.
> Could not get resource https://jcenter.bintray.com/commons-io/
commons-io/2.4/commons-io-2.4.pom
> Received status code 500 from server: Internal Server Error

14. Day 0
Start Project Docker!

15. Docker is so awesome!
• We can control docker content
• CI builds are reproducible locally
• Builds do not affect each other
• We can cache stuff in docker

16. Docker
Small intro

17. Dockerfile
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get -y install
gcc ccache cppcheck

18. Docker commands
docker build
docker push
docker pull
docker run my_image

19. Day 1
Something went wrong
Our expectations didn’t meet reality

20. New image - new pain
docker pull my_product:latest
docker pull test:latest
sha256:12d30ce421ad530494d588f87b2328ddc3ca
Status: Downloaded newer image for test:latest
… 5 minutes later…
docker pull test:latest
sha256:01a21daf124543213d1a0514523612345198
Status: Downloaded newer image for test:latest

21. Testing new images in pre-commit
• Versioning is mandatory (no
“latest” anymore!)
• Actual version is defined in config
file (pre-submit testable now)

22. Timeouts
“docker pull” times out
docker pull my_image:1.0
b6f892c0043a: Downloading [===========> ] XX MB/YY MB
55010f332b02: Downloading [============> ] XX MB/YY MB
2955fb827c93: Downloading [=============> ] XX MB/YY MB

23. Timeouts
New feature in Timeout Plugin ->
Step with timeout
All images are backed in AMI itself

24. Docker stucks
--rm doesn’t guarantee much
docker run ——rm my_image:1.0 /bin/bash
do_work.sh

25. Docker stucks: trap for docker!
Add trap for $DOCKER_TAG
docker run ——rm —name=${BUILD_TAG} my_image:1.0 /bin/bash
do_work.sh
trap "{
docker ps -aq --filter name=$BUILD_TAG |
xargs --no-run-if-empty docker rm -f --volumes || true;
} &> /dev/null" EXIT

26. Lightweight docker image?!
docker images have trend to
become bigger and bigger
from 500 MB… up to 3.2 GB

27. Let’s share common stuff
Base images

28. Let’s share common stuff
Base images
FROM base:1.0
RUN apt-get install
gcc-4.9 python
FROM base:1.0
RUN apt-get install
gcc-4.9 nodejs

29. Let’s share common stuff
Base images
FROM ubuntu:16.04
RUN apt-get install gcc-4.9
FROM base:1.0
RUN apt-get install python
FROM base:1.0
RUN apt-get install nodejs

30. Day 2
Chaos
Duplicated code is not worst duplicate problem

31. Docker image should do one thing only
Need something? Just put to the
basic image and enjoy!

32. Docker image should do one thing only
Split base image to test and build
images

33. Mandatory reviews
Too many images -> too easy to
copy/paste

34. Mandatory reviews
Restrict permissions to repository

35. Too many projects
Hard to review:
• Explain same things multiple
times
• Argue

36. Simplify review process
Static analyzes:
• versions
• number of layers
• etc.

37. Day 3
Still not perfect
But already much better!

38. Images are still big
Hard to explain best practices each
time:
• no-install-recommends
• rm -rf /var/lib/apt/lists
• apt-get clean

39. etc/apt/apt.conf.d/docker-no-cache
Dpkg {
# Don't keep copies of packages after download
Cache "";
Cache::archives "";
Post-Invoke { "rm -rf /var/lib/apt/lists"; };
}
APT {
Install-Recommends "false";
}
DSELECT::Clean "always”;

40. etc/apt/apt.conf.d/docker-no-cache
Dpkg {
# Don't keep copies of packages after download
Cache "";
Cache::archives "";
# Always delete list of packages
Post-Invoke { "rm -rf /var/lib/apt/lists"; };
}
APT {
Install-Recommends "false";
}
DSELECT::Clean "always”;

41. etc/apt/apt.conf.d/docker-no-cache
Dpkg {
# Don't keep copies of packages after download
Cache "";
Cache::archives "";
# Always delete list of packages
Post-Invoke { "rm -rf /var/lib/apt/lists"; };
}
APT {
Install-Recommends "false";
}
DSELECT::Clean "always”;

42. etc/apt/apt.conf.d/docker-no-cache
Dpkg {
# Don't keep copies of packages after download
Cache "";
Cache::archives "";
# Always delete list of packages
Post-Invoke { "rm -rf /var/lib/apt/lists"; };
}
APT {
Install-Recommends "false";
}
DSELECT::Clean "always”;

43. External dependency
It’s not safe to query Internet in
pre-commit
Still.

44. Restrict external resources
--net=none
—net=container:$DOCKER_TAG
are only allowed in pre-submit tests
And a little bit more in builds

45. Restrict resources
All tests must be equal
• no thread starvation
• no oom-killer
• prevent regressions

46. Restrict external resources
standard profiles and recommend
values:
--cpus
--memory

47. Docker registry returns 500
Docker registry is down
• everything is blocked
• nothing is really needed from
registry

48. Docker registry returns 500
Don’t do `docker pull` if it’s not
needed
• check existing images
• exclude “:latest”
• not our registry

49. Day X
New day - new challenges
Learn fast

50. Monitoring
We monitor:
- uptime for `docker run`
- parameters
- infra issues

51. Monitoring
Docker issues per day

52. Monitoring
FluentD Plugin
Build Failure Analyzer Plugin
Groovy Event Listener Plugin

53. Morale

54. Morale
• There is no silver bullet
• Consider Dockerfile as a source code
• Build monitoring for your CI
• Docker really helps to stabilize CI
pipelines

55. Thank you
Contact
Alexander Akbashev
HERE
Invalidenstraße 116
10115 Berlin
GitHub: Jimilian
alexander.akbashev@here.com