1.
Continuous Delivery
with Containers
Nick Gauthier @ngauthier
Developer @codeship

2.
What is Continuous Delivery?

3.
Continuous Delivery (CD) is a software
engineering approach in which teams keep
producing valuable software in short cycles
and ensure that the software can be reliably
released at any time.
Chen, Lianping (2015). "Continuous Delivery: Huge Benefits, but
Challenges Too". IEEE Software 32

4.
“valuable software”
I think we can all agree here!

5.
“short cycles”
ship early, ship often!

6.
“reliably released at any time”
don’t break the build!

7.
What are our needs?

8.
Core CD System Attributes
1. Speed
2. Parity
3. Reliability
4. Flexibility

9.
Speed: The CD Cycle
1. Check out code
2. Run tests
3. Build deployment artifact
4. Ship deployment artifact
5. Start deployment artifact

10.
Deployment Artifact?
1. Binary
2. “slug”
3. WAR
4. VM
5. AMI
6. Container
7. tar
8. etc...

11.
Parity
“it worked for me”

12.
Reliability
“try running it again…”

13.
Flexibility
“I need X”

14.
What are our CD options?

15.
PaaS
1. Speed ✔
2. Parity ✘
3. Reliability ✔
4. Flexibility ✘

16.
Config Management
(puppet, chef, ansible, salt)
1. Speed ✘
2. Parity ✘ (✔ if virtualized locally)
3. Reliability ✔?
4. Flexibility ✔

17.
VM
1. Speed ✘
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

18.
Container
1. Speed ✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

19.
Docker
1. Speed ✔✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

20.
Why Docker?

21.
Cache layers
rebuilds are fast!

22.
Dockerfile
simple, understandable, centralized

23.
Official + Community Images
don’t reinvent the wheel
unless you want to

24.
Hosting Choices
low lock-in, many new options
including diy

25.
Docker CD
What are our options?

26.
#1 Local

27.
1. Run tests
2. Build images
3. Push images
4. Trigger host pull
LOCAL

28.
1. Run tests (hopefully? same env?)
2. Build images (slow, cpu + net)
3. Push images (slow, race?)
4. Trigger host pull (race?)
LOCAL

29.
#2 Hosted Solution

30.
HOSTED SOLUTION
1. Build CI Container
2. Run tests in CI Container
3. Build Production Container
4. Push Production Container
5. Trigger host pull

31.
HOSTED SOLUTION
1. Build CI Container (caching? multiple?)
2. Run tests in CI Container (prod parity?)
3. Build Production Container (build artifacts? mult?)
4. Push Production Container
5. Trigger host pull

32.
#3 Custom Solution

33.
Custom Solution
???

34.
What are we missing?

35.
Centralized CD
Our goals

36.
Parity

37.
CI Container == Prod Container
PARITY

38.
CI Container ~= Prod Container
PARITY

39.
Same FROM
PARITY

40.
Same Packages
(+ a few)
PARITY

41.
Linked Services
(not monoliths)
PARITY

42.
Hosting “Stack” = Docker
PARITY

43.
Orchestration

44.
Running tasks + linking services
ORCHESTRATION

45.
Docker compose run
ORCHESTRATION

46.
Multiple Compositions?
(unit + integration + acceptance)
ORCHESTRATION

47.
Speed

48.
Speed
(caching)

49.
Base Images
SPEED + CACHING

50.
Layers from “last time”
SPEED + CACHING

51.
Layers from “last time” = registries!
SPEED + CACHING

52.
Parallelism

53.
Parallelism
@#$%!

54.
Docker isolation!
PARALLELISM

55.
Ochestration
PARALLELISM

56.
Ochestration of
Orchestration
PARALLELISM

57.
Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
Build P1 Build P2
Push P1 Push P2
Deploy P1 Deploy P2
PASS!

58.
Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
Build P1 Build P2
Push P1 Push P2
Deploy P1 Deploy P2
PASS!
DB Cache
API APP + Tests
Compose

59.
Let’s build it!
a.k.a. lessons learned a.k.a. what went wrong :)

60.
Step 1: Define Containers

61.
Official Language Containers

62.
Official Language Containers
if you can

63.
Official Language Containers
dictate O.S. (often debian)

64.
Official Language Containers
big, pull = slow, lots of disk
(esp w/ docker machine + vbox)

65.
Containers for CI

66.
Containers for CI
code changes frequently

67.
Containers for CI
code changes frequently
=
lots of rebuilds

68.
Containers for CI
dependencies
=
majority of time

69.
Containers for CI
dependencies
=
change occasionally

70.
Hack #1: Layer Dependencies

71.
CODESNIPPET
# Classic
ADD .
RUN make deps / bundle / ...
CMD [“run my tests”]
# Layered
ADD Godeps + Makefile / Gemfile + .lock / …
RUN make deps / bundle / …
ADD .
CMD [“run my tests”]

72.
Dependency Layer
Cached on “manifest” files

73.
Dependency Layer
Great on CI
Fantastic on dev machine

74.
Hack #2: CI Registries

75.
Pull app:branch before CI

76.
Pull app:branch before CI
Primes all layers from last time

77.
Push app:branch after CI
Updates layers in registry

78.
Run registry near CI
You can use docker hub
but you don’t have to

79.
Production Containers
Same as CI (-pkgs)

80.
Production Containers
Layering + Common bases help here too!

81.
Production Containers
Huge = slow push = slow host pull
= slow deploy

82.
Hack #3: Static Containers

83.
CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM debian:latest
RUN apt-get update &&
apt-get install some-dyn-lib
ADD prod-binary /
CMD [“/prod-binary”]

84.
Static Container
1. Minimal Packages
2. No language packages
3. Only works for compiled languages
(but also gains for vm languages)

85.
Hack #3.a: Scratch Containers

86.
Scratch?
empty tarball

87.
CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM scratch
ADD prod-binary /
CMD [“/prod-binary”]

88.
Hilariously Tiny
single digits mb

89.
Scratch Containers
1. Only works for statically compiled binaries
2. Worst CI + Prod parity
3. Near Instant push+pull+boot+deploy

90.
Minimal Guests

91.
Minimal Guest
1. Bare O.S.
2. Minimal toolchain
3. Minimal Packaging + DIY
4. Small containers for the dynamic crowd

92.
Promising Project
gilderlabs/docker-alpine

93.
CODESNIPPET
# 16mb mysql container
FROM gliderlabs/alpine:3.1
RUN apk --update add mysql-client
ENTRYPOINT ["mysql"]

94.
alpine is just one
hopefully we will have many options!

95.
Orchestration

96.
docker compose

97.
CODESNIPPET
# docker-compose.yml
app:
build: .
links:
- db
ports:
- "8000:8000"
db:
image: postgres
# running ci
docker compose run app make test

98.
docker compose problems
Specify different Dockerfile

99.
docker compose problems
Can’t tell when services are running

100.
docker compose problems
ENV encryption

101.
docker compose problems
only 1 at a time

102.
Docker CI
manually run and link :(

103.
CODESNIPPET
docker run -d --name u1-postgres postgres
sleep magicnumber # :(
docker run --name u1-app --link u1-postgres:postgres
app make test
docker kill u1-postgres

104.
Parallelism

105.
Parallelism
Manually run and link and namespace

106.
Parallelism
run your own agent
split work
monitor containers

107.
Parallelism
@#$%!

108.
Parallelism
we’ll figure it out!

109.
Docker Swarm

110.
Docker Swarm
if all we use to run CI is Docker,
Swarm gives us machine parallelism

111.
Deploy

112.
Deployment Styles
1. Push code + build on host
(e.g. Deis, EB, …)
2. Build + push image + trigger host pull
(e.g. EB, ECS, GAE, …)
3. Build + crossbuild for host + push
(e.g. Heroku docker release)

113.
Build on Host
1. Removes burden from CI
2. Must be single-Dockerfile app
3. Must be deployable in isolation
4. Can’t leverage local cache

114.
Build + Push + Trigger Pull
1. Use local cache
2. CI must do production builds
3. Push + Pull adds time
4. Can push many and launch many

115.
Build + Crossbuild
1. ???

116.
Hack #4: use the same registry

117.
Use the same registry
1. Put registry near CI
2. Push CI layers to registry a la Hack #2
3. Minimal layers pushed with small changes
4. Put registry near host
5. Minimal pull onto host with small changes

118.
Keep common bases hot

119.
Try to use similar bases

120.
CI + CD + Hosting
keep them close

121.
CI + CD + Hosting
keep them close
(a.k.a. Amazon all the things)

122.
What’s next?

123.
Better parallel tooling

124.
Better development environments
(non virtualized drivers)

125.
Better machine provisioning
(docker machine as a library)

126.
Codeship Private Beta
super docker funtimes!
chat with me or visit our booth

127.
Thanks!

128.
Questions?
also:
@ngauthier