Continuous Delivery
with Containers
Nick Gauthier @ngauthier
Developer @codeship

What is Continuous Delivery?

Continuous Delivery (CD) is a software
engineering approach in which teams keep
producing valuable software in short cycles
and ensure that the software can be reliably
released at any time.
Chen, Lianping (2015). "Continuous Delivery: Huge Benefits, but
Challenges Too". IEEE Software 32

“valuable software”
I think we can all agree here!

“short cycles”
ship early, ship often!

“reliably released at any time”
don’t break the build!

What are our needs?

Core CD System Attributes
1. Speed
2. Parity
3. Reliability
4. Flexibility

Speed: The CD Cycle
1. Check out code
2. Run tests
3. Build deployment artifact
4. Ship deployment artifact
5. Start deployment artifact

Deployment Artifact?
1. Binary
2. “slug”
3. WAR
4. VM
5. AMI
6. Container
7. tar
8. etc...

Parity
“it worked for me”

Reliability
“try running it again…”

Flexibility
“I need X”

What are our CD options?

PaaS
1. Speed ✔
2. Parity ✘
3. Reliability ✔
4. Flexibility ✘

Config Management
(puppet, chef, ansible, salt)
1. Speed ✘
2. Parity ✘ (✔ if virtualized locally)
3. Reliability ✔?
4. Flexibility ✔

VM
1. Speed ✘
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

Container
1. Speed ✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

Docker
1. Speed ✔✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔

Why Docker?

Cache layers
rebuilds are fast!

Dockerfile
simple, understandable, centralized

Official + Community Images
don’t reinvent the wheel
unless you want to

Hosting Choices
low lock-in, many new options
including diy

Docker CD
What are our options?

#1 Local

1. Run tests
2. Build images
3. Push images
4. Trigger host pull
LOCAL

1. Run tests (hopefully? same env?)
2. Build images (slow, cpu + net)
3. Push images (slow, race?)
4. Trigger host pull (race?)
LOCAL

#2 Hosted Solution

HOSTED SOLUTION
1. Build CI Container
2. Run tests in CI Container
3. Build Production Container
4. Push Production Container
5. Trigger host pull

HOSTED SOLUTION
1. Build CI Container (caching? multiple?)
2. Run tests in CI Container (prod parity?)
3. Build Production Container (build artifacts? mult?)
4. Push Production Container
5. Trigger host pull

#3 Custom Solution

Custom Solution
???

What are we missing?

Centralized CD
Our goals

Parity

CI Container == Prod Container
PARITY

CI Container ~= Prod Container
PARITY

Same FROM
PARITY

Same Packages
(+ a few)
PARITY

Linked Services
(not monoliths)
PARITY

Hosting “Stack” = Docker
PARITY

Orchestration

Running tasks + linking services
ORCHESTRATION

Docker compose run
ORCHESTRATION

Multiple Compositions?
(unit + integration + acceptance)
ORCHESTRATION

Speed

Speed
(caching)

Base Images
SPEED + CACHING

Layers from “last time”
SPEED + CACHING

Layers from “last time” = registries!
SPEED + CACHING

Parallelism

Parallelism
@#$%!

Docker isolation!
PARALLELISM

Ochestration
PARALLELISM

Ochestration of
Orchestration
PARALLELISM

Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
Build P1 Build P2
Push P1 Push P2
Deploy P1 Deploy P2
PASS!

Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
Build P1 Build P2
Push P1 Push P2
Deploy P1 Deploy P2
PASS!
DB Cache
API APP + Tests
Compose

Let’s build it!
a.k.a. lessons learned a.k.a. what went wrong :)

Step 1: Define Containers

Official Language Containers

Official Language Containers
if you can

Official Language Containers
dictate O.S. (often debian)

Official Language Containers
big, pull = slow, lots of disk
(esp w/ docker machine + vbox)

Containers for CI

Containers for CI
code changes frequently

Containers for CI
code changes frequently
=
lots of rebuilds

Containers for CI
dependencies
=
majority of time

Containers for CI
dependencies
=
change occasionally

Hack #1: Layer Dependencies

CODESNIPPET
# Classic
ADD .
RUN make deps / bundle / ...
CMD [“run my tests”]
# Layered
ADD Godeps + Makefile / Gemfile + .lock / …
RUN make deps / bundle / …
ADD .
CMD [“run my tests”]

Dependency Layer
Cached on “manifest” files

Dependency Layer
Great on CI
Fantastic on dev machine

Hack #2: CI Registries

Pull app:branch before CI

Pull app:branch before CI
Primes all layers from last time

Push app:branch after CI
Updates layers in registry

Run registry near CI
You can use docker hub
but you don’t have to

Production Containers
Same as CI (-pkgs)

Production Containers
Layering + Common bases help here too!

Production Containers
Huge = slow push = slow host pull
= slow deploy

Hack #3: Static Containers

CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM debian:latest
RUN apt-get update &&
apt-get install some-dyn-lib
ADD prod-binary /
CMD [“/prod-binary”]

Static Container
1. Minimal Packages
2. No language packages
3. Only works for compiled languages
(but also gains for vm languages)

Hack #3.a: Scratch Containers

Scratch?
empty tarball

CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM scratch
ADD prod-binary /
CMD [“/prod-binary”]

Hilariously Tiny
single digits mb

Scratch Containers
1. Only works for statically compiled binaries
2. Worst CI + Prod parity
3. Near Instant push+pull+boot+deploy

Minimal Guests

Minimal Guest
1. Bare O.S.
2. Minimal toolchain
3. Minimal Packaging + DIY
4. Small containers for the dynamic crowd

Promising Project
gilderlabs/docker-alpine

CODESNIPPET
# 16mb mysql container
FROM gliderlabs/alpine:3.1
RUN apk --update add mysql-client
ENTRYPOINT ["mysql"]

alpine is just one
hopefully we will have many options!

Orchestration

docker compose

CODESNIPPET
# docker-compose.yml
app:
build: .
links:
- db
ports:
- "8000:8000"
db:
image: postgres
# running ci
docker compose run app make test

docker compose problems
Specify different Dockerfile

docker compose problems
Can’t tell when services are running

docker compose problems
ENV encryption

docker compose problems
only 1 at a time

Docker CI
manually run and link :(

CODESNIPPET
docker run -d --name u1-postgres postgres
sleep magicnumber # :(
docker run --name u1-app --link u1-postgres:postgres
app make test
docker kill u1-postgres

Parallelism

Parallelism
Manually run and link and namespace

Parallelism
run your own agent
split work
monitor containers

Parallelism
@#$%!

Parallelism
we’ll figure it out!

Docker Swarm

Docker Swarm
if all we use to run CI is Docker,
Swarm gives us machine parallelism

Deploy

Deployment Styles
1. Push code + build on host
(e.g. Deis, EB, …)
2. Build + push image + trigger host pull
(e.g. EB, ECS, GAE, …)
3. Build + crossbuild for host + push
(e.g. Heroku docker release)

Build on Host
1. Removes burden from CI
2. Must be single-Dockerfile app
3. Must be deployable in isolation
4. Can’t leverage local cache

Build + Push + Trigger Pull
1. Use local cache
2. CI must do production builds
3. Push + Pull adds time
4. Can push many and launch many

Build + Crossbuild
1. ???

Hack #4: use the same registry

Use the same registry
1. Put registry near CI
2. Push CI layers to registry a la Hack #2
3. Minimal layers pushed with small changes
4. Put registry near host
5. Minimal pull onto host with small changes

Keep common bases hot

Try to use similar bases

CI + CD + Hosting
keep them close

CI + CD + Hosting
keep them close
(a.k.a. Amazon all the things)

What’s next?

Better parallel tooling

Better development environments
(non virtualized drivers)

Better machine provisioning
(docker machine as a library)

Codeship Private Beta
super docker funtimes!
chat with me or visit our booth

Thanks!

Questions?
also:
@ngauthier