Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Continuous Delivery
with Containers
Nick Gauthier @ngauthier
Developer @codeship
What is Continuous Delivery?
Continuous Delivery (CD) is a software
engineering approach in which teams keep
producing valuable software in short cycle...
“valuable software”
I think we can all agree here!
“short cycles”
ship early, ship often!
“reliably released at any time”
don’t break the build!
What are our needs?
Core CD System Attributes
1. Speed
2. Parity
3. Reliability
4. Flexibility
Speed: The CD Cycle
1. Check out code
2. Run tests
3. Build deployment artifact
4. Ship deployment artifact
5. Start deplo...
Deployment Artifact?
1. Binary
2. “slug”
3. WAR
4. VM
5. AMI
6. Container
7. tar
8. etc...
Parity
“it worked for me”
Reliability
“try running it again…”
Flexibility
“I need X”
What are our CD options?
PaaS
1. Speed ✔
2. Parity ✘
3. Reliability ✔
4. Flexibility ✘
Config Management
(puppet, chef, ansible, salt)
1. Speed ✘
2. Parity ✘ (✔ if virtualized locally)
3. Reliability ✔?
4. Fle...
VM
1. Speed ✘
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔
Container
1. Speed ✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔
Docker
1. Speed ✔✔
2. Parity ✔
3. Reliability ✔
4. Flexibility ✔
Why Docker?
Cache layers
rebuilds are fast!
Dockerfile
simple, understandable, centralized
Official + Community Images
don’t reinvent the wheel
unless you want to
Hosting Choices
low lock-in, many new options
including diy
Docker CD
What are our options?
#1 Local
1. Run tests
2. Build images
3. Push images
4. Trigger host pull
LOCAL
1. Run tests (hopefully? same env?)
2. Build images (slow, cpu + net)
3. Push images (slow, race?)
4. Trigger host pull (r...
#2 Hosted Solution
HOSTED SOLUTION
1. Build CI Container
2. Run tests in CI Container
3. Build Production Container
4. Push Production Contai...
HOSTED SOLUTION
1. Build CI Container (caching? multiple?)
2. Run tests in CI Container (prod parity?)
3. Build Production...
#3 Custom Solution
Custom Solution
???
What are we missing?
Centralized CD
Our goals
Parity
CI Container == Prod Container
PARITY
CI Container ~= Prod Container
PARITY
Same FROM
PARITY
Same Packages
(+ a few)
PARITY
Linked Services
(not monoliths)
PARITY
Hosting “Stack” = Docker
PARITY
Orchestration
Running tasks + linking services
ORCHESTRATION
Docker compose run
ORCHESTRATION
Multiple Compositions?
(unit + integration + acceptance)
ORCHESTRATION
Speed
Speed
(caching)
Base Images
SPEED + CACHING
Layers from “last time”
SPEED + CACHING
Layers from “last time” = registries!
SPEED + CACHING
Parallelism
Parallelism
@#$%!
Docker isolation!
PARALLELISM
Ochestration
PARALLELISM
Ochestration of
Orchestration
PARALLELISM
Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
...
Pipelines
PARALLELISM
Checkout
Static Analysis
Lint Style Security
CI
Build A Build B Build C
U1 U3U2 U4 I1 I1I1 A1 A1
CD
...
Let’s build it!
a.k.a. lessons learned a.k.a. what went wrong :)
Step 1: Define Containers
Official Language Containers
Official Language Containers
if you can
Official Language Containers
dictate O.S. (often debian)
Official Language Containers
big, pull = slow, lots of disk
(esp w/ docker machine + vbox)
Containers for CI
Containers for CI
code changes frequently
Containers for CI
code changes frequently
=
lots of rebuilds
Containers for CI
dependencies
=
majority of time
Containers for CI
dependencies
=
change occasionally
Hack #1: Layer Dependencies
CODESNIPPET
# Classic
ADD .
RUN make deps / bundle / ...
CMD [“run my tests”]
# Layered
ADD Godeps + Makefile / Gemfile + ...
Dependency Layer
Cached on “manifest” files
Dependency Layer
Great on CI
Fantastic on dev machine
Hack #2: CI Registries
Pull app:branch before CI
Pull app:branch before CI
Primes all layers from last time
Push app:branch after CI
Updates layers in registry
Run registry near CI
You can use docker hub
but you don’t have to
Production Containers
Same as CI (-pkgs)
Production Containers
Layering + Common bases help here too!
Production Containers
Huge = slow push = slow host pull
= slow deploy
Hack #3: Static Containers
CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM deb...
Static Container
1. Minimal Packages
2. No language packages
3. Only works for compiled languages
(but also gains for vm l...
Hack #3.a: Scratch Containers
Scratch?
empty tarball
CODESNIPPET
# Commands during CI
make prod-binary
docker build -f Dockerfile.production .
# Dockerfile.production
FROM scr...
Hilariously Tiny
single digits mb
Scratch Containers
1. Only works for statically compiled binaries
2. Worst CI + Prod parity
3. Near Instant push+pull+boot...
Minimal Guests
Minimal Guest
1. Bare O.S.
2. Minimal toolchain
3. Minimal Packaging + DIY
4. Small containers for the dynamic crowd
Promising Project
gilderlabs/docker-alpine
CODESNIPPET
# 16mb mysql container
FROM gliderlabs/alpine:3.1
RUN apk --update add mysql-client
ENTRYPOINT ["mysql"]
alpine is just one
hopefully we will have many options!
Orchestration
docker compose
CODESNIPPET
# docker-compose.yml
app:
build: .
links:
- db
ports:
- "8000:8000"
db:
image: postgres
# running ci
docker co...
docker compose problems
Specify different Dockerfile
docker compose problems
Can’t tell when services are running
docker compose problems
ENV encryption
docker compose problems
only 1 at a time
Docker CI
manually run and link :(
CODESNIPPET
docker run -d --name u1-postgres postgres
sleep magicnumber # :(
docker run --name u1-app --link u1-postgres:p...
Parallelism
Parallelism
Manually run and link and namespace
Parallelism
run your own agent
split work
monitor containers
Parallelism
@#$%!
Parallelism
we’ll figure it out!
Docker Swarm
Docker Swarm
if all we use to run CI is Docker,
Swarm gives us machine parallelism
Deploy
Deployment Styles
1. Push code + build on host
(e.g. Deis, EB, …)
2. Build + push image + trigger host pull
(e.g. EB, ECS,...
Build on Host
1. Removes burden from CI
2. Must be single-Dockerfile app
3. Must be deployable in isolation
4. Can’t lever...
Build + Push + Trigger Pull
1. Use local cache
2. CI must do production builds
3. Push + Pull adds time
4. Can push many a...
Build + Crossbuild
1. ???
Hack #4: use the same registry
Use the same registry
1. Put registry near CI
2. Push CI layers to registry a la Hack #2
3. Minimal layers pushed with sma...
Keep common bases hot
Try to use similar bases
CI + CD + Hosting
keep them close
CI + CD + Hosting
keep them close
(a.k.a. Amazon all the things)
What’s next?
Better parallel tooling
Better development environments
(non virtualized drivers)
Better machine provisioning
(docker machine as a library)
Codeship Private Beta
super docker funtimes!
chat with me or visit our booth
Thanks!
Questions?
also:
@ngauthier
Upcoming SlideShare
Loading in …5
×

ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)

433 views

Published on

Slides from Nick Gauthier's talk "Continuous Delivery with Containers" at ContainerDays Boston 2015: http://dynamicinfradays.org/events/2015-boston/programme.html#cdwithcontainers

Published in: Technology
  • Be the first to comment

ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)

  1. 1. Continuous Delivery with Containers Nick Gauthier @ngauthier Developer @codeship
  2. 2. What is Continuous Delivery?
  3. 3. Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time. Chen, Lianping (2015). "Continuous Delivery: Huge Benefits, but Challenges Too". IEEE Software 32
  4. 4. “valuable software” I think we can all agree here!
  5. 5. “short cycles” ship early, ship often!
  6. 6. “reliably released at any time” don’t break the build!
  7. 7. What are our needs?
  8. 8. Core CD System Attributes 1. Speed 2. Parity 3. Reliability 4. Flexibility
  9. 9. Speed: The CD Cycle 1. Check out code 2. Run tests 3. Build deployment artifact 4. Ship deployment artifact 5. Start deployment artifact
  10. 10. Deployment Artifact? 1. Binary 2. “slug” 3. WAR 4. VM 5. AMI 6. Container 7. tar 8. etc...
  11. 11. Parity “it worked for me”
  12. 12. Reliability “try running it again…”
  13. 13. Flexibility “I need X”
  14. 14. What are our CD options?
  15. 15. PaaS 1. Speed ✔ 2. Parity ✘ 3. Reliability ✔ 4. Flexibility ✘
  16. 16. Config Management (puppet, chef, ansible, salt) 1. Speed ✘ 2. Parity ✘ (✔ if virtualized locally) 3. Reliability ✔? 4. Flexibility ✔
  17. 17. VM 1. Speed ✘ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  18. 18. Container 1. Speed ✔ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  19. 19. Docker 1. Speed ✔✔ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  20. 20. Why Docker?
  21. 21. Cache layers rebuilds are fast!
  22. 22. Dockerfile simple, understandable, centralized
  23. 23. Official + Community Images don’t reinvent the wheel unless you want to
  24. 24. Hosting Choices low lock-in, many new options including diy
  25. 25. Docker CD What are our options?
  26. 26. #1 Local
  27. 27. 1. Run tests 2. Build images 3. Push images 4. Trigger host pull LOCAL
  28. 28. 1. Run tests (hopefully? same env?) 2. Build images (slow, cpu + net) 3. Push images (slow, race?) 4. Trigger host pull (race?) LOCAL
  29. 29. #2 Hosted Solution
  30. 30. HOSTED SOLUTION 1. Build CI Container 2. Run tests in CI Container 3. Build Production Container 4. Push Production Container 5. Trigger host pull
  31. 31. HOSTED SOLUTION 1. Build CI Container (caching? multiple?) 2. Run tests in CI Container (prod parity?) 3. Build Production Container (build artifacts? mult?) 4. Push Production Container 5. Trigger host pull
  32. 32. #3 Custom Solution
  33. 33. Custom Solution ???
  34. 34. What are we missing?
  35. 35. Centralized CD Our goals
  36. 36. Parity
  37. 37. CI Container == Prod Container PARITY
  38. 38. CI Container ~= Prod Container PARITY
  39. 39. Same FROM PARITY
  40. 40. Same Packages (+ a few) PARITY
  41. 41. Linked Services (not monoliths) PARITY
  42. 42. Hosting “Stack” = Docker PARITY
  43. 43. Orchestration
  44. 44. Running tasks + linking services ORCHESTRATION
  45. 45. Docker compose run ORCHESTRATION
  46. 46. Multiple Compositions? (unit + integration + acceptance) ORCHESTRATION
  47. 47. Speed
  48. 48. Speed (caching)
  49. 49. Base Images SPEED + CACHING
  50. 50. Layers from “last time” SPEED + CACHING
  51. 51. Layers from “last time” = registries! SPEED + CACHING
  52. 52. Parallelism
  53. 53. Parallelism @#$%!
  54. 54. Docker isolation! PARALLELISM
  55. 55. Ochestration PARALLELISM
  56. 56. Ochestration of Orchestration PARALLELISM
  57. 57. Pipelines PARALLELISM Checkout Static Analysis Lint Style Security CI Build A Build B Build C U1 U3U2 U4 I1 I1I1 A1 A1 CD Build P1 Build P2 Push P1 Push P2 Deploy P1 Deploy P2 PASS!
  58. 58. Pipelines PARALLELISM Checkout Static Analysis Lint Style Security CI Build A Build B Build C U1 U3U2 U4 I1 I1I1 A1 A1 CD Build P1 Build P2 Push P1 Push P2 Deploy P1 Deploy P2 PASS! DB Cache API APP + Tests Compose
  59. 59. Let’s build it! a.k.a. lessons learned a.k.a. what went wrong :)
  60. 60. Step 1: Define Containers
  61. 61. Official Language Containers
  62. 62. Official Language Containers if you can
  63. 63. Official Language Containers dictate O.S. (often debian)
  64. 64. Official Language Containers big, pull = slow, lots of disk (esp w/ docker machine + vbox)
  65. 65. Containers for CI
  66. 66. Containers for CI code changes frequently
  67. 67. Containers for CI code changes frequently = lots of rebuilds
  68. 68. Containers for CI dependencies = majority of time
  69. 69. Containers for CI dependencies = change occasionally
  70. 70. Hack #1: Layer Dependencies
  71. 71. CODESNIPPET # Classic ADD . RUN make deps / bundle / ... CMD [“run my tests”] # Layered ADD Godeps + Makefile / Gemfile + .lock / … RUN make deps / bundle / … ADD . CMD [“run my tests”]
  72. 72. Dependency Layer Cached on “manifest” files
  73. 73. Dependency Layer Great on CI Fantastic on dev machine
  74. 74. Hack #2: CI Registries
  75. 75. Pull app:branch before CI
  76. 76. Pull app:branch before CI Primes all layers from last time
  77. 77. Push app:branch after CI Updates layers in registry
  78. 78. Run registry near CI You can use docker hub but you don’t have to
  79. 79. Production Containers Same as CI (-pkgs)
  80. 80. Production Containers Layering + Common bases help here too!
  81. 81. Production Containers Huge = slow push = slow host pull = slow deploy
  82. 82. Hack #3: Static Containers
  83. 83. CODESNIPPET # Commands during CI make prod-binary docker build -f Dockerfile.production . # Dockerfile.production FROM debian:latest RUN apt-get update && apt-get install some-dyn-lib ADD prod-binary / CMD [“/prod-binary”]
  84. 84. Static Container 1. Minimal Packages 2. No language packages 3. Only works for compiled languages (but also gains for vm languages)
  85. 85. Hack #3.a: Scratch Containers
  86. 86. Scratch? empty tarball
  87. 87. CODESNIPPET # Commands during CI make prod-binary docker build -f Dockerfile.production . # Dockerfile.production FROM scratch ADD prod-binary / CMD [“/prod-binary”]
  88. 88. Hilariously Tiny single digits mb
  89. 89. Scratch Containers 1. Only works for statically compiled binaries 2. Worst CI + Prod parity 3. Near Instant push+pull+boot+deploy
  90. 90. Minimal Guests
  91. 91. Minimal Guest 1. Bare O.S. 2. Minimal toolchain 3. Minimal Packaging + DIY 4. Small containers for the dynamic crowd
  92. 92. Promising Project gilderlabs/docker-alpine
  93. 93. CODESNIPPET # 16mb mysql container FROM gliderlabs/alpine:3.1 RUN apk --update add mysql-client ENTRYPOINT ["mysql"]
  94. 94. alpine is just one hopefully we will have many options!
  95. 95. Orchestration
  96. 96. docker compose
  97. 97. CODESNIPPET # docker-compose.yml app: build: . links: - db ports: - "8000:8000" db: image: postgres # running ci docker compose run app make test
  98. 98. docker compose problems Specify different Dockerfile
  99. 99. docker compose problems Can’t tell when services are running
  100. 100. docker compose problems ENV encryption
  101. 101. docker compose problems only 1 at a time
  102. 102. Docker CI manually run and link :(
  103. 103. CODESNIPPET docker run -d --name u1-postgres postgres sleep magicnumber # :( docker run --name u1-app --link u1-postgres:postgres app make test docker kill u1-postgres
  104. 104. Parallelism
  105. 105. Parallelism Manually run and link and namespace
  106. 106. Parallelism run your own agent split work monitor containers
  107. 107. Parallelism @#$%!
  108. 108. Parallelism we’ll figure it out!
  109. 109. Docker Swarm
  110. 110. Docker Swarm if all we use to run CI is Docker, Swarm gives us machine parallelism
  111. 111. Deploy
  112. 112. Deployment Styles 1. Push code + build on host (e.g. Deis, EB, …) 2. Build + push image + trigger host pull (e.g. EB, ECS, GAE, …) 3. Build + crossbuild for host + push (e.g. Heroku docker release)
  113. 113. Build on Host 1. Removes burden from CI 2. Must be single-Dockerfile app 3. Must be deployable in isolation 4. Can’t leverage local cache
  114. 114. Build + Push + Trigger Pull 1. Use local cache 2. CI must do production builds 3. Push + Pull adds time 4. Can push many and launch many
  115. 115. Build + Crossbuild 1. ???
  116. 116. Hack #4: use the same registry
  117. 117. Use the same registry 1. Put registry near CI 2. Push CI layers to registry a la Hack #2 3. Minimal layers pushed with small changes 4. Put registry near host 5. Minimal pull onto host with small changes
  118. 118. Keep common bases hot
  119. 119. Try to use similar bases
  120. 120. CI + CD + Hosting keep them close
  121. 121. CI + CD + Hosting keep them close (a.k.a. Amazon all the things)
  122. 122. What’s next?
  123. 123. Better parallel tooling
  124. 124. Better development environments (non virtualized drivers)
  125. 125. Better machine provisioning (docker machine as a library)
  126. 126. Codeship Private Beta super docker funtimes! chat with me or visit our booth
  127. 127. Thanks!
  128. 128. Questions? also: @ngauthier

×