Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)

436 views

Published on

Slides from Nick Gauthier's talk "Continuous Delivery with Containers" at ContainerDays Boston 2015: http://dynamicinfradays.org/events/2015-boston/programme.html#cdwithcontainers

Published in: Technology
  • Be the first to comment

ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)

  1. 1. Continuous Delivery with Containers Nick Gauthier @ngauthier Developer @codeship
  2. 2. What is Continuous Delivery?
  3. 3. Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time. Chen, Lianping (2015). "Continuous Delivery: Huge Benefits, but Challenges Too". IEEE Software 32
  4. 4. “valuable software” I think we can all agree here!
  5. 5. “short cycles” ship early, ship often!
  6. 6. “reliably released at any time” don’t break the build!
  7. 7. What are our needs?
  8. 8. Core CD System Attributes 1. Speed 2. Parity 3. Reliability 4. Flexibility
  9. 9. Speed: The CD Cycle 1. Check out code 2. Run tests 3. Build deployment artifact 4. Ship deployment artifact 5. Start deployment artifact
  10. 10. Deployment Artifact? 1. Binary 2. “slug” 3. WAR 4. VM 5. AMI 6. Container 7. tar 8. etc...
  11. 11. Parity “it worked for me”
  12. 12. Reliability “try running it again…”
  13. 13. Flexibility “I need X”
  14. 14. What are our CD options?
  15. 15. PaaS 1. Speed ✔ 2. Parity ✘ 3. Reliability ✔ 4. Flexibility ✘
  16. 16. Config Management (puppet, chef, ansible, salt) 1. Speed ✘ 2. Parity ✘ (✔ if virtualized locally) 3. Reliability ✔? 4. Flexibility ✔
  17. 17. VM 1. Speed ✘ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  18. 18. Container 1. Speed ✔ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  19. 19. Docker 1. Speed ✔✔ 2. Parity ✔ 3. Reliability ✔ 4. Flexibility ✔
  20. 20. Why Docker?
  21. 21. Cache layers rebuilds are fast!
  22. 22. Dockerfile simple, understandable, centralized
  23. 23. Official + Community Images don’t reinvent the wheel unless you want to
  24. 24. Hosting Choices low lock-in, many new options including diy
  25. 25. Docker CD What are our options?
  26. 26. #1 Local
  27. 27. 1. Run tests 2. Build images 3. Push images 4. Trigger host pull LOCAL
  28. 28. 1. Run tests (hopefully? same env?) 2. Build images (slow, cpu + net) 3. Push images (slow, race?) 4. Trigger host pull (race?) LOCAL
  29. 29. #2 Hosted Solution
  30. 30. HOSTED SOLUTION 1. Build CI Container 2. Run tests in CI Container 3. Build Production Container 4. Push Production Container 5. Trigger host pull
  31. 31. HOSTED SOLUTION 1. Build CI Container (caching? multiple?) 2. Run tests in CI Container (prod parity?) 3. Build Production Container (build artifacts? mult?) 4. Push Production Container 5. Trigger host pull
  32. 32. #3 Custom Solution
  33. 33. Custom Solution ???
  34. 34. What are we missing?
  35. 35. Centralized CD Our goals
  36. 36. Parity
  37. 37. CI Container == Prod Container PARITY
  38. 38. CI Container ~= Prod Container PARITY
  39. 39. Same FROM PARITY
  40. 40. Same Packages (+ a few) PARITY
  41. 41. Linked Services (not monoliths) PARITY
  42. 42. Hosting “Stack” = Docker PARITY
  43. 43. Orchestration
  44. 44. Running tasks + linking services ORCHESTRATION
  45. 45. Docker compose run ORCHESTRATION
  46. 46. Multiple Compositions? (unit + integration + acceptance) ORCHESTRATION
  47. 47. Speed
  48. 48. Speed (caching)
  49. 49. Base Images SPEED + CACHING
  50. 50. Layers from “last time” SPEED + CACHING
  51. 51. Layers from “last time” = registries! SPEED + CACHING
  52. 52. Parallelism
  53. 53. Parallelism @#$%!
  54. 54. Docker isolation! PARALLELISM
  55. 55. Ochestration PARALLELISM
  56. 56. Ochestration of Orchestration PARALLELISM
  57. 57. Pipelines PARALLELISM Checkout Static Analysis Lint Style Security CI Build A Build B Build C U1 U3U2 U4 I1 I1I1 A1 A1 CD Build P1 Build P2 Push P1 Push P2 Deploy P1 Deploy P2 PASS!
  58. 58. Pipelines PARALLELISM Checkout Static Analysis Lint Style Security CI Build A Build B Build C U1 U3U2 U4 I1 I1I1 A1 A1 CD Build P1 Build P2 Push P1 Push P2 Deploy P1 Deploy P2 PASS! DB Cache API APP + Tests Compose
  59. 59. Let’s build it! a.k.a. lessons learned a.k.a. what went wrong :)
  60. 60. Step 1: Define Containers
  61. 61. Official Language Containers
  62. 62. Official Language Containers if you can
  63. 63. Official Language Containers dictate O.S. (often debian)
  64. 64. Official Language Containers big, pull = slow, lots of disk (esp w/ docker machine + vbox)
  65. 65. Containers for CI
  66. 66. Containers for CI code changes frequently
  67. 67. Containers for CI code changes frequently = lots of rebuilds
  68. 68. Containers for CI dependencies = majority of time
  69. 69. Containers for CI dependencies = change occasionally
  70. 70. Hack #1: Layer Dependencies
  71. 71. CODESNIPPET # Classic ADD . RUN make deps / bundle / ... CMD [“run my tests”] # Layered ADD Godeps + Makefile / Gemfile + .lock / … RUN make deps / bundle / … ADD . CMD [“run my tests”]
  72. 72. Dependency Layer Cached on “manifest” files
  73. 73. Dependency Layer Great on CI Fantastic on dev machine
  74. 74. Hack #2: CI Registries
  75. 75. Pull app:branch before CI
  76. 76. Pull app:branch before CI Primes all layers from last time
  77. 77. Push app:branch after CI Updates layers in registry
  78. 78. Run registry near CI You can use docker hub but you don’t have to
  79. 79. Production Containers Same as CI (-pkgs)
  80. 80. Production Containers Layering + Common bases help here too!
  81. 81. Production Containers Huge = slow push = slow host pull = slow deploy
  82. 82. Hack #3: Static Containers
  83. 83. CODESNIPPET # Commands during CI make prod-binary docker build -f Dockerfile.production . # Dockerfile.production FROM debian:latest RUN apt-get update && apt-get install some-dyn-lib ADD prod-binary / CMD [“/prod-binary”]
  84. 84. Static Container 1. Minimal Packages 2. No language packages 3. Only works for compiled languages (but also gains for vm languages)
  85. 85. Hack #3.a: Scratch Containers
  86. 86. Scratch? empty tarball
  87. 87. CODESNIPPET # Commands during CI make prod-binary docker build -f Dockerfile.production . # Dockerfile.production FROM scratch ADD prod-binary / CMD [“/prod-binary”]
  88. 88. Hilariously Tiny single digits mb
  89. 89. Scratch Containers 1. Only works for statically compiled binaries 2. Worst CI + Prod parity 3. Near Instant push+pull+boot+deploy
  90. 90. Minimal Guests
  91. 91. Minimal Guest 1. Bare O.S. 2. Minimal toolchain 3. Minimal Packaging + DIY 4. Small containers for the dynamic crowd
  92. 92. Promising Project gilderlabs/docker-alpine
  93. 93. CODESNIPPET # 16mb mysql container FROM gliderlabs/alpine:3.1 RUN apk --update add mysql-client ENTRYPOINT ["mysql"]
  94. 94. alpine is just one hopefully we will have many options!
  95. 95. Orchestration
  96. 96. docker compose
  97. 97. CODESNIPPET # docker-compose.yml app: build: . links: - db ports: - "8000:8000" db: image: postgres # running ci docker compose run app make test
  98. 98. docker compose problems Specify different Dockerfile
  99. 99. docker compose problems Can’t tell when services are running
  100. 100. docker compose problems ENV encryption
  101. 101. docker compose problems only 1 at a time
  102. 102. Docker CI manually run and link :(
  103. 103. CODESNIPPET docker run -d --name u1-postgres postgres sleep magicnumber # :( docker run --name u1-app --link u1-postgres:postgres app make test docker kill u1-postgres
  104. 104. Parallelism
  105. 105. Parallelism Manually run and link and namespace
  106. 106. Parallelism run your own agent split work monitor containers
  107. 107. Parallelism @#$%!
  108. 108. Parallelism we’ll figure it out!
  109. 109. Docker Swarm
  110. 110. Docker Swarm if all we use to run CI is Docker, Swarm gives us machine parallelism
  111. 111. Deploy
  112. 112. Deployment Styles 1. Push code + build on host (e.g. Deis, EB, …) 2. Build + push image + trigger host pull (e.g. EB, ECS, GAE, …) 3. Build + crossbuild for host + push (e.g. Heroku docker release)
  113. 113. Build on Host 1. Removes burden from CI 2. Must be single-Dockerfile app 3. Must be deployable in isolation 4. Can’t leverage local cache
  114. 114. Build + Push + Trigger Pull 1. Use local cache 2. CI must do production builds 3. Push + Pull adds time 4. Can push many and launch many
  115. 115. Build + Crossbuild 1. ???
  116. 116. Hack #4: use the same registry
  117. 117. Use the same registry 1. Put registry near CI 2. Push CI layers to registry a la Hack #2 3. Minimal layers pushed with small changes 4. Put registry near host 5. Minimal pull onto host with small changes
  118. 118. Keep common bases hot
  119. 119. Try to use similar bases
  120. 120. CI + CD + Hosting keep them close
  121. 121. CI + CD + Hosting keep them close (a.k.a. Amazon all the things)
  122. 122. What’s next?
  123. 123. Better parallel tooling
  124. 124. Better development environments (non virtualized drivers)
  125. 125. Better machine provisioning (docker machine as a library)
  126. 126. Codeship Private Beta super docker funtimes! chat with me or visit our booth
  127. 127. Thanks!
  128. 128. Questions? also: @ngauthier

×