[Many thanks to Nginx for sponsoring this O'Reilly webinar!] Implementing a continuous delivery (CD) pipeline is not trivial, and the introduction of container technology to the development stack can introduce additional challenges and requirements. In this webcast we will look at the steps that are essential for creating an effective pipeline for creating and deploying containerized applications. Topic covered include: - The impact of containers on CD - Creating a container pipeline (including functional and nonfunctional testing) - Lessons learned the hard way A supporting O'Reilly report "Containerizing Continuous Delivery in Java" is also available, and this contains instructions and code for how to create a Jenkins-based continuous delivery pipeline that takes a series of Java applications and containerizes them, ready for functional and nonfunctional testing, and ultimately, deployment.

1. Continuous Delivery for Containerized Applications:
The Trials and Tribulations
Daniel Bryant
@danielbryantuk

2. Setting the scene…
• Continuous delivery is a large topic
• Focusing on the process and tooling
• No live coding today
• Mini-book contains more details
• “Building a CD pipeline” by Adrian and Kevin
15/12/2016 @danielbryantuk

3. TL;DR – Containers and CD
• Container image becomes the build pipeline ‘single binary’
• Adding metadata to containers images is vital
• Must validate container constraints (NFRs)
• Cultivate containerised ‘mechanical sympathy’
15/12/2016 @danielbryantuk

4. @danielbryantuk
• Chief Scientist at OpenCredo, CTO at SpectoLabs
• Agile, architecture, CI/CD, DevOps
• Java, Go, JS, microservices, cloud, containers
• Leading change through the application of technology and teams
15/12/2016 @danielbryantuk

5. Continuous Delivery
15/12/2016 @danielbryantuk

6. Continuous Delivery
• Produce valuable and robust software in short cycles
• Optimising for feedback and learning
• Not (necessarily) Continuous Deployment
15/12/2016 @danielbryantuk

7. Creation of a build pipeline is mandatory for continuous delivery
15/12/2016 @danielbryantuk

8. 15/12/2016 @danielbryantuk

9. The Impact of containers on CD
15/12/2016 @danielbryantuk

10. Container technology (and CD)
• OS-level virtualisation
• cgroups, namespaces, rootfs
• Package and execute software
• Container image == ‘single binary’
15/12/2016 @danielbryantuk

11. 15/12/2016 @danielbryantuk

12. 15/12/2016 @danielbryantuk

13. Creating a pipeline for containers
15/12/2016 @danielbryantuk

14. 15/12/2016 @danielbryantuk

15. Make your dev environment like production
• Develop locally or copy/code in container
• Use base images from production
• Must build/test containers locally
• Perform (at least) happy path tests
• All tests should be runnable locally
15/12/2016 @danielbryantuk

16. Lesson learned: Dockerfile content is super important
• OS choice
• Configuration
• Build artifacts
• Exposing ports
• Java
• JDK vs JRE and Oracle vs OpenJDK
• Golang
• Statically compiled binary
• Python
• Virtualenv
15/12/2016 @danielbryantuk

17. Please talk to the sysadmin people:
Their operational knowledge is invaluable
15/12/2016 @danielbryantuk

18. Different prod and test containers?
• Create “test” version of container
• Full OS (e.g. Ubuntu)
• Test tools and data
• Easy to see app/configuration drift
• Use test sidecar containers instead
• ONTEST proposal by Alexi Ledenev
15/12/2016 @danielbryantuk
http://blog.terranillius.com/post/docker_testing/

19. 15/12/2016 @danielbryantuk

20. Building images with Jenkins
• My report covers this
• Build as usual…
• Build Docker Image
• Cloudbees Docker Build and Publish Plugin
• Push image to registry
15/12/2016 @danielbryantuk

21. Storing in an image registry (DockerHub)
15/12/2016 @danielbryantuk

22. Lesson learned: Metadata is valuable
• Application metadata
• Version / GIT SHA
• Build metadata
• Build date
• Image name
• Vendor
• Quality metadata
• QA control
• Security audited etc
15/12/2016 @danielbryantuk

23. Metadata – Beware of “latest” Docker Tag
• Beware of the ‘latest’ Docker tag
• “Latest” simply means
• the last build/tag that ran without
a specific tag/version specified
• Ignore “latest” tag
• Version your tags, every time
• Danielbryantuk/test:2.4.1
15/12/2016 @danielbryantuk

24. Metadata - Adding Labels at build time
• Docker Labels
• Add key/value data to image
15/12/2016 @danielbryantuk

25. Metadata - Adding Labels at build time
• Microscaling Systems’ Makefile
• Labelling automated builds on
DockerHub (h/t Ross Fairbanks)
• Create file /hooks/build
• label-schema.org
• microbadger.com
15/12/2016 @danielbryantuk

26. Metadata - Adding Labels at runtime
15/12/2016 @danielbryantuk
$ docker run -d --label
uk.co.danielbryant.lbname=frontdoor nginx
• Can ’docker commit’, but creates new image
• Not possible to update running container
• Docker Proposal: Update labels #21721

27. 15/12/2016 @danielbryantuk

28. Component testing
15/12/2016 @danielbryantuk

29. Testing: Jenkins Pipeline (as code)
15/12/2016 @danielbryantuk

30. 15/12/2016 @danielbryantuk

31. Testing individual containers
15/12/2016 @danielbryantuk

32. Integration testing
15/12/2016 @danielbryantuk

33. Introducing Docker Compose
15/12/2016 @danielbryantuk

34. Docker Compose & Jenkins Pipeline
15/12/2016 @danielbryantuk

35. Mechanical sympathy: Docker and Java
• Watch for cgroup limits (and cgroup awareness)
• getAvailableProcessors issue (bugs.openjdk.java.net/browse/JDK-8140793)
• Default fork/join thread pool sizes (based from host CPU count)
• Set container memory appropriately
• JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead
• Account for native thread requirements e.g. thread stack size (Xss)
• Entropy
• Host entropy can soon be exhausted by crypto operations
15/12/2016 @danielbryantuk | @spoole167 35

36. Mechanical sympathy: Docker and security
15/12/2016 @danielbryantuk

37. Containers are not a silver bullet
15/12/2016 @danielbryantuk

38. Containers: Expectations versus reality
15/12/2016 @danielbryantuk
“DevOps”

39. Containerise an existing (monolithic) app?
• For
• We know the monolith well
• Allows homogenization of the
pipeline and deployment platform
• Can be a demonstrable win for
tech and the business
• Against
• Can be difficult (100+ line scripts)
• Often not designed for operation
within containers, nor cloud native
• Putting lipstick on a pig?
15/12/2016 @danielbryantuk

40. Whatever you decide…
push it through the pipeline ASAP!
15/12/2016 @danielbryantuk

41. Key lessons learned
• Conduct an architectural review
• Architecture for Developers, by Simon Brown
• Architecture Interview, by Susan Fowler
• Look for data ingress/egress
• File system access
• Support resource constraints/transience
• Optimise for quick startup and shutdown
• Evaluate approach to concurrency
• Store configuration (secrets) remotely
15/12/2016 @danielbryantuk

42. New design patterns
15/12/2016 @danielbryantuk
bit.ly/2efe0TP

43. Microservices…
Containers and microservices are
complementary
Testing and deployment change
15/12/2016 @danielbryantuk
https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

44. 15/12/2016 @danielbryantuk

45. 15/12/2016 @danielbryantuk

46. 15/12/2016 @danielbryantuk

47. Microservice architectural impact on CD
• Application decomposition
• Bounded context
• Change cadence
• Risk
• Performance
• Scalability
• Team location
h/t Matthew Skelton, Adam Tornhill
• Worth knowing about:
• Consumer-based contracts
• Service virtualisation
• Synthetic transactions and
semantic monitoring
15/12/2016 @danielbryantuk

48. Using containers does not obviate the need for
good architectural practices
15/12/2016 @danielbryantuk

49. 15/12/2016 @danielbryantuk
https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

50. Summary
15/12/2016 @danielbryantuk

51. In summary
• Continuous delivery is vitally important in modern architectures/ops
• Container images must be the (single) source of truth within pipeline
• Mechanical sympathy is important (assert properties in the pipeline)
• We’re now bundling more responsibility into our artifact (e.g. an OS)
• Not all developers are operationally aware
• The tooling is now becoming stable/mature
• We need to re-apply old CD practices with new technologies/tooling
15/12/2016 @danielbryantuk

52. Bedtime reading
15/12/2016 @danielbryantuk

53. Thanks for listening
• Any questions?
• Feel free to contact me
• @danielbryantuk
• daniel.bryant@opencredo.com
15/12/2016 @danielbryantuk