Continuous Delivery the Hard Way with Kubernetes
•
5 likes•1,343 views
How do you version control your deployment config + automate the delivery of your software through a CI/CD pipeline? We will cover open source tools that facilitate: • Simultaneously updating configuration and performing releases to Kubernetes • Rolling back and pinning releases • Having different release policies to different environments Read the blog: https://www.weave.works/blog/continuous-delivery-the-hard-way Visit Weave Cloud: https://www.weave.works/product/cloud/ For more free talks, join our Weave Online User Group: https://www.meetup.com/Weave-User-Group/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Continuous Delivery the Hard Way with Kubernetes
Similar to Continuous Delivery the Hard Way with Kubernetes (20)
More from Weaveworks
More from Weaveworks (20)
Recently uploaded
Recently uploaded (20)
Continuous Delivery the Hard Way with Kubernetes
- 1. Continuous Delivery the hard way with Kubernetes Jeff Hoffer, Developer Experience github.com/eudaimos
- 2. Agenda 1. Why should I deliver continuously? 2. Kubernetes primer 3. GitLab primer 4. “OK, so we’ve got these pieces, how are we going to put them together?” 5. Let’s iterate on a design! 6. Conclusions
- 3. Agenda 1. Why should I deliver continuously? 2. Kubernetes primer 3. GitLab primer 4. “OK, so we’ve got these pieces, how are we going to put them together?” 5. Let’s iterate on a design! 6. Conclusions
- 4. Why should I continuously deliver? • Microservices • Conway’s law • Scaling project, scaling team • Velocity!
- 5. Kubernetes: all you need to know Pods containers ServicesDeployments Container Image Docker container image, contains your application code in an isolated environment. Pod A set of containers, sharing network namespace and local volumes, co- scheduled on one machine. Mortal. Has pod IP. Has labels. Deployment Specify how many replicas of a pod should run in a cluster. Then ensures that many are running across the cluster. Has labels. Service Names things in DNS. Gets virtual IP. Two types: ClusterIP for internal services, NodePort for publishing to outside. Routes based on labels.
- 6. GitLab primer • Or you can use GitHub, Travis, Circle, Docker Hub, Quay.io, GCR… CI system Docker registry GitLab Version controlled code Version controlled code
- 7. Version controlled code These are the things that we’ve got Version controlled code CI system Docker registry Kubernetes clusterCode Docker image Kubernetes YAML
- 8. Version controlled code These are the things that we’ve got Version controlled code CI system Docker registry Kubernetes clusterCode Docker image Kubernetes YAML git git + shell docker registry API kubernetes API
- 9. These are the things that we’ve got Version controlled code CI system Docker registry Kubernetes clusterCode Docker image Kubernetes YAML
- 12. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterkubectl apply -f service.yaml
- 13. V1 Deploy update (with CI system)
- 15. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clustergit push master
- 16. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterdocker build :a1b2c3
- 17. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterdocker push :a1b2c3
- 18. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterkubectl set image :a1b2c3
- 19. V1 Rollback
- 21. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes cluster git checkout master git revert HEAD git push
- 22. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterdocker build :b2c3d4
- 23. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterdocker push :b2c3d4
- 24. Version controlled code V1 architecture Version controlled code CI system Docker registry Kubernetes clusterkubectl set image :b2c3d4
- 26. Downsides • Building & pushing containers is slow (disk I/O, network), shouldn’t need to this when rolling back • Branch per environment required per microservice (explosion of branches, hard to manage & scale) • Only a matter of time until you get a git merge mess • Better to decouple version of code at HEAD from version deployed…
- 27. Version controlled configuration • users service • code for users service • Kubernetes YAML • orders service • code for orders service • Kubernetes YAML • config repo • Kubernetes YAML for users • Kubernetes YAML for orders • Version controlled config should be the source of truth for your whole app (all the microservices)
- 28. Decoupling versions from releases Code versions (branches, tags) Environments & releases • users service • master • feature_A • feature_B • orders service • master • feature_A • feature_B • … • production • users -> master @ t1 • orders -> master @ t1 • staging • orders -> master @ t2 • orders -> master @ t2 conflating per- service code branches with environments in each repo is a hack, and doesn’t scale well
- 29. V2 Put all the yamels in one place
- 30. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 31. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 32. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 33. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 34. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 35. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 36. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 37. Version controlled code V2 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you
- 38. Version controlled code Container builder demo Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Code Docker image Kubernetes YAML Have the CI system update the yamels automatically for you Builder used as CD system
- 39. Now you can recreate your production environment from the central YAML repository even if your entire production cluster gets deleted
- 40. Demo!
- 41. Downsides • The CI system is responsible for a lot now (design smell – overloaded) • You can only trigger the CI system by pushing code (we wanted to be able to rollback without pushing code) • If you rollback out of band (directly with kubectl), you have to remember to update the central configuration repo as well • Parallel builds can tread on eachothers’ toes, not atomic: race between git checkout and git push (need a global lock) • Scripting updates of yamels can be a pain… it mangles your yamels • Developers start asking for more release management features (rollback, pinning, automation for some envs and manual gating for others, and your once-simple script keeps growing…)
- 42. V3 Refactor architecture Add “release manager”
- 43. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 44. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 45. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 46. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 47. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 48. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 49. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 50. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy
- 51. V3 Rollback doesn’t go via CI
- 52. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy rollback!
- 53. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy rollback!
- 54. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy rollback!
- 55. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy rollback!
- 56. Version controlled code V3 architecture Version controlled code CI system Docker registry Kubernetes cluster Version controlled config Release manager Code Docker image Kubernetes YAML policy rollback!
- 57. What does the release manager do? • Watches for changes in a container registry (output of CI system) • Makes commits for you to version controlled configuration (understands Kubernetes YAML) • Depending on release policy (per environment), either push changes continuously or permit manually gated releases • Allows releases to be rolled back by changing a pointer • Releases can be “locked” as a social cue
- 58. Different environments can have different release policies (no tight coupling between individual microservices repos and what’s released)
- 60. This is how we deploy Weave Cloud Weave Cloud helps devops iterate faster with: • observability & monitoring • continuous delivery • container networks & firewalls Weave Flux is a release manager for Kubernetes
- 61. Other topics Kubernetes 101 How do I monitor this stuff? (Prometheus) Network policy for isolating & firewalling different microservices We have talks & trainings on all these topics in the Weave user group!
- 62. Join the Weave user group! meetup.com/pro/Weave/ Come hang out on Slack! weave.works/help
- 63. Thanks! Questions? We are hiring! DX in San Francisco Engineers in London & SF weave.works/weave-company/hiring Check out Flux on GitHub: github.com/weaveworks/flux
Editor's Notes
- nb - this is just one way to do it Why should I deliver continuously? Kubernetes primer GitLab primer “OK, so we’ve got these pieces, how are we going to put them together?” First attempt: kubectl in CI always build & push “production” branch downsides: container images are big and slow, why do we need to rebuild them just to change a pointer? especially if you roll back? Second attempt: push all container images to repo, write a Python script to automatically clone, edit the tag, and write out the yaml downsides: it loses comments in the yaml, so i could use a regular expression, gah Feature request… developers want to be able to roll back… deploy to different environments Introducing Flux, a tool that does all this for you Flux demo (then alternatives?)
- nb - this is just one way to do it Why should I deliver continuously? Kubernetes primer GitLab primer “OK, so we’ve got these pieces, how are we going to put them together?” First attempt: kubectl in CI always build & push “production” branch downsides: container images are big and slow, why do we need to rebuild them just to change a pointer? especially if you roll back? Second attempt: push all container images to repo, write a Python script to automatically clone, edit the tag, and write out the yaml downsides: it loses comments in the yaml, so i could use a regular expression, gah Feature request… developers want to be able to roll back… deploy to different environments Introducing Flux, a tool that does all this for you Flux demo (then alternatives?)
- to scale your software to scale your team
- Deploying a new version
- Deploying a new version
- https://www.youtube.com/watch?v=C2YZnTL596Q
- Deploying a new version
- Deploying a new version
- Deploying a new version
- You should version control the config of your entire app (all the microservices) in one repo Don’t depend on CI history to reconstruct it Just like you version control your infrastructure-as-code This way you can easily reconstruct your entire application if your cluster gets wiped out Now the configuration about which versions of each service are running can live in the master branch of the config repo This is what’s deployed to prod Then you can have multiple config repos, one per environment (prod, staging) we have a whiteboard in our office “number of days since we accidentally deleted all of production” we were back up and running in an hour because we can do this
- (yamels are like camels)
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- TODO add “:prod” (actually, “:abcde”) TODO add changing colours and indicate that there’s plural of them
- https://www.katacoda.com/courses/weave/flux-training
- https://www.katacoda.com/courses/weave/kubernetes-training
- https://www.katacoda.com/courses/weave/kubernetes-training