Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Adrian Mouat - Microservice deployment techniques

46 views

Published on

Handling deployments effectively is an essential aspect of making the most of modern microservice architectures. How do you safely and quickly deploy updates? How do you test that your microservices work at scale in a full production system? How do you measure and respond to problems occurring shortly after an update?

A lot of techniques and solutions to these problems get mentioned: deployments can be ramped, blue/green or canary. System tests can take place in a staging environment or may be eschewed in favour of ““testing in production”” techniques, such as chaos monkeys and shadowing. However, whilst this gets talked about a lot, actual code examples and tools seem thinner on the ground.

In this talk, we’ll take a look at the various techniques, when you might want to use them, and provide practical examples on how to implement them.

Published in: Software
  • Be the first to comment

Adrian Mouat - Microservice deployment techniques

  1. 1. info@container-solutions.com www.container-solutions.com Microservice Deployment Techniques Adrian Mouat Devopsdays Edinburgh 2017
  2. 2. www.container-solutions.com info@container-solutions.com “70% of outages are due to changes in a live system” ■ Google Site Reliability Engineering
  3. 3. www.container-solutions.com info@container-solutions.com Reducing Outages - Google Advice ■ Implementing progressive rollouts ■ Quickly and accurately detecting problems ■ Rolling back changes safely when problems arise
  4. 4. www.container-solutions.com info@container-solutions.com Lots of talk about Deployment Techniques ■ Ramped Deployment ■ Blue/Green ■ Canaries ■ A/B ■ Feature Flags ■ Shadowing ■ ...
  5. 5. www.container-solutions.com info@container-solutions.com Less hard code... ■ Bespoke, internal solutions ■ Due to heterogeneity of systems ■ And relative newness of microservices
  6. 6. www.container-solutions.com info@container-solutions.com Kubernetes as the Platform ■ Default platform for running containers in production ■ Supported by DC/OS and Docker ■ Starting to see higher level tooling emerge
  7. 7. www.container-solutions.com info@container-solutions.com Ideal Deployment ■ Zero Downtime ■ Control over Speed ■ Control over Traffic ■ Observable ■ Easy to Roll Back ■ Automatic ■ Fearless
  8. 8. www.container-solutions.com info@container-solutions.com Horses For Courses ■ Different strategies in different circumstances ■ Financial trading or medical services ○ vs cat pictures ■ different tolerances for errors ■ amount of testing ■ rollout speed ■ cost of deployments
  9. 9. www.container-solutions.com info@container-solutions.com Kubernetes Defaults ■ Recreate ■ Ramped Deployment
  10. 10. www.container-solutions.com info@container-solutions.com Recreate ■ Delete running instances ■ Spin-up new ones ■ Implies downtime
  11. 11. www.container-solutions.com info@container-solutions.com Ramped Deployment ■ New version is rolled out slowly across instances ■ Supported out of the box in Kubernetes ■ Rollback in case of error ■ Single service updated at a time
  12. 12. www.container-solutions.com info@container-solutions.com Ramped Deployment
  13. 13. www.container-solutions.com info@container-solutions.com Prerequisites ■ Multiple versions deployed concurrently ■ Services should support 2 previous versions ○ Not easy, but very important! ■ Shutdown cleanly ■ Healthchecks ○ Readiness and Liveness
  14. 14. www.container-solutions.com info@container-solutions.com Drawbacks ■ Supporting multiple API versions is hard ■ Rollout/rollback takes time ○ Especially if long start-up time ○ No integration with metrics ○ Little control over traffic
  15. 15. www.container-solutions.com info@container-solutions.com Example apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: strategy: rollingUpdate: maxSurge: 2 maxUnavailable: 2 replicas: 5 template: ...
  16. 16. www.container-solutions.com info@container-solutions.com Blue/Green ■ Current version of system is "blue" ■ Spin up new "green" version ○ Entirely separate and parallel ■ Test green version ■ Flip switch when happy
  17. 17. www.container-solutions.com info@container-solutions.com Advantages ■ Can update multiple services together ○ Avoid versioning issue ■ Test whole system before deploying ■ Instant rollback
  18. 18. www.container-solutions.com info@container-solutions.com Drawbacks ■ Requires much more resources ■ More heavyweight ■ No control over traffic or speed ■ How to handle state?
  19. 19. www.container-solutions.com info@container-solutions.com Blue/Green & Kubernetes ■ Not obvious how to do this ■ Services and labels/selectors ○ Works well for a single service ○ Not for entire system
  20. 20. www.container-solutions.com info@container-solutions.com Canaries ■ Deploy new version to subset of users ■ Monitor performance ■ Roll-out if happy
  21. 21. www.container-solutions.com info@container-solutions.com Advantages ■ Avoid breaking changes for everyone ■ Chance to carefully monitor metrics ○ check error rate, response times ■ Doesn't require significant extra resources
  22. 22. www.container-solutions.com info@container-solutions.com Drawbacks ■ Requires intelligent load-balancer ■ Typically more work to implement ■ Slower rollout (but fast rollback)
  23. 23. www.container-solutions.com info@container-solutions.com Drawbacks ■ Requires intelligent load-balancer ■ Typically more work to implement ■ Slower rollout (but fast rollback)
  24. 24. www.container-solutions.com info@container-solutions.com Kubernetes Poor Man’s Canary ■ Start a kubernetes rollout ■ Hit kubectl rollout pause deployment ■ Monitor effect ■ kubectl rollout resume ... or resume then kubectl rollout undo...
  25. 25. www.container-solutions.com info@container-solutions.com Kubernetes Poor Man’s Canary ■ Handy in a pinch ■ Manual ■ Very limited control
  26. 26. www.container-solutions.com info@container-solutions.com Kubernetes PermaCanary ■ Blog by bitmovin ■ Multiple deployments, including "permanent canaries" ■ HAProxy routes 5% of traffic to canary ■ If no problems are seen, rollout to production
  27. 27. www.container-solutions.com info@container-solutions.com A/B Testing ■ Two (or more) versions run in parallel ■ Version to run chosen based on stats ■ Normally for feature or design evaluation ○ Measure clicks not error rates ■ Can be done with canaries ■ Or feature flags
  28. 28. www.container-solutions.com info@container-solutions.com Feature Flags ■ Multiple versions are present in the code ■ Code path selected dynamically if (useNewVersion()) { ... } else { // use old version ... }
  29. 29. www.container-solutions.com info@container-solutions.com Feature Flags ■ Enabled for certain users ○ Or a random sample ■ Also “shadowing” ○ both code paths executed ○ only old code used ○ new code verified
  30. 30. www.container-solutions.com info@container-solutions.com Related Concerns ■ API versioning ■ Databases ■ Deployment state (see Flux from Weaveworks) ■ Monitoring ■ Testing-in-production vs staging
  31. 31. www.container-solutions.com info@container-solutions.com The Future ■ More tooling building on top of Kubernetes ■ Primarily “service meshes” ○ Istio, linkerd ○ Sidecar containers providing intelligent routing ○ Out-of-the box support for canaries, blue-green ○ Better monitoring and observability
  32. 32. www.container-solutions.com info@container-solutions.com Thanks for Listening! @adrianmouat
  33. 33. www.container-solutions.com info@container-solutions.com References ■ Site Reliability Engineering ■ BitMovin Canary Deployments on Kubernetes ■ New Stack Article on Deployment Strategies by Etienne Tremel ■ Netflix Articles ○ http://techblog.netflix.com/2013/08/deploying-netflix-api.html ○ https://www.infoq.com/presentations/canary-analysis-deployment-pattern ■ Thanks to Tom Wilkie & Weave, colleagues at Container Solutions, Dave Farley, Ben Whaley, Matt Heath, Daniel Hoelbling-Inzko

×