Adrian Mouat - Microservice deployment techniques

info@container-solutions.com
www.container-solutions.com
Microservice
Deployment
Techniques
Adrian Mouat
Devopsdays Edinburgh
2017

www.container-solutions.com info@container-solutions.com
“70% of outages are due to changes
in a live system”
■ Google Site Reliability Engineering

Reducing Outages - Google Advice
■ Implementing progressive rollouts
■ Quickly and accurately detecting problems
■ Rolling back changes safely when problems arise

Lots of talk about Deployment Techniques
■ Ramped Deployment
■ Blue/Green
■ Canaries
■ A/B
■ Feature Flags
■ Shadowing
■ ...

Less hard code...
■ Bespoke, internal solutions
■ Due to heterogeneity of systems
■ And relative newness of microservices

Kubernetes as the Platform
■ Default platform for running containers in
production
■ Supported by DC/OS and Docker
■ Starting to see higher level tooling emerge

Ideal Deployment
■ Zero Downtime
■ Control over Speed
■ Control over Traffic
■ Observable
■ Easy to Roll Back
■ Automatic
■ Fearless

Horses For Courses
■ Different strategies in different circumstances
■ Financial trading or medical services
○ vs cat pictures
■ different tolerances for errors
■ amount of testing
■ rollout speed
■ cost of deployments

Kubernetes Defaults
■ Recreate
■ Ramped Deployment

Recreate
■ Delete running instances
■ Spin-up new ones
■ Implies downtime

Ramped Deployment
■ New version is rolled out slowly across instances
■ Supported out of the box in Kubernetes
■ Rollback in case of error
■ Single service updated at a time

Ramped Deployment

Prerequisites
■ Multiple versions deployed concurrently
■ Services should support 2 previous versions
○ Not easy, but very important!
■ Shutdown cleanly
■ Healthchecks
○ Readiness and Liveness

Drawbacks
■ Supporting multiple API versions is hard
■ Rollout/rollback takes time
○ Especially if long start-up time
○ No integration with metrics
○ Little control over traffic

Example
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 2
replicas: 5
template:
...

Blue/Green
■ Current version of system is "blue"
■ Spin up new "green" version
○ Entirely separate and parallel
■ Test green version
■ Flip switch when happy

Advantages
■ Can update multiple services together
○ Avoid versioning issue
■ Test whole system before deploying
■ Instant rollback

Drawbacks
■ Requires much more resources
■ More heavyweight
■ No control over traffic or speed
■ How to handle state?

Blue/Green & Kubernetes
■ Not obvious how to do this
■ Services and labels/selectors
○ Works well for a single service
○ Not for entire system

Canaries
■ Deploy new version to subset of users
■ Monitor performance
■ Roll-out if happy

Advantages
■ Avoid breaking changes for everyone
■ Chance to carefully monitor metrics
○ check error rate, response times
■ Doesn't require significant extra resources

Drawbacks
■ Requires intelligent load-balancer
■ Typically more work to implement
■ Slower rollout (but fast rollback)

Kubernetes Poor Man’s Canary
■ Start a kubernetes rollout
■ Hit kubectl rollout pause deployment
■ Monitor effect
■ kubectl rollout resume ...
or resume then kubectl rollout undo...

Kubernetes Poor Man’s Canary
■ Handy in a pinch
■ Manual
■ Very limited control

Kubernetes PermaCanary
■ Blog by bitmovin
■ Multiple deployments, including "permanent
canaries"
■ HAProxy routes 5% of traffic to canary
■ If no problems are seen, rollout to production

A/B Testing
■ Two (or more) versions run in parallel
■ Version to run chosen based on stats
■ Normally for feature or design evaluation
○ Measure clicks not error rates
■ Can be done with canaries
■ Or feature flags

Feature Flags
■ Multiple versions are present in the code
■ Code path selected dynamically
if (useNewVersion()) {
...
} else { // use old version
...
}

Feature Flags
■ Enabled for certain users
○ Or a random sample
■ Also “shadowing”
○ both code paths executed
○ only old code used
○ new code verified

Related Concerns
■ API versioning
■ Databases
■ Deployment state (see Flux from Weaveworks)
■ Monitoring
■ Testing-in-production vs staging

The Future
■ More tooling building on top of Kubernetes
■ Primarily “service meshes”
○ Istio, linkerd
○ Sidecar containers providing intelligent routing
○ Out-of-the box support for canaries, blue-green
○ Better monitoring and observability

Thanks for Listening!
@adrianmouat

References
■ Site Reliability Engineering
■ BitMovin Canary Deployments on Kubernetes
■ New Stack Article on Deployment Strategies by Etienne Tremel
■ Netflix Articles
○ http://techblog.netflix.com/2013/08/deploying-netflix-api.html
○ https://www.infoq.com/presentations/canary-analysis-deployment-pattern
■ Thanks to Tom Wilkie & Weave, colleagues at Container Solutions, Dave Farley, Ben
Whaley, Matt Heath, Daniel Hoelbling-Inzko

Adrian Mouat - Microservice deployment techniques

More Related Content

What's hot

Similar to Adrian Mouat - Microservice deployment techniques

Recently uploaded

Adrian Mouat - Microservice deployment techniques