Continuous Deployment

Continuous Deployment is the practice of shipping your code as frequently as possible.

Roots
Agile & Lean software development
Development teams work in iterations with incremental delivery
Operations has not traditionally managed to keep up with the demands of 'the business'
Cloud Computing, aka Utility Computing

Secret Sauce
Operations is the new “Secret Sauce” of many successful businesses (2007 – Adam Jacob, co-founder Opscode)
What is in the Sauce?
“ A collection of techniques to avoid sucking.” (@Kellanem – VP Engineering, Etsy)
-I'll get to these techniques in a bit...

Who does this?
You might have heard of...
IMVU
Wordpress
Etsy
Flickr
Wealthfront
37 Signals
Smart Frog
Prezi
Outbrain
Digg4
Heyo
Atlassian
Quora
Countless others...

So you might be thinking this is just for small companies or startups...

Oh, and also these guys: Google Amazon Facebook LinkedIn

So now the WHY?
Why would companies deploy to Production multiple times every day?
Smaller changes = less risk
More frequent changes = closer feedback loop to knowing when you've introduced a bug
Closer feedback from customers
Competition advantage

Stop!
That's crazy! How can you deploy to Production multiple times every day?!
What about release management, risk management, project management, service management, etc?
What about QA?
What about all of our policies and best practices?

Continuous Deployment != Deploying new features without Coordination and planning

What's Lean got to do with it?
Lean Principles:
Optimize the Whole
Build Quality In
Learn Constantly
Deliver Fast
Engage Everyone
Keep Getting Better

Lean into it...
“ Commit to frequent deploys and
the tooling to support it”

Step 1 Get Deployment Simple [Lean principles: Deliver Fast, Build Quality In, Optimize the Whole]

Step 2 Developers do deployments [Lean principles: Deliver Fast, Optimize the Whole, Engage Everyone]

Step 3 Measure Everything [Lean principles: Learn Constantly, Keep Getting Better]

Step 4 Automate Everything [Lean principles: Optimize the Whole, Build Quality In, Deliver Fast, Keep Getting Better]

What does the workstream look like? Old way Write code->Commit->CI Tests->Manual deploy to test/load environment->Manual Load Tests->Manual Regression Tests->Deploy to production->Observe metrics New way Write code->Commit->CI Tests->Auto deploy to testing/load environment->Automated Tests->Deploy to production->Observe metrics -LonelyPlanet

What about QA?
While some companies claim not to have QA, more often they have embedded QA.
Most companies have dedicated software development teams that write automated tests above unit/component tests.
-Facebook: Test Engineering Team
IMVU calls this their “Immune System”

What about Process?
“ Once you add a process, it never goes away”
“ bad process is about fear”

Metrics of CD in action:
Amazon.com
Mean time between deployments: 11.6s
Max # of deploys in a single hour: 1079
Mean # of hosts simultaneously receiving a deployment: 10,000
Max # of hosts receiving a deployment: 30,000

Metrics of this in action:
Etsy
729 deploys to Production, Nov 2010
6 change related incidents in 2010 (3500 deploys)
Deployed new big feature on Dec 1 st , 4 th busiest day of the year without FEAR

Etsy cont.
CI System – 10 machine Jenkins cluster
Types of tests: unit, integration, smoke, functional
7000 “trunk tests”
30 minutes to execute, so they don't run sequential
11 minutes in parallel
Functional tests = expensive to write and maintain, so mission critical parts of the site only
Deploy takes 20 minutes to reach Prod

Even more Etsy
Engineers deploy to Prod on Day 1 to get over deployment fear
Use config flags for dark launches
“ Core platform team”
Deployment tools
Front end and back end site performance
Large scale data migrations
Caching architecture

More Etsy. Really?
Yes. They've now opensourced
Logster
Statsd
Deployinator
Two books by John Allspaw:
Web Operations (Allspaw & Jesse Robbins)
The Art of Capacity Planning
Awesome engineering blog: http://codeascraft.etsy.com/

Other techniques
Work in Trunk, Branch in Code
Set true by config, cookie, ip address, probability
Dark launch
A/B test
Schema changes -> only on Thursdays :)

Other techniques
Operability Review
What could possibly go wrong?
What is our communication strategy?
What is our contingency plan?
Blame free post-mortems
http://velocityconf.com/velocity2011/public/schedule/detail/19766

What to measure?
Mean Time Between Failures
Mean Time To Detect
Mean Time To Repair
Changes per week
Change success rate

Ugly (legacy) metrics
Change management overhead
Number of people to provision env
Number of people to deploy
Number of people to access Prod logs
Number of manual steps to deploy
Number of manual steps to provision env
Number of manual steps to access Prod logs
How long to do any of the above?

The person who says it cannot be done should not interrupt the person doing it. --Chinese Proverb

http://a0.twimg.com	5
http://us-w1.rockmelt.com	1
http://www.linkedin.com	1
https://twitter.com	1

Continuous Deployment

by Brian Henerey

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 8

Statistics

Continuous Deployment Presentation Transcript