Continuous Deployment
Continuous Deployment is the practice of shipping your code as frequently as possible.
Roots Agile & Lean software development Development teams work in iterations with incremental delivery  Operations has not traditionally managed to keep up with the demands of 'the business' Cloud Computing, aka Utility Computing
Secret Sauce Operations is the new “Secret Sauce” of many successful businesses (2007 – Adam Jacob, co-founder Opscode) What is in the Sauce? “ A collection of techniques to avoid sucking.” (@Kellanem – VP Engineering, Etsy) -I'll get to these techniques in a bit...
Who does this? You might have heard of... IMVU Wordpress Etsy Flickr Wealthfront 37 Signals Smart Frog Prezi Outbrain Digg4 Heyo Atlassian Quora Countless others...
So you might be thinking  this is just for small companies or startups...
Oh, and also these guys: Google Amazon Facebook LinkedIn
So now the WHY? Why would companies deploy to Production multiple times every day? Smaller changes = less risk More frequent changes = closer feedback loop to knowing when you've introduced a bug Closer feedback from customers Competition advantage
Stop! That's crazy! How can you deploy to Production multiple times every day?! What about release management, risk management, project management, service management, etc? What about QA? What about all of our policies and best practices?
Continuous Deployment != Deploying new features without Coordination and planning
What's Lean got to do with it? Lean Principles: Optimize the Whole Build Quality In Learn Constantly Deliver Fast Engage Everyone Keep Getting Better
Lean into it... “ Commit to frequent deploys and  the tooling to support it”
Step 1 Get Deployment Simple [Lean principles: Deliver Fast, Build Quality In, Optimize the Whole]
Step 2 Developers do deployments [Lean principles: Deliver Fast, Optimize the Whole,  Engage Everyone]
Step 3 Measure Everything [Lean principles: Learn Constantly, Keep Getting Better]
Step 4 Automate Everything [Lean principles: Optimize the Whole, Build Quality In, Deliver Fast, Keep Getting Better]
What does the workstream look like? Old way Write code->Commit->CI Tests->Manual deploy to test/load environment->Manual Load Tests->Manual Regression Tests->Deploy to production->Observe metrics New way Write code->Commit->CI Tests->Auto deploy to testing/load environment->Automated Tests->Deploy to production->Observe metrics -LonelyPlanet
What about QA? While some companies claim not to have QA, more often they have embedded QA. Most companies have dedicated software development teams that write  automated  tests above unit/component tests. -Facebook: Test Engineering Team IMVU calls this their “Immune System”
What about Process? “ Once you add a process, it  never  goes away” “ bad process is about fear”
Metrics of CD in action: Amazon.com Mean time between deployments: 11.6s Max # of deploys in a single hour: 1079 Mean # of hosts simultaneously receiving a deployment: 10,000 Max # of hosts receiving a deployment: 30,000
Metrics of this in action: Etsy 729 deploys to Production, Nov 2010 6 change related incidents in 2010 (3500 deploys) Deployed new big feature on Dec 1 st , 4 th  busiest day of the year without FEAR
Etsy cont. CI System – 10 machine Jenkins cluster Types of tests: unit, integration, smoke, functional 7000 “trunk tests” 30 minutes to execute, so they don't run sequential 11 minutes in parallel Functional tests = expensive to write and maintain, so mission critical parts of the site only Deploy takes 20 minutes to reach Prod
Even more Etsy Engineers deploy to Prod on Day 1 to get over deployment fear Use config flags for dark launches “ Core platform team”  Deployment tools Front end and back end site performance Large scale data migrations Caching architecture
More Etsy. Really? Yes. They've now opensourced Logster Statsd Deployinator Two books by John Allspaw: Web Operations (Allspaw & Jesse Robbins) The Art of Capacity Planning Awesome engineering blog:  http://codeascraft.etsy.com/
Other techniques Work in Trunk, Branch in Code Set true by config, cookie, ip address, probability Dark launch A/B test Schema changes -> only on Thursdays :)
Other techniques Operability Review What could possibly go wrong? What is our communication strategy? What is our contingency plan? Blame free post-mortems http://velocityconf.com/velocity2011/public/schedule/detail/19766
What to measure? Mean Time Between Failures Mean Time To Detect Mean Time To Repair Changes per week Change success rate
Ugly (legacy) metrics Change management overhead Number of people to provision env Number of people to deploy  Number of people to access Prod logs Number of manual steps to deploy Number of manual steps to provision env Number of manual steps to access Prod logs How long to do any of the above?
The person who says it cannot be done should not interrupt the person doing it. --Chinese Proverb

Continuous Deployment

  • 1.
  • 2.
    Continuous Deployment isthe practice of shipping your code as frequently as possible.
  • 3.
    Roots Agile &Lean software development Development teams work in iterations with incremental delivery Operations has not traditionally managed to keep up with the demands of 'the business' Cloud Computing, aka Utility Computing
  • 4.
    Secret Sauce Operationsis the new “Secret Sauce” of many successful businesses (2007 – Adam Jacob, co-founder Opscode) What is in the Sauce? “ A collection of techniques to avoid sucking.” (@Kellanem – VP Engineering, Etsy) -I'll get to these techniques in a bit...
  • 5.
    Who does this?You might have heard of... IMVU Wordpress Etsy Flickr Wealthfront 37 Signals Smart Frog Prezi Outbrain Digg4 Heyo Atlassian Quora Countless others...
  • 6.
    So you mightbe thinking this is just for small companies or startups...
  • 7.
    Oh, and alsothese guys: Google Amazon Facebook LinkedIn
  • 8.
    So now theWHY? Why would companies deploy to Production multiple times every day? Smaller changes = less risk More frequent changes = closer feedback loop to knowing when you've introduced a bug Closer feedback from customers Competition advantage
  • 9.
    Stop! That's crazy!How can you deploy to Production multiple times every day?! What about release management, risk management, project management, service management, etc? What about QA? What about all of our policies and best practices?
  • 10.
    Continuous Deployment !=Deploying new features without Coordination and planning
  • 11.
    What's Lean gotto do with it? Lean Principles: Optimize the Whole Build Quality In Learn Constantly Deliver Fast Engage Everyone Keep Getting Better
  • 12.
    Lean into it...“ Commit to frequent deploys and the tooling to support it”
  • 13.
    Step 1 GetDeployment Simple [Lean principles: Deliver Fast, Build Quality In, Optimize the Whole]
  • 14.
    Step 2 Developersdo deployments [Lean principles: Deliver Fast, Optimize the Whole, Engage Everyone]
  • 15.
    Step 3 MeasureEverything [Lean principles: Learn Constantly, Keep Getting Better]
  • 16.
    Step 4 AutomateEverything [Lean principles: Optimize the Whole, Build Quality In, Deliver Fast, Keep Getting Better]
  • 17.
    What does theworkstream look like? Old way Write code->Commit->CI Tests->Manual deploy to test/load environment->Manual Load Tests->Manual Regression Tests->Deploy to production->Observe metrics New way Write code->Commit->CI Tests->Auto deploy to testing/load environment->Automated Tests->Deploy to production->Observe metrics -LonelyPlanet
  • 18.
    What about QA?While some companies claim not to have QA, more often they have embedded QA. Most companies have dedicated software development teams that write automated tests above unit/component tests. -Facebook: Test Engineering Team IMVU calls this their “Immune System”
  • 19.
    What about Process?“ Once you add a process, it never goes away” “ bad process is about fear”
  • 20.
    Metrics of CDin action: Amazon.com Mean time between deployments: 11.6s Max # of deploys in a single hour: 1079 Mean # of hosts simultaneously receiving a deployment: 10,000 Max # of hosts receiving a deployment: 30,000
  • 21.
    Metrics of thisin action: Etsy 729 deploys to Production, Nov 2010 6 change related incidents in 2010 (3500 deploys) Deployed new big feature on Dec 1 st , 4 th busiest day of the year without FEAR
  • 22.
    Etsy cont. CISystem – 10 machine Jenkins cluster Types of tests: unit, integration, smoke, functional 7000 “trunk tests” 30 minutes to execute, so they don't run sequential 11 minutes in parallel Functional tests = expensive to write and maintain, so mission critical parts of the site only Deploy takes 20 minutes to reach Prod
  • 23.
    Even more EtsyEngineers deploy to Prod on Day 1 to get over deployment fear Use config flags for dark launches “ Core platform team” Deployment tools Front end and back end site performance Large scale data migrations Caching architecture
  • 24.
    More Etsy. Really?Yes. They've now opensourced Logster Statsd Deployinator Two books by John Allspaw: Web Operations (Allspaw & Jesse Robbins) The Art of Capacity Planning Awesome engineering blog: http://codeascraft.etsy.com/
  • 25.
    Other techniques Workin Trunk, Branch in Code Set true by config, cookie, ip address, probability Dark launch A/B test Schema changes -> only on Thursdays :)
  • 26.
    Other techniques OperabilityReview What could possibly go wrong? What is our communication strategy? What is our contingency plan? Blame free post-mortems http://velocityconf.com/velocity2011/public/schedule/detail/19766
  • 27.
    What to measure?Mean Time Between Failures Mean Time To Detect Mean Time To Repair Changes per week Change success rate
  • 28.
    Ugly (legacy) metricsChange management overhead Number of people to provision env Number of people to deploy Number of people to access Prod logs Number of manual steps to deploy Number of manual steps to provision env Number of manual steps to access Prod logs How long to do any of the above?
  • 29.
    The person whosays it cannot be done should not interrupt the person doing it. --Chinese Proverb

Editor's Notes

  • #13 -this had this single biggest effect on releasing more frequently at Etsy
  • #15 “ Any developer can deploy their own code, whenever they want, to whatever fleet they want.” -Jon Jenkins, Director of Platform Analysis, Amazon.com
  • #23 5.7 million members. 6.5 million items listed. 775 million page views per month, ~$200milion sales