• Save
Continuous Deployment
Upcoming SlideShare
Loading in...5

Continuous Deployment



A lightning talk I gave to our development team

A lightning talk I gave to our development team



Total Views
Views on SlideShare
Embed Views



5 Embeds 10

http://a0.twimg.com 5
http://www.linkedin.com 2
http://us-w1.rockmelt.com 1
https://twitter.com 1
https://www.linkedin.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • -this had this single biggest effect on releasing more frequently at Etsy
  • “ Any developer can deploy their own code, whenever they want, to whatever fleet they want.” -Jon Jenkins, Director of Platform Analysis, Amazon.com
  • 5.7 million members. 6.5 million items listed. 775 million page views per month, ~$200milion sales

Continuous Deployment Continuous Deployment Presentation Transcript

  • Continuous Deployment
  • Continuous Deployment is the practice of shipping your code as frequently as possible.
  • Roots
    • Agile & Lean software development
    • Development teams work in iterations with incremental delivery
    • Operations has not traditionally managed to keep up with the demands of 'the business'
    • Cloud Computing, aka Utility Computing
  • Secret Sauce
    • Operations is the new “Secret Sauce” of many successful businesses (2007 – Adam Jacob, co-founder Opscode)
    • What is in the Sauce?
    • “ A collection of techniques to avoid sucking.” (@Kellanem – VP Engineering, Etsy)
    • -I'll get to these techniques in a bit...
  • Who does this?
    • You might have heard of...
    • IMVU
    • Wordpress
    • Etsy
    • Flickr
    • Wealthfront
    • 37 Signals
    • Smart Frog
    • Prezi
    • Outbrain
    • Digg4
    • Heyo
    • Atlassian
    • Quora
    • Countless others...
  • So you might be thinking this is just for small companies or startups...
  • Oh, and also these guys: Google Amazon Facebook LinkedIn
  • So now the WHY?
    • Why would companies deploy to Production multiple times every day?
    • Smaller changes = less risk
    • More frequent changes = closer feedback loop to knowing when you've introduced a bug
    • Closer feedback from customers
    • Competition advantage
  • Stop!
    • That's crazy! How can you deploy to Production multiple times every day?!
    • What about release management, risk management, project management, service management, etc?
    • What about QA?
    • What about all of our policies and best practices?
  • Continuous Deployment != Deploying new features without Coordination and planning
  • What's Lean got to do with it?
    • Lean Principles:
    • Optimize the Whole
    • Build Quality In
    • Learn Constantly
    • Deliver Fast
    • Engage Everyone
    • Keep Getting Better
  • Lean into it...
    • “ Commit to frequent deploys and
    • the tooling to support it”
  • Step 1 Get Deployment Simple [Lean principles: Deliver Fast, Build Quality In, Optimize the Whole]
  • Step 2 Developers do deployments [Lean principles: Deliver Fast, Optimize the Whole, Engage Everyone]
  • Step 3 Measure Everything [Lean principles: Learn Constantly, Keep Getting Better]
  • Step 4 Automate Everything [Lean principles: Optimize the Whole, Build Quality In, Deliver Fast, Keep Getting Better]
  • What does the workstream look like? Old way Write code->Commit->CI Tests->Manual deploy to test/load environment->Manual Load Tests->Manual Regression Tests->Deploy to production->Observe metrics New way Write code->Commit->CI Tests->Auto deploy to testing/load environment->Automated Tests->Deploy to production->Observe metrics -LonelyPlanet
  • What about QA?
    • While some companies claim not to have QA, more often they have embedded QA.
    • Most companies have dedicated software development teams that write automated tests above unit/component tests.
      • -Facebook: Test Engineering Team
    • IMVU calls this their “Immune System”
  • What about Process?
    • “ Once you add a process, it never goes away”
    • “ bad process is about fear”
  • Metrics of CD in action:
    • Amazon.com
    • Mean time between deployments: 11.6s
    • Max # of deploys in a single hour: 1079
    • Mean # of hosts simultaneously receiving a deployment: 10,000
    • Max # of hosts receiving a deployment: 30,000
  • Metrics of this in action:
    • Etsy
    • 729 deploys to Production, Nov 2010
    • 6 change related incidents in 2010 (3500 deploys)
    • Deployed new big feature on Dec 1 st , 4 th busiest day of the year without FEAR
  • Etsy cont.
    • CI System – 10 machine Jenkins cluster
    • Types of tests: unit, integration, smoke, functional
    • 7000 “trunk tests”
      • 30 minutes to execute, so they don't run sequential
      • 11 minutes in parallel
    • Functional tests = expensive to write and maintain, so mission critical parts of the site only
    • Deploy takes 20 minutes to reach Prod
  • Even more Etsy
    • Engineers deploy to Prod on Day 1 to get over deployment fear
    • Use config flags for dark launches
    • “ Core platform team”
      • Deployment tools
      • Front end and back end site performance
      • Large scale data migrations
      • Caching architecture
  • More Etsy. Really?
    • Yes. They've now opensourced
      • Logster
      • Statsd
      • Deployinator
    • Two books by John Allspaw:
      • Web Operations (Allspaw & Jesse Robbins)
      • The Art of Capacity Planning
    • Awesome engineering blog: http://codeascraft.etsy.com/
  • Other techniques
    • Work in Trunk, Branch in Code
      • Set true by config, cookie, ip address, probability
      • Dark launch
      • A/B test
    • Schema changes -> only on Thursdays :)
  • Other techniques
    • Operability Review
    • What could possibly go wrong?
    • What is our communication strategy?
    • What is our contingency plan?
    • Blame free post-mortems
    • http://velocityconf.com/velocity2011/public/schedule/detail/19766
  • What to measure?
    • Mean Time Between Failures
    • Mean Time To Detect
    • Mean Time To Repair
    • Changes per week
    • Change success rate
  • Ugly (legacy) metrics
    • Change management overhead
    • Number of people to provision env
    • Number of people to deploy
    • Number of people to access Prod logs
    • Number of manual steps to deploy
    • Number of manual steps to provision env
    • Number of manual steps to access Prod logs
    • How long to do any of the above?
  • The person who says it cannot be done should not interrupt the person doing it. --Chinese Proverb