Continuous
Deployment
        Brian Moon
       dealnews.com
       @brianlmoon
http://brian.moonspot.net/
What is it?
    It is the culture of updating production
software when individual new features or bug
         fixes are ready for production.


The process will be different from one shop to
 another, but the culture is where it counts.
Production Code Updates per Day Q1
15




10




 5




 0
      1/1   1/4   1/7   1/10 1/13 1/16 1/19 1/22 1/25 1/28 1/31    2/3   2/6   2/9   2/12 2/15 2/18 2/21 2/24 2/27   3/1   3/4   3/7   3/10 3/13 3/16 3/19 3/22 3/25



                                                                  Production Puppet Updates per Day Q1
10



 8



 6 Also  made 43 SQL
     Schema changes
 4



 2



 0
      1/1   1/4   1/7   1/10 1/13 1/16 1/19 1/22 1/25 1/28 1/31    2/3   2/6   2/9   2/12 2/15 2/18 2/21 2/24 2/27   3/1   3/4   3/7   3/10 3/13 3/16 3/19 3/22 3/25
Who Does This?

• Facebook
• Amazon
• Twitter
• Etsy
• Flickr
What is it not?

• Nightly Builds - nightly builds are not
  always ready
• Cowboy Development - rolling code for
  the sake of rolling code
Continuous Deployment

Continuous Integration (CI)



     Version Control
Developer   Developer    Developer



              Trunk      CI of all developer code




             Staging     CI with production code




            Production
Benefits
Ownership


• Developers roll code to production
• Developers have access to production
  servers and production data
Agility

• When bugs to occur, it is much quicker to
  identify what might have caused it.
• When new business requirements arise,
  time to implementation is greatly reduced.
Real Example
•   11:45:47 AM - Editor reports a bug in a critical
    tool

•   11:51:29 AM - Senior developer analyzes logs, sees
    the bad data

•   14:18:14 PM - Developer merges changes into
    trunk. Begins testing on staging.

•   14:35:21 PM - Developer rolls fix. It is live and
    working.
Visual Example
Process
Will vary, do what works for you.
Development


• Developers work in their own branch
• They merge changes into trunk when they
  are working
Testing

• Code is merged from trunk into a
  production checkout on staging
• QA testing and unit testing is performed
  there by the developer or via peer review
• When it is determined ready, it is
  committed to production branch
Deployment
• The developer responsible for the change
  logs in to a production server using a
  limited privilege user and updates the code
  via a script.
• This central server updates all production
  servers with the new code as needed.
• The developer then does a second pass of
  testing on production as needed.
Monitoring
• Graph Everything™ and alert on anomalies
• Tools:
 • Graphite
 • Cacti/Ganglia
 • Circonus/Reconnoiter
 • Loggly/HopToad
 • Nagios
 • Keynote/AlertSite/Gomez
Examples
• We send PHP Error counts to Circonus. It
  alerts us when more than 1/5/10 per
  second occur, escalating as needed
• Nagios monitors load averages of Apache
  servers and alerts when it is too high
• AlertSite monitors our page load times
  every 15 minutes from 30 locations and
  alerts when it slows down
Projects
• Critical people (marketing, management,
  etc.) as well as a senior developer discuss
  the project
• Once the big picture is clear, senior
  developers break down the work into
  discrete tasks
• Tasks are completed and rolled as they are
  ready, not waiting on all parts to be done.
Real Example

• For SEO purposes, we wanted content that
  had expired to redirect to a page of similar
  content rather than living forever as an
  expired article. But, only for certain
  categories of content.
Discrete Tasks

• Update the category maintenance tools,
  adding a flag for this behavior
• Change the code that updates our front
  end databases, adding this flag
• Change the code that generates a page to
  consider this new flag
Summary
• It is a culture, not a strict process
• However, there are common processes that
  can be copied
• Give people ownership and they will
  flourish
• Monitor after deployment
People Smarter Than Me

• John Allspaw (Etsy, formerly Flickr)
 • http://slidesha.re/jaetsy
 • http://bit.ly/10deploys
• Theo Schlossnagle (OmniTI)
 • http://bit.ly/tsopscareer
Continuous
Deployment
        Brian Moon
       dealnews.com
       @brianlmoon
http://brian.moonspot.net/

Continuous Deployment

  • 1.
    Continuous Deployment Brian Moon dealnews.com @brianlmoon http://brian.moonspot.net/
  • 2.
    What is it? It is the culture of updating production software when individual new features or bug fixes are ready for production. The process will be different from one shop to another, but the culture is where it counts.
  • 3.
    Production Code Updatesper Day Q1 15 10 5 0 1/1 1/4 1/7 1/10 1/13 1/16 1/19 1/22 1/25 1/28 1/31 2/3 2/6 2/9 2/12 2/15 2/18 2/21 2/24 2/27 3/1 3/4 3/7 3/10 3/13 3/16 3/19 3/22 3/25 Production Puppet Updates per Day Q1 10 8 6 Also made 43 SQL Schema changes 4 2 0 1/1 1/4 1/7 1/10 1/13 1/16 1/19 1/22 1/25 1/28 1/31 2/3 2/6 2/9 2/12 2/15 2/18 2/21 2/24 2/27 3/1 3/4 3/7 3/10 3/13 3/16 3/19 3/22 3/25
  • 4.
    Who Does This? •Facebook • Amazon • Twitter • Etsy • Flickr
  • 5.
    What is itnot? • Nightly Builds - nightly builds are not always ready • Cowboy Development - rolling code for the sake of rolling code
  • 6.
  • 7.
    Developer Developer Developer Trunk CI of all developer code Staging CI with production code Production
  • 8.
  • 9.
    Ownership • Developers rollcode to production • Developers have access to production servers and production data
  • 10.
    Agility • When bugsto occur, it is much quicker to identify what might have caused it. • When new business requirements arise, time to implementation is greatly reduced.
  • 11.
    Real Example • 11:45:47 AM - Editor reports a bug in a critical tool • 11:51:29 AM - Senior developer analyzes logs, sees the bad data • 14:18:14 PM - Developer merges changes into trunk. Begins testing on staging. • 14:35:21 PM - Developer rolls fix. It is live and working.
  • 12.
  • 13.
    Process Will vary, dowhat works for you.
  • 14.
    Development • Developers workin their own branch • They merge changes into trunk when they are working
  • 15.
    Testing • Code ismerged from trunk into a production checkout on staging • QA testing and unit testing is performed there by the developer or via peer review • When it is determined ready, it is committed to production branch
  • 16.
    Deployment • The developerresponsible for the change logs in to a production server using a limited privilege user and updates the code via a script. • This central server updates all production servers with the new code as needed. • The developer then does a second pass of testing on production as needed.
  • 17.
    Monitoring • Graph Everything™and alert on anomalies • Tools: • Graphite • Cacti/Ganglia • Circonus/Reconnoiter • Loggly/HopToad • Nagios • Keynote/AlertSite/Gomez
  • 18.
    Examples • We sendPHP Error counts to Circonus. It alerts us when more than 1/5/10 per second occur, escalating as needed • Nagios monitors load averages of Apache servers and alerts when it is too high • AlertSite monitors our page load times every 15 minutes from 30 locations and alerts when it slows down
  • 19.
    Projects • Critical people(marketing, management, etc.) as well as a senior developer discuss the project • Once the big picture is clear, senior developers break down the work into discrete tasks • Tasks are completed and rolled as they are ready, not waiting on all parts to be done.
  • 20.
    Real Example • ForSEO purposes, we wanted content that had expired to redirect to a page of similar content rather than living forever as an expired article. But, only for certain categories of content.
  • 21.
    Discrete Tasks • Updatethe category maintenance tools, adding a flag for this behavior • Change the code that updates our front end databases, adding this flag • Change the code that generates a page to consider this new flag
  • 22.
    Summary • It isa culture, not a strict process • However, there are common processes that can be copied • Give people ownership and they will flourish • Monitor after deployment
  • 23.
    People Smarter ThanMe • John Allspaw (Etsy, formerly Flickr) • http://slidesha.re/jaetsy • http://bit.ly/10deploys • Theo Schlossnagle (OmniTI) • http://bit.ly/tsopscareer
  • 24.
    Continuous Deployment Brian Moon dealnews.com @brianlmoon http://brian.moonspot.net/