Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Changing Etsy's Architectural Foundations with Continuous Deployment

1,841 views

Published on

Published in: Technology
  • Hey, thanks for the feedback! Glad you liked the hand-drawn graphs. One time I got sick of trying to make a chart drawing program do what I wanted, so I decided to draw them myself. I figured I could get away with it since I work at Etsy, the handmade marketplace.

    Enabling the unfeasible is about taking changes that would be too risky to do normally, and breaking them down into small enough chunks that they can be done safely.

    It is a lot of slides but I was able to get through it in ~40 minutes. Some of the slides were only up for a few seconds to get a laugh or something. And as you say, it was for a technical conference. Within the domain, this is talking about a different approach to what is otherwise a famliar topic.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Actually I forgot, you are not presenting to Etsy people, you are presenting to people familiar with architecture, so forget everything I just said. ; )
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Lots and lots of slides. Enough for a 3 hour presentation. Tip: never forget you are way way familiar with this stuff, and 95% are way way unfamiliar with it, so keep it slow and simple and fundamental, talk like you are presenting to a child, like Dean's kids Zack or Noah or even Akira. Just from looking at the slides, I think you are already aware of this, but if you try to cover all the ground you have laid out, you will almost certainly leave most of the people in the dust scratching their heads, wondering what just happened. ; )
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • A tale of 6 bugs looks like it will be both fun and instructional, and the phrase 'enable the unfeasible' makes me curious, makes me think: 'whaaaat?' ; )
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I like the humour and the handwritten graphs, it makes them more people-friendly for lower tech people and I think more digestible.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Changing Etsy's Architectural Foundations with Continuous Deployment

  1. 1. Changing EtsysArchitectural Foundation with Continuous Deployment Matt Graham Core Engineer @ Etsy Continuous Deployer#surgeconSeptember 28, 2012
  2. 2. Marketplace forHandmade GoodsGross Sales 2011: $537 millionTotal Members: 19 millionItems For Sale: 15 millionUniques / month: 40 millionPage Views / month: 1.4 billion
  3. 3. Architecture is Relative
  4. 4. Organic Architecture
  5. 5. Premature Architecture
  6. 6. Premature Architecture
  7. 7. Passing Time => Change● Scale● Product● Technology● Engineering Team
  8. 8. Passing Time => Change● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes
  9. 9. Architectural Change Antipattern
  10. 10. A Brief History of Deployment
  11. 11. The Internet
  12. 12. Agility
  13. 13. Continuous
  14. 14. What its all about● Reduce Failure Time● Start with Culture● Tools Help● Enable the Unfeasible
  15. 15. A Tale of Six Bugs
  16. 16. Six Bugs withMonthly Deploys 4 caught ---> 2 missed <--- fix live: 24 hours
  17. 17. Six Bugs withContinuous Deploys 2 caught ---> 4 missed <--- fix live: 6 hours
  18. 18. Failure Time2 Bugs * 24 Hours = 48 BH4 Bugs * 6 Hours = 24 BH Minimize BugHours 24 < 48
  19. 19. MTTR vs MTTF
  20. 20. Cost of RecoveryPhotons MinimalElectrons LowProtons & Neutrons HighHumans Prohibitive
  21. 21. Cost of Recovery$6 million in 1973 = $31m today
  22. 22. Good Excuses● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment
  23. 23. Medical Devices? No
  24. 24. NASA? No
  25. 25. Enterprise Software? Yes!
  26. 26. Print of Cards? No
  27. 27. App Store? No
  28. 28. Financial Transactions?
  29. 29. Financial Transactions? Yes!
  30. 30. Getting Started
  31. 31. Culture Before Tools● Throw out the deploy schedule● Ship changes when tested & ready● Software is stable & supported
  32. 32. Tools of Etsy Deployment
  33. 33. Jenkins● Unit Tests● Functional Tests
  34. 34. Jenkins● Unit Tests● Functional Tests● Manual Testing
  35. 35. Jenkins● Unit Tests● Functional Tests● Manual Testing
  36. 36. Nagios & Naglite2github.com/lozzd/Naglite2
  37. 37. tail -f | grep
  38. 38. github.com/etsy/deployinator
  39. 39. IRC
  40. 40. Graphs!!!
  41. 41. Ganglia
  42. 42. Graphite
  43. 43. Event Overlay
  44. 44. StatsDif ($success) { StatsD::timing(query.runtime, $time);} else { StatsD::increment(query.failure);} github.com/etsy/statsd
  45. 45. github.com/etsy/logster
  46. 46. github.com/etsy/logster
  47. 47. Practices @ Etsy Feature FlagsCustomer Communication
  48. 48. Feature FlagsDeploy != Product Launch
  49. 49. Dark Launchdef get_payment_link(): return ...
  50. 50. Dark Launchdef get_payment_link(): if enabled(creditcards): return creditcard_link() else: return check_link()
  51. 51. Dark Launchapplication_config: - creditcards: admin - NewFeatureB: off - NewFeatureC: on
  52. 52. Ramp Upapplication_config - creditcards: 1% - NewFeatureB: off - NewFeatureC: on
  53. 53. Ramp Upapplication_config - creditcards: 5% - NewFeatureB: off - NewFeatureC: on
  54. 54. Whoops!application_config: - creditcards: admin - NewFeatureB: off - NewFeatureC: on
  55. 55. Ramp Upapplication_config - creditcards: 5% - NewFeatureB: off - NewFeatureC: on
  56. 56. Ramp Upapplication_config - creditcards: 25% - NewFeatureB: off - NewFeatureC: on
  57. 57. Credit Cards are ONapplication_config - creditcards: 100% - NewFeatureB: off - NewFeatureC: on
  58. 58. AB Testing● Prove success of interface changes● Prove interest in new features
  59. 59. Community Communication
  60. 60. Forums / Message Boards
  61. 61. etsystatus.com
  62. 62. twitter.com/etsystatus
  63. 63. twitter.com/etsystatus
  64. 64. Deployment is First Class Deployment is a First Class Feature
  65. 65. Engineers are Users Too
  66. 66. Examples from Etsy● Photos From Twisted to PHP● PostgreSQL to MySQL Shards
  67. 67. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for each service● Shut down Twisted version
  68. 68. PostgreSQL to MySQL Shards● Migrate table by table● Tee writes to both DBs● Copy old data from PostgreSQL● Verify data matches● Ramp up reads from MySQL● Stop PostgreSQL writes
  69. 69. Continuous Deploy Pattern● Change in small steps● Dark launch via config● Iterations to prod while dark● Maintain old & new in parallel● Ramp up new architecture● Remove old architecture
  70. 70. Once Again● Minimize BugHours● Trash the Schedule● Iterate on the Tools● Make Big Changes
  71. 71. Mean Time To Addiction
  72. 72. Changing EtsysArchitectural Foundation with Continuous Deployment Matt Graham http://twitter.com/lapsu http://lapsu.tv Core Engineer @ Etsy Continuous Deployer http://codeascraft.etsy.com http://www.etsy.com/careers

×