Continuous Deployment of Architectural Change

2,041 views
1,919 views

Published on

Continuous deployment has proven to be a successful and even addicting part of Etsy's engineering culture. See where it's applicable, some of the tools that make it easy, and the kind of architectural change that it makes possible.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,041
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
50
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Continuous Deployment of Architectural Change

  1. 1. Continuous Deployment of Architectural Change Matt Graham Core Engineer @ Etsy Continuous Deployer#iasanySeptember 27, 2011
  2. 2. A Brief History of Deployment
  3. 3. The Internet
  4. 4. Web Applications
  5. 5. Agility
  6. 6. Continuous
  7. 7. Database Deployment @ Etsy Thursday
  8. 8. What its all about● Broadly Applicable● Tools Help● Enables the Unfeasible
  9. 9. MTTR vs MTTF
  10. 10. MTTR vs MTTF● Cheaper for electrons
  11. 11. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons
  12. 12. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons● Cost prohibitive for humans
  13. 13. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons● Cost prohibitive for humans● Necessitates low MTTD
  14. 14. The Case of Electrons● Consider a project that has 6 bugs● continuous deployment testing misses 4 & deploys fixes in 4 hours● monthly release testing misses only 2 & deploys fixes in 24 hours● 16 hours of broken < 48 hours of broken
  15. 15. Good Excuses● Infrequent Changes
  16. 16. Good Excuses● Infrequent Changes● Infrequent Executions
  17. 17. Good Excuses● Infrequent Changes● Infrequent Executions● Life and Death
  18. 18. Good Excuses● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment
  19. 19. Deployment Tools @ Etsy
  20. 20. Culture Before Tools● No planned deploys; only on demand
  21. 21. Culture Before Tools● No planned deploys; only on demand● A positive change is ready & tested
  22. 22. Culture Before Tools● No planned deploys; only on demand● A positive change is ready & tested● Software is stable & supported
  23. 23. Jenkins● Unit Tests
  24. 24. Jenkins● Unit Tests● Functional Tests
  25. 25. Jenkins● Unit Tests● Functional Tests● Manual Testing
  26. 26. Nagios & Naglite2github.com/lozzd/Naglite2
  27. 27. tail -f | grep
  28. 28. github.com/etsy/deployinator
  29. 29. IRC
  30. 30. Key ToolsCommunication Tools Graphs Feature Flags
  31. 31. Feature FlagsDeploy != Product Launch
  32. 32. Dark Launchdef get_feature_a_link(): return ...
  33. 33. Dark Launchdef get_feature_a_link(): if enabled(NewFeatureA): return new_feature_a_link() else: return old_feature_a_link()
  34. 34. Dark Launchapplication_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on
  35. 35. Ramp Upapplication_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on
  36. 36. Ramp Upapplication_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on
  37. 37. Ramp Upapplication_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on
  38. 38. Ramp Upapplication_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on
  39. 39. AB Testing● Prove success of interface changes● Prove interest in new features
  40. 40. Graphs!!!
  41. 41. Ganglia
  42. 42. Graphite
  43. 43. Event Overlay
  44. 44. StatsDif ($success) { StatsD::timing(feature.runtime, $time);} else { StatsD::increment(feature.failure);} github.com/etsy/statsd
  45. 45. github.com/etsy/logster
  46. 46. github.com/etsy/logster
  47. 47. Community Communication
  48. 48. Forums / Message Boards
  49. 49. etsystatus.com
  50. 50. twitter.com/etsystatus
  51. 51. twitter.com/etsystatus
  52. 52. Deployment is First Class Deployment is a First Class Feature
  53. 53. Engineers are Users Too
  54. 54. Enable Architectural Change
  55. 55. Architecture is Relative
  56. 56. Organic Architecture
  57. 57. Premature Architecture
  58. 58. Premature Architecture What now?
  59. 59. Passing Time => Change● Scale● Product● Technology● Engineering Team
  60. 60. Passing Time => Change● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes
  61. 61. Examples● Images From Twisted to PHP● From PostgreSQL to MySQL Shards
  62. 62. From Twisted to PHP● Run Apache/PHP on a new port
  63. 63. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP
  64. 64. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service
  65. 65. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services
  66. 66. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version
  67. 67. PostgreSQL to MySQL Shards● Migrate table by table
  68. 68. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL
  69. 69. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL
  70. 70. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs
  71. 71. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL
  72. 72. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables
  73. 73. Continuous Deploy Pattern● Change in small steps
  74. 74. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config
  75. 75. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark
  76. 76. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel
  77. 77. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture
  78. 78. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture
  79. 79. Review● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication● Enables Architectural Change● Its really fun!!!
  80. 80. Continuous Deploymentof Architectural ChangeMatt Graham http://twitter.com/lapsu http://lapsu.tvCore Engineer @ EtsyContinuous Deployer http://codeascraft.etsy.comWere Hiring! http://bit.ly/etsywantsawesome

×