Continuous Deployment of Architectural Change

1. Continuous Deployment of Architectural Change Matt Graham Core Engineer @ Etsy Continuous Deployer#iasanySeptember 27, 2011

2. A Brief History of Deployment

3. The Internet

4. Web Applications

5. Agility

6. Continuous

7. Database Deployment @ Etsy Thursday

8. What its all about● Broadly Applicable● Tools Help● Enables the Unfeasible

9. MTTR vs MTTF

10. MTTR vs MTTF● Cheaper for electrons

11. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons

12. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons● Cost prohibitive for humans

13. MTTR vs MTTF● Cheaper for ● Cheaper for electrons protons & neutrons● Cost prohibitive for humans● Necessitates low MTTD

14. The Case of Electrons● Consider a project that has 6 bugs● continuous deployment testing misses 4 & deploys fixes in 4 hours● monthly release testing misses only 2 & deploys fixes in 24 hours● 16 hours of broken < 48 hours of broken

15. Good Excuses● Infrequent Changes

16. Good Excuses● Infrequent Changes● Infrequent Executions

17. Good Excuses● Infrequent Changes● Infrequent Executions● Life and Death

18. Good Excuses● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment

19. Deployment Tools @ Etsy

20. Culture Before Tools● No planned deploys; only on demand

21. Culture Before Tools● No planned deploys; only on demand● A positive change is ready & tested

22. Culture Before Tools● No planned deploys; only on demand● A positive change is ready & tested● Software is stable & supported

23. Jenkins● Unit Tests

24. Jenkins● Unit Tests● Functional Tests

25. Jenkins● Unit Tests● Functional Tests● Manual Testing

26. Nagios & Naglite2github.com/lozzd/Naglite2

27. tail -f | grep

28. github.com/etsy/deployinator

29. IRC

30. Key ToolsCommunication Tools Graphs Feature Flags

31. Feature FlagsDeploy != Product Launch

32. Dark Launchdef get_feature_a_link(): return ...

33. Dark Launchdef get_feature_a_link(): if enabled(NewFeatureA): return new_feature_a_link() else: return old_feature_a_link()

34. Dark Launchapplication_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on

35. Ramp Upapplication_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on

39. AB Testing● Prove success of interface changes● Prove interest in new features

40. Graphs!!!

41. Ganglia

42. Graphite

43. Event Overlay

44. StatsDif ($success) { StatsD::timing(feature.runtime, $time);} else { StatsD::increment(feature.failure);} github.com/etsy/statsd

45. github.com/etsy/logster

46. github.com/etsy/logster

47. Community Communication

48. Forums / Message Boards

49. etsystatus.com

50. twitter.com/etsystatus

51. twitter.com/etsystatus

52. Deployment is First Class Deployment is a First Class Feature

53. Engineers are Users Too

54. Enable Architectural Change

55. Architecture is Relative

56. Organic Architecture

57. Premature Architecture

58. Premature Architecture What now?

59. Passing Time => Change● Scale● Product● Technology● Engineering Team

60. Passing Time => Change● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes

61. Examples● Images From Twisted to PHP● From PostgreSQL to MySQL Shards

62. From Twisted to PHP● Run Apache/PHP on a new port

63. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP

64. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service

65. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services

66. From Twisted to PHP● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version

67. PostgreSQL to MySQL Shards● Migrate table by table

68. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL

69. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL

70. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs

71. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL

72. PostgreSQL to MySQL Shards● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables

73. Continuous Deploy Pattern● Change in small steps

74. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config

75. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark

76. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel

77. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture

78. Continuous Deploy Pattern● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture

79. Review● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication● Enables Architectural Change● Its really fun!!!

80. Continuous Deploymentof Architectural ChangeMatt Graham http://twitter.com/lapsu http://lapsu.tvCore Engineer @ EtsyContinuous Deployer http://codeascraft.etsy.comWere Hiring! http://bit.ly/etsywantsawesome