Releasing To Production Every Week

2,940 views

Published on

Presented at DevTeach Vancouver 2009

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,940
On SlideShare
0
From Embeds
0
Number of Embeds
1,007
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Releasing To Production Every Week

  1. 1. Releasing to Production Every Week
  2. 2. 1 year
  3. 3. 46 releases
  4. 4. 1 rollback
  5. 5. summary
  6. 6. lessons learned
  7. 7. poll:
  8. 8. how long is your release cycle?
  9. 9. Company
  10. 10. real-time energy monitoring
  11. 11. building energy efficiency
  12. 12. save $
  13. 13. save kW
  14. 14. save
  15. 15. aggregate data
  16. 16. hosted solution
  17. 17. SaaS
  18. 18. Java
  19. 19. learn
  20. 20. Team
  21. 21. • 9 developers
  22. 22. • 9 developers • 1 product manager
  23. 23. • 9 developers • 1 product manager • 1 graphic designer
  24. 24. • 9 developers • 1 product manager • 1 graphic designer • 1 tester*
  25. 25. • 9 developers • 1 product manager • 1 graphic designer • 1 tester* * recently added
  26. 26. what’s missing?
  27. 27. operations?
  28. 28. support?
  29. 29. QA?
  30. 30. project manager?
  31. 31. DBA?
  32. 32. generalizing specialists
  33. 33. fungibility
  34. 34. rotating responsibility
  35. 35. daily support duty
  36. 36. few hand-offs
  37. 37. empowered
  38. 38. Process
  39. 39. Goal: • to surface risk as early as possible to keep problems out of production
  40. 40. maximize feedback
  41. 41. shrink time
  42. 42. Continuous Integration
  43. 43. Continuous Integration • 3-4 Commits/dev/day
  44. 44. Automated Build
  45. 45. Automated Build • 3 minute build
  46. 46. Automated Build • 3 minute build • 30-40 builds/day
  47. 47. Automated Build • 3 minute build • 30-40 builds/day
  48. 48. Automated Test Deploy
  49. 49. Automated Test Deploy • 4-5 times per day
  50. 50. Automated Test Deploy • 4-5 times per day • Scheduled nightly at 11PM
  51. 51. Daily Standup
  52. 52. Daily Standup • Weekly company standup
  53. 53. Daily Review
  54. 54. Daily Review • Quick peer feature review
  55. 55. Weekly Demo
  56. 56. Weekly Demo • 15 minute company- wide user-driven demo
  57. 57. weekly cycle Mon Tues Wed Thurs Fri 9:30AM 10:00AM Release 5pm Planning User-driven Testing Release Meeting Demo
  58. 58. “agile”
  59. 59. Scrum?
  60. 60. XP?
  61. 61. what’s missing?
  62. 62. user stories?
  63. 63. user stories?
  64. 64. user stories?
  65. 65. story boards?
  66. 66. story boards?
  67. 67. story boards?
  68. 68. estimation?
  69. 69. estimation?
  70. 70. estimation? fits or it doesn’t
  71. 71. pairing?
  72. 72. pairing?
  73. 73. pairing? as needed
  74. 74. pairing? as needed code reviews
  75. 75. TDD?
  76. 76. TDD?
  77. 77. TDD? 30% coverage
  78. 78. TDD? 30% coverage + Selenium
  79. 79. TDD? 30% coverage + Selenium
  80. 80. retrospectives?
  81. 81. retrospectives?
  82. 82. retrospectives? Release Review
  83. 83. retrospectives? Release Review 5 Whys
  84. 84. 5 Whys
  85. 85. lightweight RCA
  86. 86. just-in-time
  87. 87. within 24 hours
  88. 88. 1 Jira per question
  89. 89. Practices
  90. 90. Daily support rotation
  91. 91. 5 Whys
  92. 92. Continuous Monitoring
  93. 93. Continuous Monitoring Socket App Server LogMaster Logger WARN | ERROR
  94. 94. Continuous Monitoring
  95. 95. Continuous Monitoring • Proactive response to issues
  96. 96. Continuous Monitoring • Proactive response to issues • Clean logs
  97. 97. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling
  98. 98. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail
  99. 99. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail • Managing the signal-to-noise ratio
  100. 100. Continuous Monitoring
  101. 101. Continuous Monitoring
  102. 102. Continuous Monitoring
  103. 103. Continuous Monitoring App Server web app gmond gmetad UDP multicast RRD
  104. 104. Continuous Monitoring
  105. 105. Continuous Monitoring
  106. 106. Continuous Monitoring
  107. 107. Continuous Monitoring
  108. 108. Test mirrors Prod
  109. 109. Trust Test
  110. 110. Zero-downtime Deployment
  111. 111. Zero-downtime Deploys Load Balancer App Server App Server App Server Deploy
  112. 112. Zero-downtime Deploys
  113. 113. Zero-downtime Deploys • Fully automated deployment
  114. 114. Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity
  115. 115. Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity • Gracefully bring down servers
  116. 116. Zero-downtime Deploys What about the DB?
  117. 117. Zero-downtime Deploys Bering
  118. 118. Zero-downtime Deploys 001_create_login_table 001_add_inv_constraint 002_add_login_id_index 002_drop_alias_column 003_create_user_table 004_create_group_table Expand Contract
  119. 119. Zero-downtime Deploys
  120. 120. Zero-downtime Deploys • Database migration decoupled from the release
  121. 121. Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback
  122. 122. Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback • Some additional complexity in writing migrations
  123. 123. Incremental Rollout
  124. 124. Incremental Rollout
  125. 125. Incremental Rollout • New features are released to user subset (by role)
  126. 126. Incremental Rollout • New features are released to user subset (by role) • “Release is a marketing term”
  127. 127. Incremental Rollout • New features are released to user subset (by role) • “Release is a marketing term” • Production levers
  128. 128. Incremental Rollout • New features are released to user subset (by role) • “Release is a marketing term” • Production levers • Selective degredation
  129. 129. Production DB Restore
  130. 130. Production DB Restore
  131. 131. Production DB Restore • Nightly backups
  132. 132. Production DB Restore • Nightly backups • Automated Test DB refresh every Monday morning
  133. 133. Production DB Restore • Nightly backups • Automated Test DB refresh every Monday morning • Local DB refresh on demand
  134. 134. Production DB Restore • Nightly backups • Automated Test DB refresh every Monday morning • Local DB refresh on demand • Cleansed
  135. 135. WANGMI
  136. 136. WANGMI aka the discipline to defer
  137. 137. Single Feature Release
  138. 138. • Daily Support Rotation • 5 Whys • Continuous Monitoring • Test mirrors Prod • Zero-downtime deployment • Incremental rollout • Production DB restore • WANGMI - the discipline to defer • Single Feature Release
  139. 139. Tools
  140. 140. Questions? owen@pulseenergy.com
  141. 141. Nov 3, 4, 5 • Martin Fowler • Mary Poppendieck • Eric Evans • Michael Feathers • Michael Nygard
  142. 142. Questions? owen@pulseenergy.com
  143. 143. • Eric Ries: Continuous Deployment
  144. 144. • Daily rotating support duty • Log: 100MB/server/day • 30 KLOC
  145. 145. • ThoughtWorks • Working with clients to shorten their release cycle • “Agile” • CruiseControl.NET

×