Releasing To Production Every Week India

4,500 views

Published on

Published in: Technology, Business

Releasing To Production Every Week India

  1. 1. Releasing to Production Every Week Owen Rogers Twitter: @exortech http://exortech.com/blog
  2. 2. 1 year
  3. 3. 49 releases
  4. 4. 49 releases (excluding patch releases)
  5. 5. ~ 1 release / week
  6. 6. summary
  7. 7. lessons learned
  8. 8. process
  9. 9. practices
  10. 10. tools
  11. 11. Poll:
  12. 12. how long is your release cycle?
  13. 13. Me
  14. 14. Vancouver
  15. 15. > 15 teams
  16. 16. Agile methods
  17. 17. Company
  18. 18. start up
  19. 19. real-time energy monitoring
  20. 20. building energy efficiency
  21. 21. save $
  22. 22. save kW
  23. 23. save
  24. 24. aggregate data
  25. 25. hosted solution
  26. 26. SaaS
  27. 27. Java
  28. 28. Team
  29. 29. • 9 developers
  30. 30. • 9 developers • 1 product manager
  31. 31. • 9 developers • 1 product manager • 1 graphic designer
  32. 32. • 9 developers • 1 product manager • 1 graphic designer • 1 tester
  33. 33. what’s missing?
  34. 34. operations?
  35. 35. support?
  36. 36. QA?
  37. 37. project manager?
  38. 38. DBA?
  39. 39. generalizing specialists
  40. 40. fungibility
  41. 41. rotating responsibility
  42. 42. daily support duty
  43. 43. few hand-offs
  44. 44. empowered
  45. 45. Process
  46. 46. Goal: • to surface risk as early as possible to keep problems out of production
  47. 47. maximize feedback
  48. 48. shrink time
  49. 49. Continuous Integration
  50. 50. Continuous Integration • 3-4 Commits/dev/day
  51. 51. Automated Build
  52. 52. Automated Build • 3 minute build
  53. 53. Automated Build • 3 minute build • 30-40 builds/day
  54. 54. Automated Build • 3 minute build • 30-40 builds/day
  55. 55. Automated Test Deploy
  56. 56. Automated Test Deploy • 4-5 times per day
  57. 57. Automated Test Deploy • 4-5 times per day • Scheduled nightly at 11PM
  58. 58. Daily Standup
  59. 59. Daily Standup • Weekly company standup
  60. 60. Daily Review
  61. 61. Daily Review • Quick peer feature review
  62. 62. Weekly Demo
  63. 63. Weekly Demo • 15 minute company- wide user-driven demo
  64. 64. weekly cycle Mon Tues Wed Thurs Fri 9:30AM 10AM 11pm 5pm Planning User-driven Cut-off Release Meeting Demo Release Testing
  65. 65. Sustainable Pace
  66. 66. Sustainable Pace
  67. 67. Sustainable Pace • regular release window
  68. 68. Sustainable Pace • regular release window • quality of life and code
  69. 69. Sustainable Pace • regular release window • quality of life and code • part of culture
  70. 70. Sustainable Pace • regular release window • quality of life and code • part of culture • company heartbeat
  71. 71. “agile”
  72. 72. Scrum?
  73. 73. XP?
  74. 74. what’s missing?
  75. 75. user stories?
  76. 76. user stories?
  77. 77. user stories?
  78. 78. story boards?
  79. 79. story boards?
  80. 80. story boards?
  81. 81. estimation?
  82. 82. estimation?
  83. 83. estimation? fits or it doesn’t “muda”
  84. 84. pairing?
  85. 85. pairing?
  86. 86. pairing? as needed
  87. 87. pairing? as needed code reviews
  88. 88. pairing? as needed code reviews open workspace
  89. 89. TDD?
  90. 90. TDD?
  91. 91. TDD? 52% coverage
  92. 92. TDD? 52% coverage + Selenium
  93. 93. TDD? 52% coverage + Selenium
  94. 94. retrospectives?
  95. 95. retrospectives?
  96. 96. retrospectives? Release Review
  97. 97. retrospectives? Release Review 5 Whys
  98. 98. lightweight RCA
  99. 99. 5 Whys
  100. 100. just-in-time
  101. 101. within 24 hours
  102. 102. 1 Jira per question
  103. 103. Practices
  104. 104. Daily support rotation
  105. 105. 5 Whys
  106. 106. Continuous Monitoring
  107. 107. Continuous Monitoring Socket App Server LogMaster Logger (async) WARN | ERROR
  108. 108. Continuous Monitoring
  109. 109. Continuous Monitoring • Proactive response to issues
  110. 110. Continuous Monitoring • Proactive response to issues • Clean logs
  111. 111. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling
  112. 112. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail
  113. 113. Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail • Managing the signal-to-noise ratio
  114. 114. Continuous Monitoring
  115. 115. Continuous Monitoring
  116. 116. Continuous Monitoring
  117. 117. Continuous Monitoring App Server web app gmond gmetad UDP multicast RRD
  118. 118. Continuous Monitoring
  119. 119. Continuous Monitoring
  120. 120. Continuous Monitoring
  121. 121. Continuous Monitoring
  122. 122. Zero-downtime Deployment
  123. 123. Zero-downtime Deploys Load Balancer App Server App Server App Server Deploy
  124. 124. Zero-downtime Deploys
  125. 125. Zero-downtime Deploys • Fully automated deployment
  126. 126. Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity
  127. 127. Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity • Gracefully bring down servers
  128. 128. Zero-downtime Deploys What about the DB?
  129. 129. Zero-downtime Deploys Bering
  130. 130. Zero-downtime Deploys 012_rename_login_id_to_user_name ALTER TABLE USER RENAME COLUMN login_id user_name
  131. 131. Zero-downtime Deploys 012_rename_login_id_to_user_name 012_add_user_name_column 02_remove_login_id_column expansion contraction
  132. 132. Zero-downtime Deploys
  133. 133. Zero-downtime Deploys • expansion
  134. 134. Zero-downtime Deploys • expansion • preserves backwards compatibility
  135. 135. Zero-downtime Deploys • expansion • preserves backwards compatibility • contract
  136. 136. Zero-downtime Deploys • expansion • preserves backwards compatibility • contract • clean up
  137. 137. Zero-downtime Deploys
  138. 138. Zero-downtime Deploys • Typically safe expansion operations:
  139. 139. Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data
  140. 140. Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations:
  141. 141. Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations: • rename or remove columns or tables
  142. 142. Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations: • rename or remove columns or tables • add constraint
  143. 143. Zero-downtime Deploys /expand /... /143 001_create_login_table 002_add_login_id_index 003_create_user_table 004_create_group_table /... /contract /... /143 001_add_inv_constraint 002_drop_alias_column /...
  144. 144. Zero-downtime Deploys
  145. 145. Zero-downtime Deploys • Database migration decoupled from the release
  146. 146. Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback
  147. 147. Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback • Some additional complexity in writing migrations
  148. 148. Release Options
  149. 149. Traditional Release
  150. 150. Traditional Release
  151. 151. Traditional Release • high risk
  152. 152. Traditional Release • high risk • not scalable
  153. 153. Traditional Release • high risk • not scalable • prolonged release: “Don’t release until it’s all ready”
  154. 154. Release Options
  155. 155. Release Options
  156. 156. Release Options • release to user subset (private beta)
  157. 157. Release Options • release to user subset (private beta) • split test (experimentation)
  158. 158. Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout
  159. 159. Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading
  160. 160. Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode)
  161. 161. Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode) • defer commit
  162. 162. Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode) • defer commit • defer release
  163. 163. Release Options
  164. 164. Release Options • Features running in production for weeks before fully available
  165. 165. Release Options • Features running in production for weeks before fully available • lots of data
  166. 166. Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term”
  167. 167. Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term” • Production levers
  168. 168. Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term” • Production levers • WANGMI: the discipline to defer
  169. 169. Use Production Data
  170. 170. Use Production Data
  171. 171. Use Production Data • Upload Nightly backups
  172. 172. Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release
  173. 173. Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand
  174. 174. Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand • Cleansed
  175. 175. Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand • Cleansed • Test data is a liability
  176. 176. Single Feature Release
  177. 177. • Daily Support Rotation • 5 Whys • Continuous Monitoring • Zero-downtime deployment • Release Options • Use Production Data • Single Feature Release
  178. 178. Tools
  179. 179. Questions? owen@pulseenergy.com

×