1. Releasing to Production
Every Week
Owen Rogers
Twitter: @exortech
http://exortech.com/blog

2. 1 year

4. 49 releases

5. 49 releases
(excluding patch releases)

6. ~ 1 release / week

7. summary

8. lessons learned

9. process

10. practices

11. tools

12. Poll:

13. how long is your
release cycle?

14. Me

16. Vancouver

26. > 15 teams

27. Agile methods

30. Company

32. start up

33. real-time energy
monitoring

34. building energy
efficiency

35. save $

36. save kW

37. save

38. aggregate data

44. hosted solution

45. SaaS

46. Java

49. Team

51. • 9 developers

52. • 9 developers
• 1 product manager

53. • 9 developers
• 1 product manager
• 1 graphic designer

54. • 9 developers
• 1 product manager
• 1 graphic designer
• 1 tester

55. what’s missing?

56. operations?

57. support?

58. QA?

59. project manager?

60. DBA?

61. generalizing
specialists

62. fungibility

63. rotating responsibility

64. daily support duty

65. few hand-offs

66. empowered

67. Process

68. Goal:
• to surface risk as early
as possible to keep
problems out of
production

69. maximize feedback

70. shrink time

71. Continuous Integration

72. Continuous Integration
• 3-4 Commits/dev/day

73. Automated Build

74. Automated Build
• 3 minute build

75. Automated Build
• 3 minute build
• 30-40 builds/day

76. Automated Build
• 3 minute build
• 30-40 builds/day

77. Automated Test Deploy

78. Automated Test Deploy
• 4-5 times per day

79. Automated Test Deploy
• 4-5 times per day
• Scheduled nightly at
11PM

80. Daily Standup

81. Daily Standup
• Weekly company
standup

82. Daily Review

83. Daily Review
• Quick peer feature
review

84. Weekly Demo

85. Weekly Demo
• 15 minute company-
wide user-driven
demo

86. weekly cycle
Mon Tues Wed Thurs Fri
9:30AM 10AM 11pm 5pm
Planning User-driven Cut-off Release
Meeting Demo Release
Testing

87. Sustainable Pace

88. Sustainable Pace

89. Sustainable Pace
• regular release window

90. Sustainable Pace
• regular release window
• quality of life and code

91. Sustainable Pace
• regular release window
• quality of life and code
• part of culture

92. Sustainable Pace
• regular release window
• quality of life and code
• part of culture
• company heartbeat

93. “agile”

94. Scrum?

95. XP?

96. what’s missing?

97. user stories?

98. user stories?

99. user stories?

100. story boards?

101. story boards?

102. story boards?

104. estimation?

105. estimation?

106. estimation?
fits or it doesn’t
“muda”

107. pairing?

108. pairing?

109. pairing?
as needed

110. pairing?
as needed
code reviews

111. pairing?
as needed
code reviews
open workspace

112. TDD?

113. TDD?

114. TDD?
52% coverage

115. TDD?
52% coverage
+ Selenium

116. TDD?
52% coverage
+ Selenium

117. retrospectives?

118. retrospectives?

119. retrospectives?
Release Review

120. retrospectives?
Release Review
5 Whys

121. lightweight RCA

122. 5 Whys

123. just-in-time

124. within 24 hours

125. 1 Jira per question

126. Practices

127. Daily support rotation

128. 5 Whys

129. Continuous
Monitoring

130. Continuous Monitoring
Socket
App Server LogMaster
Logger
(async)
WARN | ERROR

131. Continuous Monitoring

132. Continuous Monitoring
• Proactive response to issues

133. Continuous Monitoring
• Proactive response to issues
• Clean logs

134. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling

135. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail

136. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
• Managing the signal-to-noise ratio

137. Continuous Monitoring

138. Continuous Monitoring

139. Continuous Monitoring

140. Continuous Monitoring
App Server web app
gmond gmetad
UDP multicast
RRD

141. Continuous Monitoring

142. Continuous Monitoring

143. Continuous Monitoring

144. Continuous Monitoring

145. Zero-downtime
Deployment

146. Zero-downtime Deploys
Load Balancer
App Server App Server App Server
Deploy

147. Zero-downtime Deploys

148. Zero-downtime Deploys
• Fully automated deployment

149. Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity

150. Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
• Gracefully bring down servers

151. Zero-downtime Deploys
What about the DB?

152. Zero-downtime Deploys
Bering

153. Zero-downtime Deploys
012_rename_login_id_to_user_name
ALTER TABLE USER RENAME
COLUMN login_id user_name

154. Zero-downtime Deploys
012_rename_login_id_to_user_name
012_add_user_name_column 02_remove_login_id_column
expansion contraction

155. Zero-downtime Deploys

156. Zero-downtime Deploys
• expansion

157. Zero-downtime Deploys
• expansion
• preserves backwards compatibility

158. Zero-downtime Deploys
• expansion
• preserves backwards compatibility
• contract

159. Zero-downtime Deploys
• expansion
• preserves backwards compatibility
• contract
• clean up

160. Zero-downtime Deploys

161. Zero-downtime Deploys
• Typically safe expansion operations:

162. Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data

163. Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:

164. Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:
• rename or remove columns or tables

165. Zero-downtime Deploys
• Typically safe expansion operations:
• add table, add column, insert data
• Typically unsafe expansion operations:
• rename or remove columns or tables
• add constraint

166. Zero-downtime Deploys
/expand
/...
/143
001_create_login_table
002_add_login_id_index
003_create_user_table
004_create_group_table
/...
/contract
/...
/143
001_add_inv_constraint
002_drop_alias_column
/...

167. Zero-downtime Deploys

168. Zero-downtime Deploys
• Database migration decoupled from the
release

169. Zero-downtime Deploys
• Database migration decoupled from the
release
• Simplified rollback

170. Zero-downtime Deploys
• Database migration decoupled from the
release
• Simplified rollback
• Some additional complexity in writing
migrations

171. Release Options

172. Traditional Release

173. Traditional Release

174. Traditional Release
• high risk

175. Traditional Release
• high risk
• not scalable

176. Traditional Release
• high risk
• not scalable
• prolonged release: “Don’t release until
it’s all ready”

177. Release Options

178. Release Options

179. Release Options
• release to user subset (private beta)

180. Release Options
• release to user subset (private beta)
• split test (experimentation)

181. Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout

182. Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading

183. Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)

184. Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)
• defer commit

185. Release Options
• release to user subset (private beta)
• split test (experimentation)
• incremental rollout
• selective enabling/downgrading
• disable feature (feature darkmode)
• defer commit
• defer release

186. Release Options

187. Release Options
• Features running in production for
weeks before fully available

188. Release Options
• Features running in production for
weeks before fully available
• lots of data

189. Release Options
• Features running in production for
weeks before fully available
• lots of data
• “Release is a marketing term”

190. Release Options
• Features running in production for
weeks before fully available
• lots of data
• “Release is a marketing term”
• Production levers

191. Release Options
• Features running in production for
weeks before fully available
• lots of data
• “Release is a marketing term”
• Production levers
• WANGMI: the discipline to defer

192. Use Production
Data

193. Use Production Data

194. Use Production Data
• Upload Nightly backups

195. Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every
release

196. Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every
release
• Local DB refresh on demand

197. Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every
release
• Local DB refresh on demand
• Cleansed

198. Use Production Data
• Upload Nightly backups
• Automated Test DB refresh after every
release
• Local DB refresh on demand
• Cleansed
• Test data is a liability

199. Single Feature
Release

200. • Daily Support Rotation
• 5 Whys
• Continuous Monitoring
• Zero-downtime deployment
• Release Options
• Use Production Data
• Single Feature Release

201. Tools

210. Questions?
owen@pulseenergy.com