This document summarizes how to use open source tools like Jenkins, Git, Ubuntu dpkg, node.js, and others to improve a node.js web application release process and ensure better release quality. Key aspects covered include continuous integration, automated testing, configuration management, and packaging and deploying code.

1. Better node.js software

2. release practice using open

3. source tool

4. ????????

5. Software release to production is such a hard practice, especially considering the business

6. impact and the quality of software . I will demonstrate how to use open source tools to

7. enhance a node.js web software release process and ensure the better release quality.

8. The open source tools I will mention will includes Jenkins, Git, Ubuntu dpkg system, node.js

9. language , ruby , rake, jshint , nodeunit, expresso , YETI, Selenium, and mongodb

10. ?, ??????? 100+ ???? = =

11. Web Software release best

12. practice

13. Agenda

14. Who am I ?

15. Release engineering in Y! Shopping

16. Process

17. Testing

18. Deploy Tools

19. Configuration Management

20. Who am I ?

21. thegiive

22. • A open source lover

23. • Release manager of Taiwan Yahoo! Shopping site

24. • Continuous study on software development, release engineering and software quality

25. • Fall in love with Ruby, VIM, Puppet and Rake

26. • Trying to do more work rather than write more blog

27. Release engineering in Y! Shopping

28. Common image about release

29. engineer

31. Yahoo! Shopping release manager

32. integration hub for more complex software development teams, sitting at the cross between

33. development, product management, quality assurance and other engineering efforts, also known

34. as DevOps

35. DevOps

37. Release???1 : ???

38. ??????????????

39. • RD Team ???? , PHP Team , .Net Team , Search Team

40. • User Team ???? , ?? Team , ?? Team , Search team

42. Release???1 : ???

43. ?????

45. Release???2 : bug????

46. From Book "Beautiful Testing" Chaptor 2

47. TestROI : ????????

49. • ACPB : Average Cost of a Production Bug

50. • ACTB : Average Cost of a Test Bug

51. • TBN : Total Bug Number

52. • TDBC : Total Defect Bug Cost

53. Taiwan Shopping ? ACPB?????? Test ROI ???

54. Test ROI ??????????? testing ????

55. Release???2 : bug????

57. Release???3 : User????

58. • ??RD?bug?QA????????SE??????RM?????

59. • User ???? monitor script , ????????????

60. ???? Release Manager , not release engineer

62. Release???4 : ??????

63. PHP Team

64. • Dev process : Scrum

65. • Source Code Management : Subversion

66. • Language : PHP and partial C++

67. • OS : Linux

68. .Net Team

69. • Dev process : Waterfall

70. • Source Code Management : VSS

71. • Language : ASP.Net and ASP

72. • OS : Windows

73. ????????????? ... ?????? 100??

74. release ?release 2000 ??

75. item

76. ??????

78. Release quality ??

80. Y axies stands for ehnancement number / release number

81. Throughput ??

82. Y axies stands for

83. release item / release number

84. User ?? , ???????

85. ??Release Quality ??????

86. • 2011 Q4 24? release ??, ? 12??(02:00 ~ 06:00) release

87. • ???? user ?? manager ?????

88. ??????????????? , ??????? release ???????, ????? 6???

89. ?? release qulaity ???user ???????????

90. • 2012 Q2 11? release ??, ? 0??(02:00 ~ 06:00) release

91. • 2012 Q2? 1??07:30 release

92. ????????

94. Process

95. Communication : No Surprise !

96. • ???????????, ?????????????????????? briefing and retrospective , ??????? 18%

97. • Communication ???? 9???

99. ???????

100. ?????????

101. • ????

102. • ???

103. • double confirm ??????????

104. ??

105. • ????

106. • ?? 30min ??, 30???????????? 20000 NTD

107. ??????

108. • ?????? sharing information ?????

109. ????????

110. ?????????

111. • Information syncing : 40%

112. • Discussion : 30%

113. • decision making : 30%

114. ?????

115. • Information sync : 5% => ?????????

116. • Discussion : 15%

117. • decision making : 10%

118. Convention over configuration

119. Release??

120. • Release ????? ???????????

121. • ??????? release ??

122. • ????????? release

123. • Release ???????? release briefing

124. • Release ??? Code Freeze

125. • ???? cost , ?????? retrospective

126. ??????????? sync ??

127. • ???? Release Plan

128. • ????????? Plan ??

129. ◦ Item ????,???? , ????

130. ◦ Item ?????? , ???? , ????

131. ◦ ??????? item ????????????

132. release briefing ?????

133. • Information sync : 5% => information double confirm

134. • Discussion : 15% => ???? sync , ??????????

135. • decision making : 10% => ??sync??????????????

136. ???????

137. • ???????? ??? release????? ????email

138. • ????????????????? , ????????????

139. • ????? , ???? surprise , ???????

141. Testing

142. Release Manager ???

143. • Unit Test : nodeunit , YETI

144. • Automation : Selenium

145. • CI ( Continuous Integration ) : Jenkins

146. ????????????????

148. Unit Test ?????

149. ?? code ???? , 100% RD ???

151. Automation

152. • ??????? SLA (Service-level agreement)

153. • ???? ?? automation coverage rate ??? SLA ? 100%

154. • ?? release ?????? automation

155. • Presentation layer ? coverage rate ???? 100% , ???????

157. Continuous Integration

158. • ??????? Unit Test and Automation , ???????

159. • CI ????????????? Unit Test , Automation

160. • ?????? CI ? auto build ??

162. Deploy Tool

163. ???SE/Ops ??? Deploy Tool

164. ??: ???? Yinst/Igor

165. Use os level package system

166. • Debian/Ubuntu : dpkg

167. • RedHat/Centos/Fedora : rpm

168. • FreeBSD : port

169. • Windows : MSI

170. Don't use language level package system , ????????? relay on ???????? server

171. • Webservr : apache, nginx

172. • Cache : memcached , redis

173. • SQL : MySQL , PosgreSQL , MongoDB

174. ????? Deploy Script

175. ?????????? script ? deploy code

176. 1 apt-get install npm

177. 2 apt-get install pip

178. 3 apt-get install rubygem

179. 4 apt-get install nginx #web server

180. 5 apt-get install memcached #cache server

181. 6 npm install -g express # Web framework

182. 7 npm install -d

183. 8 pip install pykml

184. 9 gem install amqp # node.js has no rabbitmq client , so use ruby XD

185. 10 apt-get install rabbitmq-server # rabbitmq server

186. 11 wget http://mydev.server/mycode_package.tgz # get my code from dev server

187. 12 tar zxcf mycode_package.tgz # unpack the code package

188. • ???? hybrid ?? package system , package dependency ?????

189. • SE/Ops team ?? language package system ??? , ???? RD ????

190. ?? Deploy Script

191. ????? package system , ?????? package version

192. 1 apt-get install nginx=0.1.19-1

193. 2 apt-get install memcached=1.4.13-0ubuntu2

194. 3 apt-get install node-express=2.5.9-1

195. 4 apt-get install pykml=0.0.3

196. 5 apt-get install ruby-amqp=0.9.5-1.

197. 6 apt-get install rabbitmq-server=2.8.4-1

198. 7 apt-get install mycode_package=1.2.3 # ???? code ?? package

199. ??

200. • SE/Ops team ??????? deploy ???

201. • CI server ????

202. ???? pkg version

203. Production System ? ???? pkg system ? depentency

205. How to pack your package

206. • ?? code ????? (Code,Config,template)

207. • ??? code ??? prodcut ????? (??,??????,???,????)

208. • ????????? (??lib ,??lib ,?? config , ??config )

209. • ???????????? ( 100? , 10? , 1? , ???? , ???? )

211. ???? Framework ???

213. Change Management

214. ??? code ???????

215. • mycode-1.2.3.zip : ?????? change ???? ???? readme

216. PKG list : ?? SE/Ops ?????

217. • Upgrade : functionA-config-1.2.3-2012.07.08

218. • Upgrade : Member-template-2.3.2-2012.07.11

219. • Donwgrade : functionC-code-3.2.1-2012.03.05

220. • Upgrade : ShoppingCart-lib-100W-1.2.3-2012.07.20

221. • Upgrade : WS-stock-lib-200W-3.2.1-2012.07.21

222. Package pack script ???

223. Script best practice

224. • ???

225. • ? package ???? source code version ???

226. • Version ??????

227. • Package ????? package server

228. • Readme ???????? svn log

229. Package Version 2.3.1-2012.08.12

230. [thegiive v123] add some slides change

231. [RD2 v121] change function a

232. Configuration Management

233. debconf

234. ???? config package

235. • mypkg-config.2.2.3-2012.07.17

236. • ?????? config ? ????? readme

237. ?????? config ??

238. 1 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

239. 2 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections

240. 3 echo "mypkg mypkg/title string coscup"|debconf-set-selections

241. 4 debian_frontend=noninteractive dpkg-reconfigure

242. package ??? postinst

243. 1 db_get mypkg/use_memached

244. 2 echo $RET #??? debconf ?????

245. 3 do_config_change($RET)

246. ??????SE/Ops?????

247. Cache Server ???SE?????? disable cache

248. 1 echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections

249. 2 debian_frontend=noninteractive dpkg-reconfigure

250. WS1.coscup.org ??????SE???? WS2.coscup.org

251. 1 echo "mypkg mypkg/ws_host string ws2.coscup.org"|debconf-set-selections

252. 2 debian_frontend=noninteractive dpkg-reconfigure

253. ???? title , SE ????? XD

254. 1 echo "mypkg mypkg/title string coscup.org"|debconf-set-selections

255. 2 debian_frontend=noninteractive dpkg-reconfigure

256. • ????RD ???SE?Daily work?????? RD , SE ??????

257. • ?? incident ??? , SE ????config ???? , RD??? SE?????

258. ?? mobile login server ??

259. • ?? mobile loging server ?????????????????

260. • ?????????? RD ???????RD,SE,??????

261. • ???? config change , ????? mobile login ?? Web?? login ??

263. Debconf ????...???

264. ????????deploy script

265. 1 # pkg

266. 2 export debian_frontend=noninteractive

267. 3 apt-get install -y nginx=0.1.19-1

268. 4 apt-get install -y memcached=1.4.13-0ubuntu2

269. 5 apt-get install -y node-express=2.5.9-1

270. 6 apt-get install -y pykml=0.0.3

271. 7 apt-get install -y ruby-amqp=0.9.5-1.

272. 8 apt-get install -y rabbitmq-server=2.8.4-1

273. 9 apt-get install -y mycode_package=1.2.3

274. 10 # config

275. 11 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

276. 12 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections

277. 13 echo "mypkg mypkg/title string coscup"|debconf-set-selections

278. 14 dpkg-reconfigure

279. • ??????? script ? deploy ?????????? , VM , Cloud instance

280. • script ???? non-interactive mode ? disable ??????? step

281. • ?? CI server ?? auto build , auto config

282. deploy script ?? version

283. control

284. ?? tag ??? deploy , rollback

285. 7??1? release ? tag ? v2012_07_01 , 7??2? release ? tag ? v2012_07_02

286. 1 # ???1? release

287. 2 git tag v2012_07_01

288. 3 exec_deploy_script

289. 4 # ???2? release

290. 5 git tag v2012_07_02

291. 6 exec_deploy_script

292. ? v2012_07_02 release ??????? rollback ??????(v2012_07_01)

293. 1 git checkout v2012_07_01

294. 2 exec_deploy_script

295. ?? git diff ?????? change

296. ????7??2? release ?????

297. 1 git diff v2012_07_01 v2012_07_02

298. 2 #git diff output

299. 3 .....

300. 4 -apt-get install -y mycode_package=1.2.3

301. 5 +apt-get install -y mycode_package=1.3.1

302. 6 -echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections

303. 7 +echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections

304. 8 .....

305. 9 -echo "mypkg mypkg/title string coscup"|debconf-set-selections

306. 10 +echo "mypkg mypkg/title string coscup.org"|debconf-set-selections

307. 11 .....

308. • package ????? , config management ????change ???

309. • ?? RD ??????? diff ???????????

310. ?? git branch ?????

311. • ?? git ??? branch ???????? deploy script

312. • ???????? alpha , beta , staging , production ?? branch

313. ??? alpha ???????? beta ??? git merge

314. 1 git checkout beta

315. 2 git merge alpha

316. 3 ...solve confict...

317. 4 git commit

318. One More Thing

319. Deploy System : Puppet

321. • Open source deploy system written by Ruby

322. • Client Server ????????? pure client

323. • Config is language, built by Ruby DSL !!!

324. • ??? template system ???? perfect configuration manage System

325. • ???OS ? package system , ???????? OS ????

326. • From Wikipedia ????: Google, Zynga, Sun, Wikipedia, Dell, Twitter...

327. Puppet Node config

328. In module

329. 1 class ntpd {

330. 2 package { "ntp": ensure => installed, }

331. 3 file { "/etc/ntp.conf": owner => root, group => root,

332. 4 require => package["ntp"] }

333. 5 @service { "ntpd": ensure => running, enable => true,

334. 6 subscribe => File["/etc/ntp.conf"] }

335. 7 }

336. 8 class ci{ package { "jenkins" : ensure => installed,} }

337. 9 class Webservr{ package { "nginx" : ensure => installed,} }

338. In main config

339. 1 node basenode { include ntpd }

340. 2 node 'ci.coscup.org' inherits basenode {

341. 3 include ci

342. 4 }

343. 5 node 'web.example.com' inherits basenode {

344. 6 include webserver

345. 7 }

346. Puppet Package config

347. ?? Package ??

348. 1 package {'vim': ensure => 'installed' }

349. 2 package { "apache2": ensure => '2.2.22-1ubuntu1' }

350. 3 package { 'mysql': ensure => installed,

351. 4 provider => 'msi',

352. 5 source => 'N:/packages/mysql-xxx.msi',

353. 6 install_options => { 'INSTALLDIR' => 'C:mysql-5.5' }, }

354. ??????? Deploy Script

355. • Debian / Ubuntu : Apt-get

356. • RedHat / Fedora / CentOS : yum

357. • FreeBSD : port

358. • MacOS : macport

359. • Windows : msi

360. • Gentoo : portage

361. Puppet Template System

362. Configure Management ??? : ERB

363. ??????? config file

364. 1 $server_name = 'domain.coscup.org'

365. 2 $server_port = 80

366. 3 file {"/usr/conf/nginx.conf": mode => "664",

367. 4 content => template("nginx/config.erb"), }

368. template

369. 1 server {

370. 2 listen <%= $server_port%>;

371. 3 server_name <%= $server_name %>;

372. 4 .....

373. 5 }

374. Pure ruby config menas

375. 1 Given I run "ssh localhost -p 2222" interactively

376. 2 And I type "apache2ctl configtest"

377. 3 And the exit status should be 0

378. • We can write unit test ... so we can TDD

379. • We can write rspec and cucumber ... so we can BDD

380. • We can commit our config in Hudson... so we can CI our config

381. ??? Deploy Script ??

382. 1 class web{ package { "nginx": ensure => '0.1.19-1' } }

383. 2 class cache{ package { "memcached" : ensure => '1.4.13-0ubuntu2' }}

384. 3 class kml_lib{ package { "pykml" : ensure =>'0.0.3' }}

385. 4 class message_quene{

386. 5 package { "ruby-amqp" : ensure => '0.9.5-1' }

387. 6 package { "rabbitmq-server": ensure => '2.8.4-1' }

388. 7 }

389. 8 class code_package{

390. 9 include kml_lib

391. 10 include message_quene

392. 11 package { "node-express" : ensure => '2.5.9-1' }

393. 12 package { "mycode" : ensure => '1.2.3' }

394. 13 }

395. 14 node "w1.coscup.com" {

396. 15 include web

397. 16 include cache

398. 17 include code_package

399. 18 }

400. Q&A

401. Puppet Config

402. ?? Command

403. 1 exec { 'remove file' : command => 'rm /tmp/pup/etd' , path => '/bin' }

404. ?? Windows Command

405. 1 exec { 'wincommand': command => 'C:Windows..powershell.exe -executionpolicy remotesigned -fi

406. ?? file

407. 1 file{'/tmp/pup': ensure => 'directory'}

408. 2 file{ '/tmp/pup/et': ensure => 'present', content => 'look at me',

409. 3 require => File['/tmp/pup']}

410. 4 file { "/etc/ntp.conf": owner => root, group => root,

411. 5 mode => 0644, source => "puppet:///ntpd/ntp.conf",

412. 6 require => package["ntp"]

413. 7 }