Distributed Release Management

10,803 views

Published on

Full Stack Engineering Meetup in NYC, May 27, 2014.

Published in: Technology
1 Comment
5 Likes
Statistics
Notes
No Downloads
Views
Total views
10,803
On SlideShare
0
From Embeds
0
Number of Embeds
8,231
Actions
Shares
0
Downloads
46
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide

Distributed Release Management

  1. 1. Distributed Release Management Deploying etsy.com 40+ times per day Mike Brittain Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
  2. 2. 1st Day Assignment Put your face on etsy.com/about
  3. 3. What I’m showing you tonight is the result of four years of iteration.
  4. 4. Small incremental changes to the application “Dark” features: new classes, methods, controllers Graphics, stylesheets, templates Copy/content changes ! App deploys Turning flags on, off, or % ramp up Config deploys
  5. 5. Latent bugs and security holes Traffic management, load shedding Adding and removing infrastructure ! Tweaking config flags or releasing patches. “Operating” the site
  6. 6. IRC, #push
  7. 7. /topic mbrittain | jgoulah | rsnyder | ekastner
  8. 8. /topic mbrittain, jgoulah, rsnyder | ekastner
  9. 9. Keep real people in the loop Queue, with max batch size of seven. Automated deployment run by humans
  10. 10. 4 people in this deploy. “I’ve pushed my changes to master.” “Everyone has checked in.”
  11. 11. Build QA and Pre-prod Build progress Status in #push Git SHA1 in for each env. Date, username, deploy log, changeset, link to dashboard from time of deploy
  12. 12. Reporting what’s going on in Deployinator, and who triggered Status from build cluster
  13. 13. Pre-prod (“princess”) has been deployed. ! SHA1 of the change Time it took to deploy Link to changeset in GitHub Log of the deploy script
  14. 14. Btw, there are three bots talking in channel at this point. O_o
  15. 15. Queuing for next deploy Humans talk to other humans from time to time.
  16. 16. Talking to pushbot. ! Pushbot knows some Spanish… because, ya know, why not?
  17. 17. Link to test results for CI environment, along with how long the tests took.Alerting by name.
  18. 18. 8 minutes have elapsed… We’ve built and tested our release in the CI environment (“QA”). ! QA build failed our 5 min. SLA for tests.
  19. 19. “Try” is our pre-commit testing cluster.
  20. 20. Bots help reinforce our values. This is especially helpful for new people on the team.
  21. 21. Still 8 minutes elapsed… Pre-prod has been deployed and tested. ! This ran in parallel with our QA build and tests.
  22. 22. Cross-traffic: In a separate channel (#config), our app configs files were deployed to pre-prod.
  23. 23. Cross-traffic: Ops team deployed a configuration change. And, yes… another non-human.
  24. 24. Code is live Link to dashboard.
  25. 25. 13 minutes elapsed… Code is now in production with public traffic.
  26. 26. Who committed code in the last deploy? And how many lines did each of them change?
  27. 27. Handoff for the next deploy.
  28. 28. Entire app deploy took 15 minutes. ! 4 people running the deployment 8 committers Config deploy and Chef change deployed in parallel.
  29. 29. Optimal queue size Normalized communication Improved visibility Historical record is ideal for post-mortems Organic evolution
  30. 30. Hold up the queue (.hold) Work the issue with the people available in #push Additional help always available in #sysops Buddy-system for off-hours deploys Ops-on-call, dev-on-call When something goes wrong?
  31. 31. 25 Million Items listed 60+ Million Monthly unique visitors 200 Countries with annual transactions ! 175+ Committers, everyone deploys Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
  32. 32. @mikebrittain DEPLOYMENTSPERDAY APPCODE CONFIGFILES
  33. 33. Start small. (We did.) Automated tests and production monitoring. Have a story around maintaining quality. “We can always go back to the old way.” Demonstrate value to leadership.
  34. 34. Go write your own story.
  35. 35. Thank you. Mike Brittain Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

×