Development, Deployment & Collaboration at Etsy

1,265 views

Published on

Published in: Technology

Development, Deployment & Collaboration at Etsy

  1. 1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz
  2. 2. @mrtazz Etsy Stats
  3. 3. @mrtazz Etsy Stats
  4. 4. @mrtazzItem by TheBackPackShoppe
  5. 5. http://www.flickr.com/photos/brianglanz/1095706242
  6. 6. avg 50 deploys/ day
  7. 7. avg n > m deploys/ day
  8. 8. How comfortable are you deploying a change right now?
  9. 9. @mrtazzhttp://www.flickr.com/photos/renaissancechambara/2349811492 small change
  10. 10. Config Flags Item by RocajoStudio
  11. 11. “If this is your first day at Etsy, you deploy the site”
  12. 12. Developer VMs
  13. 13. @mrtazz Developer VMs • KVM • Every engineer has one • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles
  14. 14. Continuous Integration
  15. 15. @mrtazz Continuous Integration • Run set of tests before each deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one)
  16. 16. http://www.flickr.com/photos/egfocus/6962179321
  17. 17. @mrtazz The Bobs • LXC virtualized hosts • 14/physical hosts • Spread over 3 SSDs • Most of them attached to try
  18. 18. Item by decomodwalls
  19. 19. Deployinator
  20. 20. @mrtazz Deployinator • 2 Buttons, no ambiguity • Overview of current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy
  21. 21. http://www.flickr.com/photos/jbgeronimi/6363087361
  22. 22. Monitoring
  23. 23. @mrtazz shouldigraphit.com
  24. 24. @mrtazz Monitoring • Devs do their feature monitoring • Everybody can access all the graphs • Dashboard All The Things! • Stream All The Logs!
  25. 25. On Call
  26. 26. If you are writing code, you are on-call
  27. 27. @mrtazz On-Call Schedules • ops on-call • dev on-call • payments on-call • support on-call
  28. 28. @mrtazz Dev On-Call • On-call for 3 days • All developers who are not in another rotation • L1 and L2 escalations • L1 if it’s your first time
  29. 29. Incident Response
  30. 30. @mrtazz Incident Response • “This graph looks funny” • “Hey I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!”
  31. 31. Is the site down?
  32. 32. #warroom
  33. 33. @mrtazz #warroom • only outage related conversations • coordinate investigation, communication, countermeasures and monitoring • good place to lurk for new engineers
  34. 34. Post Mortems
  35. 35. blameless
  36. 36. Everybody’s invited
  37. 37. Learning Opportunity
  38. 38. Summary
  39. 39. @mrtazz Summary • These are things that work for *us* • Culture is an on-going effort • Share everything • Encourage learning/teaching
  40. 40. @mrtazz Summary • Lunch ’n learns • DC visits • On-call for a day • Bootcamps/Senior rotations
  41. 41. codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers
  42. 42. Questions?
  43. 43. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com

×