Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Development, Deployment and Collaboration at Etsy

1,651 views

Published on

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

Published in: Technology
  • Be the first to comment

Development, Deployment and Collaboration at Etsy

  1. 1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz
  2. 2. @mrtazz
  3. 3. @mrtazz
  4. 4. Item by TheBackPackShoppe @mrtazz
  5. 5. http://www.flickr.com/photos/brianglanz/1095706242
  6. 6. avg 50 deploys/ day
  7. 7. avg n > m deploys/ day
  8. 8. How comfortable are you deploying a change right now?
  9. 9. small change @mrtazz http://www.flickr.com/photos/renaissancechambara/2349811492
  10. 10. Config Flags Item by RocajoStudio
  11. 11. “If this is your first day at Etsy, you deploy the site”
  12. 12. Developer VMs
  13. 13. Developer VMs • KVM • Every engineer has one • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles @mrtazz
  14. 14. Continuous Integration
  15. 15. Continuous Integration • Run set of tests before each deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one) @mrtazz
  16. 16. http://www.flickr.com/photos/egfocus/6962179321
  17. 17. The Bobs • LXC virtualized hosts • 14/physical hosts • Spread over 3 SSDs • Most of them attached to try @mrtazz
  18. 18. Item by decomodwalls
  19. 19. Deployinator
  20. 20. Deployinator • 2 Buttons, no ambiguity • Overview of current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy @mrtazz
  21. 21. http://www.flickr.com/photos/jbgeronimi/6363087361
  22. 22. Monitoring
  23. 23. shouldigraphit.com @mrtazz
  24. 24. Monitoring • Devs do their feature monitoring • Everybody can access all the graphs • Dashboard All The Things! • Stream All The Logs! @mrtazz
  25. 25. On Call
  26. 26. If you are writing code, you are on-call
  27. 27. On-Call Schedules • ops on-call • dev on-call • payments on-call • support on-call @mrtazz
  28. 28. Dev On-Call • Scheduled for 6 months • On-call roughly every 4 weeks for 1 week • L1 and L2 escalations • L1 if it’s your first time @mrtazz
  29. 29. Incident Response
  30. 30. Incident Response • “This graph looks funny” • “Hey I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!” @mrtazz
  31. 31. #warroom
  32. 32. #warroom • only outage related conversations • coordinate investigation, communication, countermeasures and monitoring • good place to lurk for new engineers @mrtazz
  33. 33. Post Mortems
  34. 34. blameless
  35. 35. Everybody’s invited
  36. 36. Learning Opportunity
  37. 37. Summary
  38. 38. Summary • These are things that work for *us* • Culture is an on-going effort • Share everything • Encourage learning/teaching @mrtazz
  39. 39. Summary • Lunch ’n learns • DC visits • On-call for a day • Bootcamps/Senior rotations @mrtazz
  40. 40. codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers
  41. 41. Questions?
  42. 42. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com

×