At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

1. Development, Deployment
and Collaboration at Etsy
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz

3. @mrtazz

4. @mrtazz

5. Item by TheBackPackShoppe
@mrtazz

6. http://www.flickr.com/photos/brianglanz/1095706242

7. avg 50 deploys/
day

8. avg n > m deploys/
day

9. How comfortable
are you deploying
a change right
now?

10. small change
@mrtazz
http://www.flickr.com/photos/renaissancechambara/2349811492

11. Config
Flags
Item by RocajoStudio

13. “If this is your first
day at Etsy, you
deploy the site”

14. Developer VMs

15. Developer VMs
• KVM
• Every engineer has one
• Fully Chef’d with the Etsy Stack
• Different sizes and Chef roles
@mrtazz

17. Continuous
Integration

19. Continuous Integration
• Run set of tests before each deploy
• Full QA suite
• Princess/Production smoker tests
• Try (yup, there is one)
@mrtazz

20. http://www.flickr.com/photos/egfocus/6962179321

21. The Bobs
• LXC virtualized hosts
• 14/physical hosts
• Spread over 3 SSDs
• Most of them attached to try
@mrtazz

23. Item by decomodwalls

24. Deployinator

25. Deployinator
• 2 Buttons, no ambiguity
• Overview of current state of deploy
• Links to Logwatcher and Dashboards
• Easy to add stacks for new tools to deploy
@mrtazz

26. http://www.flickr.com/photos/jbgeronimi/6363087361

28. Monitoring

29. shouldigraphit.com
@mrtazz

30. Monitoring
• Devs do their feature monitoring
• Everybody can access all the graphs
• Dashboard All The Things!
• Stream All The Logs!
@mrtazz

34. On Call

35. If you are writing
code, you are
on-call

36. On-Call Schedules
• ops on-call
• dev on-call
• payments on-call
• support on-call
@mrtazz

37. Dev On-Call
• Scheduled for 6 months
• On-call roughly every 4 weeks for 1 week
• L1 and L2 escalations
• L1 if it’s your first time
@mrtazz

38. Incident Response

39. Incident Response
• “This graph looks funny”
• “Hey I just got paged for elevated error rate
after deploys”
• “Supergrep is going crazy!!”
@mrtazz

41. #warroom

42. #warroom
• only outage related conversations
• coordinate investigation, communication,
countermeasures and monitoring
• good place to lurk for new engineers
@mrtazz

43. Post Mortems

44. blameless

45. Everybody’s invited

46. Learning Opportunity

47. Summary

48. Summary
• These are things that work for *us*
• Culture is an on-going effort
• Share everything
• Encourage learning/teaching
@mrtazz

49. Summary
• Lunch ’n learns
• DC visits
• On-call for a day
• Bootcamps/Senior rotations
@mrtazz

50. codeascraft.com
etsy.com/codeascraft/talks
etsy.github.com
etsy.com/careers

51. Questions?

52. Development,
Deployment and
Collaboration at Etsy
Daniel Schauenberg
dschauenberg@etsy.com