A Whirlwind Tour of Etsy's Monitoring Stack
•
16 likes•7,392 views
Etsy uses a variety of open source and custom tools to monitor their infrastructure including Ganglia, StatsD, Graphite, Syslog-Ng, Logster, Splunk, Logstash, Eventinator, Chef, Nagios, and Nagios Herald. They collect system, application, and log metrics and events and store them for analysis. Their monitoring setup is fully configured through Chef to provide consistent deployments across environments with over 120 monitoring recipes.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to A Whirlwind Tour of Etsy's Monitoring Stack
Similar to A Whirlwind Tour of Etsy's Monitoring Stack (20)
More from Daniel Schauenberg
Recently uploaded
Recently uploaded (20)
A Whirlwind Tour of Etsy's Monitoring Stack
- 1. A Whirlwind Tour of Etsy's Monitoring Stack Daniel Schauenberg dschauenberg@etsy.com @mrtazz
- 3. @mrtazz
- 4. @mrtazz
- 6. How comfortable are you deploying a change right now?
- 7. “If this is your first day at Etsy, you deploy the site”
- 10. @mrtazz Ganglia • System level metrics • Instance per DC/environment • > 220k RRD files • Fully configured through Chef role attributes
- 12. @mrtazz StatsD • Single instance on one server • Traffic mostly from 70 Web & 24 API servers • Node.js • Heavy Sampling • Graphite as backend
- 13. @mrtazz
- 14. @mrtazz Graphite • Application level metrics • 96G RAM, 20 Cores, 7.3T SSD RAID 10 • 525k metrics/minute • Mirrored Master/Master Setup • Functionally sharded relays
- 16. @mrtazz
- 17. @mrtazz
- 18. @mrtazz Syslog-Ng • Web, Search, Gearman, Photos, Nagios, Network, VPN • 1.2GB written/minute • Chef role attribute based config • Rule ordering!
- 20. @mrtazz github.com/etsy/logster • Extract metrics from log files • Written in Python • Runs every minute via cron
- 21. @mrtazz Splunk • Indexes all of our log files • Easy search for patterns • Saved searches for interesting ones • Basically using it as a glorified grep
- 22. @mrtazz Logstash • Experiment status • Makes it easier integrate different sources • Easy to set up in dev environment • Trying to figure out where/how it fits into our infrastructure
- 23. @mrtazz Eventinator • Tracks all events in our infrastructure • Chef runs and changes • DNS changes • Network • Deploys • Server provisioning and decommissioning • ~ 12 million events in the last 2 years
- 24. @mrtazz
- 25. @mrtazz Chef • rules everything around me • Same cookbooks on prod and dev • every node runs Chef every 10 minutes • ton of knife plugins and handlers
- 26. @mrtazz
- 28. @mrtazz
- 29. @mrtazz Nagios
- 30. @mrtazz Nagios • 2 instances in each DC/environment • Fully Chef generated configuration • Service checks and contacts in git • Notifications via email->SMS gateway • ~75% ops on-call
- 32. @mrtazz
- 33. @mrtazz
- 34. @mrtazz
- 35. @mrtazz Nagios Herald • Add context to nagios alerts • What are the first 5 things you do when you get paged? • You already have the phone in your hand • nagios notification handler
- 36. @mrtazz
- 37. @mrtazz The Toys are real
- 38. @mrtazz There’s another side of heaven
- 41. @mrtazz Summary • Set of trusted tools • Enhance where they come short • Try out new things • Write tools where applicable • Continuous monitoring and adaptation
- 44. A Whirlwind Tour of Etsy's Monitoring Stack Daniel Schauenberg dschauenberg@etsy.com @mrtazz