Downloaded 72 times
Uploaded byDaniel Schauenberg
A Whirlwind Tour of Etsy's Monitoring Stack
Etsy uses a variety of open source and custom tools to monitor their infrastructure including Ganglia, StatsD, Graphite, Syslog-Ng, Logster, Splunk, Logstash, Eventinator, Chef, Nagios, and Nagios Herald. They collect system, application, and log metrics and events and store them for analysis. Their monitoring setup is fully configured through Chef to provide consistent deployments across environments with over 120 monitoring recipes.
More Related Content
Similar to A Whirlwind Tour of Etsy's Monitoring Stack
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
A Whirlwind Tour of Etsy's Monitoring Stack
- 1. A Whirlwind Tourof Etsy's Monitoring Stack Daniel Schauenberg dschauenberg@etsy.com @mrtazz
- 3.
- 4.
- 5.
- 6. How comfortable are youdeploying a change right now?
- 7. “If this isyour first day at Etsy, you deploy the site”
- 10. @mrtazz Ganglia • System levelmetrics • Instance per DC/environment • > 220k RRD files • Fully configured through Chef role attributes
- 11.
- 12. @mrtazz StatsD • Single instanceon one server • Traffic mostly from 70 Web & 24 API servers • Node.js • Heavy Sampling • Graphite as backend
- 13.
- 14. @mrtazz Graphite • Application levelmetrics • 96G RAM, 20 Cores, 7.3T SSD RAID 10 • 525k metrics/minute • Mirrored Master/Master Setup • Functionally sharded relays
- 15.
- 16.
- 17.
- 18. @mrtazz Syslog-Ng • Web, Search,Gearman, Photos, Nagios, Network, VPN • 1.2GB written/minute • Chef role attribute based config • Rule ordering!
- 20. @mrtazz github.com/etsy/logster • Extract metricsfrom log files • Written in Python • Runs every minute via cron
- 21. @mrtazz Splunk • Indexes allof our log files • Easy search for patterns • Saved searches for interesting ones • Basically using it as a glorified grep
- 22. @mrtazz Logstash • Experiment status •Makes it easier integrate different sources • Easy to set up in dev environment • Trying to figure out where/how it fits into our infrastructure
- 23. @mrtazz Eventinator • Tracks allevents in our infrastructure • Chef runs and changes • DNS changes • Network • Deploys • Server provisioning and decommissioning • ~ 12 million events in the last 2 years
- 24.
- 25. @mrtazz Chef • rules everythingaround me • Same cookbooks on prod and dev • every node runs Chef every 10 minutes • ton of knife plugins and handlers
- 26.
- 27.
- 28.
- 29.
- 30. @mrtazz Nagios • 2 instancesin each DC/environment • Fully Chef generated configuration • Service checks and contacts in git • Notifications via email->SMS gateway • ~75% ops on-call
- 31.
- 32.
- 33.
- 34.
- 35. @mrtazz Nagios Herald • Addcontext to nagios alerts • What are the first 5 things you do when you get paged? • You already have the phone in your hand • nagios notification handler
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. @mrtazz Summary • Set oftrusted tools • Enhance where they come short • Try out new things • Write tools where applicable • Continuous monitoring and adaptation
- 42.
- 43.
- 44. A Whirlwind Tourof Etsy's Monitoring Stack Daniel Schauenberg dschauenberg@etsy.com @mrtazz