StatsD Workshop Monitorama 2013

1,821 views

Published on

StatsD workshop from Monitorama 2013

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,821
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

StatsD Workshop Monitorama 2013

  1. 1. StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazzFriday, March 29, 13
  2. 2. • What is this StatsD? • Metrics types • Backends • Graphite and operating StatsD • Send all the metrics • Write all the backendsFriday, March 29, 13
  3. 3. Friday, March 29, 13
  4. 4. Metrics processing module Receiver loop BackendsFriday, March 29, 13
  5. 5. legacyNamespace: falseFriday, March 29, 13
  6. 6. Counters foo.bar:1|c stats.counters.foo.bar.rate stats.counters.foo.bar.countFriday, March 29, 13
  7. 7. Gauges foo.gar:30|g foo.gar:-5|g foo.gar:+5|g stats.gauges.foo.garFriday, March 29, 13
  8. 8. Sets foo.sar:30|s foo.sar:50|s stats.sets.foo.sarFriday, March 29, 13
  9. 9. Timers foo.tar:30|ms stats.timers.foo.tar.mean_90 stats.timers.foo.tar.std stats.timers.foo.tar.sum_90Friday, March 29, 13
  10. 10. Histograms histogram: [{metric: "foo.tar", bins: [10,20,30,50]}] stats.timers.foo.tar.histogram.bin_10 stats.timers.foo.tar.histogram.bin_20Friday, March 29, 13
  11. 11. Friday, March 29, 13
  12. 12. Pluggable backends function MyBackend(startupTime, config, emitter){   var self = this;   this.lastFlush = startupTime;   this.lastException = startupTime;   this.config = config.mybackendconfig || {};   // what to do with metrics   emitter.on(flush, function(timestamp, metrics) {}); // update your status   emitter.on(status, function(callback) {}); // handle raw packet emitter.on(packet, function(packet, rinfo) {}); };Friday, March 29, 13
  13. 13. Management Interface > telnet 127.0.0.1 8126 > stats > counters > delcounters > helpFriday, March 29, 13
  14. 14. Graphite + StatsD • default backend • whisper archives should match flush interval • sum aggregation for raw counters • docs/graphite.mdFriday, March 29, 13
  15. 15. The single-thread dilemmaFriday, March 29, 13
  16. 16. Tuning UDP # increase default core memory sizes net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.ipv4.udp_wmem_min = 67108864 net.ipv4.udp_rmem_min = 67108864 net.ipv4.udp_mem = 4648512 6198016 9297024Friday, March 29, 13
  17. 17. Sampling • default sampling of 0.1 on all metrics • most sent keys (log file) are sampled 0.01Friday, March 29, 13
  18. 18. Quis monitoret ipsos monitores?Friday, March 29, 13
  19. 19. Friday, March 29, 13
  20. 20. Friday, March 29, 13
  21. 21. Friday, March 29, 13
  22. 22. Friday, March 29, 13
  23. 23. Let’s do thisFriday, March 29, 13
  24. 24. Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.jsFriday, March 29, 13
  25. 25. Send some stats > echo -n "foo.bar:1|c" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lerl:3|g" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:33|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:43|ms" | nc -w0 -u 127.0.0.1 8125 > echo -n "foo.lol:53|ms" | nc -w0 -u 127.0.0.1 8125Friday, March 29, 13
  26. 26. Some actual stats Disk space: df -m | grep "/var" | awk {print "diskspace.var:"$4"|g"} Apache HTTPD processes: echo -n "httpd.procs:$(ps aux | grep [h]ttpd | wc -l)|g" Logfile size: du -s /var/log/chef.log | awk {print "cheflog:"$1"|g"}Friday, March 29, 13
  27. 27. Is this real life? • Set up a web page • Have it send stats to StatsD • page load time • counters when page was accessed • look in the examples/ directory for codeFriday, March 29, 13
  28. 28. Histograms • You should have timers now • Configure histograms for them • { graphite: { histograms: [] }Friday, March 29, 13
  29. 29. Level up!Friday, March 29, 13
  30. 30. Write a new backend • Have it HTTP POST to a web server • write metrics to a file • send a txt message • repeat all UDP packets to TCPFriday, March 29, 13
  31. 31. Hackathon ideas • Metrics path validation in backend • Pluggable receiver frontends • Forked backends • Configurable timer metricsFriday, March 29, 13
  32. 32. • #statsd on Freenode IRC • statsd@librelist.com • Ask meFriday, March 29, 13
  33. 33. Basic setup • git clone https://github.com/etsy/statsd.git • curl https://gist.github.com/mrtazz/5270376 > test_config.js • nc -k -l 2003 > output.log • node stats.js test_config.jsFriday, March 29, 13
  34. 34. StatsD workshop @ Monitorama 2013 Daniel Schauenberg Software Engineer, Etsy Infrastructure @mrtazzFriday, March 29, 13

×