A talk about monitoring using Graphite and StatsD I gave at a local Ruby meetup.

1. Measure everything
Ruby Underground, August 2012
@arikfr
Tuesday, February 12, 13

2. Today’s plan
• Why measure
• How to measure
• Graphite / StatsD
• Using Graphite/StatsD with Ruby and Rails
Tuesday, February 12, 13

3. Questions to answer
• How fast is my system?
• Is it faster than last month?
• Did our last deploy affect database
performance?
• How much time do we spend calling
external web services?
Tuesday, February 12, 13

4. More questions
• How many errors do we have a day?
• How many failed logins?
• How many successful logins?
• How many orders without house number?
Tuesday, February 12, 13

5. And more questions!
• How many orders did we have today?
• How many orders did we have today from
Android version 2.056?
• How many rejected orders did we have?
• How many rejected orders due to lack of
coverage vs. lack of taxis?
Tuesday, February 12, 13

6. To answer all of this,
you need a way to
track different
numbers.
Tuesday, February 12, 13

7. So now we know the
why.
But how...?
Tuesday, February 12, 13

8. Tuesday, February 12, 13

9. But that’s not enough
• Not real time enough
• Hard to control what’s being collected
• Pricey for big deployments
Tuesday, February 12, 13

10. The Alternative
• Graphite (Whisper, Carbon, Graphite Web)
• StatsD
• CollectD
• (there are other options -- OpenTSDB,
Liberato, home grown)
Tuesday, February 12, 13

11. Benefits
• Easy to install
• Highly scalable
• Practically zero cost to measure anything:
• efficient storage
• UDP packets to send data in
• Ecosystem
Tuesday, February 12, 13

12. Whisper
• Default settings:
• 6 hours of 10 second data
• 1 week of 1 minute data
• 5 years of 10 minute data
• That’s amounts to ~3.2MB per metric.
• Configurable.
Tuesday, February 12, 13

13. Types of data
• Counters - number of orders per sec
• Gauges - total orders today
• Timers - time to make an order
• (with additional values, such as: count,
mean, 90th percentile, max, min, etc)
Tuesday, February 12, 13

14. Sending a number to
Graphite
• metric number timestamp
• example.ruby.under_ground 20 1346075634
• echo "example.ruby_underground 20 `date
+%s`" | nc graphite.yourcorp.com 2003
Tuesday, February 12, 13

15. Sending a number to
StatsD
./statsd-client.sh 'my_metric:100|g'
Tuesday, February 12, 13

16. Naming Convention
• Whatever makes sense to you, just
remember that it’s a tree.
• We use:
• {env}.{metric}.{region}.{hostname}
• You can use globs, when querying:
• app.orders.daily.completed.israel.*
• app.orders.daily.completed.*.*
Tuesday, February 12, 13

17. Ruby Examples
Tuesday, February 12, 13

18. $ gem install statsd-ruby
(https://github.com/reinh/statsd)
Tuesday, February 12, 13

19. Track Deployments
Tuesday, February 12, 13

20. Measure controllers
Tuesday, February 12, 13

21. Measure Specific stuff
Tuesday, February 12, 13

22. Getting Data out
Tuesday, February 12, 13

23. Graphite web
Tuesday, February 12, 13

24. Tasseo
Tuesday, February 12, 13

25. Gdash
Tuesday, February 12, 13

26. Graphene
Tuesday, February 12, 13

27. Down sides
• Hard to track user analytics
• You can tell how many orders were done
today
• You can’t tell (easily) how many unique
users did those orders
• The tree of metrics is sometimes annoying
Tuesday, February 12, 13

28. Questions?
Tuesday, February 12, 13