Performance metrics + Nagios traffic + other sources + Datadog in the cloud = real time graphs + analytics

1. A Deep Dive into
Nagios Analytics
Alexis Lê-Quôc (@alq)
http://datadoghq.com

2. A Deep Dive into
Nagios Analytics
Alexis Lê-Quôc (@alq)
http://datadoghq.com

3. @alq
Dev & Ops
Nagios user since 2008
Datadog co-founder

4. A little survey

5. Top 3 failed checks

6. That woke me up
That I responded to
last week
That I responded to
5 weeks ago
Top 3 failed checks
That most of my team
responded to at least once
That impacts our business
the most?

7. That woke me up
That I responded to
last week
That I responded to
5 weeks ago
Top 3 failed checks
That most of my team
responded to at least once
That impacts our business
the most?

8. At best, finding local optimums
Using memory to
prioritize remediation...
At worst, brownian motion

9. Analytics

10. Performance Metrics Nagios Traffic Other Sources
In the “Cloud”
Real-time graphs + analytics

11. Aggregation

12. Real-time Analytics
(Nagios et al.)

13. Real-time Analytics

14. Nagios Traffic
In the “Cloud”
Real-time graphs + analytics

15. Nagios a “chatty” source
out of 40+ Datadog supports

16. One example

18. Almost 13000 Nagios “events”
over past week

19. Constant stream

20. 86 notifications!

21. Pattern

22. Pattern

23. More data?
More questions.

24. A diNaolto a sgci ewntifiict shtud ydata

25. 6
4
2
0
0 250 500 750
Host count
Population
factor(quartile)
1
2
3
4
Nagios samples
Population
25%
50%
75%
100%
20
93
322
904

26. Does size matter?

27. 40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
0 250 500 750 1000
Nagios alert per host
count per week
Weekly count per host split by quartile

28. 40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
0 250 500 750 1000
Nagios alert per host
count per week
Weekly count per host split by quartile
Outliers
Sick hosts,
silenced checks

29. Notifications

30. Notifications
1-3% of alerts notify
Little difference per quartile

31. Does time of day
matter?

32. ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
12
8
4
12
8
4
12
8
4
12
8
4
1 2 3 4
0 5 10 15 20
Hour of Day (UTC)
Alerts per hour

33. ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Mean about the same
across quartiles
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
12
8
4
12
8
4
12
8
4
12
8
4
1 2 3 4
0 5 10 15 20
Hour of Day (UTC)
Alerts per hour
Time-based deviation?

34. Does the day of week
matter?

35. 40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
Sun Mon Tue Wed Thu Fri Sat
Day of week
Alerts per hour
Notifying Alerts per Day

36. 40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
Sun Mon Tue Wed Thu Fri Sat
Day of week
Alerts per hour
Notifying Alerts per Day
Not really

37. Squeaky wheels?
(checks)

38. 30
20
10
0
30
20
10
0
30
20
10
0
30
20
10
0
1 2 3 4
0 50 100 150 200 250
Checks ranked by noise
Alerts per hour
Noisiest checks (overall)
Outlier

39. ●
●
●
● ●
● ●
● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ●
30
20
10
0
0 20 40
Checks ranked by noise
Alerts per hour
Noisiest checks (outlier)
Outlier in more detail

40. 8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
1 2 3 4
0 50 100 150 200
Checks ranked by noise
Alerts per hour
Noisiest checks (without outlier)
Long Tail

41. Squeaky wheel?
(hosts)

42. 30
20
10
0
30
20
10
0
30
20
10
0
30
20
10
0
1 2 3 4
0 50 100 150 200
Hosts ranked by noise
Alerts per hour
Noisiest hosts (overall)
Same outlier

43. ●
●
● ●
●
●
●
●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
●
30
20
10
0
3
0 20 40 60
Hosts ranked by noise
Alerts per hour
Noisiest hosts (outlier)
Similar pattern as checks

44. 8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
1 2 3 4
0 50 100 150 200
Checks ranked by noise
Alerts per hour
Noisiest checks (without outlier)
Long Tail

45. Recurring alerts

46. ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
● ●
● ●
● ●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●●
●●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
● ●
●
●
● ● ●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●●
●●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●● Happens
150
Often
100
50
0
0 100 200 300
Age between earliest and latest occurrence
Number of days occurring
factor(quartile)
●
●
●
●
1
2
3
4
Alert age & frequency of occurrence
Young Old
Seldom
happens

47. Occur often, for a long time Tolerated
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
● ●
● ●
● ●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●●
●●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
● ●
●
●
● ● ●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●●
●●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●● 150
100
50
0
0 100 200 300
Age between earliest and latest occurrence
Number of days occurring
factor(quartile)
●
●
●
●
1
2
3
4
Alert age & frequency of occurrence
Happen once in a while

48. More data?
More questions.

49. HOWTO?

50. Awk
R
ggplot2
Find out tomorrow!
Postgres
d3

51. Presentation matters

53. Take-away?

54. Take-aways
• Don’t rely on your memory to prioritize
• Your Nagios logs are a treasure trove
• Have a dialog with your data
• Presentation matters

55. Curious about Datadog?
Like cute logos?
http://dtdg.co/nagios2012