This document discusses moving beyond the limitations of Nagios for infrastructure monitoring. It summarizes that while Nagios is an industry standard, its configuration can be daunting and not user-friendly. It then describes common problems with Nagios like being overwhelmed by alerts, entering a "spiral of death" by adding more checks that lead to more alerts, and trying to improve coverage through more checks but resulting in the "trough of despair." The document recommends ways to improve the situation like measuring data, looking for patterns in alerts, putting alerts in context visually, and focusing on business impact.

1. Beyond Nagios
NYC DevOps 2011/07/21
Alexis Lê-Quôc - alq@datadoghq.com

2. Beyond Nagios
NYC DevOps 2011/07/21
Alexis Lê-Quôc - alq@datadoghq.com

3. What I’m Going To Talk About
• Super-quick Nagios summary
• Monitoring/Alerting Pathologies
• How to fix it

4. What Is
• “Industry Standard in IT Infrastructure Monitoring”
• For once it’s true...
• Scheduler & Notification server

5. (+) Robust, Mature code-base
(-) Configuration can be daunting
(-) Not human-friendly

6. “OVERWHELMING”

7. A “NORMAL” HOUR

8. THE “OTHER” NAGIOS UI

9. Process alerts
& Fix things
Receive alerts Add more checks
THE HAPPY START

10. Missed alerts
Ignore Alerts Add more checks
THE SPIRAL OF DEATH

11. Quality
of life
Few checks
Few alerts
More checks
Too many alerts
# of alerts
FIGHT OR FLIGHT

12. Effective Checks n^2
Coverage Fault-tolerant
Less urgency
Few checks
Few alerts
Every host counts
More checks
Too many alerts
Every host still counts Scale
Complexity
THE TROUGH OF DESPAIR

13. Effective
Coverage
Scale
IF ONLY I ADDED MORE
CHECKS...

14. Reset!

15. Way Out
‣Breathe!
‣Measure
‣Look for Patterns
‣Put Alerts in Context
‣Focus on the Business

16. Turn Nagios logs into structured data
Analyze
day | success_pct | warning_pct | error_pct | events
---------------------+-------------+-------------+-----------+--------
2011-07-12 00:00:00 | 89 | 0| 2 | 9628
2011-07-13 00:00:00 | 90 | 0| 2 | 9210
2011-07-14 00:00:00 | 90 | 0| 2 | 9735
2011-07-15 00:00:00 | 89 | 0| 2 | 9531
MEASURE

17. day | success_pct | warning_pct | error_pct | events
---------------------+-------------+-------------+-----------+--------
2011-07-12 00:00:00 | 89 | 0| 2 | 9628
2011-07-13 00:00:00 | 90 | 0| 2 | 9210
2011-07-14 00:00:00 | 90 | 0| 2 | 9735
2011-07-15 00:00:00 | 89 | 0| 2 | 9531
VISUALIZATION MATTERS

18. In Time
Flapping
LOOK FOR PATTERNS

19. PUT ALERTS IN CONTEXT
https://app.datad0g.com/dash/dash/1000#/date_range/1310682467000.0-1310684267000.0

20. Ultimate (hard) question
‣Does this alert impact the business?
‣If so by how much?
‣Assumes that you track business metrics...
‣And they can be accessed programatically
FOCUS ON THE BUSINESS

21. What applies to Nagios...
Applies to other sources too
etc...

22. Thanks
http://datadoghq.com