The document discusses various aspects of monitoring systems including: - What monitoring involves such as collecting data, processing it, and displaying metrics. - Why monitoring is important for analyzing trends, comparing performance, and debugging. - Some advantages and disadvantages of the Nagios monitoring tool and why it may be time to stop using it. - What makes for good state and metric monitoring in terms of being actionable, automated, proactive, and easy for operators. - Potential monitoring tools and strategies including Icinga2, Prometheus, Elasticsearch, PagerDuty, and New Relic.

1. Monitoring
Deeper dive

2. Who am I
Robert Kubiś
DevOps Engineer
https://www.linkedin.com/in/robertkubis89

3. Mikey
Dickerson
Hierarchy of
Needs

5. Monitoring
Collecting, processing, aggregating, and displaying real-time quantitative data
about a system, such as query counts and types, error counts and types,
processing times, and server lifetimes.
● White-box monitoring
● Black-box monitoring
● Dashboard
● Alert
● Root cause
● Push
● Node and machine

6. Why Monitor?
● Analyzing long-term trends
● Comparing over time or experiment groups
● Alerting
● Building dashboards
● Conducting ad hoc retrospective analysis (i.e., debugging)

7. Please stop using nagios (Andy Sykes)
So we can die peacefully…..
Who use it?
Why did you choose it?

8. Please stop using nagios (Andy Sykes)
So we can die peacefully…..
Who use it?
Why did you choose it?
Advantages:
● Incredible simple plugins model.
● Simple to use
● Many people know it.
● On the top in google and everybody
use it :)

9. Please stop using nagios (Andy Sykes)
So we can die peacefully…..
Who use it?
Why did you choose it?
Advantages:
● Incredible simple plugins model.
● Simple to use
● Many people know it.
● On the top in google and everybody
use it :)
Disadvantages:
● Doesn’t scale - cannot be clustering -
Thruk hack
● Millions lines of configuration -
check_mk hack
● Horrible interface
● Only for static infrastructure
● Stupid format of clients - hacks
● Perfdata…
● Doesn’t have API - livestatus hack
● Always need to hack….

10. Nagios

13. When your monitoring suck...
- Improve the quality of alerts
- Improve monitoring tools, or even change them
Wait a minute…. Before you start to solve them...

14. UNDERSTAND PROBLEMS AND MEASURE THEM!!!
“To measure is to know”
“If you can not measure it,
you can not improve it”
Lord William Thomson
(aka Baron Kelvin)

17. Over-monitoring and alarm fatigue: For whom do the
bells toll? Hospitals in USA
- Ignoring Alarms notification
- “Yeah that is no important”
- 72–99% false alerts
- Young parents vs nurses in hospital
- Monitoring means more money
- More is not better
- Patient could died
- Telemetry as a means of preventing, detecting, and improving
Source:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4926996/

18. What to do?
So what to use for monitoring?

19. What monitoring should be?
Actionable
Compatible
Essential - only alerts which are needed
Fully Automated
Proactive - should predict failures
Easy for operators

20. State monitoring what it should be like?
State or blackbox monitoring now has the most of sense in VMs and bare-metals
What should be monitored with those kind of tools?
● Health endpoints
● Service states (like systemctl status *)
What could be monitored?
● Specific endpoints (using for example satellite node) with http/tcp checks

21. Icinga2 - Nagios fork but rewritten in many places, has scaling scenarios
(multimaster, with 3 levels of nodes - masters, satellites(ie. Supervisor per DC),
clients(check executors)), plugins - like InfluxDB metric exporter, livestatus etc.
What we can get from Icinga2?
● High Available and distributed setup
● Nice and good documented REST API
● (dynamic inventory)
● Decrease amount of time needed for implement features

22. Metrics
Metric tools could be used in two ways:
1. Failure prediction
2. Graphing the data for humans - for humans it means SIMPLE
First case is quite simple - rules for detecting anomalies like more traffic than
usual and alert if it can make an impact on other clients
Second case is also simple - just graphs for debugging and better understanding
what’s happening with applications
Not every metric should have alert (and notifications)!

23. Prometheus
Circa 120 ready to use dashboards in Grafana repository(ie. MySQL board by
Percona)
Many useful features in one tool - Prometheus has a rich query language, Alert
manager, support for PagerDuty etc.
Plenty of exporters (collectors) for standard tools: MySQL, HAProxy, NGINX,
Pagespeed, BIND, Jenkins, scollector
Third party project support for Prometheus: GitLab, Kubernetes, etcd, telegraf,
jmx-exporter, collectd)

24. Logs
Servers, application, network and security devices generate log files.
Errors, problems, and more information is constantly logged and saved for
analysis.
Once an event is detected, the monitoring system will send alert, either to a
person or to another software/hardware system.

25. Elasticsearch stack

27. Monitoring strategy
Icinga 2 for state monitoring on bare metal, VMs and VMs in cloud.
Prometheus for metrics and data from Kubernetes (or other container) clusters.
ELK stack for logs

28. Is that enough?
What should be next step?

29. What is Pager Duty?

30. Users Settings

31. Notification Roules

32. Schedules

33. Escalation Policies

34. Services

35. Integrations
Integrations list
Connect with any
tool that provides
incoming event
data.

36. Extensions
Extensions list
Extend the PagerDuty
workflow to your existing tools.

37. Good practices for alerts
● Notify before accident
● Actionable alarms
● Value of measure things
● Documentation - not only one-liners in on call wiki
● Reduce number of tools
● Terraform

38. Let’s say that you’re rich :)

39. New Relic

51. STACKDRIVER
● Full-Stack Monitoring, Powered by Google
● For Cloud Platform, AWS, and Hybrid Deployments
● Identify Trends, Prevent Issues
● Reduce Monitoring Overhead
● Improve Signal-to-Noise
● Fix Problems Faster

52. Stackdriver heatmap

53. STACKDRIVER MONITORING FEATURES
● Debugger
● Error reporting
● Rapid discovery
● Uptime monitoring
● Integrations
● Smart defaults
● Alerts
● Tracing
● Logging
● Dashboards
● Profiling

54. MONITORING = PEOPLE
Not only tools...

55. Team
To make sense currently and in the future changing the monitoring infrastructure
should be supported by development and "reacting" teams.
Reacting team:
● 24/7 people for looking on boards and reacting on issues work shifts
● Incident manager taking decisions and investigating tuning of monitoring
● People with “programming” skills responsible for deploy proposals of IM
(writing new checks, adding some pieces of code)

56. Plan your work