The document discusses integrating Icinga and Prometheus for system monitoring. It describes the key features and concepts of Prometheus including its data model, metric types, jobs and instances. The presentation covers configuring Prometheus servers, scraping metrics using exporters and the Pushgateway, and implementing custom metrics using the Node Exporter. Combining Icinga and Prometheus allows leveraging Icinga for configuration, alerting and notifications while using Prometheus for collection and querying of metrics.

1. https://bit.ly/2Cs2ql4

2. Thank you
Hosted by Sponsored by

3. Today
● Agentless monitoring with Icinga and Prometheus by Diogo Machado
● Coffee break and networking

4. Agentless monitoring with Icinga and
Prometheus
Diogo Machado
dgm@eurotux.com
04/11/2019
DevOps Braga #15

5. Agenda
● From Icinga to Prometheus
● Prometheus Basic Concepts
● Prometheus Server Configuration
● Getting data into Prometheus
● Implement custom metrics
● How to integrate Icinga with Prometheus?

6. From Icinga to Prometheus - Introduction
● Open-source computer system and network
monitoring application;
● Monitor the availability of hosts and
services;
● Distributed Monitoring;
● Agent Based Monitoring;
● Notifications and Downtimes;
● Written in PHP and C++;
● Open-source systems monitoring and
alerting toolkit;
● Monitoring of highly service-oriented
architectures;
● Collect metrics with Exporters;
● Store aggregated metrics;
● Written in Go;

7. From Icinga to Prometheus - Comparison
● Alerting based on the exit codes of Checks;
● Host-based;
● There is no notion of labels or query language.
● No storage per-se, beyond the check state;
● All configuration are made via file;
● Monitoring of small and/or static systems
where blackbox probing is sufficient.
● Multi-dimensional data model with time
series data;
● Rule-based alerts;
● Prometheus Query Language - PromQL;
● Centralized data store;
● Suitable for dynamic or cloud based
environment - Whitebox monitoring;

8. From Icinga to Prometheus - Comparison
If there are both to system monitoring….
Why not choose only one?
What does Prometheus have that Icinga doesn't have?
Why should we combine both?

9. From Icinga to Prometheus - Conclusion
So … Why should we combine both?
● Prometheus have:
○ Exporters to third-party systems and applications;
○ Centralized control and HTTP API;
● Icinga have:
○ Easy configuration of host and services (Icinga Director);
○ Good alerting system with notifications and schedule downtimes;
Combine to get the best of each one:
● Scrape metrics with Prometheus
● Configure and Alert with Icinga

10. Prometheus's main features are:
● multi-dimensional data model with time series data identified by metric name and key/value pairs;
● PromQL, a flexible query language to leverage this dimensionality;
● no reliance on distributed storage - single server nodes are autonomous;
● time series collection happens via a pull model over HTTP;
● pushing time series is supported via an intermediary gateway;
● targets are discovered via service discovery or static configuration;
● support for graph and dashboard mode.
Prometheus Basic Concepts

11. Prometheus Basic Concepts - Architecture
The Prometheus ecosystem consists of
multiple components:
● the main Prometheus server which
scrapes and stores time series data;
● a Pushgateway for supporting
short-lived jobs;
● specific exporters for services like
HAProxy, StatsD, Graphite, etc;
● an Alertmanager to handle alerts;
● data visualization tools like Grafana;

12. Prometheus Basic Concepts - Data Model
● Data is stored as time series: streams of
timestamped values belonging to the same
metric and the same set of labeled dimensions;
● Key Value Data Model:
○ Key - Metric name and a set of labels;
○ Value - Metric measuring;

13. Prometheus Basic Concepts - Metric Types
Counter
● Cumulative metric;
● Monotonically increasing
counter;
● Examples:
○ Nº requests served;
○ Nº tasks completed;
○ Number of errors;
Gauge
● Numerical value that can
arbitrarily go up or down;
● Examples:
○ Temperature;
○ Memory usage;
○ Nº concurrent requests;
Histogram
● Values are aggregated in
buckets;
● Expose total sum of all
observed values;
● Count of events that have
been observed;
● Example:
○ Request latency;
Summary
● Similar to a histogram;
● Calculates configurable
quantiles over a sliding
time window;
● Examples:
○ Request durations;
○ Response sizes;
requests_time_seconds_bucket{app=”projectx”,le=”0.005"} 2343340162
… (buckets)
requests_time_seconds_sum{app=”projectx”} 5.366133242442994e+07
requests_time_seconds_count{app=”projectx”} 3973861256
go_gc_duration_seconds{quantile="0"} 4.274e-05
… (quantiles)
go_gc_duration_seconds_sum 0.467543895
go_gc_duration_seconds_count 92
Histogram Summary

14. Prometheus Basic Concepts - Jobs and Instances
● An endpoint you can scrape is called an Instance;
● A collection of instances with the same purpose is called a Job;
● Prometheus scrapes a Target and attaches labels automatically: job name, instance host and port;
● Example of job with 3 instances:
job: 1:
instance 1: 128.0.0.1:9030
instance 2: 128.0.0.2:9030
instance 3: 128.0.0.3:9030

15. Prometheus Basic Concepts - PromQL
Expression language data types:
● Instant vector - a set of time series containing a single sample for each time series;
○ Example: http_requests_total{environment=~"staging|development",method!="GET"}
● Range vector - a set of time series containing a range of data points for each time series;
○ Example: http_requests_total{job="prometheus"}[5m]
● Scalar - a simple numeric floating point value;
○ Example: -2.43
● String - a simple string value;
○ Example: 'these are unescaped: n t'

16. Prometheus Basic Concepts - REST API
● Response format is JSON: ● Methods: GET, POST
● Endpoints:
○ /api/v1/query
○ /api/v1/query_range
○ /api/v1/series
○ /api/v1/labels
○ /api/v1/targets
○ /api/v1/rules
○ /api/v1/targets/metadata
○ /api/v1/status/config
○ /api/v1/status/flags

17. Prometheus Basic Concepts - Scrape Metrics
● Prometheus works essentially pulling metrics metadata from targets;
● Pulling over HTTP offers a number of advantages:
○ Easily tell if a target is down;
○ Manually inspect target health with a web browser.
● However, push model can be implemented with Pushgateway:
○ Intermediary service which allows to push metrics from jobs that cannot be scraped;
○ Disadvantages:
■ Pushgateway becomes both a single point of failure (SPOF) and a potential
bottleneck;
■ Lose Prometheus's automatic instance health monitoring via the UP metric.

18. Prometheus Server - Installation
● Using pre-compiled binaries;
● From source (Makefile);
● Using Docker (Quay.io or Docker Hub)
● Using configuration management systems:
○ Ansible
○ Chef
○ Puppet
Docker command:
docker run -p 9090:9090 -v /prometheus-data
prom/prometheus --config.file=prometheus.yml
Dockerfile:
FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/
Build:
docker build -t my-prometheus .
docker run -p 9090:9090 my-prometheus

19. Prometheus Server - Configuration
● Prometheus is configured via command-line flags and
a configuration file: prometheus.yml;
● Prometheus default port is 9090;
● Prometheus can reload its configuration at runtime:
○ SIGHUP to the Prometheus process;
○ HTTP POST request to the “/-/reload” endpoint (when
the --web.enable-lifecycle flag is enabled).
● Recording rules and Alerting rules should be written in
a YAML file (rule_files);
● It’s possible to use service discovery mechanism to
automatically update scrape target list;
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
external_labels:
environment: tst
rule_files:
- /opt/prometheus/rules/*.rules
scrape_configs:
- job_name: node
scrape_interval: 30s
metrics_path: /metrics
scheme: http
ec2_sd_configs:
- endpoint: ""
region: eu-west-3
refresh_interval: 1m
port: 9100
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
scheme: http
timeout: 10s
api_version: v1

20. Prometheus Server
Configuration
Consult Prometheus
Configuration via Web browser

21. Prometheus Server
Targets and Rules
Targets and Rules defined on
Prometheus configuration

22. Getting data into Prometheus
● Data can be export to Prometheus using Pushgateway or via Exporter;
● Prometheus have multiple Exporters, of which stand out:
○ Node/system metrics exporter;
○ MySQL server exporter;
○ Memcached exporter;
○ Kafka exporter;
○ HAProxy exporter;
○ Tomcat exporter;
○ AWS CloudWatch exporter;
● Each Exporter has a default port allocated and a route (‘/metrics’’);
● Besides official and community Exporters, it’s possible to write custom Exporters;
● Most exporters can be run as a service or using Docker;

23. ● Pushgateway isn’t an event store but an intermediary service to push metrics from jobs that can’t be
scraped;
● Deploy using binary or Docker image:
○ Example: docker run -d -p 9091:9091 prom/pushgateway
● To change the listen address, use the “--web.listen-address” flag;
● By default, Pushgateway doesn’t persist metrics. To cache them, use “--persistence.file” option;
● All pushes are done via HTTP. The interface is REST-like. Metrics are available on ‘/metrics’ route ;
● Examples:
○ echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/job1
○ curl -X DELETE http://localhost:9091/metrics/job/job1
Getting data into Prometheus - Pushgateway

24. Getting data into Prometheus - Node Exporter
● Expose metrics of Hardware and Operating System based on collectors, usually running on port 9100;
● By default, there is a specific set of collectors for each operating system: cpu, diskstats, filesystem,
netstat, nfs, textfile, …
● Others can be enabled with collector option: ntp, processes, systemd, …
○ Example: --collector.processes
● Exporter can be run using third-party repository for RHEL/CentOS/Fedora (Corp), using the source
code or using Docker;
○ Example:
docker run -d --net=”host” --pid=”host” -v “/:/host:ro” prom/node-exporter --path.rootfs=/host --collector.ntp
--collector.processes --collector.textfile.directory /var/lib/node_exporter/textfile_collector --cap-add=SYS_TIME

25. Implement custom metrics on Node Exporter
● Custom metrics can be implemented with textfile collector;
● Textfile collector is similar to Pushgateway, in that it allows exporting of statistics from batch jobs;
● Pushgateway should be used for service-level metrics, while textfile should be used for machine metrics;
● The collector will parse all files in textfile directory matching the glob *.prom using the text format;
● Example to automatically push logged in users on machine:
echo node_users_logged_in $(who /host/var/run/utmp | wc -l) > /var/lib/node_exporter/textfile_collector/users.prom.$$
mv /var/lib/node_exporter/textfile_collector/users.prom.$$ /var/lib/node_exporter/textfile_collector/users.prom

26. Getting data into
Prometheus - Node
Exporter
Example of metrics available at
endpoint “/metrics”

27. Prometheus query
Node Exporter Data
Example of a query in
PromQL to check CPU usage

28. How to integrate Icinga with Prometheus?
● Icinga can be integrated with Prometheus via Nagitheus (Claranet);
● Nagitheus is a Nagios plugin for querying Prometheus, written in Go;
● Nagitheus process vector or scalar results and return an exit code, according with warning/critical
values and comparison method (ge, gt, le, lt);
● Allows basic authentication on Prometheus with username and password (-u and -pw options);
● Example:
/usr/lib64/nagios/plugins/nagitheus -H http://localhost:9090 -i 10.0.0.10 -p 9100 -l 'Check CPU' -d yes -q
'(avg by (mode) (irate(node_cpu_seconds_total{instance="", mode!="idle"}[5m])) * 100)' -m ge -w 70 -c 80

29. How to integrate Icinga with Prometheus?
● Basic Linux Checks:
○ CPU
○ Disk
○ Load
○ Memory
○ Total procs
○ Ntp
● Steps to implement a check:
○ Identify metrics to use on query;
○ Create query considering PromQL operators and metric types;
○ Test query on Promehteus and define label, method and critical/warning values;
○ Implement icinga command using the query and values specified for each option.

30. How to integrate Icinga with Prometheus?
● Check Disk (Percentage of disk free):
1. Metrics to use on check:
○ node_filesystem_free_bytes
○ node_filesystem_size_bytes
2. Data types: Instant vectores;
3. PromQL operators:
○ / (division)
○ * (multiplication)
4. Query:
(node_filesystem_free_bytes / node_filesystem_size_bytes{fstype!~"none|tmpfs|sysfs", mountpoint=~"/var/.*", instance=”172.27.68.163:9030”} )* 100
5. Analyze query result and define method, critical and warning value
# HELP node_filesystem_free_bytes Filesystem free space in bytes.
# TYPE node_filesystem_free_bytes gauge
node_filesystem_free_bytes{device="shm",fstype="ext4",mountpoint="/var"} 1.01808578e+10
…
# HELP node_filesystem_size_bytes Filesystem size in bytes.
# TYPE node_filesystem_size_bytes gauge
node_filesystem_size_bytes{device="shm",fstype="ext4",mountpoint="/var"} 1.05017712e+10
…

31. How to integrate Icinga with Prometheus?
● Query Result
Method: le Warning: 20% Critical:10%
● Icinga Command:
/usr/lib64/nagios/plugins/nagitheus -H http://localhost:9090 -i 172.27.68.163 -p 9030 -l 'Check disk' -d yes -m le -w 20 -c 10 -q
'(node_filesystem_free_bytes / node_filesystem_size_bytes{fstype!~"none|tmpfs|sysfs", mountpoint=~"/var/.*"} )* 100'
Warning
(20) Critical
(10)

32. Questions?