Monitoring with Prometheus

#DOPPA17
Prometheus: Monitoring
Pravin Magdum
9th September 2017

#DOPPA17
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media
marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us)
• Devops evangelist @ Crevise Technology
• Developer turned into Devops evangelist
• 9 + years of development and project management exp in
various technology.
• Love to resolve tech problems,debug issues.
Who Am I?

#DOPPA17
What is and why Monitoring ?
• Continuously keep track of the status of the system
• Continuously keep track of deployed applications
• Earliest warning of failures, defects or problems and to improve them
• Trending to see over time - help with upgrade /downgrade infra resources
• To know when things go wrong
• If issue persists, analysed data to debug issue and prevent it in future
• Black box monitoring
• Whitebox monitoring

#DOPPA17
Black box monitoring
• Just like smoke testing
• Examples - Ping ,http requests
• To check if server is up and working etc
• When - when system broken and to test from outside n/w
• Won’t get info -whats going inside machine

#DOPPA17
White box monitoring
• Complementary to black box testing
• Get info -whats inside going in system
• Example - check CPU usage, n/w usage

#DOPPA17
What to Monitor ?
• It is best to, first, understand what holds business value to
you and your customers.
• CPU, Memory, IO, storage - typical metrics
• Application monitoring - to make application run in cluster
depending on these metrics
• Predicate resource utilization to avoid downtime.

#DOPPA17
Prometheus
• Inspired from Google’ Borgmon monitoring system
• Mainly written in GO , publicly launched in 2015
• Open source Monitoring and alerting system with active Eco
system
• Used by Docker, Digital ocean, Core Os to name few

Prometheus Offers
Prometheus Offers -
• Multi-dimensional data model(time series data)
– No strings like “doppa.pune”
– Key value pairs {event=Doppa, city=pune}
• Powerful Queries - To leverage this dimensionality
• Precise alerting
• Pull model over HTTP
• Scalable
• Dashboards
• Efficient -
– Single server can handle - Millions of metrics

#DOPPA17
Lets Understand with simple
diagram

#DOPPA17
Components
• Prometheus Server - scrapes and stores time series data
• Exporters - to get metrics from resources
• Alet Rules - define alert rules
• Alert Manager - to notify on different communication channel
about alerts

#DOPPA17
Powerful Queries
• Can multiply ,join,add,aggregate ,predict in same query
• Can evaluate current as well as backdated data
• E.g.
• Which are top 3 services who are consuming CPU most or more
than 80% ?
• Will my storage get full in next 4 hours?

#DOPPA17
Some Query examples
• CPU: 100 - (avg by (instance)
(irate(node_cpu{instance="node1:9100",job="node",mode="idle"}[1m])) *
100)
• Memory: node_memory_MemTotal{job=‘node’,instance=‘node1:9100’} -
node_memory_MemFree{job=‘node’,instance=‘node1:9100’} -
node_memory_Buffers{job=‘node’,instance=‘node1:9100’} -
node_memory_Cached{job=‘node’,instance=‘node1:9100’}
• Disk Write : irate(node_disk_bytes_written[60s]) / 1024

Out of box feature
• The textfile collector is similar to the Pushgateway, in that it allows
exporting of statistics from batch jobs,shell scripts.
• Metrics not exported by node-exporter
• You can still have such metrics in prometheus with the help of Textfile
Collector
• Produce output that is compatible with Prometheus text output format
• Write your own exporters to feed prometheus
• ./node_exporter --collector.textfile.directory=Metrics

#DOPPA17
Running prometheus
• Download prometheus
https://prometheus.io/download/#prometheus
• Extract and Run - done.
• Let’s hit http://localhost:9090
• Let’s see in action

#DOPPA17
Node Exporter installation
• Again two steps
• Go to https://prometheus.io/download/#node_exporter
• And download node exporter ,Extract and Run - done
• Exports metrics at port : 9100
• let’s hit http://localhost:9100

#DOPPA17
Configuration
• It’s time to tell prometheus to pull metrics from node exporter
• Edit Prometheus.yml file - configuration file for prometheus
• Scrape interval -15 sec
• Scrape_configs: what are we scraping
• targets: nodes Ip/hostname to monitor
• Labels: logical group of hosts

#DOPPA17
Sample configuration file
Global:
scrape_interval: 15s
Alert.Rules:
-’CriticalAlert.Rules’
scrape_configs:
job_name: node
static_configs:
labels:
Group: ‘QA-Env'
targets:
- "IP:9100"

#DOPPA17
Reload prometheus
curl -X POST
http://localhost:9090/-/reload
# above curl command will reload prometheus server with new
configuration without restart

#DOPPA17
Alert Rules and manager
• Define alert rules with powerful prom queries
• Predicate about linear changes at nodes
• Send Alerts on your choice of communication channel
• e.g. slack , pagerduty , email ,sms etc

#DOPPA17
Setting up Alert Manager
• Alertmanager can be configured to send prometheus alerts to
your mailbox,slack,get automated calls in critical situation etc.
• Download Alertmanager from
https://prometheus.io/download/#alertmanager
• Extract and configure
• Run

#DOPPA17
Alert rules - instance is up and
running ?ALERT InstanceDown
IF up == 0
FOR 10m
LABELS { severity = "CRITICAL" }
ANNOTATIONS {
summary = "Instance down",
description = "{{ $labels.group }}-{{$labels.instance}} - instance has been down for more than 10 minute." }

#DOPPA17
Alert Rule - does my cpu usage
going beyond 75% ?
ALERT NodeCPUUsage
IF (100 - (avg by (instance) (irate(node_cpu{job="node",mode="idle"}[1m])) * 100)) > 75
FOR 2m
LABELS { severity="CRITICAL"}
ANNOTATIONS {
SUMMARY = "{{ $labels.group }}-{{$labels.instance}}: High CPU usage detected",
DESCRIPTION = " CPU usage is above 75% (current value is: {{ $value }})"
}

#DOPPA17
Alert Rule- storage packed 80 % ?
ALERT filesystem_threshold_exceeded
IF 100 *(1 - (node_filesystem_free{mountpoint="/"} / node_filesystem_size{ mountpoint="/"}) ) > 80
LABELS {severity="CRITICAL" }
ANNOTATIONS {
summary = "{{ $labels.group }}-{{ $labels.instance }} High filesystem usage is detected",
description = "This device's filesystem usage has exceeded the threshold with a value of {{ $value }}.",
}

#DOPPA17
AlertManager.yml
Alert manager Configuration file
global:
route:
repeat_interval: 4h
routes:
- receiver: email-QA
match:
group: 'QA-trad'
- receiver: email-prod

#DOPPA17
Receivers
receivers:
- name: "email-Prod"
email_configs:
- to: 'Prodsupport@crevise.com'
from: 'no-reply@crevise.com'
smarthost: 'smtp.office365.com:587'
auth_username: 'no-reply@crevise.com'
auth_identity: 'no-reply@crevise.com'
auth_password: 'fXXXX'

#DOPPA17
Slack Alerts

#DOPPA17
Email Alerts

#DOPPA17
Set up Grafana
• Download tar from official site,Extract it and run binary
• Check URL http://<Server-IP>:3000
• Add Prometheus as datasource
• Go to http://<Server-IP>:3000
• Enter the username admin and password admin, and then click “Log In”.
• Click “Data Sources” on the left menu
• Click “Add new” on the top menu

#DOPPA17
Thank you !!
Questions ?
Reachable at
pravin.magdum@crevise.com
Twitter - @pravin_magdum

Monitoring with Prometheus

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Monitoring with Prometheus

Similar to Monitoring with Prometheus (20)

Recently uploaded

Recently uploaded (20)

Monitoring with Prometheus