Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working

1. Presentation on Monitoring Tool
GRAFANA
Dhruba Mandal
Email : dhruvmandal@gmail.com
Mobile : +91-8800884197

2. Introduction To Grafana
✓Grafana is an open-source platform for data visualization, monitoring and
analysis.
✓It is most commonly used for visualizing time series data for infrastructure
and application analytics.
✓In my company, we use it with primarily Elastic search and InfluxDB, though
Grafana also supports other data sources Prometheus, MySQL, Etc
✓ For each data source, Grafana has a customized query editor and specific
syntax.

3. Features of Grafana
✓Open source : It’s completely open source, and can be easily install on any platform.
✓Visualization : From heat maps to histograms. Graphs to geomaps. Grafana has a plethora
of visualization options to help you understand your data, beautifully.
✓Alerting : Seamlessly define alerts (i.e. smoothly and continuously, with no apparent gaps
between one part and the next )where it makes sense — while you’re in the data. Define
thresholds visually, and get notified via Slack, Pager Duty, and more
✓Unifying : make or become united, uniform, or whole (i.e. Bring your data together to get
better context. Grafana supports dozens of databases, natively. Mix them together in the
same Dashboard)
✓Extended Plugins : Discover hundreds of dashboards and plug-in in the official library.
✓Collaboration : Bring everyone together, and share data and dashboards across teams.

4. Installation of Grafana On Linux
(Redhat/Centos)
1. We can install Grafana using Yum directly by below Command
sudo yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-
4.6.1-1.x86_64.rpm
2. Once configured Start the server (via systemd).
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
3. Enabling the systemd service to start at boot
sudo systemctl enable grafana-server.service

5. 4. Login into the Grafana by UrL on port 3000 (3000 is default port of Grafana)
http://localhost:3000/
The default user name and password is (admin /admin ) but it gives option to change the credentials at
first login

6. Environment file of Grafana
✓The systemd service file and init.d script both use the file located at
/etc/sysconfig/grafana-server
✓By default Grafana will log to (i.e. login log will be found ) at
/var/log/grafana
✓The default configuration of database located at
/var/lib/grafana/grafana.db
✓The configuration file is located at
/etc/grafana/grafana.ini

7. Adding data sources to Grafana
Different data sources used with Grafana are
• Graphite
• InfluxDB
• OpenTSDB
• Prometheus
As We go on comparing pros and cons of different data sources based on various
dimension we find that Prometheus is one of the best data sources for Grafana as
The Prometheus monitoring system includes :
✓ A rich, multidimensional data model, a concise and powerful query language
called PromQL,
✓An efficient embedded time series database and
✓Over 150 integrations with third-party systems.

8. Introduction To Prometheus
Prometheus is an open-source systems monitoring and alerting toolkit originally built at
Sound Cloud. Since its inception in 2012, many companies and organizations have
adopted Prometheus, and the project has a very active developer and user community
Prometheus works well for recording any purely numeric time series.
It fits both machine-centric monitoring (i.e. Information in the form of numbers which
can be understood by a computer or microchip) , as well as monitoring of highly dynamic
service-oriented architectures
Its support for multi-dimensional data collection and querying is a particular strength.

9. Architecture of Prometheus

10. Components of Prometheus
The Prometheus ecosystem consists of multiple components, many of which are
optional:
The main Prometheus server which scrapes and stores time series data
• Client libraries for instrumenting application code
• A push gateway for supporting short-lived jobs
• Special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
• An alert manager to handle alerts and
• various support tools
Note : Most Prometheus components are written in Go, making them easy to build
and deploy as static binaries.

11. Features of Prometheus
Prometheus's main features are:
• A multi-dimensional data model with time series data identified by metric name and
key/value pairs
• PromQL, a flexible query language to leverage this dimensionality
• No reliance on distributed storage; single server nodes are autonomous
• Time series collection happens via a pull model over HTTP
• Pushing time series is supported via an intermediary gateway
• Targets are discovered via service discovery or static configuration
• Multiple modes of graphing and dash boarding support

12. Downloading and Installing Prometheus
1. Download and Unpack Prometheus latest release of Prometheus by below command
sudo apt-get update && apt-get upgrade
wgethttps://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz
tar xfz prometheus-*.tar.gz
cd prometheus-*
The following two binaries are in the directory:
Prometheus - Prometheus main binary file
promtool
2 . Copy the binary files into the /usr/local/bin/directory:
sudo cp ./prometheus /usr/local/bin/
sudo cp ./promtool /usr/local/bin/

13. 3 . Set the ownership of these files to the prometheus user previously created:
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
The following two folders (which contain the web interface, configuration files examples
and the license) are in the directory:
• consoles
• console libraries
4 . Copy the consoles and console libraries directories to /etc/prometheus:
sudo cp -r ./consoles /etc/prometheus
sudo cp -r ./console_libraries /etc/prometheus
5 . Set the ownership of the two folders, as well as of all files that they contain, to our
prometheus user:
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
6 . In our home folder, remove the source files that are not needed anymore:
cd .. && rm -rf prometheus-*

14. Configuring Prometheus
Prior to using Prometheus, we need to create a configuration file named prometheus.yml
1 . Open the file prometheus.yml in a text editor:
sudo nano /etc/prometheus/prometheus.yml
There are main 3 blocks in Prometheus.yml
• Global Part
• Rules Files
• Scrap Configs
In the global part we can find the general configuration of Prometheus:
The rule files block contains information of the location of any rules we want the
Prometheus server to load.
In the scrape_configs part we have defined our first exporter. It is Prometheus that
monitors itself.

15. global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
sudo nano /etc/prometheus/prometheus.yml
General Configuration of Promethesus.yml

16. 2 . Set the ownership of the file to our Prometheus user:
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
3. Prometheus server is ready to run to Run the which executes the binary file as our
Prometheus user
sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --
storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --
web.console.libraries=/etc/prometheus/console_libraries
The server starts displaying multiple status messages and the information that the server
has started: Else any error will be displayed if any of configuration is wrong

17. Boot configuration for Prometheus server
Our Prometheus server is running Now but it cannot yet be launched automatically at
boot. To achieve this, we have to create a new systemd configuration file that will tell
your OS which services should it launch automatically during the boot process
sudo nano /etc/systemd/system/prometheus.service
1. Copy the following information in the file and save it, then exit the editor:
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target After=network-online.target
[Service] User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus
--config.file /etc/prometheus/prometheus.yml
--storage.tsdb.path /var/lib/prometheus/
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
ExecReload=/bin/kill -HUP $MAINPID
[Install] WantedBy=multi-user.target

18. .2. To use the new service, reload systemd:
sudo systemctl daemon-reload
3 . We enable the service so that it will be loaded automatically during boot:
sudo systemctl enable prometheus
6 . Start Prometheus:
sudo systemctl start prometheus
Browse the prometheus server with below web URL
http://your.server.ip:9000

19. Browsing the Prometheus on Web Interface
Note : by default port of Prometheus is 9090

20. Prometheus data source on Grafana
Open the Grafana by URL http://your.server.ip:3000
Login with Credentials by Default admin /admin
➢ Click on the Grafana logo to open the sidebar.
➢ Click on “Data Sources” in the sidebar.
➢ Choose “Add New”.
➢ Select “Prometheus” as the data source.
➢ Set the Prometheus server URL (in our case: http://localhost:9090/)
➢ Click “Add” to test the connection and to save the new data source.
Screenshot of same is in Next Page

22. Node Exporter
Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable
metric collectors.
As our Prometheus is only capable of collecting metrics, we want to extend its
capabilities by adding Node Exporter, a tool that collects information about the system
including CPU, disk, and memory usage and exposes them for scraping.
We need to configure the node exporter on every machine Which needed to be
Monitored on Grafana

23. Downloading and Installing Node Exporter
1. Download the latest version of Node Exporter:
wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-
amd64.tar.gz
2 . Unpack the downloaded archive. This will create a directory node_exporter-
0.16.0.linux-amd64, containing the executable, a readme and license file:
tar xvf node_exporter-0.16.0.linux-amd64.tar.gz
3 . Copy the binary file into the directory /usr/local/bin and set the ownership to the user
you have created in step previously
sudo cp node_exporter-0.16.0.linux-amd64/node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

24. 4 . Remove the leftover files of Node Exporter, as they are not needed any longer:
rm -rf node_exporter-0.16.0.linux-amd64.tar.gz node_exporter-0.16.0.linux-amd64
6. Copy the following information in the service file, save it and exit nano:
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install] WantedBy=multi-user.target
5 . To run Node Exporter automatically on each boot, a Systemd service file is
required. Create the following file by opening it in Nano:
sudo nano /etc/systemd/system/node_exporter.service

25. 7 . Collectors are used to gather information about the system. By default a set of
collectors is activated. It can be disabled by providing a --no-collector.<name> flag.
8 . Reload Systemd to use the newly defined service:
sudo systemctl daemon-reload
9 . Run Node Exporter by typing the following command:
sudo systemctl start node_exporter
10. Verify that the software has been started successfully:
sudo systemctl status node_exporter
You will see an output like this, showing you the status active (running) as well as the
main PID of the application:
11 . If everything is working, enable Node Exporter to be started on each boot of the
server:
sudo systemctl enable node_exporter

26. Prometheus Tomcat Exporter
1. To run tomcat exporter we have to add below file to library of tomcat
➢ simpleclient
➢ simpleclient_common
➢ simpleclient_servlet
➢ simpleclient_hotspot
➢ tomcat_exporter_client
You can download it from below Git Hub address URL
https://github.com/nlighten/tomcat_exporter
2. Download the tomcat_exporter_servlet war from same above URL and
rename it to metrics.war and add it to the webapps
3. After restart of tomcat you should be able to access metrics via the
/metrics/ endpoint on prometheus

27. Below are some of Tomcat related metrics
➢ Thread pool metrics
➢ Session metrics
➢ Request processor metrics
➢ Database connection pool metrics
➢ Tomcat version info
➢ Servlet response time metrics
➢ Database response time metrics

28. Alert Manager
The Alert manager handles alerts sent by client applications such as the Prometheus
server. It takes care of de-duplicating, grouping, and routing them to the correct receiver
integration such as email, Pager Duty
Features of alert Manager
Grouping : - Grouping categorizes alerts of similar nature into a single notification.
Inhibition : - Inhibition is a concept of suppressing notifications for certain alerts if
certain other alerts are already firing.
For ex - An alert is firing that informs that an entire cluster is not reachable. Alert
manager can be configured to mute all other alerts concerning this cluster if that
particular alert is firing
Silences : - Silences are a straightforward way to simply mute alerts for a given time .
Incoming alerts are checked whether they match all the equality or regular expression
matchers of an active silence. If they do, no notifications will be sent out for that alert.

29. Downloading and Installing Alert Manager
1. Download the latest version of Alert manager inside /opt/ from below site
https://github.com/prometheus/alertmanager/releases
alertmanager-0.18.0.linux-amd64.tar.gz
2. Extract the tar.gz file with command
tar –zxvf alertmanager-0.18.0.linux-amd64.tar.gz
3. Go inside the cd /opt/alertmanager-0.18.0.linux-amd64
Now move alert manager from alertmanager-0.18.0.linux-amd6 to /usr/local/bin/
mv alertmanager-0.18.0.linux-amd64/ alertmanager /usr/local/bin
4. create a alert manager folder inside /etc
cd /etc/
mkdir alert manager

30. 5. Create a alertmanager.yml file inside the directory /etc/alertmanager/
sudo nano /etc/alertmanager/alertmanager.yml
global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'user-emailid'
smtp_auth_username: 'user-name '
smtp_auth_password: 'password'
smtp_auth_identity: 'user-email id '
route:
receiver: 'team-X-mails'
group_by: ['alertname']
#group_wait: 30s
#group_interval: 5m
#repeat_interval: 6h
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
# Apply inhibition if the alertname is the same.
equal: ['alertname']
receivers:
- name: 'team-X-mails'
email_configs:
- to: 'receivermail- id ‘

31. 6 . Now to set up alert manager service create a alertmanager.service file
sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=AlertManager server service
Wants=network-online.target
After=network-online.target
[Service]
user=root
Group=root
#Type=Simple
ExecStart=/usr/local/bin/alertmanager
--config.file=/etc/alertmanager/alertmanager.yml
[Install]
WantedBy=multi-user.target
Configuration of alertmanager.service

32. 7. Now go to the prometheus .yml file and map the path of alert manager
sudo nano /etc/prometheus/prometheus.yml
8 . Add the alert manager configuration path to this prometheus.yml file
# Alertmanager configuration
alerting:
alertmanagers:
- scheme: http
# - path_prefix: "alertmanager"
static_configs:
- targets:
- 10.0.205.50:9093
- 10.0.205.53:9093
9 . Now reload the configuration and start alert manager
sudo systemctl daemon-reload
sudo systemctl start alertmanager

33. 10 . After startup check the status of alert manager by below command
sudo systemctl status alertmanager
Will display active (running) if started successfully or any error can be seen here
Note : by default port of alert manager is 9093

34. 10 . Enter the URL http://ip:9093/ will display the alert manager as below

35. References
• https://grafana.com/grafana
• https://grafana.com/docs/installation/debian/
• https://www.scaleway.com/en/docs/configure-prometheus-monitoring-with-grafana/
• https://grafana.com/docs/plugins/installation/
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
• https://www.youtube.com/watch?v=GiaYg19-OTM
• https://github.com/nlighten/tomcat_exporter
• https://github.com/prometheus/prometheus/wiki/Default-port-allocations
• https://github.com/prometheus/alertmanager/blob/master/doc/examples/simple.yml
• https://github.com/prometheus/alertmanager
• https://prometheus.io/docs/alerting/alertmanager/
• https://itnext.io/prometheus-with-alertmanager-f2a1f7efabd6