MySQL Monitoring using Prometheus & Grafana

1. MySQL Monitoring
using Prometheus & Grafana

2. 2
dba.kim@gmail.com
https://kr.linkedin.com/in/youngheon-roy-kim-23a11181
About me

3. MySQL Monitoring Dashboard
3
Alert using slack
Grafana Dashboard

4. Architecture
4
DB Servers
Monitoring Server
node_exporter
mysql_exporter
prometheus
alert manager
playbook
node_exporter
mysql_exporter
•
•
•
Deploy Server

5. Install & config prometheus on monitoring server
5
1. Install Prometheus
Download .tar file
https://prometheus.io/download/
$tar –xvf prometheus-0.17.0rc2.linux-amd64.tar.gz –C /opt
$mv prometheus-0.17.0rc2.linux-amd64 prometheus

6. Install & config prometheus on monitoring server
6
2. Config prometheus.yml file
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: linux_hostname1
target_groups:
- targets: [‘ip_addr:9100']
labels:
alias: hostname1
- job_name: mysql_hostname1
target_groups:
- targets: [‘ip_addr:9104']
labels:
alias: hostname1
/opt/prometheus/prometheus.yml
continue right side…
scrape_configs :
you can add server information.
It can be added automatically using ansible playbook.

7. Install & config alertmanager on monitoring server
7
1. Install alertmanager
Download .tar file
https://prometheus.io/download/
$tar –xvf alertmanager-0.1.1.linux-amd64.tar.gz –C /opt
$mv alertmanager-0.1.1.linux-amd64 alertmanager
2. make a slack Webhook url
You can make a slack webhook url for sending a alert message.
https://slack.com/apps/A0F7XDUAZ-incoming-webhooks

8. Install & config alertmanager on monitoring server
8
3. Config simple.yml
/opt/alertmanager/simple.yml
global:
slack_api_url: {{ webhook url}}
route:
receiver: 'slack-notifications'
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 1m
repeat_interval: 3m
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: {{ slack channel name}}
continue right side…

9. 9
4. Config rule file(1/4)
/opt/alertmanager/alert.rules
ALERT LinuxLoad1m
IF node_load1 >15
FOR 1m
ANNOTATIONS {summary="Linux Load is at 10"}
ALERT LinuxMemory
IF ((node_memory_MemTotal - (node_memory_MemFree + node_memory_Buffers +
node_memory_Cached))/node_memory_MemTotal) * 100 >85
FOR 1m
ANNOTATIONS {summary="Linux Memory Usage is at 85%"}
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server

10. 10
4. Config rule file(2/4)
/opt/alertmanager/alert.rules
ALERT LinuxDiskUsage
IF ((node_filesystem_size{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"} -
node_filesystem_avail{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"})
/node_filesystem_size{ fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"}) * 100 > 90
FOR 5m
ANNOTATIONS {summary="Linux Disk Usage over 90%"}
ALERT MySQLSlaveLag
IF mysql_slave_lag_seconds > 300
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "Slave lag is too high.", severity="warning" }
Set a alert rule file
Install & config alertmanager on monitoring server
continue next page…

11. 11
4. Config rule file(3/4)
/opt/alertmanager/alert.rules
ALERT MySQLReplicationSQLThreadStatus
IF mysql_slave_status_slave_sql_running==0
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "SQL thread stop", severity="warning"}
ALERT MySQLReplicationIOThreadStatus
IF mysql_slave_status_slave_io_running==0
FOR 1m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "IO thread stop", severity="warning"}
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server

12. 12
4. Config rule file(4/4)
/opt/alertmanager/alert.rules
ALERT MySQLstatus
IF mysql_up==0
FOR 30s
LABELS { severity = "warning" }
ANNOTATIONS { summary = "Mysql Process Down" }
continue next page…
Set a alert rule file
Install & config alertmanager on monitoring server

13. 13
5. Add rule file information on prometheus.yml
/opt/prometheus/prometheus.yml
You can add following information between global and scrape_configs section.
global:
rule_files:
- /opt/alertmanager/alert.rules
scrape_configs:
Set a alert rule file
Install & config alertmanager on monitoring server

14. 14
6. Start alertmanager and prometheus
Alertmanager start
#./alertmanager –config.file=simple.yml
Prometheus start
#./prometheus –config.file prometheus.yml –alertmanager.url=http://10.xxx.xxx.xxx:9093
start alertmanager and prometheus
Install & config alertmanager on monitoring server

15. Install & config alertmanager on monitoring server
15
7. Check status of alertmanager and prometheus
Prometheus status
http://10.xxx.xxx.xxx:9090/status

16. Install & config alertmanager on monitoring server
16
7. Check status of alertmanager and prometheus
Alertmanager status
http://10.xxx.xxx.xxx:9093/#/status

17. Auto install & config exporters using Ansible
17
1. Install Ansible
On iOS
$brew install ansible
On Linux
$yum install ansible

18. 1818
2. Consist of Directory & Files
- /etc/hosts
- ~/ansible_hosts
- ~/ansible/playbooks/prometheus_exporter.yml
- ~/ansible/playbooks/files/prometheus/node_exporter-0.12.0.linux-amd64.tar.gz
- ~/ansible/playbooks/files/prometheus/mysqld_exporter-0.8.1.linux-amd64.tar.gz
- ~/ansible/playbooks/files/prometheus/start_node_exporter.sh
- ~/ansible/playbooks/files/prometheus/start_mysqld_exporter.sh
- ~/ansible/playbooks/files/prometheus/.my.cnf
- ~/ansible/playbooks/files/prometheus/prometheus.yml.temp
Auto install & config exporters using Ansible

19. 19
/etc/hosts
~/ansible_hosts
#DB server lists
10.xxx.xxx.xxx dbhost01
10.xxx.xxx.xxx dbhost02
10.xxx.xxx.xxx dbhost03
[Group1]
dbhost01
dbhost02
dbhost03
system hosts file
ansible hosts file
You should set following information on profile.
export ANSIBLE_INVENTORY=~/ansible_hosts
Auto install & config exporters using Ansible

20. 20
~/ansible/playbooks/prometheus_exporter.yml
- name: Prometheus Install and Configuration
user : user1
hosts: Group1
sudo: yes
tasks:
- name: make directory
file: path=/opt/prometheus_exporters/ state=directory
- name: Copy tar.gz files
copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/{{ item }}
with_items:
- node_exporter-0.12.0.linux-amd64.tar.gz
- mysqld_exporter-0.8.1.linux-amd64.tar.gz
continue next page…
ansible playbook file
for install and config
prometheus exporter
Auto install & config exporters using Ansible

21. 21
~/ansible/playbooks/prometheus_exporter.yml
- name: Uncompress gz files
command: tar -xzvf /opt/prometheus_exporters/{{ item }} -C /opt/prometheus_exporters
with_items:
- node_exporter-0.12.0.linux-amd64.tar.gz
- mysqld_exporter-0.8.1.linux-amd64.tar.gz
- name: Rename node_exporter Directory name
command: mv /opt/prometheus_exporters/node_exporter-0.12.0.linux-amd64 /opt/prometheus_exporters/node_exporter
- name: Rename mysqld_exporter Directory name
command: mv /opt/prometheus_exporters/mysqld_exporter-0.8.1.linux-amd64 /opt/prometheus_exporters/mysqld_exporter
- name: change directory owner
file: path=/opt/prometheus_exporters/{{ item }} state=directory owner=root group=root mode=640
with_items:
- node_exporter
- mysqld_exporter
continue next page…
Auto install & config exporters using Ansible

22. 22
~/ansible/playbooks/prometheus_exporter.yml
- name: Copy start_mysqld_exporter.sh
copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/mysqld_exporter
with_items:
- start_mysqld_exporter.sh
- name: Copy start_node_exporter.sh
copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/node_exporter
with_items:
- start_node_exporter.sh
- name: .my.cnf coyp
template: src=files/prometheus/.my.cnf dest=/opt/prometheus_exporters/mysqld_exporter
owner=root group=root mode=640
Auto install & config exporters using Ansible
continue next page…

23. 23
~/ansible/playbooks/prometheus_exporter.yml
- name: .my.cnf coyp
template: src=files/prometheus/.my.cnf dest=/opt/prometheus_exporters/mysqld_exporter
owner=root group=root mode=640
- name: Execute node exporter
command: /bin/sh /opt/prometheus_exporters/node_exporter/start_node_exporter.sh
- name: Execute mysql exporter
command: /bin/sh /opt/prometheus_exporters/mysqld_exporter/start_mysqld_exporter.sh
- name: move new host info on prometheus server
template: src=files/prometheus/prometheus.yml.temp dest=/opt/prometheus/prometheus.yml.{{ansible_hostname}}
owner=root group=root mode=640
delegate_to: monitoring_server_hostname
- name: add new host info on prometheus.yml
shell: cat /opt/prometheus/prometheus.yml.{{ansible_hostname}} >> /opt/prometheus/prometheus.yml
delegate_to: monitoring_server_hostname
Auto install & config exporters using Ansible

24. 24
~/ansible/playbooks/files/prometheus/
node_exporter-0.12.0.linux-amd64.tar.gz
mysqld_exporter-0.8.1.linux-amd64.tar.gz
start_node_exporter.sh
start_mysqld_exporter.sh
: node_exporter install file
: mysqld_exporter install file
: node_exporter start script
nohup /opt/prometheus_exporters/node_exporter/node_exporter &
nohup /opt/prometheus_exporters/mysqld_exporter/mysqld_exporter
-config.my-cnf="/opt/prometheus_exporters/mysqld_exporter/.my.cnf" &
Auto install & config exporters using Ansible
: mysqld_exporter start script

25. 25
~/ansible/playbooks/files/prometheus/
.my.cnf
[client]
host= {{ansible_default_ipv4.address}}
user=mon_user # db user for monitoring
password=mon_user_passwd # db user password
Auto install & config exporters using Ansible
: mysqld_exporter config file
mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO ‘mon_user'@'127.0.0.1' identified by ‘mon_user_passwd';
mysql> GRANT SELECT ON performance_schema.* TO ‘mon_user'@'127.0.0.1';
You have to add db user for monitoring on db server

26. 26
~/ansible/playbooks/files/prometheus/
prometheus.yml.temp
- job_name: linux_{{ansible_hostname}}
target_groups:
- targets: ['{{ansible_default_ipv4.address}}:9100']
labels:
alias: {{ansible_hostname}}
- job_name: mysql_{{ansible_hostname}}
target_groups:
- targets: ['{{ansible_default_ipv4.address}}:9104']
labels:
alias: {{ansible_hostname}}
Auto install & config exporters using Ansible
: prometheus config information

27. Install & config grafana on monitoring server
27
1. Install Grafana
Download rpm file
http://grafana.org/download/
#rpm –ivh grafana-3.1.1-1470047149.x86_64.rpm
#/etc/init.d/grafana start
2. Grafana connect on web browser
http://10.xxx.xxx.xxx:3000/

28. 28
Make grafana dashboard
CPU Usage
Disk
Bandwidth
Disk space
MySQL
Active
Threads
Load
Average
MySQL QPS
Network
Traffic
Memory

29. Make grafana dashboard
29
1. Register Datasource

30. Make grafana dashboard
30
2. Create new dashboard and templating

31. 31
Make grafana dashboard
3. Create a templating

32. 32
Make grafana dashboard
4. Add Graph-Load average(1)

33. 33
Make grafana dashboard
4. Add Graph-Load average(2)

34. 34
Make grafana dashboard
4. Add Graph-Load average(3)

35. 35
Make grafana dashboard
4. Add Graph-Load average(4)

36. 36
Make grafana dashboard
4. Add Graph-Load average(5)

37. 37
Make grafana dashboard
4. Add Graph-Load average(6)

38. 38
Make grafana dashboard
4. Add Graph-Load average(7)

39. 39
Make grafana dashboard
5. Add Graph-CPU Usage
6. Add Graph-Disk Bandwidth

40. 40
Make grafana dashboard
7. Add Graph-Disk space
Query Detail
((node_filesystem_size{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"} -
node_filesystem_avail{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"})
/node_filesystem_size{ fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"}) * 100

41. 41
Make grafana dashboard
8. Add Graph-MySQL Active Threads
9. Add Graph-MySQL QPS

42. 42
Make grafana dashboard
10. Add Graph-Network Traffic

43. 43
Make grafana dashboard
11. Add Graph-Memory

44. 44
Make grafana dashboard
You can get more various kinds of dashboard.
https://github.com/percona/grafana-dashboards

45. Thank You