This document discusses setting up MySQL auditing using the Percona Audit Plugin and ELK (Elasticsearch, Logstash, Kibana) stack to retrieve and analyze MySQL logs. Key steps include installing the Percona Audit Plugin on MySQL servers, configuring it to log to syslog, installing and configuring rsyslog/syslog-ng on database and ELK servers to forward logs, and installing and configuring the ELK stack including Elasticsearch, Logstash, and Kibana to index and visualize the logs. Examples are provided of creating searches, graphs, and dashboards in Kibana for analyzing the MySQL audit logs.

1. MySQL Audit
using percona audit plugin & ELK

2. About me
dba.kim@gmail.com

3. Object
3
We need
- log to MySQL connection information
- log to MySQL query log
- retrieve these logs using various search conditions
- download search results

4. Architecture
Percona Audit
Plugin
4
DB Servers ELK Server
logstash elasticsearch

5. Audit Plugin
Which audit plugin will we use? 5
Percona Audit Plugin MariaDB Audit Plugin
- support various log format(csv, json, xml)
- most of parameter can’t modify online
- support various filtering condition
(user, query type : ddl, dml, dcl)
- most of parameters can modify online
- stability issue(db crash)
https://mariadb.atlassian.net/browse/MDEV-5145
http://mysql.az/audit-mysql-no-just-crash-it/
- easy install & config
- can log at extra log file or syslog file
We chose Percona Audit Plugin

6. 6
Percona Audit Plugin Install & Config
6
mysql>install plugin audit_log soname ‘audit_log.so’;
Ref URL : https://www.percona.com/doc/percona-server/5.5/management/audit_log_plugin.html
$/etc/init.d/mysql restart
$vi my.cnf
# Server Audit
audit_log_format = JSON
audit_log_policy = ALL
audit_log_syslog_facility = LOG_LOCAL1
audit_log_handler = SYSLOG
1. Percona Audit Plugin Install (over MySQL Ver 5.5.39) – on DB servers
2. Parameter configuration
3. MySQL restart

7. rsyslog config
7
1. rsyslog.conf – on DB servers
$ vi /etc/rsyslog.conf
# mysql logging
local1.* @10.xxx.xxx.xxx # ELK server ip
2. rsyslog restart
$ /etc/init.d/rsyslog restart
3. confirm log messages on syslog file
$cat messages
Jan 29 15:08:30 testdbsvr01 percona-audit: {"audit_record":{"name":"Query","record":"3683778651_1970-01-
01T00:00:00","timestamp":"2016-01-29T06:08:30
UTC","command_class":"select","connection_id":"455338789","status":0,"sqltext":"SELECT * FROM test_table
WHERE status='Done'","user":"app[app] @ [10.xxx.xxx.xxx]","host":"","os_user":"","ip":"10.xxx.xxx.xxx"}}

8. 8
rsyslog-ng config
1. rsyslog.conf – on ELK servers
$ vi /etc/syslog-ng/syslog-ng.conf
source s_sys {
file ("/proc/kmsg" program_override("kernel: "));
unix-stream ("/dev/log");
internal();
udp(ip(0.0.0.0) port(514)); ## uncomment this line
};
destination d_myaudit { file("/var/log/myaudit/myaudit.${HOST}.${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) dir-perm(0755) perm(0644) ); };
filter f_myaudit { facility(local1); };
log { source(s_sys); filter(f_myaudit); destination(d_myaudit); };
2. syslog-ng restart
$ /etc/init.d/syslog-ng restart

9. Install ELK
Elasticsearch 9
1. elasticsearch install – on ELK server
$ yum install elasticsearch
2. configuration
$ vi /etc/elasticsearch/elasticsearch.yml
cluster.name : my_cluster # cluster name
node.name : my_node01 # node name
network.host : 10.xxx.xxx.xxx # server’s ip
You can download from
https://www.elastic.co/downloads/elasticsearch.
It needs a recent version of java before install elasticsearch.

10. Install ELK
Elasticsearch 10
3. start elasticsearch
$/etc/init.d/elasticsearch start
4. Head plugin install
$cd /usr/share/elasticsearch/bin
$./plugin install mobz/elasticsearch-head
5. Check plugin install
http://10.xxx.xxx.xxx:9200/_plugin/head/

11. 1111
Install ELK
Logstash 11
1. Install Logstash – on ELK server
$ rpm –ivh logstash-2.1.1-1.noarch.rpm
2-1. Configure ( input plugin )
$ sudo vi /etc/logstash/conf.d/01-myaudit-input.conf
input {
file {
type => "myaudit"
path => ["/var/log/myaudit/*.log"]
start_position => "beginning"
codec => 'json'
}
}
You can download from
https://www.elastic.co/downloads/logstash

12. 12
Install ELK
Logstash
2-2. Configure ( filter plugin )
$ sudo vi /etc/logstash/conf.d/10-myaudit.conf
filter {
grok {
match => { "message" =>
"%{SYSLOGTIMESTAMP:sys_timestamp}%{SPACE}%{HOSTNAME:host_name}
%{SPACE} percona-audit: %{GREEDYDATA:json_data}"}
}
json {
source => "json_data"
}
}

13. 13
Install ELK
Logstash
2-3. Configure ( output plugin )
$ sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => "10.xxx.xxx.xxx"
}
}
3. Start logstash
$ sudo /etc/init.d/logstash start

14. 14
Install ELK
Kibana 14
1. Kibana install – on ELK server
$ tar –xvf kibana-4.3.1-linux-x64.tar.gz
2. configuration
$ vi ./config/kibana.yml
host: “10.xxx.xxx.xxx” # kibana server ip
elasticsearch_url: “http://10.xxx.xxx.xxx:9200” # elasticsearch server ip
4. Check
You can download from
https://www.elastic.co/downloads/kibana
http://10.xxx.xxx.xxx:5601
3. Start Kibana
$ ./bin/kibana
It needs a space between “host:” and IP.

15. 15
Install ELK
Kibana
5. Config an index pattern

16. 16
Make Kibana Dashboard
http://10.xxx.xxx.xxx:5601

17. Search condition create(1/3) 17
Make Kibana Dashboard
1
2

18. Search condition create(2/3) 18
Make Kibana Dashboard
3
4

19. 19
Search condition create(3/3)
Make Kibana Dashboard
5
6

20. 20
1
Make Graph: line chart(1/6)
Make Kibana Dashboard

21. 21
2
Make Kibana Dashboard
Make Graph: line chart(2/6)

22. 22
3
4
Make Kibana Dashboard
Make Graph: line chart(3/6)

23. 23
5
6
Make Kibana Dashboard
Make Graph: line chart(4/6)

24. 24
7
8
Make Kibana Dashboard
Make Graph: line chart(5/6)

25. 25
9
Make Kibana Dashboard
Make Graph: line chart(6/6)

26. 26
Make Kibana Dashboard
Make Graph: Pie chart(1/4)
1

27. 27
Make Kibana Dashboard
Make Graph: Pie chart(2/4)
2

28. 28
Make Kibana Dashboard
Make Graph: Pie chart(3/4)
3
4

29. 29
Make Kibana Dashboard
Make Graph: Pie chart(4/4)
5

30. 30
Make Kibana Dashboard
Markdown widget(1/3)
1

31. 31
Make Kibana Dashboard
Markdown widget(2/3)
1
###Menu: // label, the number of “#” determines a size of character.
[Main]: //label
(/#dashboard/Main): // link dashboard. “Main” is name of dashboard.
You have to make dashboard before markdown widget.
2

32. 32
Make Kibana Dashboard
Markdown widget(3/3)
3

33. 33
Make Dashboard
Make Kibana Dashboard
1

34. 34
2
Make Dashboard
Make Kibana Dashboard

35. 35
3
4
Make Dashboard
Make Kibana Dashboard

36. Dashboard-sample
36
Markdown widget
Line chart
Search condition

37. Dashboard-sample
37
Markdown widget
Pie chart

38. Elasticsearch Plugin – Head(1/2)
38
http://10.xxx.xxx.xxx:9200/_plugin/head/
1
2
3
4
search query

39. Elasticsearch Plugin – Head(2/2)
39
{
"_source": [
"@timestamp",
"host_name", "audit_record.user", "audit_record.sqltext", "audit_record.name" ],
"sort": { "@timestamp": "asc" },
"size": 100,
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "2016-01-16T00:00:00.000",
"lte": "2016-01-16T02:59:59.999" }
} },
{"term": { "host_name": “host_name123" } },
{"term": { "audit_record.name": "query"} }
] } }}
Search condition of string type uses a lower character
only.
ex) “audit_record.name”:”Query” (x)
“audit_record.name”:”query” (o)

40. Thank You