The document discusses open source tools for monitoring and auditing databases at scale. It describes commercial monitoring products and their limitations in scaling and functionality. It then summarizes several open source auditing options for MySQL and MongoDB databases, including plugins, log files, and network sniffing tools. These tools provide visibility into database queries and operations with varying levels of reliability and overhead. Combining tools can provide more complete auditing while reducing individual tool limitations.

1. OPEN SOURCE DATABASE MONITORING AT SCALE
#eko10

2. What is this talk about?
● Auditing tools available for databases, commercial and otherwise
● What problems they have
● How we can make them scale as much as we might need

3. About us
● Juan Berner
o @89berner / 89berner@gmail.com
o Hobbies = [‘Movies/Series','Reading','Programming']
o Mostly Blue Team
o http://secureandscalable.wordpress.com/
● Pablo Garbossa
o @pgarbossa / pablo.garbossa@gmail.com
o Fully Blue Team

4. About MercadoLibre
● Devops culture (everyone and their mothers can access the boxes)
● Different DBs technologies
● Hybrid Cloud
● Database servers > 1K && Servers > 15K
● More than 100000 qps

5. Commercial products
● Expensive
● Lots of functionalities you might not need
● Don’t scale so well
● Will make you choose what to log

6. Audit options
● Inline / TAP / sensors or agents
● Plugin based
● Sniffers
● Client loggers

7. Mysql Audit Options
● Commercial products
● Mysql General Log
● MySQL Enterprise Audit Log Plugin
● Mysql audit plugins
● Mysql sniffer

8. Mysql General Log
● Easy to activate, by default in mysql
● Can be customized by modifying the log table to a degree
● As of 5.1 can be activated on the fly
● Less freedom than audit plugins

9. MySQL Enterprise Audit
Log Plugin
● Available for Mysql Enterprise
● Uses the open MySQL Audit API
● Does not log triggers or prepared statements
● Allows asynchronous or synchronous logging

10. Mysql Audit Plugin (1)
● Works using API created by Mysql to replace the general log
● Available in Github
● Flexibility to choose objects to inspect, types of queries to log or users to
whitelist
● Similar restrictions as Mysql Enterprise Plugin

11. Mysql Audit Plugin (2)
● Steps to audit
o Download the plugin from github
o Move the library to /usr/lib/mysql/plugin/
o Enable with INSTALL PLUGIN AUDIT SONAME 'libaudit_plugin.so';
o Either send it to a file or to a socket

12. Mysql Audit Plugin (3)
● The log now looks like:
"msg-type":"activity","date":"1414531661274","thread-id":"72","query-id":"
1600563","user":"workshop","priv_user":"workshop","host":"ip-172-31-32-202.us-west-
2.compute.internal","ip":"172.31.32.202","cmd":"show_fields","query":"show_fields"

13. Mysql Audit Plugin (4)
We can parse it with logstash into:
{
….
"@timestamp": "2014-10-29T04:10:37.000Z",
"type": "mysqlplugin",
"host": "0.0.0.0",
"path": "/var/log/mysqlplugin-2014-10-29.log",
"date": "Oct 29 04:10:37",
"agent": "54.200.106.239",
"user": "workshop",
"priv_user": "workshop",
"srcip": "54.69.169.73",
"command": "show_fields","",
"query": "show_fields"
}
}

14. Mysql Audit Plugin DEMO

15. Problems with the Plugin
● Generates overhead on the host
● Can’t log all events (audit api limitations)
● Not available in sniffing only situations

16. Sniffing options
● Span port which sends you the traffic
● Sniffing and parsing from the server
● Forwarding the traffic from the agent (ie: iptables)
● Using agents to sniff traffic and forward it to repeaters (which repeat locally
the traffic with the original address)

17. Mysql Sniffer
● Client / Server architecture
● Sniffs for common queries (select/insert/update/delete)
● Beta phase
● Has to keep up with protocol changes

18. Mysql Sniffer Agent
● It will sniff traffic on Mysql port 3306 and send it elsewhere
● Small use of resources
● Must be tweaked to work in high load situations (ie: increase buffer for
packets to be processed)
./agent eth0 3306 DESTINATION 9200 1000 5000 5

19. Mysql Sniffer Repeater
● Application that listens at a tcp port for connections
● Receives packets and does a local replay of them
● Packets are seen as coming from the original client
./repeater 9200

20. Mysql Sniffer Parser
● Listens to traffic on the interface for the mysql port
● Parsers queries and keeps track of connections
● Writes output to logfile:
Wed Oct 29 00:20:24,54.69.169.73,55981,172.31.32.202,workshop,test,
select,"select * from test"

21. Mysql Sniffer DEMO

22. Mysql Sniffer Problems
● Not reliable
● Depends on the protocol not to change or something weird not to happen
● Only a limit subset of types of queries which represent most queries
● Shouldn’t be used on databases with small activity

23. You can use a combo
● Mysql sniffer to audit common queries without giving overhead to the
mysql server
● Mysql Audit Plugin to audit all other queries or specific objects with more
reliability
● Be creative

24. MongoDB
● Document oriented database
● Great scaling capabilities
● Bson Data Store
● Most popular NoSQL (according to wikipedia)

25. MongoDB Operations
● Insert: db.scores.save({a:99})
● Delete: db.scores.remove({server: 999});
● Update: db.scores.update({a: 5}, {server:999});
● Query: db.scores.find();

26. MongoDB auditing options
● Server log
● MongoDB Enterprise Auditing
● Query to the oplog
● Mongosniff

27. MongoDB Mongosniff
● Gives you detailed output of operations in MongoDB
● Does not come in the default package, you need to compile it
● Uses the mongo libraries to parse the commands
● Sample output:
111.22.33.44:6612 <<– 22.33.44.55:42947 262 bytes id:6a89eb 6982123 –
308293
reply n:4 cursorId: 0
{ _id: “db”, partitioned: false, primary: “Segmon_RS1″ }

28. MongoDB Mongosniff
(Modified)
● Some pcap tweaks to reduce dropped packets
● Minor bug fixes
● Different output format:
172.31.36.172:56228,54.68.230.224:6612,test.$cmd,,query,{ authenticate: 1, nonce:
"745ad1e4a6075a25", user: "workshop", key: "869c8d69703e2d1bb9394ddf4c116dcb" }
ntoreturn: 1 ntoskip: 0AAAAAAA

29. MongoDB Mongosniff
Wrapper
● Ruby wrapper
● Handles extra functions we would need without modifying mongosniff
● Output format:
Oct 29 03:43:11,workshop,54.68.230.224,workshop,test,172.31.36.172:56231, 54.68.230.224:6612,
test.cmd,,query,{ isMaster: 1.0, forShell: 1.0 } ntoreturn: -1 ntoskip: 0

30. MongoDB Mongosniff
Architecture

31. Mongo Sniffer DEMO

32. MongoDB Sniffer Problems
● No support for packet fragmentation
● Not 100% reliable
● Not it’s intended use

33. References
● https://github.com/89berner/MysqlAudit
● https://github.com/mcafee/mysql-audit

34. thank you!
we’re hiring ;)