La majeure partie du SI d'OVH repose sur des bases de données relationnelles (PostgreSQL, MySQL, MariaDB). En termes de volumétrie cela représente 400 bases pesants plus de 20To de données réparties sur 60 clusters dans deux zones géographiques le tout propulsant 3000 applications. Comment tout voir dans notre parc ? Mieux encore, comment faire pour que tout le monde puisse suivre l'activité de sa base de données ? C'est le challenge que nous nous sommes fixés, un an après nous pouvons partager notre expérience. Et si l'observability n'était pas juste un buzzword, mais avait un réel impact sur la production ?

1. 28 Mars 2019 - Bordeaux

2. Improve your SQL workload
with observability
WILFRIED ROSET
Team leader Core database @ OVH

3. AGENDA
• Why we need to improve our SQL workload?
• How to be efficient?
• Reporting
• Logs
• Metrics
• Take out after 1 year

4. Everyone use SQL
Directly or Indirectly:
• CEO/CTO/CXO
• DEV/DEVOPS/SRE
• Support & Run

5. Is this happening to you?
"The website is slow
because of the DB!"
"The API is slow
because of the DB!"
"We need a bigger
DB cluster, ASAP!"

6. Our infra
Metrics Logs
Grafana Graylog
Or on-premises
servers,
other cloud providers…
DB DB DB
WWW
Customers
OVH DB team

7. Bad Monito: the ostrich

8. Ugly Monito: unexploitable

9. Good Monitoring!

10. Our internal DB perimeter
• Total Volumetry: 20TB
• # Clusters: ~60
• # Applications: ~3k
• # Users: ~700
• # Databases: ~400
• 24 hours a day
• 2 DevOps & DBAs

11. How to isolate A SLOW query
SELECT username
FROM customer
WHERE username LIKE '_wilfried%';

12. Internal DBs Infra
Master
ReplicaBackup
Prod
Dev Replica
Dev
Write
Read
Replication
Read/Write
24-
hour
restore
point
Archive

13. Old request process
Team DB
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

14. Easier, Better, Faster, Stronger
• Self-service, for complete autonomy
• Meaningful KPIs
• Capacity planning
• Ownership and proactivity

15. Prerequisites
As-a-service
• OVH Logs
• OVH Metrics
• UI
Open source
• Data collection
• Reporting

16. First things first
Observability is
not about how
to collect data but
what to do with it

17. DBMS Configuration
PostgreSQL
• Everything happens in postgresql.conf
• Format log to produce report
• log_line_prefix = '%t [%p]: [%l-1] db=%d,user=%u,app=%a,client=%h '
• Log slow queries
• log_min_duration_statement = '1000'

18. DBMS Configuration
MySQL / MariaDB
• Everything happens in my.cnf
• Log slow queries
• slow_query_log = 1
• slow_query_log_file = /var/log/mysql/slow.log

19. Give meaning to logs
• Pgbadger and pt-query-digest
• Process log files and produce fully readable reports
website
PostgreSQL MySQL

20. Monitoring: step 1/4
Team DB
Reporting
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

21. Output example

22. Logs, logs, logs
Logs are useful only if they are usable
• <3 grep, less, …
• Avoid ssh

23. Logs pipeline
SQL Servers Logstash Clusters ElasticSearch indexes

24. Let’s grok
# User@Host: app_user @ [1.2.3.4]
^# User@Host:
%{WORD:query_user}s*@s*[%{IP:query_ip}?].*
{
"query_user": "app_user",
"query_ip": "1.2.3.4"
}

25. Sending Data
Use whatever is Efficient && Easy to setup for your case
• Filebeat
• Syslog

26. Filebeat
# /etc/filebeat /filebeat.yml
output.logstash:
hosts: ["graX.logs.ovh.com:6514"]
ssl.certificate_authorities: ["/etc/filebeat/cert.pem"]
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/postgresql/postgresql.log

27. Syslog
destination d_pg_ldp {
tcp( "graX.logs.ovh.com",
port(6514),
ts_format("iso"),
keep-alive(yes),
so_keepalive(yes),
log-fifo-size(10000),
);
};

28. Monitoring: step 2/4
Team DB
Reporting
Logs
Syslog/Filebeat
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

29. Graylog

30. Metrics
Logs are useful when you can use them
But we need wider view …

31. It’s not about how
Metrics is about
which tools to use
But more about
what to do with your data

32. Collect your metrics
There are several options for doing this:
• Homemade  please don’t do that
• Collectl, Collectd, Statsd...
• Telegraf, Prometheus…

33. Telegraf’s plug-ins
Plug-ins:
• System-related
• cpu, disk, diskio, system, network ...
• Dbms
• PostgreSQL, MySQL
• Homemade
• Exec
There are a lot more plug-ins in telegraf

34. Let’s push some metrics
# /etc/telegraf/telegraf.conf
[agent]
interval = "30s"
flush_interval = "30s"
[[outputs.influxdb]]
urls = ["https://influxdb.graXXX.metrics.ovh.net"]
timeout = "15s"
username = "telegraf"
password = "write.token.from.metrics.manager"

35. Monitoring: step 3/4
Team DB
Reporting
Logs
Metrics
Syslog/Logstash
Telegraf "Why is my DB
slow?"
"Can I have an
extract from my
logs?"

36. System dashboard

37. I/O dashboard

38. DBMS DASHBOARD

39. Grafana is beautiful!

40. Monitoring: step 4/4
Reporting
LogsSyslog/Logstash
Telegraf
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"
Metrics

41. Slow queries count/cumul. time

42. Query types and slowness

43. Spot bugs

44. Spot bug fixes

45. Wake up, it’s the best part

46. 80/20
• Don’t try to fix the world…
• Focus where it matters

47. Talk…
Hey guys, I’ve open 60 JIRA
for your databases slow
queries and errors.
XOXO <3
Don’t do
that!

48. Talk, again…
Guys, come on! What the F…
Don’t do
that!

49. Always (louder) talk…
That’s it guys. I’m done with
you. No more prod for you
until you fixe everything
Don’t do
that!

50. … But do it gently
Ok guys, my bad. Let’s try
again.

51. Gamification
Make it fun
Make it interesting
Turn it to a game
Each game needs a ….

52. Leader board
Hello,
You will find information which can help you identify your queries in our welcome guide:
< insert documentation link >
Tldr:
• Database 94 did great because of blablabla
• Database 91 is our #1 producer of slow queries
• Blablabla
…

53. Leader board
Please find below last week sql report.
MySQL slow queries/user
PostgreSQL slow queries/database
PostgreSQL invalid syntax/database
…

54. What we’ve learned
Choose the right tools for the job
• Grafana for dashboard (mix and match sources)
• Graylog for search
An infrastructure can be set up in less than a week
1 year down the road, we can expect…
• A widely-adopted gamechanger
• Inspire others to do the same kind of report <3
• 4 times less slow queries
KPIs

55. Remember This SLOW query
SELECT username
FROM customer
WHERE username LIKE '_wilfried%';

56. A Few words about our offerS
Right now:
- OVH Cloud Databases: managed DB, up to 4GB RAM, no
cluster
- … New Lab starting! For your high-end needs.
• Bare metal performance
• Clustered (minimum four hosts)
• Horizontal scaling
• Include Logs and Metrics
• Starting with PostgreSQL
OVHLabs 

57. PS: We are hiring !
• Opensource database expert
• Site Reliability Engineers (Private cloud, Openstack, DNS, Deploy,
Observability)
• Software engineers (containers, baremetal, webhosting)
• Back-end developers (go, python)
• Engineering manager webhosting
• … And a lot more

58. Questions ?