Troubleshooting performance issues can be a bit tricky, especially when you’re given a broad statement that the database is slow. Learn to direct your attention to the correct moving pieces and fix what needs your attention. Learn how all this is done at Percona, what we monitor and track, and the tools we use.

1. © 2017 Percona1
Roma Novikov
Percona
Best Practices for MySQL
Performance & Troubleshooting
with the aid of Percona
Monitoring and Management

2. © 2017 Percona2
About me
Roma Novikov
▪Positions
• Percona - Director of Platform Engineering (since 2017)
• TemplateMonster.com (since 2002) - Web developer,
Lead/Architect, Manager,CTO
▪Interests
• web, highload, monitoring and observability.

3. © 2017 Percona3
Why does this talk exist?

4. © 2017 Percona4
Why does this talk exist?
▪Troubleshooting performance issues can be a bit tricky,
especially when you’re given a broad statement that
the database is slow.

5. © 2017 Percona5
Why does this talk exist?
▪Troubleshooting performance issues can be a bit tricky,
especially when you’re given a broad statement that
the database is slow.
▪Learn to direct your attention to the correct moving
pieces and fix what needs your attention.

6. © 2017 Percona6
Why does this talk exist?
▪Troubleshooting performance issues can be a bit tricky,
especially when you’re given a broad statement that
the database is slow.
▪Learn to direct your attention to the correct moving
pieces and fix what needs your attention.
▪Learn how all this is done at Percona, what we monitor
and track, and the tools we use.

7. © 2017 Percona7
Agenda
▪What the Problem?
▪What’s Percona Solution?
▪How to use it and solve problems
▪Typical Cases
▪Extend with your components

8. © 2017 Percona8
What the Problem?
what we should remember about it

9. © 2017 Percona9
My database is slow !?!

10. © 2017 Percona10
My database is slow !?!
▪What is ‘normal’ ? Establishing a baseline

11. © 2017 Percona11
My database is slow !?!
▪What is ‘normal’ ? Establishing a baseline
▪Are other factors impacting your database
performance?
• Hardware/virtualisation layer
• Network
• Application behaviour

12. © 2017 Percona12
My database is slow !?!
▪What is ‘normal’ ? Establishing a baseline
▪Are other factors impacting your database
performance?
• Hardware/virtualisation layer
• Network
• Application behaviour
▪But I already have “top” and “SHOW
PROCESSLIST”, isn't that sufficient?

13. © 2017 Percona13
What DOES Matter?

14. © 2017 Percona14
What DOES Matter?
Application Performance Always Matters!

15. © 2017 Percona15
What DOES Matter?
Application Performance Always Matters!
▪Take Away
• Performance Problems might not be DataBase
• Performance Solutions might not be with DataBase

16. © 2017 Percona16
Performance Stakeholders

17. © 2017 Percona17
Performance Stakeholders
▪Internal

18. © 2017 Percona18
Performance Stakeholders
▪Internal
• DEV

19. © 2017 Percona19
Performance Stakeholders
▪Internal
• DEV
• QA

20. © 2017 Percona20
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS

21. © 2017 Percona21
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External

22. © 2017 Percona22
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External
• Managers

23. © 2017 Percona23
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External
• Managers
• C-Level

24. © 2017 Percona24
What to use?

25. © 2017 Percona25
What to use - U.S.E.?
USE
▪ For every resource, check utilization, saturation, and errors.
• utilization: the average time that the resource was busy
servicing work
• saturation: the degree to which the resource has extra work
which it can't service, often queued
• errors: the count of error events
Brendan D. Gregg : http://www.brendangregg.com/usemethod.html

26. © 2017 Percona26
What to use - R.E.D.?
RED
▪ The RED Method defines the three key metrics you should
measure for every microservice in your architecture. Those
metrics are:
• (Request) Rate - the number of requests, per second, you services are serving.
• (Request) Errors - the number of failed requests per second.
• (Request) Duration - distributions of the amount of time each request takes.
Tom Wilkie
https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-ar
chitecture/

27. © 2017 Percona27
What the solution?

28. © 2017 Percona28
What we've done
▪Our own
▪On premise / host your own
▪Free and Open source

29. © 2017 Percona29
Architecture
Main Components
● Prometheus
● Grafana
● Percona Dashboards
● Consul
● Query Analytics
● CLI tool
● prometheus exporters

30. © 2017 Percona30
Technical details
Distribution methods
• Docker
• AMI
• OVA
What is supported
▪MySQL Databases (MySQL, MariaDB, Percona Server,
AWS/Aurora)
▪NoSQL Databases(MongoDB, Percona Server for
MongoDB)
▪OS (Linux,..)

31. © 2017 Percona31
Product Demo
https://pmmdemo.percona.com

32. © 2017 Percona32
Crosserver

33. © 2017 Percona33
System Overview

34. © 2017 Percona34
MySQL Overview

35. © 2017 Percona35
MySQL Overview VS Query
Analytics

36. © 2017 Percona36
Query Analytics Tools
▪Total Table
▪Query details and metrics
▪Fingerprint VS Example
▪Explain
▪Table info
• Create
• Status
• Indexes

37. © 2017 Percona37
Update with lock

38. © 2017 Percona38
InnoDB wait

39. © 2017 Percona39
What to focus on?

40. © 2017 Percona40
Performance Optimization
focused

41. © 2017 Percona41
Performance Optimization
focused
▪Making queries run faster

42. © 2017 Percona42
Performance Optimization
focused
▪Making queries run faster
▪Using less resources

43. © 2017 Percona43
Performance Optimization
focused
▪Making queries run faster
▪Using less resources
▪Scaling better

44. © 2017 Percona44
Performance Optimization
focused
▪Making queries run faster
▪Using less resources
▪Scaling better
▪Specific Application Transaction
• Find out which queries it runs
• Optimize them

45. © 2017 Percona45
Things to Consider

46. © 2017 Percona46
Things to Consider
▪Do not look at the average case only
• But avoid focusing only on outliers

47. © 2017 Percona47
Things to Consider
▪Do not look at the average case only
• But avoid focusing only on outliers
▪Look at trends over time
• Consider daily, weekly, monthly cycles

48. © 2017 Percona48
Things to Consider
▪Do not look at the average case only
• But avoid focusing only on outliers
▪Look at trends over time
• Consider daily, weekly, monthly cycles
▪Think about future performance
• Data size change? Cardinality?

49. © 2017 Percona49
Query Tips

50. © 2017 Percona50
Query Tips
▪Do less queries –latency and overhead

51. © 2017 Percona51
Query Tips
▪Do less queries –latency and overhead
▪Read less data; Modify less data

52. © 2017 Percona52
Query Tips
▪Do less queries –latency and overhead
▪Read less data; Modify less data
▪Less data processing on the fly

53. © 2017 Percona53
Query Tips
▪Do less queries –latency and overhead
▪Read less data; Modify less data
▪Less data processing on the fly
▪How much data is traversed vs sent

54. © 2017 Percona54
Query Tips
▪Do less queries –latency and overhead
▪Read less data; Modify less data
▪Less data processing on the fly
▪How much data is traversed vs sent
▪How much data is sent vs used by app

55. © 2017 Percona55
How to Extend?
Adding your App metrics

56. © 2017 Percona56
How to Extend
▪For MySQL/MongoDB
• create your dashboard (Grafana + PromQL)
▪For your app
• Create your App specific Prometheus exporter
▪ https://prometheus.io/docs/instrumenting/writing_exporters/
• Connect it to PMM (will be in 1.4 release - Oct 20)
• Create dashboard

57. © 2017 Percona57

58. © 2017 Percona58
Thanks!
QA

59. DATABASE
PERFORMANCE
MATTERS
Database Performance MattersDatabase Performance MattersDatabase Performance MattersDatabase Performance MattersDatabase Performance Matters