Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Monitoring MySQL at SCALE
Who We Are
Ilan Rabinovitch
Dir. Technical Community
Datadog
Ovais Tariq
Storage SRE
Uber
(formerly at Lithium & Percona)
Agenda
1. About Lithium and MySQL
2. Background: Monitoring Challenges in a Dynamic World
3. Theory: Monitoring 101
4. Pra...
About Lithium Technologies
Lithium’s platform helps brands connect,
engage and understand their customers
MySQL Architecture / Data Flow
•Multi-Tenant SaaS applications
•Typical Master-slave replication setup
•MySQL running
○ On...
Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis
DevOps Day LA
Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis
DevOps Day LA
You’re in the cloud and it's everything
you dreamed of!
Autoscaling Infinite StorageManaged
Databases
Container
Orchestrat...
Collecting data is cheap;
not having it when you
need it can be expensive
Instrument all the things!
Operational Complexity Increases with..
• Number of things to measure
• Velocity of change
How much we measure?
1 instance
• 10 metrics from CloudWatch
1 operating system (e.g., Linux)
• 100 metrics
MySQL Instance...
460
metrics per host
46,000
100
instances
•Earlier - typical Nagios and Cacti setup
•Static config and lack of context
•No correlation between alerts and
graphs
•No...
When to let a sleeping
engineer lie?
Recurse until you find root cause
• Query Time
• Queries Per Second
Data Sources
• Performance Schema
• MySQL Status Variables
• Query Time
• Queries Per Second
Sources:
• Performance Schema
• Disk Space Usage
• Threads_connected
• Threads_running
• Connection_errors_ internal
• Aborted_connects
• Connection_err...
• Configuration Change
• Code Deployment
• Service Started / Stopped
• MySQL Upgrades
• Failovers
• etc
Change in workload without an increase in
workload affected the schema ‘groupecasino’
• Workload characteristics change to...
Monitoring 101: Alerting
https://www.datadoghq.com/blog/monitoring-101-alerting/
Monitoring 101: Collecting the Right Data...
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Monitoring MySQL at scale
Upcoming SlideShare
Loading in …5
×

of

Monitoring MySQL at scale Slide 1 Monitoring MySQL at scale Slide 2 Monitoring MySQL at scale Slide 3 Monitoring MySQL at scale Slide 4 Monitoring MySQL at scale Slide 5 Monitoring MySQL at scale Slide 6 Monitoring MySQL at scale Slide 7 Monitoring MySQL at scale Slide 8 Monitoring MySQL at scale Slide 9 Monitoring MySQL at scale Slide 10 Monitoring MySQL at scale Slide 11 Monitoring MySQL at scale Slide 12 Monitoring MySQL at scale Slide 13 Monitoring MySQL at scale Slide 14 Monitoring MySQL at scale Slide 15 Monitoring MySQL at scale Slide 16 Monitoring MySQL at scale Slide 17 Monitoring MySQL at scale Slide 18 Monitoring MySQL at scale Slide 19 Monitoring MySQL at scale Slide 20 Monitoring MySQL at scale Slide 21 Monitoring MySQL at scale Slide 22 Monitoring MySQL at scale Slide 23 Monitoring MySQL at scale Slide 24 Monitoring MySQL at scale Slide 25 Monitoring MySQL at scale Slide 26 Monitoring MySQL at scale Slide 27 Monitoring MySQL at scale Slide 28 Monitoring MySQL at scale Slide 29 Monitoring MySQL at scale Slide 30 Monitoring MySQL at scale Slide 31 Monitoring MySQL at scale Slide 32 Monitoring MySQL at scale Slide 33 Monitoring MySQL at scale Slide 34 Monitoring MySQL at scale Slide 35 Monitoring MySQL at scale Slide 36 Monitoring MySQL at scale Slide 37 Monitoring MySQL at scale Slide 38 Monitoring MySQL at scale Slide 39 Monitoring MySQL at scale Slide 40 Monitoring MySQL at scale Slide 41 Monitoring MySQL at scale Slide 42 Monitoring MySQL at scale Slide 43 Monitoring MySQL at scale Slide 44 Monitoring MySQL at scale Slide 45 Monitoring MySQL at scale Slide 46 Monitoring MySQL at scale Slide 47
Upcoming SlideShare
DevOps, continuous delivery, & the new composable enterprise
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Monitoring MySQL at scale

Download to read offline

Percona Live 2016 presentation on best practices to monitor large scale MySQL deployments.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Monitoring MySQL at scale

  1. 1. Monitoring MySQL at SCALE
  2. 2. Who We Are Ilan Rabinovitch Dir. Technical Community Datadog Ovais Tariq Storage SRE Uber (formerly at Lithium & Percona)
  3. 3. Agenda 1. About Lithium and MySQL 2. Background: Monitoring Challenges in a Dynamic World 3. Theory: Monitoring 101 4. Practical: Triaging a Real Incident at Lithium
  4. 4. About Lithium Technologies Lithium’s platform helps brands connect, engage and understand their customers
  5. 5. MySQL Architecture / Data Flow •Multi-Tenant SaaS applications •Typical Master-slave replication setup •MySQL running ○ On bare metal ○ In AWS public cloud ○ In OpenStack
  6. 6. Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA
  7. 7. Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA
  8. 8. You’re in the cloud and it's everything you dreamed of! Autoscaling Infinite StorageManaged Databases Container Orchestration Private Clouds
  9. 9. Collecting data is cheap; not having it when you need it can be expensive
  10. 10. Instrument all the things!
  11. 11. Operational Complexity Increases with.. • Number of things to measure • Velocity of change
  12. 12. How much we measure? 1 instance • 10 metrics from CloudWatch 1 operating system (e.g., Linux) • 100 metrics MySQL Instance • 350~ metrics
  13. 13. 460 metrics per host 46,000 100 instances
  14. 14. •Earlier - typical Nagios and Cacti setup •Static config and lack of context •No correlation between alerts and graphs •No self-service for developers •In-house tooling has high cost
  15. 15. When to let a sleeping engineer lie?
  16. 16. Recurse until you find root cause
  17. 17. • Query Time • Queries Per Second Data Sources • Performance Schema • MySQL Status Variables
  18. 18. • Query Time • Queries Per Second Sources: • Performance Schema
  19. 19. • Disk Space Usage • Threads_connected • Threads_running • Connection_errors_ internal • Aborted_connects • Connection_errors_ max_connections Sources: ● Server Status Variables
  20. 20. • Configuration Change • Code Deployment • Service Started / Stopped • MySQL Upgrades • Failovers • etc
  21. 21. Change in workload without an increase in workload affected the schema ‘groupecasino’ • Workload characteristics change to make it more CPU bound • No increase in IO activity • Increase in number of read operations • No change in types of read operations • Similar number of range queries reading more rows
  22. 22. Monitoring 101: Alerting https://www.datadoghq.com/blog/monitoring-101-alerting/ Monitoring 101: Collecting the Right Data https://www.datadoghq.com/blog/monitoring-101-collecting-data/ Monitoring 101: Investigating performance issues https://www.datadoghq.com/blog/monitoring-101-investigation/ Monitoring MySQL Performance Metrics https://www.datadoghq.com/blog/monitoring-mysql-performance-metrics/ Collecting MySQL Metrics https://www.datadoghq.com/blog/collecting-mysql-statistics-and-metrics/
  • SrikantDalai

    Aug. 25, 2021

Percona Live 2016 presentation on best practices to monitor large scale MySQL deployments.

Views

Total views

2,063

On Slideshare

0

From embeds

0

Number of embeds

69

Actions

Downloads

18

Shares

0

Comments

0

Likes

1

×