Successfully reported this slideshow.
Your SlideShare is downloading. ×

Performance Analysis and Troubleshooting Methodologies for Databases

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 44 Ad

Performance Analysis and Troubleshooting Methodologies for Databases

Download to read offline

Have you heard about the USE Method (Utilization - Saturation - Errors), RED (Rate - Errors - Duration), or Golden Signals (Latency - Traffic - Errors - Saturations)?

In this presentation, we will talk briefly about these different, but similar “focuses” and discuss how we can apply them to the data infrastructure performance analysis troubleshooting, and monitoring.

We will use MySQL as an example but most of the talk will apply to other database technologies as well.

Outline to use if needed.
- Introduce the Challenge of Troubleshooting by Random Googling (1min)
- Introduce USE Method, how it applies to databases (5 min)
- Introduce RED Method, how it applies to databases (5 min)
- Introduce Golden Signals (4 min)
- Provide a High-Level Comparison of Methods as a takeaway (4 min).

Have you heard about the USE Method (Utilization - Saturation - Errors), RED (Rate - Errors - Duration), or Golden Signals (Latency - Traffic - Errors - Saturations)?

In this presentation, we will talk briefly about these different, but similar “focuses” and discuss how we can apply them to the data infrastructure performance analysis troubleshooting, and monitoring.

We will use MySQL as an example but most of the talk will apply to other database technologies as well.

Outline to use if needed.
- Introduce the Challenge of Troubleshooting by Random Googling (1min)
- Introduce USE Method, how it applies to databases (5 min)
- Introduce RED Method, how it applies to databases (5 min)
- Introduce Golden Signals (4 min)
- Provide a High-Level Comparison of Methods as a takeaway (4 min).

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Performance Analysis and Troubleshooting Methodologies for Databases (20)

Advertisement

More from ScyllaDB (20)

Advertisement

Recently uploaded (20)

Performance Analysis and Troubleshooting Methodologies for Databases

  1. 1. Brought to you by Performance Analysis and Troubleshooting Methodologies for Databases Peter Zaitsev CEO at
  2. 2. <Peter Zaitsev> CEO and Co-Founder ■ Founder, CEO | Open Source Advocate | Technologist | Board Member & Advisor ■ Expert on Open Source strategy and databases optimization. ■ Co-author of High Performance MySQL: Optimization, Backups, and Replication, one of the most popular books on MySQL performance.
  3. 3. Databases and Performance Databases are Frequent Troublemakers!
  4. 4. NON LINEAR SCALABILITY COMPLEX OFTEN POORLY UNDERSTOOD BY DEVELOPERS Why?
  5. 5. Performance Related Tasks Troubleshooting Capacity Planning Cost and Efficiency Optimization Change Management
  6. 6. Two Points of View • Application Developer • Sees Database as Blackbox • Database Operations (DBA, DBRE etc) • Sees Database as a Whitebox
  7. 7. “Developer Point of View” DATABASE AS A BLACKBOX I CAN CONNECT TO THE DATABASE I QUERY IT FROM MY APPLICATION IT RESPONDS QUICKLY AND CORRECTLY
  8. 8. Load Resource Utilization Index Usage Contention System/Hardware Problems Scaling/Capacity Planning Ops Point of View
  9. 9. Specific Methodologies
  10. 10. Most Typical Finding Solutions by Random Googling
  11. 11. Problems with Random Googling HARD TO ASSURE OUTCOME HARD TO TRAIN PEOPLE HARD TO AUTOMATE
  12. 12. There are Better Ways! USE (Utilization, Saturation, Errors) Method by Brendan Gregg RED (Rate, Errors, Duration) Method by Tom Wilkie Golden Signals (Latency, Traffic, Errors, Saturations) Method by Rob Ewaschuk
  13. 13. USE Method
  14. 14. USE Metod Basics Developed to Troubleshoot System Performance Issues Resolve 80% of Problems with 5% of Effort Operating System Specific Checklists Available
  15. 15. USE Method in one Sentence “For Every Resource check Utilization, Saturation and Errors” 15
  16. 16. USE Method Terminology Resource all physical server functional components (CPUs, disks, busses, ...) Utilization the average time that the resource was busy servicing work Saturation the degree to which the resource has extra work which it can't service, often queued Errors the count of error events
  17. 17. Resources CPUs sockets, cores, hardware threads (virtual CPUs) Memory Capacity Network Interfaces And their buffers Storage Devices I/O, Capacity Controllers Storage, Network Cards, Interconnects CPUs, Memory, I/O
  18. 18. USE Method and Software Same Basic Resources Apply Additional Software Resources Apply Mutex Locks File Descriptors Connections
  19. 19. USE Method Benefits Proven Track Record Broad Applicability Detailed Checklists Available
  20. 20. USE Method Drawbacks REQUIRES A GOOD UNDERSTANDING OF SYSTEM ARCHITECTURE REQUIRED ACCESS TO LOW LEVEL RESOURCE MONITORING HARD TO APPLY IN “AS-A-SERVICE” ENVIRONMENTS
  21. 21. USE Method and Linux 21 http://www.brendangregg.com/USEmethod/use-linux.html
  22. 22. Meet Percona Monitoring and Management (PMM) • Purpose Build Platform for Open Source Database Troubleshooting and Performance Optimization • 100% Free and Open Source • Easy to Setup • Based on Leading technologies in Open Source Observability Space • http://per.co.na/PMM
  23. 23. CPU Saturation
  24. 24. Memory Usage
  25. 25. Disk Saturation
  26. 26. Disk Saturation through Latency Changes
  27. 27. MySQL Utilization
  28. 28. MySQL Saturation
  29. 29. RED Method
  30. 30. RED Method Focus Microservices “Cattle not Pets” Mapping to Resources can be fluid As-a-Service Model – Internal details may be unknown
  31. 31. RED Method For every Service Request check these are within Service Level Objective (SLO) • Rate • Error (Rate) • Duration (Distribution)
  32. 32. Red Method for Databases Looking at Service Level Looking at Individual Database Server Can be applied to Specific Applications/Users Can be applied to Individual Queries/Transactions
  33. 33. RED Method Benefits Easily Maps to what Developers Care About 1 Does not require as deep understanding of Architecture 2 Does not need access to low level resource monitoring 3
  34. 34. RED Method Drawbacks Tools and checklists are less developed More focused answering WHAT is wrong rather than WHY
  35. 35. Different Views
  36. 36. Custom RED Method Dashboard https://grafana.com/grafana/dashboards/12470
  37. 37. Four Golden Signals
  38. 38. Focus From “Monitoring Distributed Systems” Chapter in SRE Book Can be used for Alerting, Troubleshooting, Trend Analyses
  39. 39. Four Golden Signals Distribution not just Average; Latency for Successful requests vs Errors Latency How much Demand is being placed on the System Traffic Error Codes are Easy; Bad Content is hard Errors How Full your system “capacity”. Forecast when Possible. Saturation
  40. 40. For Golden Signals And Databases Latency Query Response Time by Database Instance and Query Type Traffic Number of Queries of Specific Type Served Errors Connection Failures and Query Error Codes Saturation How much Load Database Can take vs Max ? Also Connections, Disk Space etc
  41. 41. Golden Signal Benefits Methodology now well tested by SREs Good Resource Book, Many Resources Available
  42. 42. Golden Signals Drawbacks FRAMEWORK FOR MONITORING FIRST OF ALL NO SPECIFIC TROUBLESHOOTING CHECKLISTS AVAILABLE
  43. 43. Take Aways ■ Multiple Methods are Available ■ Pick what Makes sense for your circumstances ■ Or invent your own Method combining those and others ■ Just do not settle for Random Googling
  44. 44. Brought to you by Peter Zaitsev https://www.linkedin.com/in/peterzaitsev/ @peterzaitsev

×