SlideShare a Scribd company logo
1 of 35
Download to read offline
Copyright 2016 Severalnines AB
1
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the Questions
section of this application or via the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com during
or after the webinar.
Copyright 2016 Severalnines AB
2
About Severalnines and ClusterControl
Copyright 2016 Severalnines AB
3
What we do
Manage Scale
MonitorDeploy
Copyright 2016 Severalnines AB
4
ClusterControl Automation & Management
! Provisioning
! Deploy a cluster in minutes
! On-premises or in the cloud (AWS)
! Monitoring
! Systems view
! 1sec resolution
! DB / OS stats & performance advisors
! Configurable dashboards
! Query Analyzer
! Real-time / historical
! Management
! Multi cluster/data-center
! Automate repair/recovery
! Database upgrades
! Backups
! Configuration management
! Cloning
! One-click scaling
Copyright 2016 Severalnines AB
5
Supported Databases
! MySQL (Oracle/Percona/
MariaDB)
! MySQL Galera Cluster
! MySQL Cluster (NDB)
! MariaDB Cluster
! MySQL Replication 5.6
! Standalone MySQL/MariaDB
! Postgres 9.4
! Percona TokuMX Cluster
! MongoDB Sharded Cluster
! MongoDB Replica Set
Copyright 2016 Severalnines AB
6
Customers
Copyright 2016 Severalnines AB
MySQL Query Tuning - Process & Tools
August 30, 2016
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
7
Copyright 2016 Severalnines AB
8
Agenda
! Query tuning process
! Build
! Collect
! Analyze
! Tune
! Test
! Tools
! tcpdump
! pt-query-digest
Copyright 2016 Severalnines AB
9
The process
Copyright 2016 Severalnines AB
10
The process
! A short guide to query performance review:
! Build test environment
! Collect your data
! Process your data
! Analyze your data
! Tune SQL and schemas
! Test changes
! Apply changes on production systems
Copyright 2016 Severalnines AB
11
The process - build test environment
! You need a deterministic, test environment to make sure you can measure the impact of the
changes
! Environment should mirror production as close as possible, to make it more relevant
! Hardware makes a difference, especially I/O - you’ll see different performance if your I/O
subsystem is slow
! Dataset makes even more significant difference - you should test your new queries against
production dataset
! If you need to obfuscate your data, your results may be different than on production systems
Copyright 2016 Severalnines AB
12
The process - build test environment
! The best way to build a test system is to grab a backup of your production systems
! Restore it on a host which matches hardware configuration of the production
! You are all set
! Ideally, you have such QA/staging/dev environment up and running all the time and use it to
test your new code and SQL
! If not, build it at least couple times per year to run such a review
! If you are using ClusterControl and Galera Cluster, take advantage of the cloning feature
Copyright 2016 Severalnines AB
13
The process - collect your data
! Native method - slow query log
! Collects the most important data about the query:
Copyright 2016 Severalnines AB
14
The process - collect your data
! It can collect even more data on Percona Server and MariaDB:
! Even more data with log_slow_verbosity='full,profiling_use_getrusage,profiling'
Copyright 2016 Severalnines AB
15
The process - collect your data
! Another method - tcpdump
! Capture TCP traffic containing MySQL queries
! Gives limited information about the query:
! Query execution time
! Query size
! Main advantage - not necessary to execute it on the MySQL host - this reduces the impact
! Proxy
! Application host
Copyright 2016 Severalnines AB
16
The process - collect your data
Copyright 2016 Severalnines AB
17
The process - collect your data
Copyright 2016 Severalnines AB
18
The process - analyze your data
! Thousands of lines in the log - can’t parse it manually
! You can build your own tool to aggregate data from slow query log
! Or you can use industry standard - pt-query-digest:
! wget http://percona.com/get/pt-query-digest && chmod u+x ./pt-query-digest
! Run pt-query-digest against slow query log or tcpdump
! But not on the database host - it can be CPU and memory intensive
! It’ll provide you with a summary of the traffic along with detailed information about each and
every query
Copyright 2016 Severalnines AB
19
The process - tune SQL and schemas
! Use EXPLAIN to check the query execution plan
! Rewrite queries to your liking
! Test new indexes, remove obsolete ones
! Make sure the new query execution plan works better
! Leverage Performance Schema for deep dive into where query execution time goes
Copyright 2016 Severalnines AB
20
The process - apply changes on production systems
! Make sure you know what impact your
changes will have
! If you run detailed tests in the test
environment, you should be good
! Execute your changes
! Directly
! Using pt-online-schema-change
! Make sure you monitor the system after the
change
! Have a rollback plan
! A list of ALTERs which will revert your
changes can be really handy when
things go awry
! Make sure you monitor how things unravel
for at least couple of days - some processes
may be executed on a weekly basis (or
even less often)
Copyright 2016 Severalnines AB
21
The tools
Copyright 2016 Severalnines AB
22
The tools - tcpdump
! Collects a snapshot of the TCP traffic
! When you capture MySQL traffic, it contains
enough data to calculate:
! Query execution time
! Query size
! Not much of details but enough to pinpoint
slow queries
! You can always check details later, via
EXPLAIN
! tcpdump -s 65535 -x -nn -q -tttt -i any port
3306 > mysql.tcp.txt
! Run it on the MySQL host
! Run it on the application server
! Run it on the proxy node
! Great flexibility - you can easily avoid most
of the overhead caused by the tool
Copyright 2016 Severalnines AB
23
The tools - pt-query-digest
! Amazing tool which is designed to
aggregate data located in slow query log
! It can work with (among others) slow log
and tcpdump output
! Presents the data in a very nice way
! Summary of all of the queries
! Each query presented in a detailed way
(if there’s enough data for it, that is)
! EXPLAIN ready for copy&paste
! Report can be modified to user’s liking
! Sort queries by numerous attributes
! Aggregate queries differently
! Ability to store data in the database helps
to build a solid review process
! You can also compare old query
performance data with current one - to
confirm performance has not changed
Copyright 2016 Severalnines AB
24
The tools - pt-query-digest
! pt-query-digest --limit=100% /var/lib/mysql/slow.log > ptqd1.out
! Report starts with a summary:
! How many queries?
! How many distinct queries?
Copyright 2016 Severalnines AB
25
The tools - pt-query-digest
! Large amount of summarized data
! Helps to understand 

query patterns
! Helps to understand

how workload has 

changed
Copyright 2016 Severalnines AB
26
The tools - pt-query-digest
! Queries in the report are sorted - by default, by total query execution time.
! Each query has a Query ID assigned - a hash of the query digest
Copyright 2016 Severalnines AB
27
The tools - pt-query-digest
! Extensive information about every query. Extra data for Percona Server and MariaDB.
Copyright 2016 Severalnines AB
28
The tools - pt-query-digest
! Detailed InnoDB stats:
! All the data: I/O operations, lock waits, pages accessed
Copyright 2016 Severalnines AB
29
The tools - pt-query-digest
! Query histogram
! EXPLAIN
! SHOW commands
Copyright 2016 Severalnines AB
30
The tools - pt-query-digest
! Data from tcpdump can’t be 

as detailed as from slow log
! It still can be used to generate 

important statistics
! You can always analyze such query

manually later
Copyright 2016 Severalnines AB
31
The tools - ClusterControl Query Monitor
! ClusterControl provides a continuous view into query performance
! Query Monitor presents query performance data based on Performance Schema (preferred) or
slow query log (if Performance Schema is not available)
! May not be suitable for a “grand” review, but will help significantly on a day-to-day basis
! Query Histogram section lists queries with variable execution time - something definitely worth
looking into
Copyright 2016 Severalnines AB
32
The tools - ClusterControl Query Monitor
Copyright 2016 Severalnines AB
33
The tools - ClusterControl Query Monitor
Copyright 2016 Severalnines AB
34
The tools - ClusterControl Query Monitor
Copyright 2016 Severalnines AB
35
Thank You!
! Blog posts covering query tuning process:
! http://severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process
! http://severalnines.com/blog/become-mysql-dba-blog-series-analyzing-your-sql-workload-using-pt-query-digest
! http://severalnines.com/blog/become-mysql-dba-blog-series-deep-dive-sql-workload-analysis-using-pt-query-digest
! Register for the next part of our trilogy:
! http://severalnines.com/upcoming-webinars
! Install ClusterControl:
! http://severalnines.com/getting-started
! Contact: jj@severalnines.com

More Related Content

What's hot

phoenix-on-calcite-nyc-meetup
phoenix-on-calcite-nyc-meetupphoenix-on-calcite-nyc-meetup
phoenix-on-calcite-nyc-meetup
Maryann Xue
 

What's hot (20)

Optimizing Your Amazon Redshift Cluster for Peak Performance - AWS Summit Syd...
Optimizing Your Amazon Redshift Cluster for Peak Performance - AWS Summit Syd...Optimizing Your Amazon Redshift Cluster for Peak Performance - AWS Summit Syd...
Optimizing Your Amazon Redshift Cluster for Peak Performance - AWS Summit Syd...
 
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache CalciteCost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
 
Apache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAse
 
Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14
 
Apache Calcite overview
Apache Calcite overviewApache Calcite overview
Apache Calcite overview
 
Apache Phoenix + Apache HBase
Apache Phoenix + Apache HBaseApache Phoenix + Apache HBase
Apache Phoenix + Apache HBase
 
Strongly Consistent Global Indexes for Apache Phoenix
Strongly Consistent Global Indexes for Apache PhoenixStrongly Consistent Global Indexes for Apache Phoenix
Strongly Consistent Global Indexes for Apache Phoenix
 
Loading Data into Redshift
Loading Data into RedshiftLoading Data into Redshift
Loading Data into Redshift
 
hive HBase Metastore - Improving Hive with a Big Data Metadata Storage
hive HBase Metastore - Improving Hive with a Big Data Metadata Storagehive HBase Metastore - Improving Hive with a Big Data Metadata Storage
hive HBase Metastore - Improving Hive with a Big Data Metadata Storage
 
Loading Data into Redshift: Data Analytics Week SF
Loading Data into Redshift: Data Analytics Week SFLoading Data into Redshift: Data Analytics Week SF
Loading Data into Redshift: Data Analytics Week SF
 
phoenix-on-calcite-nyc-meetup
phoenix-on-calcite-nyc-meetupphoenix-on-calcite-nyc-meetup
phoenix-on-calcite-nyc-meetup
 
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data WarehouseApache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
 
DBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should KnowDBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should Know
 
Apache hive
Apache hiveApache hive
Apache hive
 
Calcite meetup-2016-04-20
Calcite meetup-2016-04-20Calcite meetup-2016-04-20
Calcite meetup-2016-04-20
 
Cost-based Query Optimization
Cost-based Query Optimization Cost-based Query Optimization
Cost-based Query Optimization
 
EMR and DynamoDB
EMR and DynamoDBEMR and DynamoDB
EMR and DynamoDB
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
 
What's New in Apache Spark 3.0 !!
What's New in Apache Spark 3.0 !!What's New in Apache Spark 3.0 !!
What's New in Apache Spark 3.0 !!
 
Apache Phoenix Query Server
Apache Phoenix Query ServerApache Phoenix Query Server
Apache Phoenix Query Server
 

Similar to Webinar replay: MySQL Query Tuning Trilogy: Query tuning process and tools

Presentation mongo db munich
Presentation mongo db munichPresentation mongo db munich
Presentation mongo db munich
MongoDB
 
Step by-step-lsmw-tutorial-101208040548-phpapp02
Step by-step-lsmw-tutorial-101208040548-phpapp02Step by-step-lsmw-tutorial-101208040548-phpapp02
Step by-step-lsmw-tutorial-101208040548-phpapp02
johnbryan26
 

Similar to Webinar replay: MySQL Query Tuning Trilogy: Query tuning process and tools (20)

A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
 
Webinar slides: Top 9 Tips for building a stable MySQL Replication environment
Webinar slides: Top 9 Tips for building a stable MySQL Replication environmentWebinar slides: Top 9 Tips for building a stable MySQL Replication environment
Webinar slides: Top 9 Tips for building a stable MySQL Replication environment
 
Scaling up Near Real-time Analytics @Uber &LinkedIn
Scaling up Near Real-time Analytics @Uber &LinkedInScaling up Near Real-time Analytics @Uber &LinkedIn
Scaling up Near Real-time Analytics @Uber &LinkedIn
 
Agentless System Crawler - InterConnect 2016
Agentless System Crawler - InterConnect 2016Agentless System Crawler - InterConnect 2016
Agentless System Crawler - InterConnect 2016
 
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
 
Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...
Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...
Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
 
Headaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsHeadaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous Applications
 
Presentation mongo db munich
Presentation mongo db munichPresentation mongo db munich
Presentation mongo db munich
 
Maximizing Database Tuning in SAP SQL Anywhere
Maximizing Database Tuning in SAP SQL AnywhereMaximizing Database Tuning in SAP SQL Anywhere
Maximizing Database Tuning in SAP SQL Anywhere
 
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
 
Thing you didn't know you could do in Spark
Thing you didn't know you could do in SparkThing you didn't know you could do in Spark
Thing you didn't know you could do in Spark
 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
 
Apache Spark Streaming -Real time web server log analytics
Apache Spark Streaming -Real time web server log analyticsApache Spark Streaming -Real time web server log analytics
Apache Spark Streaming -Real time web server log analytics
 
Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019
Troubleshooting Tips and Tricks for Database 19c - EMEA Tour  Oct 2019Troubleshooting Tips and Tricks for Database 19c - EMEA Tour  Oct 2019
Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019
 
Resilient Predictive Data Pipelines (QCon London 2016)
Resilient Predictive Data Pipelines (QCon London 2016)Resilient Predictive Data Pipelines (QCon London 2016)
Resilient Predictive Data Pipelines (QCon London 2016)
 
SAP
SAPSAP
SAP
 
Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016
 
Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016
 
Step by-step-lsmw-tutorial-101208040548-phpapp02
Step by-step-lsmw-tutorial-101208040548-phpapp02Step by-step-lsmw-tutorial-101208040548-phpapp02
Step by-step-lsmw-tutorial-101208040548-phpapp02
 

More from Severalnines

Webinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDBWebinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDB
Severalnines
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Severalnines
 
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Severalnines
 
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Severalnines
 
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Severalnines
 
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLWebinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
Severalnines
 
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBWebinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Severalnines
 
Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?
Severalnines
 
Webinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High AvailabilityWebinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High Availability
Severalnines
 

More from Severalnines (20)

Cloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaSCloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaS
 
Tips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudTips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloud
 
Working with the Moodle Database: The Basics
Working with the Moodle Database: The BasicsWorking with the Moodle Database: The Basics
Working with the Moodle Database: The Basics
 
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDBSysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
 
Webinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDBWebinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDB
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
 
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
 
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
 
Disaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDBDisaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDB
 
MariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseMariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash Course
 
Performance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDBPerformance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDB
 
Advanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona ServerAdvanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona Server
 
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket KnifePolyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
 
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
 
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLWebinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
 
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance TuningWebinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
 
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBWebinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
 
Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?
 
Webinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High AvailabilityWebinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High Availability
 

Recently uploaded

Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
ChloeMeadows1
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
lolsDocherty
 

Recently uploaded (16)

Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
 
Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirts
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
Pvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdf
 
The Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyThe Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case Study
 
Topology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdfTopology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdf
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdf
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirts
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's Guide
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdf
 
How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?
 

Webinar replay: MySQL Query Tuning Trilogy: Query tuning process and tools

  • 1. Copyright 2016 Severalnines AB 1 Your host & some logistics I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: jj@severalnines.com during or after the webinar.
  • 2. Copyright 2016 Severalnines AB 2 About Severalnines and ClusterControl
  • 3. Copyright 2016 Severalnines AB 3 What we do Manage Scale MonitorDeploy
  • 4. Copyright 2016 Severalnines AB 4 ClusterControl Automation & Management ! Provisioning ! Deploy a cluster in minutes ! On-premises or in the cloud (AWS) ! Monitoring ! Systems view ! 1sec resolution ! DB / OS stats & performance advisors ! Configurable dashboards ! Query Analyzer ! Real-time / historical ! Management ! Multi cluster/data-center ! Automate repair/recovery ! Database upgrades ! Backups ! Configuration management ! Cloning ! One-click scaling
  • 5. Copyright 2016 Severalnines AB 5 Supported Databases ! MySQL (Oracle/Percona/ MariaDB) ! MySQL Galera Cluster ! MySQL Cluster (NDB) ! MariaDB Cluster ! MySQL Replication 5.6 ! Standalone MySQL/MariaDB ! Postgres 9.4 ! Percona TokuMX Cluster ! MongoDB Sharded Cluster ! MongoDB Replica Set
  • 7. Copyright 2016 Severalnines AB MySQL Query Tuning - Process & Tools August 30, 2016 Krzysztof Książek Severalnines krzysztof@severalnines.com 7
  • 8. Copyright 2016 Severalnines AB 8 Agenda ! Query tuning process ! Build ! Collect ! Analyze ! Tune ! Test ! Tools ! tcpdump ! pt-query-digest
  • 9. Copyright 2016 Severalnines AB 9 The process
  • 10. Copyright 2016 Severalnines AB 10 The process ! A short guide to query performance review: ! Build test environment ! Collect your data ! Process your data ! Analyze your data ! Tune SQL and schemas ! Test changes ! Apply changes on production systems
  • 11. Copyright 2016 Severalnines AB 11 The process - build test environment ! You need a deterministic, test environment to make sure you can measure the impact of the changes ! Environment should mirror production as close as possible, to make it more relevant ! Hardware makes a difference, especially I/O - you’ll see different performance if your I/O subsystem is slow ! Dataset makes even more significant difference - you should test your new queries against production dataset ! If you need to obfuscate your data, your results may be different than on production systems
  • 12. Copyright 2016 Severalnines AB 12 The process - build test environment ! The best way to build a test system is to grab a backup of your production systems ! Restore it on a host which matches hardware configuration of the production ! You are all set ! Ideally, you have such QA/staging/dev environment up and running all the time and use it to test your new code and SQL ! If not, build it at least couple times per year to run such a review ! If you are using ClusterControl and Galera Cluster, take advantage of the cloning feature
  • 13. Copyright 2016 Severalnines AB 13 The process - collect your data ! Native method - slow query log ! Collects the most important data about the query:
  • 14. Copyright 2016 Severalnines AB 14 The process - collect your data ! It can collect even more data on Percona Server and MariaDB: ! Even more data with log_slow_verbosity='full,profiling_use_getrusage,profiling'
  • 15. Copyright 2016 Severalnines AB 15 The process - collect your data ! Another method - tcpdump ! Capture TCP traffic containing MySQL queries ! Gives limited information about the query: ! Query execution time ! Query size ! Main advantage - not necessary to execute it on the MySQL host - this reduces the impact ! Proxy ! Application host
  • 16. Copyright 2016 Severalnines AB 16 The process - collect your data
  • 17. Copyright 2016 Severalnines AB 17 The process - collect your data
  • 18. Copyright 2016 Severalnines AB 18 The process - analyze your data ! Thousands of lines in the log - can’t parse it manually ! You can build your own tool to aggregate data from slow query log ! Or you can use industry standard - pt-query-digest: ! wget http://percona.com/get/pt-query-digest && chmod u+x ./pt-query-digest ! Run pt-query-digest against slow query log or tcpdump ! But not on the database host - it can be CPU and memory intensive ! It’ll provide you with a summary of the traffic along with detailed information about each and every query
  • 19. Copyright 2016 Severalnines AB 19 The process - tune SQL and schemas ! Use EXPLAIN to check the query execution plan ! Rewrite queries to your liking ! Test new indexes, remove obsolete ones ! Make sure the new query execution plan works better ! Leverage Performance Schema for deep dive into where query execution time goes
  • 20. Copyright 2016 Severalnines AB 20 The process - apply changes on production systems ! Make sure you know what impact your changes will have ! If you run detailed tests in the test environment, you should be good ! Execute your changes ! Directly ! Using pt-online-schema-change ! Make sure you monitor the system after the change ! Have a rollback plan ! A list of ALTERs which will revert your changes can be really handy when things go awry ! Make sure you monitor how things unravel for at least couple of days - some processes may be executed on a weekly basis (or even less often)
  • 21. Copyright 2016 Severalnines AB 21 The tools
  • 22. Copyright 2016 Severalnines AB 22 The tools - tcpdump ! Collects a snapshot of the TCP traffic ! When you capture MySQL traffic, it contains enough data to calculate: ! Query execution time ! Query size ! Not much of details but enough to pinpoint slow queries ! You can always check details later, via EXPLAIN ! tcpdump -s 65535 -x -nn -q -tttt -i any port 3306 > mysql.tcp.txt ! Run it on the MySQL host ! Run it on the application server ! Run it on the proxy node ! Great flexibility - you can easily avoid most of the overhead caused by the tool
  • 23. Copyright 2016 Severalnines AB 23 The tools - pt-query-digest ! Amazing tool which is designed to aggregate data located in slow query log ! It can work with (among others) slow log and tcpdump output ! Presents the data in a very nice way ! Summary of all of the queries ! Each query presented in a detailed way (if there’s enough data for it, that is) ! EXPLAIN ready for copy&paste ! Report can be modified to user’s liking ! Sort queries by numerous attributes ! Aggregate queries differently ! Ability to store data in the database helps to build a solid review process ! You can also compare old query performance data with current one - to confirm performance has not changed
  • 24. Copyright 2016 Severalnines AB 24 The tools - pt-query-digest ! pt-query-digest --limit=100% /var/lib/mysql/slow.log > ptqd1.out ! Report starts with a summary: ! How many queries? ! How many distinct queries?
  • 25. Copyright 2016 Severalnines AB 25 The tools - pt-query-digest ! Large amount of summarized data ! Helps to understand 
 query patterns ! Helps to understand
 how workload has 
 changed
  • 26. Copyright 2016 Severalnines AB 26 The tools - pt-query-digest ! Queries in the report are sorted - by default, by total query execution time. ! Each query has a Query ID assigned - a hash of the query digest
  • 27. Copyright 2016 Severalnines AB 27 The tools - pt-query-digest ! Extensive information about every query. Extra data for Percona Server and MariaDB.
  • 28. Copyright 2016 Severalnines AB 28 The tools - pt-query-digest ! Detailed InnoDB stats: ! All the data: I/O operations, lock waits, pages accessed
  • 29. Copyright 2016 Severalnines AB 29 The tools - pt-query-digest ! Query histogram ! EXPLAIN ! SHOW commands
  • 30. Copyright 2016 Severalnines AB 30 The tools - pt-query-digest ! Data from tcpdump can’t be 
 as detailed as from slow log ! It still can be used to generate 
 important statistics ! You can always analyze such query
 manually later
  • 31. Copyright 2016 Severalnines AB 31 The tools - ClusterControl Query Monitor ! ClusterControl provides a continuous view into query performance ! Query Monitor presents query performance data based on Performance Schema (preferred) or slow query log (if Performance Schema is not available) ! May not be suitable for a “grand” review, but will help significantly on a day-to-day basis ! Query Histogram section lists queries with variable execution time - something definitely worth looking into
  • 32. Copyright 2016 Severalnines AB 32 The tools - ClusterControl Query Monitor
  • 33. Copyright 2016 Severalnines AB 33 The tools - ClusterControl Query Monitor
  • 34. Copyright 2016 Severalnines AB 34 The tools - ClusterControl Query Monitor
  • 35. Copyright 2016 Severalnines AB 35 Thank You! ! Blog posts covering query tuning process: ! http://severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process ! http://severalnines.com/blog/become-mysql-dba-blog-series-analyzing-your-sql-workload-using-pt-query-digest ! http://severalnines.com/blog/become-mysql-dba-blog-series-deep-dive-sql-workload-analysis-using-pt-query-digest ! Register for the next part of our trilogy: ! http://severalnines.com/upcoming-webinars ! Install ClusterControl: ! http://severalnines.com/getting-started ! Contact: jj@severalnines.com