The optimizer trace provides a detailed log of the actions taken by the query optimizer. It traces the major stages of query optimization including join preparation, join optimization, and join execution. During join optimization, it records steps like condition processing, determining table dependencies, estimating rows for plans, considering different execution plans, and choosing the best join order. The trace helps understand why certain query plans are chosen and catch differences in plans that may occur due to factors like database version changes.
The document summarizes new features in the query optimizer in MariaDB 10.4, including:
1) An optimizer trace that provides insight into the query planning process.
2) Using sampling for histogram collection during ANALYZE TABLE to improve performance.
3) Rowid filtering that pushes qualifying conditions into joins to filter out non-matching rows earlier.
4) Updated default settings that make better use of statistics and condition selectivity.
Optimizer features in recent releases of other databasesSergey Petrunya
The document summarizes several recent optimizer features introduced in MySQL 8.0 and PostgreSQL versions:
- MySQL 8.0 introduced an iterator-based executor, hash joins, EXPLAIN ANALYZE, and optimizations for anti-joins, semi-joins, and subqueries.
- PostgreSQL improved query parallelism, added multi-column statistics, parallel index creation, and optimized non-recursive common table expressions.
- Both databases have focused on join algorithms, statistics gathering, and parallel query processing to improve performance. MySQL continues to adopt features from other databases in recent releases.
- The document discusses histograms used for data statistics in MariaDB, MySQL, and PostgreSQL. Histograms provide compact summaries of column value distributions to help query optimizers estimate condition selectivities.
- MariaDB stores histograms in the mysql.column_stats table and collects them via full table scans. PostgreSQL collects histograms using random sampling and stores statistics in pg_stats including histograms and most common values lists.
- While both use height-balanced histograms, PostgreSQL additionally tracks most common values to improve selectivity estimates for frequent values.
EXPLAIN ANALYZE is a new query profiling tool first released in MySQL 8.0.18. This presentation covers how this new feature works, both on the surface and on the inside, and how you can use it to better understand your queries, to improve them and make them go faster.
This presentation is for everyone who has ever had to understand why a query is executed slower than anticipated, and for everyone who wants to learn more about query plans and query execution in MySQL.
Improving MariaDB’s Query Optimizer with better selectivity estimatesSergey Petrunya
The document discusses improving selectivity estimates in MariaDB's query optimizer. It begins with background on selectivity estimates and how the query optimizer uses statistics like cardinalities and selectivities. It then covers computing selectivity for local and join conditions, including techniques like histograms. The document discusses different types of histograms used in various databases and ongoing work in MariaDB to improve its histograms. It concludes with discussing computing selectivity for multiple conditions.
Using histograms to provide better query performance in MariaDB. Histograms capture the distribution of values in columns to help the query optimizer select better execution plans. The optimizer needs statistics on data distributions to estimate query costs accurately. Histograms are not enabled by default but can be collected using ANALYZE TABLE with the PERSISTENT option. Making histograms available improves the performance of queries that have selective filters or ordering on non-indexed columns.
The document summarizes new features in the query optimizer in MariaDB 10.4, including:
1) An optimizer trace that provides insight into the query planning process.
2) Using sampling for histogram collection during ANALYZE TABLE to improve performance.
3) Rowid filtering that pushes qualifying conditions into joins to filter out non-matching rows earlier.
4) Updated default settings that make better use of statistics and condition selectivity.
Optimizer features in recent releases of other databasesSergey Petrunya
The document summarizes several recent optimizer features introduced in MySQL 8.0 and PostgreSQL versions:
- MySQL 8.0 introduced an iterator-based executor, hash joins, EXPLAIN ANALYZE, and optimizations for anti-joins, semi-joins, and subqueries.
- PostgreSQL improved query parallelism, added multi-column statistics, parallel index creation, and optimized non-recursive common table expressions.
- Both databases have focused on join algorithms, statistics gathering, and parallel query processing to improve performance. MySQL continues to adopt features from other databases in recent releases.
- The document discusses histograms used for data statistics in MariaDB, MySQL, and PostgreSQL. Histograms provide compact summaries of column value distributions to help query optimizers estimate condition selectivities.
- MariaDB stores histograms in the mysql.column_stats table and collects them via full table scans. PostgreSQL collects histograms using random sampling and stores statistics in pg_stats including histograms and most common values lists.
- While both use height-balanced histograms, PostgreSQL additionally tracks most common values to improve selectivity estimates for frequent values.
EXPLAIN ANALYZE is a new query profiling tool first released in MySQL 8.0.18. This presentation covers how this new feature works, both on the surface and on the inside, and how you can use it to better understand your queries, to improve them and make them go faster.
This presentation is for everyone who has ever had to understand why a query is executed slower than anticipated, and for everyone who wants to learn more about query plans and query execution in MySQL.
Improving MariaDB’s Query Optimizer with better selectivity estimatesSergey Petrunya
The document discusses improving selectivity estimates in MariaDB's query optimizer. It begins with background on selectivity estimates and how the query optimizer uses statistics like cardinalities and selectivities. It then covers computing selectivity for local and join conditions, including techniques like histograms. The document discusses different types of histograms used in various databases and ongoing work in MariaDB to improve its histograms. It concludes with discussing computing selectivity for multiple conditions.
Using histograms to provide better query performance in MariaDB. Histograms capture the distribution of values in columns to help the query optimizer select better execution plans. The optimizer needs statistics on data distributions to estimate query costs accurately. Histograms are not enabled by default but can be collected using ANALYZE TABLE with the PERSISTENT option. Making histograms available improves the performance of queries that have selective filters or ordering on non-indexed columns.
MariaDB Optimizer - further down the rabbit holeSergey Petrunya
The document summarizes new features in the MariaDB 10.4 query optimizer including:
1) New default optimizer settings that take more factors into account for condition selectivity and use histograms by default.
2) Faster histogram collection using Bernoulli sampling rather than analyzing the whole data set.
3) Two new types of condition pushdown - from HAVING clauses into WHERE clauses, and into materialized IN subqueries.
Introduction into MySQL Query Tuning for Dev[Op]sSveta Smirnova
Percona Live Online 2021 talk: https://www.percona.com/resources/videos/introduction-mysql-query-tuning-for-devops
In this talk I will show how to get started with MySQL Query Tuning. I will make a short introduction into physical table structure and demonstrate how it may influence query execution time.
Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite queries or change table structure to achieve better performance.
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
Tutorial delivered at Percona MySQL Conference Live London 2013.
It doesn't matter what new SSD technologies appear, or what are the latest breakthroughs in flushing algorithms: the number one cause for MySQL applications being slow is poor execution plan of SQL queries. While the latest GA version provided a huge amount of transparent optimizations -specially for JOINS and subqueries- it is still the developer's responsibility to take advantage of all new MySQL 5.6 features.
In this tutorial we will propose the attendants a sample PHP application with bad response time. Through practical examples, we will suggest step-by-step strategies to improve its performance, including:
* Checking MySQL & InnoDB configuration
* Internal (performance_schema) and external tools for profiling (pt-query-digest)
* New EXPLAIN tools
* Simple and multiple column indexing
* Covering index technique
* Index condition pushdown
* Batch key access
* Subquery optimization
The document discusses Cassandra Storage Engine (Cassandra SE) in MariaDB, which allows MariaDB to access Cassandra data. Cassandra SE provides a SQL view of Cassandra data by mapping Cassandra concepts like column families to MariaDB tables. It supports basic operations like SELECT, INSERT, and UPDATE between the two systems. The document outlines use cases, benchmarks Cassandra SE performance, and discusses future directions like supporting additional Cassandra features.
MySQL is a relational database management system. It provides tools for managing data, including creating, querying, updating and deleting data in databases. Some key features include:
- Creating, altering and dropping databases, tables, indexes, users and more.
- Inserting, selecting, updating and deleting data with SQL statements.
- Backup and restore capabilities using mysqldump to backup entire databases or tables.
- Security features including user accounts and privileges to control access.
- Performance optimization using indexes, partitioning, query tuning and more.
- Data types for different kinds of data like numbers, dates, text, JSON and more.
Common Table Expressions in MariaDB 10.2 (Percona Live Amsterdam 2016)Sergey Petrunya
- Common Table Expressions (CTEs) allow temporary named results to be stored and reused within a single SQL statement.
- There are two types of CTEs: non-recursive and recursive. Non-recursive CTEs can refer to other CTEs but not recursively, while recursive CTEs define a recursive relationship between rows.
- Optimizations for non-recursive CTEs in MariaDB include merging CTEs into joins and pushing conditions down into CTEs to filter groups. Recursive CTEs are used to query hierarchical or graph-like recursive relationships in a table.
MariaDB: Engine Independent Table Statistics, including histogramsSergey Petrunya
EITS (Engine Independent Table Statistics) provides new statistics for MariaDB query optimization, including histograms for column value distributions. These statistics must be manually collected with ANALYZE TABLE and enabled to improve estimates for queries using non-indexed columns and ranges. With EITS, the optimizer can better estimate row counts and filtering for joins and WHERE clauses.
- Common Table Expressions (CTEs) allow for temporary results to be stored and reused within the same SQL statement, similar to derived tables or views.
- CTEs can be non-recursive or recursive. Non-recursive CTEs are optimized by merging into joins or pushing conditions down, while recursive CTEs compute results through iterative steps until a fixed point is reached.
- The document discusses optimizations for non-recursive CTEs in MariaDB and provides examples of using CTEs for common queries involving things like hierarchical or network data.
This presentation focuses on optimization of queries in MySQL from developer’s perspective. Developers should care about the performance of the application, which includes optimizing SQL queries. It shows the execution plan in MySQL and explain its different formats - tabular, TREE and JSON/visual explain plans. Optimizer features like optimizer hints and histograms as well as newer features like HASH joins, TREE explain plan and EXPLAIN ANALYZE from latest releases are covered. Some real examples of slow queries are included and their optimization explained.
This document summarizes new features in the MariaDB 10.0 query optimizer, including:
1. Engine-independent statistics like histograms that are collected via ANALYZE TABLE instead of random sampling.
2. New subquery optimizations that convert EXISTS subqueries to inner joins and trivially correlated EXISTS to IN.
3. EXPLAIN improvements like SHOW EXPLAIN to see EXPLAIN plans for running queries, and logging EXPLAIN output in the slow query log.
The document discusses various query optimization techniques used in database management systems including MariaDB, MySQL, PostgreSQL, and SQL Server. Specifically, it covers the use of histograms to estimate query selectivity, derived table merging, condition pushdown including through window functions, and split grouping optimizations. Histograms help query planners estimate the number of rows filtered by query conditions. Derived table merging and condition pushdown help push conditions earlier in query execution. Split grouping allows computing groupings for a subset of rows instead of all rows.
This document provides an agenda and overview for a MySQL Query Tuning 101 presentation. The summary includes:
1. The agenda covers topics like identifying slow queries, using indexes, the EXPLAIN tool, and other optimization techniques.
2. When queries run slow, the presenter will discuss using indexes to improve performance by allowing MySQL to access data more efficiently.
3. The EXPLAIN tool is covered as a way to estimate query execution and see how MySQL utilizes indexes. Different EXPLAIN output will be demonstrated using examples from an employees database.
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
The document discusses techniques for identifying and addressing problems with a database query optimizer. It describes old and new tools for catching slow queries, such as the slow query log, SHOW PROCESSLIST, and the Performance Schema. It also provides examples of using these tools to analyze query plans, identify inefficient plans, and determine if optimizer settings or query structure need to be modified to address performance issues.
What SQL functionality was added in the past year or so. The presentation covers default expressions, functional key parts, lateral derived tables, CHECK constraints, JSON and spatial improvements. Also some other small SQL and other improvements.
The document discusses MySQL indexing and optimization. It provides an overview of indexing in MySQL, including what indexes are, how they are created, and how MySQL uses indexes to optimize queries. The document also discusses best practices for indexing, including identifying missing indexes, avoiding unnecessary indexes, and ensuring indexes have good selectivity to maximize query performance.
The document summarizes several new features in the MariaDB and MySQL query optimizers, including:
1) Subquery optimizations that improved subquery performance from orders of magnitude slower to on par with other databases. These optimizations are available in MariaDB 5.3/5.5 and MySQL 5.6.
2) Batched Key Access, which speeds up large IO-bound joins by accessing keys in batches rather than randomly, improving performance by orders of magnitude.
3) Index Condition Pushdown, which pushes SELECT conditions into indexes to filter records before reading the table, improving performance for IO-bound queries similar to "Using index".
The document discusses adaptive query optimization in Oracle 12c. It begins by describing drawbacks of the optimizer in pre-12c versions, such as insufficient statistics triggering dynamic sampling. It then outlines the key features of adaptive query optimization in 12c, including adaptive/dynamic plans using techniques like adaptive parallel distribution and adaptive joins. It also discusses automatic re-optimization using feedback from initial executions. The document provides illustrations of these techniques using example queries and optimizer statistics.
The MySQL Query Optimizer Explained Through Optimizer Traceoysteing
The document discusses the MySQL query optimizer. It begins by explaining how the optimizer works, including analyzing statistics, determining optimal join orders and access methods. It then describes how the optimizer trace can provide insight into why a particular execution plan was selected. The remainder of the document provides details on the various phases the optimizer goes through, including logical transformations, cost-based optimizations like range analysis and join order selection.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.
The document discusses various techniques for optimizing query performance in MySQL, including using indexes appropriately, avoiding full table scans, and tools like EXPLAIN, Performance Schema, and pt-query-digest for analyzing queries and identifying optimization opportunities. It provides recommendations for index usage, covering indexes, sorting and joins, and analyzing slow queries.
MariaDB Optimizer - further down the rabbit holeSergey Petrunya
The document summarizes new features in the MariaDB 10.4 query optimizer including:
1) New default optimizer settings that take more factors into account for condition selectivity and use histograms by default.
2) Faster histogram collection using Bernoulli sampling rather than analyzing the whole data set.
3) Two new types of condition pushdown - from HAVING clauses into WHERE clauses, and into materialized IN subqueries.
Introduction into MySQL Query Tuning for Dev[Op]sSveta Smirnova
Percona Live Online 2021 talk: https://www.percona.com/resources/videos/introduction-mysql-query-tuning-for-devops
In this talk I will show how to get started with MySQL Query Tuning. I will make a short introduction into physical table structure and demonstrate how it may influence query execution time.
Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite queries or change table structure to achieve better performance.
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
Tutorial delivered at Percona MySQL Conference Live London 2013.
It doesn't matter what new SSD technologies appear, or what are the latest breakthroughs in flushing algorithms: the number one cause for MySQL applications being slow is poor execution plan of SQL queries. While the latest GA version provided a huge amount of transparent optimizations -specially for JOINS and subqueries- it is still the developer's responsibility to take advantage of all new MySQL 5.6 features.
In this tutorial we will propose the attendants a sample PHP application with bad response time. Through practical examples, we will suggest step-by-step strategies to improve its performance, including:
* Checking MySQL & InnoDB configuration
* Internal (performance_schema) and external tools for profiling (pt-query-digest)
* New EXPLAIN tools
* Simple and multiple column indexing
* Covering index technique
* Index condition pushdown
* Batch key access
* Subquery optimization
The document discusses Cassandra Storage Engine (Cassandra SE) in MariaDB, which allows MariaDB to access Cassandra data. Cassandra SE provides a SQL view of Cassandra data by mapping Cassandra concepts like column families to MariaDB tables. It supports basic operations like SELECT, INSERT, and UPDATE between the two systems. The document outlines use cases, benchmarks Cassandra SE performance, and discusses future directions like supporting additional Cassandra features.
MySQL is a relational database management system. It provides tools for managing data, including creating, querying, updating and deleting data in databases. Some key features include:
- Creating, altering and dropping databases, tables, indexes, users and more.
- Inserting, selecting, updating and deleting data with SQL statements.
- Backup and restore capabilities using mysqldump to backup entire databases or tables.
- Security features including user accounts and privileges to control access.
- Performance optimization using indexes, partitioning, query tuning and more.
- Data types for different kinds of data like numbers, dates, text, JSON and more.
Common Table Expressions in MariaDB 10.2 (Percona Live Amsterdam 2016)Sergey Petrunya
- Common Table Expressions (CTEs) allow temporary named results to be stored and reused within a single SQL statement.
- There are two types of CTEs: non-recursive and recursive. Non-recursive CTEs can refer to other CTEs but not recursively, while recursive CTEs define a recursive relationship between rows.
- Optimizations for non-recursive CTEs in MariaDB include merging CTEs into joins and pushing conditions down into CTEs to filter groups. Recursive CTEs are used to query hierarchical or graph-like recursive relationships in a table.
MariaDB: Engine Independent Table Statistics, including histogramsSergey Petrunya
EITS (Engine Independent Table Statistics) provides new statistics for MariaDB query optimization, including histograms for column value distributions. These statistics must be manually collected with ANALYZE TABLE and enabled to improve estimates for queries using non-indexed columns and ranges. With EITS, the optimizer can better estimate row counts and filtering for joins and WHERE clauses.
- Common Table Expressions (CTEs) allow for temporary results to be stored and reused within the same SQL statement, similar to derived tables or views.
- CTEs can be non-recursive or recursive. Non-recursive CTEs are optimized by merging into joins or pushing conditions down, while recursive CTEs compute results through iterative steps until a fixed point is reached.
- The document discusses optimizations for non-recursive CTEs in MariaDB and provides examples of using CTEs for common queries involving things like hierarchical or network data.
This presentation focuses on optimization of queries in MySQL from developer’s perspective. Developers should care about the performance of the application, which includes optimizing SQL queries. It shows the execution plan in MySQL and explain its different formats - tabular, TREE and JSON/visual explain plans. Optimizer features like optimizer hints and histograms as well as newer features like HASH joins, TREE explain plan and EXPLAIN ANALYZE from latest releases are covered. Some real examples of slow queries are included and their optimization explained.
This document summarizes new features in the MariaDB 10.0 query optimizer, including:
1. Engine-independent statistics like histograms that are collected via ANALYZE TABLE instead of random sampling.
2. New subquery optimizations that convert EXISTS subqueries to inner joins and trivially correlated EXISTS to IN.
3. EXPLAIN improvements like SHOW EXPLAIN to see EXPLAIN plans for running queries, and logging EXPLAIN output in the slow query log.
The document discusses various query optimization techniques used in database management systems including MariaDB, MySQL, PostgreSQL, and SQL Server. Specifically, it covers the use of histograms to estimate query selectivity, derived table merging, condition pushdown including through window functions, and split grouping optimizations. Histograms help query planners estimate the number of rows filtered by query conditions. Derived table merging and condition pushdown help push conditions earlier in query execution. Split grouping allows computing groupings for a subset of rows instead of all rows.
This document provides an agenda and overview for a MySQL Query Tuning 101 presentation. The summary includes:
1. The agenda covers topics like identifying slow queries, using indexes, the EXPLAIN tool, and other optimization techniques.
2. When queries run slow, the presenter will discuss using indexes to improve performance by allowing MySQL to access data more efficiently.
3. The EXPLAIN tool is covered as a way to estimate query execution and see how MySQL utilizes indexes. Different EXPLAIN output will be demonstrated using examples from an employees database.
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
The document discusses techniques for identifying and addressing problems with a database query optimizer. It describes old and new tools for catching slow queries, such as the slow query log, SHOW PROCESSLIST, and the Performance Schema. It also provides examples of using these tools to analyze query plans, identify inefficient plans, and determine if optimizer settings or query structure need to be modified to address performance issues.
What SQL functionality was added in the past year or so. The presentation covers default expressions, functional key parts, lateral derived tables, CHECK constraints, JSON and spatial improvements. Also some other small SQL and other improvements.
The document discusses MySQL indexing and optimization. It provides an overview of indexing in MySQL, including what indexes are, how they are created, and how MySQL uses indexes to optimize queries. The document also discusses best practices for indexing, including identifying missing indexes, avoiding unnecessary indexes, and ensuring indexes have good selectivity to maximize query performance.
The document summarizes several new features in the MariaDB and MySQL query optimizers, including:
1) Subquery optimizations that improved subquery performance from orders of magnitude slower to on par with other databases. These optimizations are available in MariaDB 5.3/5.5 and MySQL 5.6.
2) Batched Key Access, which speeds up large IO-bound joins by accessing keys in batches rather than randomly, improving performance by orders of magnitude.
3) Index Condition Pushdown, which pushes SELECT conditions into indexes to filter records before reading the table, improving performance for IO-bound queries similar to "Using index".
The document discusses adaptive query optimization in Oracle 12c. It begins by describing drawbacks of the optimizer in pre-12c versions, such as insufficient statistics triggering dynamic sampling. It then outlines the key features of adaptive query optimization in 12c, including adaptive/dynamic plans using techniques like adaptive parallel distribution and adaptive joins. It also discusses automatic re-optimization using feedback from initial executions. The document provides illustrations of these techniques using example queries and optimizer statistics.
The MySQL Query Optimizer Explained Through Optimizer Traceoysteing
The document discusses the MySQL query optimizer. It begins by explaining how the optimizer works, including analyzing statistics, determining optimal join orders and access methods. It then describes how the optimizer trace can provide insight into why a particular execution plan was selected. The remainder of the document provides details on the various phases the optimizer goes through, including logical transformations, cost-based optimizations like range analysis and join order selection.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.
The document discusses various techniques for optimizing query performance in MySQL, including using indexes appropriately, avoiding full table scans, and tools like EXPLAIN, Performance Schema, and pt-query-digest for analyzing queries and identifying optimization opportunities. It provides recommendations for index usage, covering indexes, sorting and joins, and analyzing slow queries.
How to use histograms to get better performanceMariaDB plc
Sergei Petrunia and Varun Gupta, software engineers MariaDB, show how histograms can be used to improve query performance. They begin by introducing histrograms and explaining why they’re needed by the query optimizer. Next, they discuss how to determine whether or not histrograms are needed, and if so, how to determine which tables and columns they should be applied. Finally, they cover best practices and recent improvements to histograms.
The optimizer is the brain of any DBMS system, This presentation helps to understand MongoDB explain plan metrics and How MongoDB Optimiser scores for each query plan and choosing the proper query plan for the execution.
Query Optimization with MySQL 5.6: Old and New TricksMYXPLAIN
The document discusses query optimization techniques for MySQL 5.6, including both established techniques and new features in 5.6. It provides an overview of tools for profiling queries such as EXPLAIN, the slow query log, and the performance schema. It also covers indexing strategies like compound indexes and index condition pushdown.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema provides detailed information for troubleshooting and optimizing MySQL. It collects instrumentation data on server operations, statements, memory usage, locks and connections. The data can be used to identify slow queries, statements not using indexes, memory consumption trends over time, and more. Configuration and enabling specific instruments allows controlling the level of detail collected.
In this session Max Mether, VP of Product Management at MariaDB, provides an introduction to MariaDB Platform X3 and the new features in MariaDB Server 10.3 and MariaDB MaxScale 2.3. He then turns his focus to what’s coming in MariaDB Server 10.4, including instant DROP COLUMN, the INTERVAL data type and advanced security features like account locking.
This document summarizes several performance improvements in MySQL 5.6 including index condition pushdown, multi-range read, batched key access, and persistent optimizer statistics. It provides examples of query execution with and without these optimizations, showing significant performance gains in MySQL 5.6 for queries that push index conditions down to the storage engine, read multiple index ranges sequentially, and access keys in batches for joins.
The document discusses how the PostgreSQL query planner works. It explains that a query goes through several stages including parsing, rewriting, planning/optimizing, and execution. The optimizer or planner has to estimate things like the number of rows and cost to determine the most efficient query plan. Statistics collected by ANALYZE are used for these estimates but can sometimes be inaccurate, especially for n_distinct values. Increasing the default_statistics_target or overriding statistics on columns can help address underestimation issues. The document also discusses different plan types like joins, scans, and aggregates that the planner may choose between.
MySQLinsanity! This document provides an overview of Stanley Huang's MySQL performance tuning experience and expertise. It begins with introductions and background on Stanley Huang. It then discusses the typical phases of MySQL performance tuning projects, including SQL tuning and RDBMS tuning. Specific tips are provided around topics like slow query logging, index usage, partitioning, and server configuration. The document concludes with an invitation for questions.
Performance improvements in PostgreSQL 9.5 and beyondTomas Vondra
This document discusses several performance improvements made in PostgreSQL versions 9.5 and beyond. Some key improvements discussed include:
- Faster sorting through allowing sorting by inlined functions, abbreviated keys for VARCHAR/TEXT/NUMERIC, and Sort Support benefits.
- Improved hash joins through reduced palloc overhead, smaller NTUP_PER_BUCKET, and dynamically resizing the hash table.
- Index improvements like avoiding index tuple copying, GiST and bitmap index scan optimizations, and block range tracking in BRIN indexes.
- Aggregate functions see speedups through using 128-bit integers for internal state instead of NUMERIC in some cases.
- Other optimizations affect PL/pgSQL performance,
*If you see the screen is not good condition, downloading please.*
MariaDB Optimizer
- Query execution process
- Optimizer
- Statistics
- Histogram-based Statistics
- Join Optimizer
- Analysis of the execution plan
- Optimizer Hint
- Tuning Point
The document describes various features of MySQL Performance Schema. It discusses how Performance Schema provides visibility into SQL statements, prepared statements, stored routines and locks. It provides examples of using Performance Schema tables and views to diagnose issues such as slow queries, full table scans, and locks preventing DDL statements from completing. Hands-on exercises are suggested to practice analyzing statements, prepared statements and stored routines using Performance Schema.
How Database Convergence Impacts the Coming Decades of Data ManagementSingleStore
How Database Convergence Impacts the Coming Decades of Data Management by Nikita Shamgunov, CEO and co-founder of MemSQL.
Presented at NYC Database Month in October 2017. NYC Database Month is the largest database meetup in New York, featuring talks from leaders in the technology space. You can learn more at http://www.databasemonth.com.
New Features
● Developer and SQL Features
● DBA and Administration
● Replication
● Performance
By Amit Kapila at India PostgreSQL UserGroup Meetup, Bangalore at InMobi.
http://technology.inmobi.com/events/india-postgresql-usergroup-meetup-bangalore
Design and develop with performance in mind
Establish a tuning environment
Index wisely
Reduce parsing
Take advantage of Cost Based Optimizer
Avoid accidental table scans
Optimize necessary table scans
Optimize joins
Use array processing
Consider PL/SQL for “tricky” SQL
Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...InfluxData
Dean will provide practical tips and techniques learned from helping hundreds of customers deploy InfluxDB and InfluxDB Enterprise. This includes hardware and architecture choices, schema design, configuration setup, and running queries.
New optimizer features in MariaDB releases before 10.12Sergey Petrunya
The document discusses new optimizer features in recent and upcoming MariaDB releases. MariaDB 10.8 introduced JSON histograms and support for reverse-ordered indexes. JSON produced by the optimizer is now valid and processible. MariaDB 10.9 added SHOW EXPLAIN FORMAT=JSON and SHOW ANALYZE can return partial results. MariaDB 10.10 enabled table elimination for derived tables and improved optimization of many-table joins. Future releases will optimize queries on stored procedures and show optimizer timing in EXPLAIN FORMAT=JSON.
MariaDB's join optimizer: how it works and current fixesSergey Petrunya
The document discusses improvements to MariaDB's join optimizer. It describes how the optimizer currently works, including join order search, pruning techniques, and greedy search. It then outlines several patches and improvements made to better prune join order search spaces and find optimal plans more quickly. This includes handling "edge tables", improving heuristics for key dependencies and model tables, pre-sorting tables during search, and exploring eq_ref chaining to further reduce search space for attribute tables.
This document discusses improvements to histograms in MariaDB. It provides background on how query optimizers use histograms to estimate condition selectivity. It describes the basic equi-width and improved equi-height histograms. It outlines how MariaDB 10.8 introduces a new JSON-based histogram type that stores exact bucket endpoints to improve accuracy, especially for popular values. The new type fixes issues the previous approaches had with inaccurate selectivity estimates for certain conditions. Overall, the document presents histograms as an important tool for query optimization and how MariaDB is enhancing their implementation.
JSON Support in MariaDB: News, non-news and the bigger pictureSergey Petrunya
This document summarizes JSON support features in MariaDB, including JSON Path and JSON_TABLE. It discusses MariaDB and MySQL's implementation of the SQL:2016 JSON Path language, noting limitations compared to other databases. JSON_TABLE is explained as a way to convert JSON data to tabular form using column definitions. Examples are provided and features like handling nested paths and errors are covered. JSON support in MariaDB is still being developed to implement more of the standard and address current limitations.
This document discusses MyRocks, a storage engine for MariaDB that uses RocksDB as its backend. It begins by explaining the limitations of InnoDB that MyRocks aims to address, such as high write and space amplification. It then describes how RocksDB uses log-structured merge trees to reduce these issues. The document outlines how MyRocks implements the MySQL storage engine interface on top of RocksDB. It concludes by covering best practices for using MyRocks, including installation, migration, tuning for replication and backups.
This document discusses new query optimization features in MariaDB 10.3. It describes how MariaDB 10.3 improves on condition pushdown from 10.2 by allowing conditions to be pushed through window functions. It also explains a new "split grouping" optimization where grouping is done separately for each relevant group, rather than computing all groups at once, allowing indexes to be leveraged more efficiently. These optimizations can improve performance by filtering out unnecessary rows earlier in query execution.
The document discusses MyRocks being included in MariaDB. Some key points:
- MyRocks is a storage engine that combines RocksDB with MySQL/MariaDB for better performance.
- MyRocks is now included in MariaDB 10.2 as an alpha plugin, with binaries/packages available. Many features work but some like binlog/replication are still in progress.
- MariaDB will continue merging updates from the MyRocks upstream project and work to increase the plugin's maturity level.
- Future plans include finishing core features like binlog/replication support, packaging backup tools, and ensuring compatibility with MariaDB features like global variables and GTID replication.
This document provides an overview of MyRocks, a storage engine for MySQL/MariaDB that uses the RocksDB key-value store. It discusses the write amplification issues with InnoDB, how LSM trees in RocksDB address these issues through log-structured merging, and benchmarks showing the size, write amplification, and performance improvements MyRocks provides over InnoDB. It also outlines the process of integrating MyRocks into MariaDB, current status as an alpha plugin, and plans to improve support and testing.
This document discusses porting MyRocks, a storage engine that combines RocksDB with MySQL, to MariaDB. It provides an overview of MyRocks, the tasks involved in porting it to MariaDB, the current status, and future plans. Key points include porting MyRocks from Facebook's MySQL to MariaDB, building packages, releasing it as part of a MariaDB version, addressing failing tests and missing features, and improving integration with MariaDB capabilities like binlogging. The goal is to get MyRocks adopted more broadly by adding it to MariaDB and expanding the community around it.
The document discusses window functions in MariaDB. It begins with an overview and plan, then covers basic window functions like row_number(), rank(), dense_rank(), and ntile(). It discusses frames for window functions, including examples using RANGE frames. It provides examples of problems that can be solved using window functions, such as smoothing noisy data, generating account balance statements, and finding sequences with no missing numbers ("islands").
MyRocks: табличный движок для MySQL на основе RocksDBSergey Petrunya
MyRocks: табличный движок для MySQL на основе RocksDB.
Презентация с HighLoad++ 2015.
Рассказывается о принципах работы LSM-Trees, их реализации в RocksDB, зачем и как был сделан MyRocks, с какими проблемами столкнулись и как их решили.
MariaDB: ANALYZE for statements (lightning talk)Sergey Petrunya
The document describes a new ANALYZE statement in MariaDB 10.1 that provides execution statistics for a SQL statement. ANALYZE runs the statement and collects statistics, similar to EXPLAIN ANALYZE in PostgreSQL. It produces an EXPLAIN plan with additional columns showing real rows, filtering percentages, and time spent. The FORMAT=JSON option outputs the results as a JSON document containing detailed timing and resource usage statistics for each step. This allows more complete analysis of how a query plan was executed versus global counters.
How mysql handles ORDER BY, GROUP BY, and DISTINCTSergey Petrunya
This document discusses how MySQL handles ORDER BY, GROUP BY, and DISTINCT clauses. It describes the different strategies MySQL uses to produce ordered or grouped result sets, including using an index, filesort, or temporary table. It also covers some special cases and limitations. The goal is for the query optimizer to better account for sorting and grouping costs when choosing a query execution plan.
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Project Management: The Role of Project Dashboards.pdfKarya Keeper
Project management is a crucial aspect of any organization, ensuring that projects are completed efficiently and effectively. One of the key tools used in project management is the project dashboard, which provides a comprehensive view of project progress and performance. In this article, we will explore the role of project dashboards in project management, highlighting their key features and benefits.
8 Best Automated Android App Testing Tool and Framework in 2024.pdfkalichargn70th171
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
Unveiling the Advantages of Agile Software Development.pdfbrainerhub1
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
Microservice Teams - How the cloud changes the way we workSven Peters
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...XfilesPro
Wondering how X-Sign gained popularity in a quick time span? This eSign functionality of XfilesPro DocuPrime has many advancements to offer for Salesforce users. Explore them now!
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
How Can Hiring A Mobile App Development Company Help Your Business Grow?ToXSL Technologies
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfVALiNTRY360
Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems.
For more info visit us https://valintry360.com/solutions/health-life-sciences
2. 2
Optimizer Trace
●
Prototyped before MariaDB existed
●
Initial release: MySQL 5.6
●
Release in MariaDB: 10.4 (June 2019)
− After that, added more tracing
− Got experience with using it
3. 3
Optimizer Trace goals
●
“Show details about what goes on in the optimizer”
EXPLAIN ANALYZEOptimizer trace
Optimization
Query
PlanSQL Execution
●
There is a lot going on there
− rewrites (e.g. view merging)
− WHERE analysis, finding ways to read rows (t.key_column < 'abc')
− Search for query plan
● *Some* of possible plans are considered
− Plan refinement
4. 4
Optimizer Trace goals
●
“Show details about what goes on in the optimizer”
EXPLAIN ANALYZEOptimizer trace
Optimization
Query
PlanSQL Execution
●
Answers questions
− Why query plan X is [not] chosen
● Can strategy Y be used for index Z?
● Is restriction on indexed column C sargable?
● ...
− Why are query plans different across db instances
● different db version (upgrades)
● different statistics or “minor” DDL changes
5. 5
Optimizer Trace goals (2)
●
“Show details about what goes on in the optimizer”
EXPLAIN ANALYZEOptimizer trace
Optimization
Query
PlanSQL Execution
●
Answers questions (2)
− Asking for help or reporting an issue to the development team
● Uploading the whole dataset is not always possible
● EXPLAIN, Table definitions, etc – are not always sufficient
● The trace is text, so one can remove any sensitive info.
12. 12
Query processing
Optimizer trace
●
Mostly covers Query Optimization
●
Covers a bit of Name Resolution
− Because some optimizations are or
were done there
●
Covers a bit of execution
− (More of it in MySQL)
Parsing
SQLSQL
Parse
Tree
Name Resolution
“Parsed
Query”
Query Optimization
Query
Plan
Execution
13. 13
Query processing in optimizer trace
TRACE: {
"steps": [
{
"join_preparation": {
"select_id": 1,
"steps": [
{
"expanded_query": "select t1.col1 AS
col1,t1.col2 AS col2,t1.col3 AS col3 from t1 where t1.col1
< 3 and t1.col2 = 'foo'"
}
]
}
},
{
"join_optimization": {
"select_id": 1,
"steps": [
...
...
...
]
}
},
{
"join_execution": {
"select_id": 1,
"steps": []
}
}
]
}
Parsing
SQLSQL
Parse
Tree
Name Resolution
“Parsed
Query”
Query Optimization
Query
Plan
Execution
select * from t1 where col1 <3 and col2 ='foo'
14. 14
Query Optimization
●
Each SELECT is optimized separately
− Except when it is merged, converted to
semi-join, etc
●
Optimization phases are roughly as shown
in the diagram
− With some exceptions
“Parsed
Query”
SELECT Optimization
Query
Plan
Do rewrites/
transformations
Collect table info/stats
Join order choice
Handle ORDER/GROUP
Do other fix-ups
22. 22
Loose index scan
create table t (
kp1 INT,
kp2 INT,
kp3 INT,
...
KEY (kp1, kp2, kp3)
);
●
Loose index scan is a useful optimization
− The choice whether to use it is cost-based (depends also on ANALYZE TABLE)
− It has complex applicability criteria
select
min(kp2)
from t
where
kp2>=10
group by
kp1
select
min(kp2)
from t
where
kp2>=10 and kp3<10
group by
kp1
●
Try two very similar queries
− This one is using Loose Scan − This one is NOT using Loose Scan
37. 37
Optimizer Trace in MySQL
●
Similar to MariaDB’s
− User interface
− Trace elements and structure
●
There are differences
− Optimizers are different
− Slightly different set of things traced
− The tracing module has extra
features
●
Let’s review
− Anything to learn?
{
"steps": [
{
"join_preparation": {
"select#": 1,
"steps": [
{
"expanded_query": "/* select#1 */ select `t1`.`col1` AS
`col1`,`t1`.`col2` AS `col2`,`t1`.`col3` AS `col3` from `t1` where
((`t1`.`col1` < 3) and (`t1`.`col2` = 'foo'))"
}
]
}
},
{
"join_optimization": {
"select#": 1,
"steps": [
{
"condition_processing": {
"condition": "WHERE",
"original_condition": "((`t1`.`col1` < 3) and (`t1`.`col2` =
'foo'))",
"steps": [
{
"transformation": "equality_propagation",
"resulting_condition": "((`t1`.`col1` < 3) and
(`t1`.`col2` = 'foo'))"
},
{
"transformation": "constant_propagation",
"resulting_condition": "((`t1`.`col1` < 3) and
(`t1`.`col2` = 'foo'))"
},
{
"transformation": "trivial_condition_removal",
"resulting_condition": "((`t1`.`col1` < 3) and
(`t1`.`col2` = 'foo'))"
}
]
}
},
{
"substitute_generated_columns": {
}
},
{
"table_dependencies": [
{
"table": "`t1`",
"row_may_be_null": false,
"map_bit": 0,
"depends_on_map_bits": [
]
}
]
},
{
"ref_optimizer_key_uses": [
]
},
38. 38
Extras in MySQL
●
Can capture multiple traces for nested statements
− Controlled with @@optimizer_trace_{offset,limit}
● Can set to save first/last N traces.
− Allows to trace statements inside stored procedure/function calls
− The issue is that other support for nested statements is not present:
● Can’t get EXPLAINs for sub-statements.
● Can’t get statement_time* (*- in some cases, can infer #rows_examined and statement_time from
PERFORMANCE_SCHEMA.
● Is this useful?
●
Can omit parts of trace
− Controlled by @@optimizer_trace_features
greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
− Why not use JSON_EXTRACT to get the part of trace you need, instead?
− Would be useful for global settings e.g. “[Don’t] print cost values everywhere”.
39. 39
Extras in MySQL (2)
●
Can control JSON formatting
− SET @@optimizer_trace='enabled=on,one_line=on'
− SET @@end_markers_in_json=on;
● Note: the output is not a valid
JSON anymore, can’t be processed.
"filesort_priority_queue_optimization": {
"usable": false,
"cause": "not applicable (no LIMIT)"
},
"filesort_execution": [
],
"filesort_summary": {
"memory_available": 998483,
"key_size": 1022,
"row_size": 66561,
"max_rows_per_buffer": 14,
"num_rows_estimate": 215,
"num_rows_found": 0,
"num_initial_chunks_spilled_to_disk": 0,
"peak_memory_used": 0,
"sort_algorithm": "none",
"sort_mode": "<fixed_sort_key,
packed_additional_fields>"
}
●
Tracing covers some parts of query execution
− Because MySQL didn’t have EXPLAIN
ANALYZE back then?
− Can produce a lot of repetitive JSON
− In MariaDB, this kind of info is shown in
ANALYZE FORMAT=JSON output.
"join_preparation": {
"select#": 1,
"steps": [
...
] /* steps */
} /* join_preparation */
40. 40
Extra in MariaDB
●
Conditions are readable
●
Better JSON formatting
"attached_conditions_summary": [
{
"table": "t1",
"attached": "t1.col1 < 3 and t1.col2 = 'foo'"
}
"attached_conditions_summary": [
{
"table": "`t1`",
"attached": "((`t1`.`col1` < 3) and (`t1`.`col2` = 'foo'))"
}
●
Date constants are printed as dates, not hex
●
etc
●
The output is correct
− Caution: https://bugs.mysql.com/bug.php?id=95824
“More polish”
vs
41. 41
Conclusions
●
Optimizer Trace is available in MariaDB
●
Shows details about query optimization
− Analyze it yourself
− Submit traces when discussing/reporting optimizer issues
●
MySQL has a similar feature
− Has some extras but they don’t seem important
●
Future
− Print more useful info in the trace
− ...