DS
Uploaded byDave Stokes
PPTX, PDF202 views

Confoo 2021 - MySQL Indexes & Histograms

The document discusses various strategies to optimize MySQL queries, focusing on the use of indexes, histograms, and the query optimizer's functionality to enhance performance. It highlights the importance of proper data normalization, the structure of different types of indexes, and the role of histograms in aiding the optimizer. Additionally, it explains how to utilize the 'EXPLAIN' command to analyze query execution plans and the impact of index management, including the introduction of invisible indexes.

Embed presentation

Downloaded 11 times
1 / 105
2 / 105
3 / 105
4 / 105
5 / 105
6 / 105
7 / 105
8 / 105
9 / 105
10 / 105
11 / 105
12 / 105
13 / 105
14 / 105
15 / 105
16 / 105
17 / 105
18 / 105
19 / 105
20 / 105
21 / 105
22 / 105
23 / 105
24 / 105
25 / 105
26 / 105
27 / 105
28 / 105
29 / 105
30 / 105
31 / 105
32 / 105
33 / 105
34 / 105
35 / 105
36 / 105
37 / 105
38 / 105
39 / 105
40 / 105
41 / 105
42 / 105
43 / 105
44 / 105
45 / 105
46 / 105
47 / 105
48 / 105
49 / 105
50 / 105
51 / 105
52 / 105
53 / 105
54 / 105
55 / 105
56 / 105
57 / 105
58 / 105
59 / 105
60 / 105
61 / 105
62 / 105
63 / 105
64 / 105
65 / 105
66 / 105
67 / 105
68 / 105
69 / 105
70 / 105
71 / 105
72 / 105
73 / 105
74 / 105
75 / 105
76 / 105
77 / 105
78 / 105
79 / 105
80 / 105
81 / 105
82 / 105
83 / 105
84 / 105
85 / 105
86 / 105
87 / 105
88 / 105
89 / 105
90 / 105
91 / 105
92 / 105
93 / 105
94 / 105
95 / 105
96 / 105
97 / 105
98 / 105
99 / 105
100 / 105
101 / 105
102 / 105
103 / 105
104 / 105
105 / 105
MySQL Indexes, Histograms, And other ways To Speed Up Your Queries Dave Stokes @Stoker david.stokes @Oracle.com Community Manager, North America MySQL Community Team Copyright © 2019 Oracle and/or its affiliates. Confoo 2021
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
Warning! MySQL 5.6 End of Life is February of 2021!! 3 WAS
Test Drive MySQL Database Service For Free Today Get $300 in credits and try MySQL Database Service free for 30 days. https://www.oracle.com/cloud/free/ Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted 4
What Is This Session About? Nobody ever complains that the database is too fast! Speeding up queries is not a ‘dark art’ But understanding how to speed up queries is often treated as magic So we will be looking at the proper use of indexes, histograms, locking options, and some other ways to speed queries up. 5
Yes, this is a very dry subject! ● How dry? ○ Very dry ● Lots of text on screen ○ Download slides and use as a reference later ■ slideshare.net/davidmstokes ○ Do not try to absorb all at once ■ Work on most frequent query first, then second most, … ■ And optimizations may need to change over time ● No Coverage today of ○ System configuration ■ OS ■ MySQL ○ Hardware ○ Networking Slides are posted to https://slideshare.net/davestokes 6
Normalize Your Data (also not covered today) ● Can not build a skyscraper on a foundation of sand ● Third normal form or better ○ Use JSON for ‘stub’ table data, avoiding repeated unneeded index/table accesses ● Think about how you will use your data ○ Do not use a fork to eat soup; How do you consume your data Badly normalized data will hurt the performance of your queries. No matter how much training you give it, a dachshund will not be faster than a thoroughbred horse! 7
The Optimizer ● Consider the optimizer the brain and nervous system of the system. ● Query optimization is a feature of many relational database management systems. ● The query optimizer attempts to determine the most efficient way to execute a given query by considering the possible query plans. Wikipedia 8 One of the hardest problems in query optimization is to accurately estimate the costs of alternative query plans. Optimizers cost query plans using a mathematical model of query execution costs that relies heavily on estimates of the cardinality, or number of tuples, flowing through each edge in a query plan. Cardinality estimation in turn depends on estimates of the selection factor of predicates in the query. Traditionally, database systems estimate selectivities through fairly detailed statistics on the distribution of values in each column, such as histograms.
The query optimizer evaluates the options ● The optimizer wants to get your data the cheapest way (least amount of very expensive disk reads) possible. ● Like a GPS, the cost is built on historical statistics. And these statistics can change while the optimizer is working. So like a traffic jam, washed out road, or other traffic problem, the optimizer may be making poor decisions for the present situation. ● The final determination from the optimizer is call the query plan. ● MySQL wants to optimize each query every time it sees it – there is no locking down the query plan like Oracle. {watch for optimizer hints later in this presentation} You will see how to obtain a query plan later in this presentation. 9
120! If your query has five joins then the optimizer may have to evaluate 120 different options 5! (5 * 4 * 3 * 2 * 1) 10
EXPLAIN Explaining EXPLAIN requires a great deal of explanation 11
EXPLAIN Syntax 12 Query optimization is covered in Chapter 8 of the MySQL Manual EXPLAIN reports how the server would process the statement, including information about how tables are joined and in which order.
But now for something completely different The tools for looking at queries ● EXPLAIN ○ EXPLAIN FORMAT= ■ JSON ■ TREE ○ ANALYZE ● VISUAL EXPLAIN 13
EXPLAIN Example 14 EXPLAIN is used to obtain a query execution plan (an explanation of how MySQL would execute a query) and should be considered an ESTIMATE as it does not run the query. QUERY QUERY PLAN DETAILS
VISUAL EXPLAIN from MySQL Workbench 15
EXPLAIN FORMAT=TREE EXPLAIN FORMAT=TREE SELECT * FROM City JOIN Country ON (City.Population = Country.Population); -> Inner hash join (city.Population = country.Population) (cost=100154.05 rows=100093) -> Table scan on City (cost=0.30 rows=4188) -> Hash -> Table scan on Country (cost=29.90 rows=239) 16
EXPLAIN FORMAT=JSON EXPLAIN FORMAT=JSON SELECT * FROM City JOIN Country ON (City.Population = Country.Population)G *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "100154.05" }, "nested_loop": [ { "table": { "table_name": "Country", "access_type": "ALL", "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": "100.00", "cost_info": { "read_cost": "6.00", "eval_cost": "23.90", "prefix_cost": "29.90", "data_read_per_join": "61K" }, 17 "used_columns": [ "Code", "Name", "Continent", "Region", "SurfaceArea", "IndepYear", "Population", "LifeExpectancy", "GNP", "GNPOld", "LocalName", "GovernmentForm", "HeadOfState", "Capital", "Code2" ] } }, { "table": { "table_name": "City", "access_type": "ALL", "rows_examined_per_scan": 4188, "rows_produced_per_join": 100093, "filtered": "10.00", "using_join_buffer": "hash join", "cost_info": { "read_cost": "30.95", "eval_cost": "10009.32", "prefix_cost": "100154.05", "data_read_per_join": "6M" }, "used_columns": [ "ID", "Name", "CountryCode", "District", "Population" ], "attached_condition": "(`world`.`city`.`Population` = `world`.`country`.`Population`)" } } ] } }
EXPLAIN ANALYZE – MySQL 8.0.18 18 EXPLAIN ANALYZE SELECT * FROM City WHERE CountryCode = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=80.76 rows=81) (actual time=0.153..0.178 rows=81 loops=1) 1 row in set (0.0008 sec) MySQL 8.0.18 introduced EXPLAIN ANALYZE, which runs the query and produces EXPLAIN output along with timing and additional, iterator-based, information about how the optimizer's expectations matched the actual execution. Real numbers not an estimate!
More on using EXPLAIN … later 19
Indexes 20
Indexes A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed. Indexes can be created using one or more columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records. - - https://en.wikipedia.org/wiki/Database_index 21
Think of an Index as a Table With Shortcuts to another table! Or a model of some of your data in it’s own table! And the more tables you have the read to get to the data the slower things run! (And the more memory is taken up) 22
Many Types of Indexes
24
25 clustered index The InnoDB term for a primary key index. InnoDB table storage is organized based on the values of the primary key columns, to speed up queries and sorts involving the primary key columns. For best performance, choose the primary key columns carefully based on the most performance-critical queries. Because modifying the columns of the clustered index is an expensive operation, choose primary columns that are rarely or never updated. In the Oracle Database product, this type of table is known as an index-organized table
Think of an index as a small, fast table pointing to a bigger table 26 Index 3 6 12 27 5001 Indexed Column column 1 column x column n 5001 3 6 27 12
How Secondary Indexes Relate to the Clustered Index All indexes other than the clustered index are known as secondary indexes. In InnoDB, each record in a secondary index contains the primary key columns for the row, as well as the columns specified for the secondary index. InnoDB uses this primary key value to search for the row in the clustered index. If the primary key is long, the secondary indexes use more space, so it is advantageous to have a short primary key. 27
28 Creating a table with a PRIMARY KEY (index) CREATE TABLE t1 ( c1 INT NOT NULL AUTO_INCREMENT PRIMARY KEY, c2 VARCHAR(100), c3 VARCHAR(100) ); An index is a list of KEYs. And you will hear ‘key’ and ‘index’ used interchangeably
PRIMARY KEY This is an key for the index that uniquely defined for a row, should be immutable. InnoDB needs a PRIMARY KEY (and will make one up if you do not specify) No Null Values Monotonically increasing - use UUID_To_BIN() if you must use UUIDS, otherwise avoid them 29
Indexing on a prefix of a column CREATE INDEX part_of_name ON customer (name(10)); Only the first 10 characters are indexed in this examples and this can save space/speed 30
Multi-column index - last_name, first_name CREATE TABLE test ( id INT NOT NULL, last_name CHAR(30) NOT NULL, first_name CHAR(30) NOT NULL, PRIMARY KEY (id), INDEX name (last_name,first_name) ); 31 This index will work on (last_name,first_name) and (last_name) but not (first_name) {Use: left to right} Put highest cardinality field first
Hashing values - an alternative SELECT * FROM tbl_name WHERE hash_col=MD5(CONCAT(val1,val2)) AND col1=val1 AND col2=val2; As an alternative to a composite index, you can introduce a column that is “hashed” based on information from other columns. If this column is short, reasonably unique, and indexed, it might be faster than a “wide” index on many columns. 32
Some other types of indexes Unique Indexes - only one row per value Full-Text Indexes - search string data Covering index - includes all columns needed for a query Secondary index - another column in the table is indexes Spatial Indexes - geographical information 33
Functional Indexes are defined on the result of a function applied to one or more columns of a single table CREATE TABLE t1 (col1 INT, col2 INT, INDEX func_index ((ABS(col1)))); CREATE INDEX idx1 ON t1 ((col1 + col2)); CREATE INDEX idx2 ON t1 ((col1 + col2), (col1 - col2), col1); ALTER TABLE t1 ADD INDEX ((col1 * 40) DESC); 34
Multi-value indexes You can now have more index pointers than index keys! ○ Very useful for JSON arrays mysql> SELECT 3 MEMBER OF('[1, 3, 5, 7, "Moe"]'); +--------------------------------------+ | 3 MEMBER OF('[1, 3, 5, 7, "Moe"]') | +--------------------------------------+ | 1 | +--------------------------------------+ 35
MySQL has Two Main Types of Index Structures B-Tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions. 36 Hash are more efficient than nested loops joins, except when the probe side of the join is very small but joins can only be used to compute equijoins.
Please Keep in mind ... If there is a choice between multiple indexes, MySQL normally uses the index that finds the smallest number of rows (the most selective index). MySQL can use indexes on columns more efficiently if they are declared as the same type and size. ● In this context, VARCHAR and CHAR are considered the same if they are declared as the same size. For example, VARCHAR(10) and CHAR(10) are the same size, but VARCHAR(10) and CHAR(15) are not. For comparisons between nonbinary string columns, both columns should use the same character set. For example, comparing a utf8 column with a latin1 column precludes use of an index. 37
NULL A few words on something that may not be there! 38
NULL ● NULL is used to designate a LACK of data. False 0 True 1 Don’t know NULL 39
Indexing NULL values really drives down the performance of INDEXes - Cardinal Values and then a ‘junk drawer’ of nulls -> hard to search 40
Invisible Indexes 41
Before Invisible Indexes 1. Doubt usefulness of index 2. Check using EXPLAIN 3. Remove that Index 4. Rerun EXPLAIN 5. Get phone/text/screams from power user about slow query 6. Suddenly realize that the index in question may have had no use for you but the rest of the planet seems to need that dang index! 7. Take seconds/minutes/hours/days/weeks rebuilding that index 42
After Invisible Indexes 1. Doubt usefulness of index 2. Check using EXPLAIN 3. Make index invisible – optimizer can not see that index! 4. Rerun EXPLAIN 5. Get phone/text/screams from power user about slow query 6. Make index visible 7. Blame problem on { network | JavaScript | GDPR | Slack | Cloud } Sys schema will show you which indexes have not been used that may be candidates for removal -- but be cautious! 43
How to use INVISIBLE INDEX ALTER TABLE t1 ALTER INDEX i_idx INVISIBLE; ALTER TABLE t1 ALTER INDEX i_idx VISIBLE; 44
Histograms What is a histogram? It is not a gluten free, keto friendly biscuit. 45
Histograms 46
Histograms What is a histogram? Wikipedia declares a histogram is an accurate representation of the distribution of numerical data. For RDBMS, a histogram is an approximation of the data distribution within a specific column. So in MySQL, histograms help the optimizer to find the most efficient Query Plan to fetch that data. 47
Histograms What is a histogram? A histogram is a distribution of data into logical buckets. There are two types: ● Singleton ● Equi-Height Maximum number of buckets is 1024 48
Histogram Histogram statistics are useful primarily for non-indexed columns. A histogram is created or updated only on demand, so it adds no overhead when table data is modified. On the other hand, the statistics become progressively more out of date when table modifications occur, until the next time they are updated. 49
The two reasons for why you might consider a histogram instead of an index: Maintaining an indexes have a cost. If you have an index, every ● Every INSERT/UPDATE/DELETE causes the index to be updated. This will have an impact on your performance. A histogram on the other hand is created once and never updated unless you explicitly ask for it. It will thus not hurt your INSERT/UPDATE/DELETE-performance. ● The optimizer will make “index dives” to estimate the number of records in a given range. This might become too costly if you have for instance very long IN-lists in your query. Histogram statistics are much cheaper in this case, and might thus be more suitable. 50
The Optimizer Occasionally the query optimizer fails to find the most efficient plan and ends up spending a lot more time executing the query than necessary. The optimizer assumes that the data is evenly distributed in the column. This can be the old ‘assume’ makes an ‘ass out of you and me’ joke brought to life. The main reason for this is often that the optimizer doesn’t have enough knowledge about the data it is about to query: ● How many rows are there in each table? ● How many distinct values are there in each column? ● How is the data distributed in each column? 51
Two-types of histograms Equi-height: One bucket represents a range of values. This type of histogram will be created when distinct values in the column are greater than the number of buckets specified in the analyze table syntax. Think A-G H-L M-T U-Z Singleton: One bucket represents one single value in the column and is the most accurate and will be created when the number of distinct values in the column is less than or equal to the number of buckets specified in the analyze table syntax. 52
Frequency Histogram Each distinct value has its own bucket. 53 101 102 104 insert into freq_histogram (x) values (101),(101),(102),(102),(102),(104);
Frequency Histogram - an Example select id,x from freq_histogram ; +----+-----+ | id | x | +----+-----+ | 1 | 101 | | 2 | 101 | | 3 | 102 | | 4 | 102 | | 5 | 102 | | 6 | 104 | +----+-----+ analyze table freq_histogram update histogram on x with 3 buckets; 54 select JSON_PRETTY(HISTOGRAM->>"$") from information_schema.column_statistics where table_name='freq_histogram' and column_name='x'G *************************** 1. row *************************** JSON_PRETTY(HISTOGRAM->>"$"): { "buckets": [ [ 101, 0.3333333333333333 ], [ 102, 0.8333333333333333 ], [ 104, 1.0 ] ], "data-type": "int", "null-values": 0.0, "collation-id": 8, "last-updated": "2020-06-16 19:54:21.033017", "sampling-rate": 1.0, "histogram-type": "singleton", "number-of-buckets-specified": 3 }
The statistics SELECT (SUBSTRING_INDEX(v, ':', -1)) value, concat(round(c*100,1),'%') cumulfreq, CONCAT(round((c - LAG(c, 1, 0) over()) * 100,1), '%') freq FROM information_schema.column_statistics, JSON_TABLE(histogram->'$.buckets','$[*]' COLUMNS(v VARCHAR(60) PATH '$[0]', c double PATH '$[1]')) hist WHERE schema_name = 'demo' and table_name = 'freq_histogram' and column_name = 'x'; +-------+-----------+-------+ | value | cumulfreq | freq | +-------+-----------+-------+ | 101 | 33.3% | 33.3% | | 102 | 83.3% | 50.0% | | 104 | 100.0% | 16.7% | +-------+-----------+-------+ 55
Creating and removing Histograms ANALYZE TABLE t UPDATE HISTOGRAM ON c1, c2, c3 WITH 10 BUCKETS; ANALYZE TABLE t UPDATE HISTOGRAM ON c1, c3 WITH 10 BUCKETS; ANALYZE TABLE t DROP HISTOGRAM ON c2; Note the first statement creates three different histograms on c1, c2, and c3. 56
Information about Histograms mysql> SELECT TABLE_NAME, COLUMN_NAME, HISTOGRAM->>'$."data-type"' AS 'data-type', JSON_LENGTH(HISTOGRAM->>'$."buckets"') AS 'bucket-count' FROM INFORMATION_SCHEMA.COLUMN_STATISTICS; +-----------------+-------------+-----------+--------------+ | TABLE_NAME | COLUMN_NAME | data-type | bucket-count | +-----------------+-------------+-----------+--------------+ | country | Population | int | 226 | | city | Population | int | 1024 | | countrylan | Language | string | 457 | +-----------------+-------------+-----------+--------------+ 57
Where Histograms Shine create table h1 (id int unsigned auto_increment, x int unsigned, primary key(id)); insert into h1 (x) values (1),(1),(2),(2),(2),(3),(3),(3),(3); select x, count(x) from h1 group by x; +---+----------+ | x | count(x) | +---+----------+ | 1 | 2 | | 2 | 3 | | 3 | 4 | +---+----------+ 3 rows in set (0.0011 sec) 58
Without Histogram EXPLAIN SELECT * FROM h1 WHERE x > 0G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: h1 partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 9 filtered: 33.32999801635742 – The optimizer estimates about 1/3 of the data Extra: Using where will go match the ‘X > 0’ condition 1 row in set, 1 warning (0.0007 sec) 59 The filtered column indicates an estimated percentage of table rows that will be filtered by the table condition. The maximum value is 100, which means no filtering of rows occurred. Values decreasing from 100 indicate increasing amounts of filtering. rows shows the estimated number of rows examined and rows × filtered shows the number of rows that will be joined with the following table. For example, if rows is 1000 and filtered is 50.00 (50%), the number of rows to be joined with the following table is 1000 × 50% = 500.
Where Histograms Shine ANALYZE TABLE h1 UPDATE HISTOGRAM ON x WITH 3 BUCKETSG *************************** 1. row *************************** Table: demox.h1 Op: histogram Msg_type: status Msg_text: Histogram statistics created for column 'x'. 1 row in set (0.0819 sec) 60
Where Histograms Shine EXPLAIN SELECT * FROM h1 WHERE x > 0G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: h1 partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 9 filtered: 100 – all rows!!! Extra: Using where 1 row in set, 1 warning (0.0007 sec) 61
With EXPLAIN ANALYSE EXPLAIN analyze SELECT * FROM h1 WHERE x > 0G *************************** 1. row *************************** EXPLAIN: -> Filter: (h1.x > 0) (cost=1.15 rows=9) (actual time=0.027..0.034 rows=9 loops=1) -> Table scan on h1 (cost=1.15 rows=9) (actual time=0.025..0.030 rows=9 loops=1) 62
Performance is not just Indexes and Histograms ● There are many other ‘tweaks’ that can be made to speed things up ● Use explain to see what your query is doing? ○ File sorts, full table scans, using temporary tables, etc. ○ Does the join order look right ○ Buffers and caches big enough ○ Do you have enough memory ○ Disk and I/O speeds sufficient 63
Locking Options MySQL added two locking options to MySQL 8.0: ● NOWAIT ○ A locking read that uses NOWAIT never waits to acquire a row lock. The query executes immediately, failing with an error if a requested row is locked. ● SKIP LOCKED ○ A locking read that uses SKIP LOCKED never waits to acquire a row lock. The query executes immediately, removing locked rows from the result set. 64
Locking Examples – Buying concert tickets START TRANSACTION; SELECT seat_no, row_no, cost FROM seats s JOIN seat_rows sr USING ( row_no ) WHERE seat_no IN ( 3,4 ) AND sr.row_no IN ( 5,6 ) AND booked = 'NO' FOR UPDATE OF s SKIP LOCKED; Let’s shop for tickets in rows 5 or 6 and seats 3 & 4 but we skip any locked records! 65
LOCK NOWAIT START TRANSACTION; SELECT seat_no FROM seats JOIN seat_rows USING ( row_no ) WHERE seat_no IN (3,4) AND seat_rows.row_no IN (12) AND booked = 'NO' FOR UPDATE OF seats SKIP LOCKED FOR SHARE OF seat_rows NOWAIT; Without NOWAIT, this query would have waited for innodb_lock_wait_timeout (default: 50) seconds while attempting to acquire the shared lock on seat_rows. With NOWAIT an error is immediately returned ERROR 3572 (HY000): Do not wait for lock. 66
Other Fast Ways Resource Groups Optimizer hints Partitioning Multi Value Indexes 67
Resource groups – setting and using CREATE RESOURCE GROUP Batch TYPE = USER VCPU = 2-3 -- assumes a system with at least 4 CPUs THREAD_PRIORITY = 10; SET RESOURCE GROUP Batch; or INSERT /*+ RESOURCE_GROUP(Batch) */ INTO t2 VALUES(2); 68
Optimizer Hints SELECT /*+ JOIN_ORDER(t1, t2) */ ... FROM t1, t2; Optimizer hints can be specified within individual statements. Because the optimizer hints apply on a per-statement basis, they provide finer control over statement execution plans than can be achieved using optimizer_switch. For example, you can enable an optimization for one table in a statement and disable the optimization for a different table. 69
Partitioning ● In MySQL 8.0, partitioning support is provided by the InnoDB and NDB storage engines. ● Partitioning enables you to distribute portions of individual tables across a file system according to rules which you can set largely as needed. In effect, different portions of a table are stored as separate tables in different locations. 70
The Query EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 71
The Query – What we want EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 72
The Query – Where we want it from EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 73
The Query – How the two tables relate EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 74
The Query – And any filters EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 75
76 EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: const possible_keys: PRIMARY key: PRIMARY key_len: 3 ref: const rows: 1 filtered: 100 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: City partitions: NULL type: ref possible_keys: CountryCode key: CountryCode key_len: 3 ref: const rows: 81 filtered: 100 Extra: NULL 2 rows in set, 1 warning (0.0013 sec)
The Actual Query Plan select `world`.`city`.`Name` AS `name`, 'United Kingdom' AS `Name` from `world`.`city` join `world`.`country` where (`world`.`city`.`CountryCode` = 'GBR') 77
Optimizer substituted Country.name select `world`.`city`.`Name` AS `name`, 'United Kingdom' AS `Name` from `world`.`city` join `world`.`country` where (`world`.`city`.`CountryCode` = 'GBR') 78
Optimizer substituted Country.name explain format=tree SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=26.85 rows=81) So the optimizer ‘knows’ just to grab the ‘GBR’ records from the City table and does not need to read the Country table at all. 79
Optimizer substituted Country.name Explain ANALYZE SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=26.85 rows=81) (actual time=0.123..0.138 rows=81 loops=1) 1 row in set (0.0009 sec) Explain ANALYZE runs the query and the estimate is pretty good! 80
81 EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: const possible_keys: PRIMARY key: PRIMARY key_len: 3 ref: const This is a CONSTANT rows: 1 filtered: 100 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: City partitions: NULL type: ref possible_keys: CountryCode key: CountryCode key_len: 3 ref: const rows: 81 There are 81 records that match in the index filtered: 100 Extra: NULL 2 rows in set, 1 warning (0.0013 sec)
82
Some General Rules ● Look at indexing/histogramming columns on right side of WHERE clause ● Maybe index SORT BY columns (test) ● JOIN on like type and size columns ○ i.e. No VARCHAR(32) to DECIMAL matches ● Can the data be found in an index? 83
Slightly More Complex SELECT CONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 84
And yes the output of EXPLAIN can be hard to read | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | 1 | SIMPLE | rental | NULL | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | customer | NULL | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | 1 | SIMPLE | address | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | 1 | SIMPLE | inventory | NULL | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | 1 | SIMPLE | film | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 85
Visual Explain 86
Tree format is often little better Limit: 5 row(s) -> Sort: <temporary>.title, limit input to 5 row(s) per chunk -> Stream results -> Nested loop inner join (cost=3866.17 rows=1601) -> Nested loop inner join (cost=3305.89 rows=1601) -> Nested loop inner join (cost=2745.61 rows=1601) -> Nested loop inner join (cost=2185.33 rows=1601) -> Filter: (rental.return_date is null) (cost=1625.05 rows=1601) -> Table scan on rental (cost=1625.05 rows=16008) -> Single-row index lookup on customer using PRIMARY (customer_id=rental.customer_id) (cost=0.25 rows=1) -> Single-row index lookup on address using PRIMARY (address_id=customer.address_id) (cost=0.25 rows=1) -> Single-row index lookup on inventory using PRIMARY (inventory_id=rental.inventory_id) (cost=0.25 rows=1) -> Filter: ((rental.rental_date + interval film.rental_duration day) < <cache>(curdate())) (cost=0.25 rows=1) -> Single-row index lookup on film using PRIMARY (film_id=inventory.film_id) (cost=0.25 rows=1) 87
Back to the old tabular EXPLAIN | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | 1 | SIMPLE | rental | NULL | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | customer | NULL | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | 1 | SIMPLE | address | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | 1 | SIMPLE | inventory | NULL | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | 1 | SIMPLE | film | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 88
Back to the old tabular EXPLAIN, cleaned up | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 89
First glance – what indexes are being used? | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 90 Anytime the type column shows all it means the entire file/table must be read, which is slow. If it does have to read all the rows is it because there are no indexes available? Or does it have to read all the table because is processing the entire table?
Slightly More Complex SELECT CONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 91
Quiz: Why so slow if it only has to return FIVE RECORDS?!?!?!?! SELECT CONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 92
Second glance – what EXTRAS are being used? | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY 100 | NULL | | film | eq_ref | PRIMARY 93 The where; Using temporary; Using filesort informs that the output has to be sorted (part of the sort clause) and that a temporary table needed to be used.
MySQL 8.0 Temporary Table much faster ● Previously temporary tables were size limited and when they reached that limit: ○ The processing was halted ○ The data was copied to InnoDB ○ The processing continued with the InnoDB copy of the data ○ And the above was slow 94
Is there something unindexed that need to be indexed? ● Or something we can use to build a histogram? ● Is there a better way to make the key? 95
SHOW INDEX FROM rental ● PRIMARY ● Rental_date – rental_date, inventory_id, customer_id ● Idx_fk_inventory_id – inventory_id, cutomer_id ● Idx_fk_staff_id 96
SHOW INDEX FROM film ● PRIMARY ● Idx_title ● Idx_fk_language_id ● Idx_fk_original_language_id 97
A Functional Index? Or Generated Column? WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; Could this part in red be reduced to a functional index/generated column? AND return_date < CURRENT_DATE() film.rental_duration + rental_date Would want to store this in the record with the rental data? 98
Yes, let us add that new column ● ALTER TABLE operations can be expensive ● How do we seed data ● Do we use a GENERATED COLUMN? Can we? ● Do we add a stub table? ○ Extra index/table dives ○ How much code do we have to change? ○ Other considerations 99
Where Else To Look for Information ● MySQL Manual ● Forums.MySQL.com ● MySQLcommunity.slack.com 100
New Book ● YOU NEED THESE BOOKS!! ● Author’s others work are also outstanding and well written ● Amazing detail!! 101
Great Book ● Getting very dated ● Make sure you get the 3rd edition ● Can be found in used book stores 102
My Book ● A guide to using JSON and MySQL ○ Programming Examples ○ Best practices ○ Twice the length of the original ○ NoSQL Document Store how-to 103
“We have saved around 40% of our costs and are able to reinvest that back into the business. And we are scaling across EMEA, and that’s basically all because of Oracle.” —Asser Smidt CEO and Cofounder, BotSupply Startups get cloud credits and a 70% discount for 2 years, global exposure via marketing, events, digital promotion, and media, plus access to mentorship, capital and Oracle’s 430,000+ customers Customers meet vetted startups in transformative spaces that help them stay ahead of their competition Oracle stays at the competitive edge of innovation with solutions that complement its technology stack Oracle for Startups - enroll at oracle.com/startup A Virtuous Cycle of Innovation, Everybody Wins.
Thank you ● David.Stokes @Oracle.com ● @Stoker ● PHP Related blog https://elephantdolphin.blogspot.com/ ● MySQL Latest Blogs https://planet.mysql.com/ ● Where to ask questions https://forums.mysql.com/ ● Slack https://mysqlcommunity.slack.com slides -> slideshare.net/davestokes 105

More Related Content

PPTX
PHP UK 2020 Tutorial: MySQL Indexes, Histograms And other ways To Speed Up Yo...
byDave Stokes
 
PDF
Dutch PHP Conference 2021 - MySQL Indexes and Histograms
byDave Stokes
 
PPTX
cPanel now supports MySQL 8.0 - My Top Seven Features
byDave Stokes
 
PDF
Longhorn PHP - MySQL Indexes, Histograms, Locking Options, and Other Ways to ...
byDave Stokes
 
PDF
Json within a relational database
byDave Stokes
 
PPTX
Open Source 1010 and Quest InSync presentations March 30th, 2021 on MySQL Ind...
byDave Stokes
 
PDF
Datacon LA - MySQL without the SQL - Oh my!
byDave Stokes
 
PPTX
A Step by Step Introduction to the MySQL Document Store
byDave Stokes
 
PHP UK 2020 Tutorial: MySQL Indexes, Histograms And other ways To Speed Up Yo...
byDave Stokes
 
Dutch PHP Conference 2021 - MySQL Indexes and Histograms
byDave Stokes
 
cPanel now supports MySQL 8.0 - My Top Seven Features
byDave Stokes
 
Longhorn PHP - MySQL Indexes, Histograms, Locking Options, and Other Ways to ...
byDave Stokes
 
Json within a relational database
byDave Stokes
 
Open Source 1010 and Quest InSync presentations March 30th, 2021 on MySQL Ind...
byDave Stokes
 
Datacon LA - MySQL without the SQL - Oh my!
byDave Stokes
 
A Step by Step Introduction to the MySQL Document Store
byDave Stokes
 

What's hot

PDF
JavaScript and Friends August 20th, 20201 -- MySQL Shell and JavaScript
byDave Stokes
 
PDF
MySQL 8.0 New Features -- September 27th presentation for Open Source Summit
byDave Stokes
 
PDF
MySQL Replication Update - DEbconf 2020 presentation
byDave Stokes
 
PPTX
Discover the Power of the NoSQL + SQL with MySQL
byDave Stokes
 
PPTX
Confoo 2021 -- MySQL New Features
byDave Stokes
 
PDF
Advanced MySQL Query Optimizations
byDave Stokes
 
PDF
Open Source World June '21 -- JSON Within a Relational Database
byDave Stokes
 
PPTX
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)
byDave Stokes
 
PDF
MySQL 8.0 Features -- Oracle CodeOne 2019, All Things Open 2019
byDave Stokes
 
PDF
MySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
byDave Stokes
 
PDF
MySQL 8.0 Operational Changes
byDave Stokes
 
PDF
MySQL Baics - Texas Linxufest beginners tutorial May 31st, 2019
byDave Stokes
 
PPTX
MySQL Replication Evolution -- Confoo Montreal 2017
byDave Stokes
 
PDF
Developers’ mDay 2021: Bogdan Kecman, Oracle – MySQL nekad i sad
bymCloud
 
PDF
MySQL Indexes and Histograms - RMOUG Training Days 2022
byDave Stokes
 
PPTX
OTN TOUR 2016 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
PPTX
OOW16 - Oracle Database 12c - The Best Oracle Database 12c New Features for D...
byAlex Zaballa
 
PPTX
Oracle Database 12c Release 2 - New Features On Oracle Database Exadata Expre...
byAlex Zaballa
 
PPTX
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
PPTX
Postgresql
byNexThoughts Technologies
 
JavaScript and Friends August 20th, 20201 -- MySQL Shell and JavaScript
byDave Stokes
 
MySQL 8.0 New Features -- September 27th presentation for Open Source Summit
byDave Stokes
 
MySQL Replication Update - DEbconf 2020 presentation
byDave Stokes
 
Discover the Power of the NoSQL + SQL with MySQL
byDave Stokes
 
Confoo 2021 -- MySQL New Features
byDave Stokes
 
Advanced MySQL Query Optimizations
byDave Stokes
 
Open Source World June '21 -- JSON Within a Relational Database
byDave Stokes
 
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)
byDave Stokes
 
MySQL 8.0 Features -- Oracle CodeOne 2019, All Things Open 2019
byDave Stokes
 
MySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
byDave Stokes
 
MySQL 8.0 Operational Changes
byDave Stokes
 
MySQL Baics - Texas Linxufest beginners tutorial May 31st, 2019
byDave Stokes
 
MySQL Replication Evolution -- Confoo Montreal 2017
byDave Stokes
 
Developers’ mDay 2021: Bogdan Kecman, Oracle – MySQL nekad i sad
bymCloud
 
MySQL Indexes and Histograms - RMOUG Training Days 2022
byDave Stokes
 
OTN TOUR 2016 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
OOW16 - Oracle Database 12c - The Best Oracle Database 12c New Features for D...
byAlex Zaballa
 
Oracle Database 12c Release 2 - New Features On Oracle Database Exadata Expre...
byAlex Zaballa
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
Postgresql
byNexThoughts Technologies
 

Similar to Confoo 2021 - MySQL Indexes & Histograms

PDF
MySQL 8.0 Optimizer Guide
byMorgan Tocker
 
PPTX
Optimizing MySQL queries
byGMO-Z.com Vietnam Lab Center
 
PDF
MySQL Query Optimisation 101
byFederico Razzoli
 
PDF
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
PDF
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
PDF
Introduction to MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
PDF
Introduction into MySQL Query Tuning
bySveta Smirnova
 
PPTX
What Your Database Query is Really Doing
byDave Stokes
 
PPTX
Guide To Mastering The MySQL Query Execution Plan
byOptimiz DBA
 
PDF
Why Use EXPLAIN FORMAT=JSON?
bySveta Smirnova
 
PDF
Zurich2007 MySQL Query Optimization
byHiệp Lê Tuấn
 
PDF
Zurich2007 MySQL Query Optimization
byHiệp Lê Tuấn
 
PDF
MySQL Performance Optimization
byMindfire Solutions
 
PDF
How to analyze and tune sql queries for better performance vts2016
byoysteing
 
PDF
query optimization
byDimara Hakim
 
PDF
How to analyze and tune sql queries for better performance
byoysteing
 
PPTX
MySQL performance tuning
byAnurag Srivastava
 
PDF
Optimizer overviewoow2014
byMysql User Camp
 
PDF
MySQL Query Optimization
byMorgan Tocker
 
PDF
Optimized cluster index generation
byRutvik Pensionwar
 
MySQL 8.0 Optimizer Guide
byMorgan Tocker
 
Optimizing MySQL queries
byGMO-Z.com Vietnam Lab Center
 
MySQL Query Optimisation 101
byFederico Razzoli
 
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
Introduction to MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
Introduction into MySQL Query Tuning
bySveta Smirnova
 
What Your Database Query is Really Doing
byDave Stokes
 
Guide To Mastering The MySQL Query Execution Plan
byOptimiz DBA
 
Why Use EXPLAIN FORMAT=JSON?
bySveta Smirnova
 
Zurich2007 MySQL Query Optimization
byHiệp Lê Tuấn
 
Zurich2007 MySQL Query Optimization
byHiệp Lê Tuấn
 
MySQL Performance Optimization
byMindfire Solutions
 
How to analyze and tune sql queries for better performance vts2016
byoysteing
 
query optimization
byDimara Hakim
 
How to analyze and tune sql queries for better performance
byoysteing
 
MySQL performance tuning
byAnurag Srivastava
 
Optimizer overviewoow2014
byMysql User Camp
 
MySQL Query Optimization
byMorgan Tocker
 
Optimized cluster index generation
byRutvik Pensionwar
 

More from Dave Stokes

PDF
Database basics for new-ish developers -- All Things Open October 18th 2021
byDave Stokes
 
PDF
Php &amp; my sql - how do pdo, mysq-li, and x devapi do what they do
byDave Stokes
 
PPTX
Validating JSON -- Percona Live 2021 presentation
byDave Stokes
 
PDF
Midwest PHP Presentation - New MSQL Features
byDave Stokes
 
PDF
Data Love Conference - Window Functions for Database Analytics
byDave Stokes
 
PPTX
Discover The Power of NoSQL + MySQL with MySQL
byDave Stokes
 
PDF
Confoo 202 - MySQL Group Replication and ReplicaSet
byDave Stokes
 
PDF
MySQL New Features -- Sunshine PHP 2020 Presentation
byDave Stokes
 
PPTX
MySQL 8.0 from December London Open Source Database Meetup
byDave Stokes
 
PPTX
MySQL 8 - UKOUG Techfest Brighton December 2nd, 2019
byDave Stokes
 
PPTX
Upgrading to MySQL 8.0 webinar slides November 27th, 2019
byDave Stokes
 
PDF
Windowing Functions - Little Rock Tech Fest 2019
byDave Stokes
 
PDF
Oracle CodeOne Foreign Keys Support in MySQL 8.0
byDave Stokes
 
PDF
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
byDave Stokes
 
PDF
MySQL 8.0 Graphical Information System - Mid Atlantic Developers Conference
byDave Stokes
 
Database basics for new-ish developers -- All Things Open October 18th 2021
byDave Stokes
 
Php &amp; my sql - how do pdo, mysq-li, and x devapi do what they do
byDave Stokes
 
Validating JSON -- Percona Live 2021 presentation
byDave Stokes
 
Midwest PHP Presentation - New MSQL Features
byDave Stokes
 
Data Love Conference - Window Functions for Database Analytics
byDave Stokes
 
Discover The Power of NoSQL + MySQL with MySQL
byDave Stokes
 
Confoo 202 - MySQL Group Replication and ReplicaSet
byDave Stokes
 
MySQL New Features -- Sunshine PHP 2020 Presentation
byDave Stokes
 
MySQL 8.0 from December London Open Source Database Meetup
byDave Stokes
 
MySQL 8 - UKOUG Techfest Brighton December 2nd, 2019
byDave Stokes
 
Upgrading to MySQL 8.0 webinar slides November 27th, 2019
byDave Stokes
 
Windowing Functions - Little Rock Tech Fest 2019
byDave Stokes
 
Oracle CodeOne Foreign Keys Support in MySQL 8.0
byDave Stokes
 
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
byDave Stokes
 
MySQL 8.0 Graphical Information System - Mid Atlantic Developers Conference
byDave Stokes
 

Recently uploaded

PDF
Self-Correction Failure Diagnostic: Detecting Drift in Complex Systems
bySystems Research Group
 
PDF
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
PPTX
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
PDF
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
PDF
GenerationAI Paris 2025 | City of Light (COL)
byapidays
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PDF
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
PDF
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PDF
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
PPTX
TravelTech Paris 2025 | Beyond the pipes: Why Channel Management isn’t really...
byapidays
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
Chapter 4 Network Security in computer security
byGetnet Tigabie Askale -(GM)
 
PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PDF
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
PDF
Configure and Manage Systemd Timers- RHCSA (RH134).pdf
byLinuxCert Guru
 
Self-Correction Failure Diagnostic: Detecting Drift in Complex Systems
bySystems Research Group
 
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
GenerationAI Paris 2025 | City of Light (COL)
byapidays
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
TravelTech Paris 2025 | Beyond the pipes: Why Channel Management isn’t really...
byapidays
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
Chapter 4 Network Security in computer security
byGetnet Tigabie Askale -(GM)
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
Configure and Manage Systemd Timers- RHCSA (RH134).pdf
byLinuxCert Guru
 

Confoo 2021 - MySQL Indexes & Histograms

  • 1.
    MySQL Indexes, Histograms, And other ways ToSpeed Up Your Queries Dave Stokes @Stoker david.stokes @Oracle.com Community Manager, North America MySQL Community Team Copyright © 2019 Oracle and/or its affiliates. Confoo 2021
  • 2.
    The following isintended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  • 3.
    Warning! MySQL 5.6 Endof Life is February of 2021!! 3 WAS
  • 4.
    Test Drive MySQL DatabaseService For Free Today Get $300 in credits and try MySQL Database Service free for 30 days. https://www.oracle.com/cloud/free/ Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted 4
  • 5.
    What Is ThisSession About? Nobody ever complains that the database is too fast! Speeding up queries is not a ‘dark art’ But understanding how to speed up queries is often treated as magic So we will be looking at the proper use of indexes, histograms, locking options, and some other ways to speed queries up. 5
  • 6.
    Yes, this isa very dry subject! ● How dry? ○ Very dry ● Lots of text on screen ○ Download slides and use as a reference later ■ slideshare.net/davidmstokes ○ Do not try to absorb all at once ■ Work on most frequent query first, then second most, … ■ And optimizations may need to change over time ● No Coverage today of ○ System configuration ■ OS ■ MySQL ○ Hardware ○ Networking Slides are posted to https://slideshare.net/davestokes 6
  • 7.
    Normalize Your Data(also not covered today) ● Can not build a skyscraper on a foundation of sand ● Third normal form or better ○ Use JSON for ‘stub’ table data, avoiding repeated unneeded index/table accesses ● Think about how you will use your data ○ Do not use a fork to eat soup; How do you consume your data Badly normalized data will hurt the performance of your queries. No matter how much training you give it, a dachshund will not be faster than a thoroughbred horse! 7
  • 8.
    The Optimizer ● Considerthe optimizer the brain and nervous system of the system. ● Query optimization is a feature of many relational database management systems. ● The query optimizer attempts to determine the most efficient way to execute a given query by considering the possible query plans. Wikipedia 8 One of the hardest problems in query optimization is to accurately estimate the costs of alternative query plans. Optimizers cost query plans using a mathematical model of query execution costs that relies heavily on estimates of the cardinality, or number of tuples, flowing through each edge in a query plan. Cardinality estimation in turn depends on estimates of the selection factor of predicates in the query. Traditionally, database systems estimate selectivities through fairly detailed statistics on the distribution of values in each column, such as histograms.
  • 9.
    The query optimizerevaluates the options ● The optimizer wants to get your data the cheapest way (least amount of very expensive disk reads) possible. ● Like a GPS, the cost is built on historical statistics. And these statistics can change while the optimizer is working. So like a traffic jam, washed out road, or other traffic problem, the optimizer may be making poor decisions for the present situation. ● The final determination from the optimizer is call the query plan. ● MySQL wants to optimize each query every time it sees it – there is no locking down the query plan like Oracle. {watch for optimizer hints later in this presentation} You will see how to obtain a query plan later in this presentation. 9
  • 10.
    120! If your queryhas five joins then the optimizer may have to evaluate 120 different options 5! (5 * 4 * 3 * 2 * 1) 10
  • 11.
    EXPLAIN Explaining EXPLAIN requiresa great deal of explanation 11
  • 12.
    EXPLAIN Syntax 12 Query optimizationis covered in Chapter 8 of the MySQL Manual EXPLAIN reports how the server would process the statement, including information about how tables are joined and in which order.
  • 13.
    But now forsomething completely different The tools for looking at queries ● EXPLAIN ○ EXPLAIN FORMAT= ■ JSON ■ TREE ○ ANALYZE ● VISUAL EXPLAIN 13
  • 14.
    EXPLAIN Example 14 EXPLAIN isused to obtain a query execution plan (an explanation of how MySQL would execute a query) and should be considered an ESTIMATE as it does not run the query. QUERY QUERY PLAN DETAILS
  • 15.
    VISUAL EXPLAIN fromMySQL Workbench 15
  • 16.
    EXPLAIN FORMAT=TREE EXPLAIN FORMAT=TREESELECT * FROM City JOIN Country ON (City.Population = Country.Population); -> Inner hash join (city.Population = country.Population) (cost=100154.05 rows=100093) -> Table scan on City (cost=0.30 rows=4188) -> Hash -> Table scan on Country (cost=29.90 rows=239) 16
  • 17.
    EXPLAIN FORMAT=JSON EXPLAIN FORMAT=JSONSELECT * FROM City JOIN Country ON (City.Population = Country.Population)G *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "100154.05" }, "nested_loop": [ { "table": { "table_name": "Country", "access_type": "ALL", "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": "100.00", "cost_info": { "read_cost": "6.00", "eval_cost": "23.90", "prefix_cost": "29.90", "data_read_per_join": "61K" }, 17 "used_columns": [ "Code", "Name", "Continent", "Region", "SurfaceArea", "IndepYear", "Population", "LifeExpectancy", "GNP", "GNPOld", "LocalName", "GovernmentForm", "HeadOfState", "Capital", "Code2" ] } }, { "table": { "table_name": "City", "access_type": "ALL", "rows_examined_per_scan": 4188, "rows_produced_per_join": 100093, "filtered": "10.00", "using_join_buffer": "hash join", "cost_info": { "read_cost": "30.95", "eval_cost": "10009.32", "prefix_cost": "100154.05", "data_read_per_join": "6M" }, "used_columns": [ "ID", "Name", "CountryCode", "District", "Population" ], "attached_condition": "(`world`.`city`.`Population` = `world`.`country`.`Population`)" } } ] } }
  • 18.
    EXPLAIN ANALYZE –MySQL 8.0.18 18 EXPLAIN ANALYZE SELECT * FROM City WHERE CountryCode = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=80.76 rows=81) (actual time=0.153..0.178 rows=81 loops=1) 1 row in set (0.0008 sec) MySQL 8.0.18 introduced EXPLAIN ANALYZE, which runs the query and produces EXPLAIN output along with timing and additional, iterator-based, information about how the optimizer's expectations matched the actual execution. Real numbers not an estimate!
  • 19.
  • 20.
  • 21.
    Indexes A database indexis a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed. Indexes can be created using one or more columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records. - - https://en.wikipedia.org/wiki/Database_index 21
  • 22.
    Think of anIndex as a Table With Shortcuts to another table! Or a model of some of your data in it’s own table! And the more tables you have the read to get to the data the slower things run! (And the more memory is taken up) 22
  • 23.
  • 24.
  • 25.
    25 clustered index The InnoDBterm for a primary key index. InnoDB table storage is organized based on the values of the primary key columns, to speed up queries and sorts involving the primary key columns. For best performance, choose the primary key columns carefully based on the most performance-critical queries. Because modifying the columns of the clustered index is an expensive operation, choose primary columns that are rarely or never updated. In the Oracle Database product, this type of table is known as an index-organized table
  • 26.
    Think of anindex as a small, fast table pointing to a bigger table 26 Index 3 6 12 27 5001 Indexed Column column 1 column x column n 5001 3 6 27 12
  • 27.
    How Secondary IndexesRelate to the Clustered Index All indexes other than the clustered index are known as secondary indexes. In InnoDB, each record in a secondary index contains the primary key columns for the row, as well as the columns specified for the secondary index. InnoDB uses this primary key value to search for the row in the clustered index. If the primary key is long, the secondary indexes use more space, so it is advantageous to have a short primary key. 27
  • 28.
    28 Creating a tablewith a PRIMARY KEY (index) CREATE TABLE t1 ( c1 INT NOT NULL AUTO_INCREMENT PRIMARY KEY, c2 VARCHAR(100), c3 VARCHAR(100) ); An index is a list of KEYs. And you will hear ‘key’ and ‘index’ used interchangeably
  • 29.
    PRIMARY KEY This isan key for the index that uniquely defined for a row, should be immutable. InnoDB needs a PRIMARY KEY (and will make one up if you do not specify) No Null Values Monotonically increasing - use UUID_To_BIN() if you must use UUIDS, otherwise avoid them 29
  • 30.
    Indexing on aprefix of a column CREATE INDEX part_of_name ON customer (name(10)); Only the first 10 characters are indexed in this examples and this can save space/speed 30
  • 31.
    Multi-column index -last_name, first_name CREATE TABLE test ( id INT NOT NULL, last_name CHAR(30) NOT NULL, first_name CHAR(30) NOT NULL, PRIMARY KEY (id), INDEX name (last_name,first_name) ); 31 This index will work on (last_name,first_name) and (last_name) but not (first_name) {Use: left to right} Put highest cardinality field first
  • 32.
    Hashing values -an alternative SELECT * FROM tbl_name WHERE hash_col=MD5(CONCAT(val1,val2)) AND col1=val1 AND col2=val2; As an alternative to a composite index, you can introduce a column that is “hashed” based on information from other columns. If this column is short, reasonably unique, and indexed, it might be faster than a “wide” index on many columns. 32
  • 33.
    Some other typesof indexes Unique Indexes - only one row per value Full-Text Indexes - search string data Covering index - includes all columns needed for a query Secondary index - another column in the table is indexes Spatial Indexes - geographical information 33
  • 34.
    Functional Indexes aredefined on the result of a function applied to one or more columns of a single table CREATE TABLE t1 (col1 INT, col2 INT, INDEX func_index ((ABS(col1)))); CREATE INDEX idx1 ON t1 ((col1 + col2)); CREATE INDEX idx2 ON t1 ((col1 + col2), (col1 - col2), col1); ALTER TABLE t1 ADD INDEX ((col1 * 40) DESC); 34
  • 35.
    Multi-value indexes You cannow have more index pointers than index keys! ○ Very useful for JSON arrays mysql> SELECT 3 MEMBER OF('[1, 3, 5, 7, "Moe"]'); +--------------------------------------+ | 3 MEMBER OF('[1, 3, 5, 7, "Moe"]') | +--------------------------------------+ | 1 | +--------------------------------------+ 35
  • 36.
    MySQL has TwoMain Types of Index Structures B-Tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions. 36 Hash are more efficient than nested loops joins, except when the probe side of the join is very small but joins can only be used to compute equijoins.
  • 37.
    Please Keep inmind ... If there is a choice between multiple indexes, MySQL normally uses the index that finds the smallest number of rows (the most selective index). MySQL can use indexes on columns more efficiently if they are declared as the same type and size. ● In this context, VARCHAR and CHAR are considered the same if they are declared as the same size. For example, VARCHAR(10) and CHAR(10) are the same size, but VARCHAR(10) and CHAR(15) are not. For comparisons between nonbinary string columns, both columns should use the same character set. For example, comparing a utf8 column with a latin1 column precludes use of an index. 37
  • 38.
    NULL A few wordson something that may not be there! 38
  • 39.
    NULL ● NULL isused to designate a LACK of data. False 0 True 1 Don’t know NULL 39
  • 40.
    Indexing NULL values reallydrives down the performance of INDEXes - Cardinal Values and then a ‘junk drawer’ of nulls -> hard to search 40
  • 41.
  • 42.
    Before Invisible Indexes 1.Doubt usefulness of index 2. Check using EXPLAIN 3. Remove that Index 4. Rerun EXPLAIN 5. Get phone/text/screams from power user about slow query 6. Suddenly realize that the index in question may have had no use for you but the rest of the planet seems to need that dang index! 7. Take seconds/minutes/hours/days/weeks rebuilding that index 42
  • 43.
    After Invisible Indexes 1.Doubt usefulness of index 2. Check using EXPLAIN 3. Make index invisible – optimizer can not see that index! 4. Rerun EXPLAIN 5. Get phone/text/screams from power user about slow query 6. Make index visible 7. Blame problem on { network | JavaScript | GDPR | Slack | Cloud } Sys schema will show you which indexes have not been used that may be candidates for removal -- but be cautious! 43
  • 44.
    How to useINVISIBLE INDEX ALTER TABLE t1 ALTER INDEX i_idx INVISIBLE; ALTER TABLE t1 ALTER INDEX i_idx VISIBLE; 44
  • 45.
    Histograms What is ahistogram? It is not a gluten free, keto friendly biscuit. 45
  • 46.
  • 47.
    Histograms What is ahistogram? Wikipedia declares a histogram is an accurate representation of the distribution of numerical data. For RDBMS, a histogram is an approximation of the data distribution within a specific column. So in MySQL, histograms help the optimizer to find the most efficient Query Plan to fetch that data. 47
  • 48.
    Histograms What is ahistogram? A histogram is a distribution of data into logical buckets. There are two types: ● Singleton ● Equi-Height Maximum number of buckets is 1024 48
  • 49.
    Histogram Histogram statistics areuseful primarily for non-indexed columns. A histogram is created or updated only on demand, so it adds no overhead when table data is modified. On the other hand, the statistics become progressively more out of date when table modifications occur, until the next time they are updated. 49
  • 50.
    The two reasonsfor why you might consider a histogram instead of an index: Maintaining an indexes have a cost. If you have an index, every ● Every INSERT/UPDATE/DELETE causes the index to be updated. This will have an impact on your performance. A histogram on the other hand is created once and never updated unless you explicitly ask for it. It will thus not hurt your INSERT/UPDATE/DELETE-performance. ● The optimizer will make “index dives” to estimate the number of records in a given range. This might become too costly if you have for instance very long IN-lists in your query. Histogram statistics are much cheaper in this case, and might thus be more suitable. 50
  • 51.
    The Optimizer Occasionally thequery optimizer fails to find the most efficient plan and ends up spending a lot more time executing the query than necessary. The optimizer assumes that the data is evenly distributed in the column. This can be the old ‘assume’ makes an ‘ass out of you and me’ joke brought to life. The main reason for this is often that the optimizer doesn’t have enough knowledge about the data it is about to query: ● How many rows are there in each table? ● How many distinct values are there in each column? ● How is the data distributed in each column? 51
  • 52.
    Two-types of histograms Equi-height:One bucket represents a range of values. This type of histogram will be created when distinct values in the column are greater than the number of buckets specified in the analyze table syntax. Think A-G H-L M-T U-Z Singleton: One bucket represents one single value in the column and is the most accurate and will be created when the number of distinct values in the column is less than or equal to the number of buckets specified in the analyze table syntax. 52
  • 53.
    Frequency Histogram Each distinct value hasits own bucket. 53 101 102 104 insert into freq_histogram (x) values (101),(101),(102),(102),(102),(104);
  • 54.
    Frequency Histogram -an Example select id,x from freq_histogram ; +----+-----+ | id | x | +----+-----+ | 1 | 101 | | 2 | 101 | | 3 | 102 | | 4 | 102 | | 5 | 102 | | 6 | 104 | +----+-----+ analyze table freq_histogram update histogram on x with 3 buckets; 54 select JSON_PRETTY(HISTOGRAM->>"$") from information_schema.column_statistics where table_name='freq_histogram' and column_name='x'G *************************** 1. row *************************** JSON_PRETTY(HISTOGRAM->>"$"): { "buckets": [ [ 101, 0.3333333333333333 ], [ 102, 0.8333333333333333 ], [ 104, 1.0 ] ], "data-type": "int", "null-values": 0.0, "collation-id": 8, "last-updated": "2020-06-16 19:54:21.033017", "sampling-rate": 1.0, "histogram-type": "singleton", "number-of-buckets-specified": 3 }
  • 55.
    The statistics SELECT (SUBSTRING_INDEX(v,':', -1)) value, concat(round(c*100,1),'%') cumulfreq, CONCAT(round((c - LAG(c, 1, 0) over()) * 100,1), '%') freq FROM information_schema.column_statistics, JSON_TABLE(histogram->'$.buckets','$[*]' COLUMNS(v VARCHAR(60) PATH '$[0]', c double PATH '$[1]')) hist WHERE schema_name = 'demo' and table_name = 'freq_histogram' and column_name = 'x'; +-------+-----------+-------+ | value | cumulfreq | freq | +-------+-----------+-------+ | 101 | 33.3% | 33.3% | | 102 | 83.3% | 50.0% | | 104 | 100.0% | 16.7% | +-------+-----------+-------+ 55
  • 56.
    Creating and removingHistograms ANALYZE TABLE t UPDATE HISTOGRAM ON c1, c2, c3 WITH 10 BUCKETS; ANALYZE TABLE t UPDATE HISTOGRAM ON c1, c3 WITH 10 BUCKETS; ANALYZE TABLE t DROP HISTOGRAM ON c2; Note the first statement creates three different histograms on c1, c2, and c3. 56
  • 57.
    Information about Histograms mysql>SELECT TABLE_NAME, COLUMN_NAME, HISTOGRAM->>'$."data-type"' AS 'data-type', JSON_LENGTH(HISTOGRAM->>'$."buckets"') AS 'bucket-count' FROM INFORMATION_SCHEMA.COLUMN_STATISTICS; +-----------------+-------------+-----------+--------------+ | TABLE_NAME | COLUMN_NAME | data-type | bucket-count | +-----------------+-------------+-----------+--------------+ | country | Population | int | 226 | | city | Population | int | 1024 | | countrylan | Language | string | 457 | +-----------------+-------------+-----------+--------------+ 57
  • 58.
    Where Histograms Shine createtable h1 (id int unsigned auto_increment, x int unsigned, primary key(id)); insert into h1 (x) values (1),(1),(2),(2),(2),(3),(3),(3),(3); select x, count(x) from h1 group by x; +---+----------+ | x | count(x) | +---+----------+ | 1 | 2 | | 2 | 3 | | 3 | 4 | +---+----------+ 3 rows in set (0.0011 sec) 58
  • 59.
    Without Histogram EXPLAIN SELECT* FROM h1 WHERE x > 0G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: h1 partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 9 filtered: 33.32999801635742 – The optimizer estimates about 1/3 of the data Extra: Using where will go match the ‘X > 0’ condition 1 row in set, 1 warning (0.0007 sec) 59 The filtered column indicates an estimated percentage of table rows that will be filtered by the table condition. The maximum value is 100, which means no filtering of rows occurred. Values decreasing from 100 indicate increasing amounts of filtering. rows shows the estimated number of rows examined and rows × filtered shows the number of rows that will be joined with the following table. For example, if rows is 1000 and filtered is 50.00 (50%), the number of rows to be joined with the following table is 1000 × 50% = 500.
  • 60.
    Where Histograms Shine ANALYZETABLE h1 UPDATE HISTOGRAM ON x WITH 3 BUCKETSG *************************** 1. row *************************** Table: demox.h1 Op: histogram Msg_type: status Msg_text: Histogram statistics created for column 'x'. 1 row in set (0.0819 sec) 60
  • 61.
    Where Histograms Shine EXPLAINSELECT * FROM h1 WHERE x > 0G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: h1 partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 9 filtered: 100 – all rows!!! Extra: Using where 1 row in set, 1 warning (0.0007 sec) 61
  • 62.
    With EXPLAIN ANALYSE EXPLAINanalyze SELECT * FROM h1 WHERE x > 0G *************************** 1. row *************************** EXPLAIN: -> Filter: (h1.x > 0) (cost=1.15 rows=9) (actual time=0.027..0.034 rows=9 loops=1) -> Table scan on h1 (cost=1.15 rows=9) (actual time=0.025..0.030 rows=9 loops=1) 62
  • 63.
    Performance is notjust Indexes and Histograms ● There are many other ‘tweaks’ that can be made to speed things up ● Use explain to see what your query is doing? ○ File sorts, full table scans, using temporary tables, etc. ○ Does the join order look right ○ Buffers and caches big enough ○ Do you have enough memory ○ Disk and I/O speeds sufficient 63
  • 64.
    Locking Options MySQL addedtwo locking options to MySQL 8.0: ● NOWAIT ○ A locking read that uses NOWAIT never waits to acquire a row lock. The query executes immediately, failing with an error if a requested row is locked. ● SKIP LOCKED ○ A locking read that uses SKIP LOCKED never waits to acquire a row lock. The query executes immediately, removing locked rows from the result set. 64
  • 65.
    Locking Examples –Buying concert tickets START TRANSACTION; SELECT seat_no, row_no, cost FROM seats s JOIN seat_rows sr USING ( row_no ) WHERE seat_no IN ( 3,4 ) AND sr.row_no IN ( 5,6 ) AND booked = 'NO' FOR UPDATE OF s SKIP LOCKED; Let’s shop for tickets in rows 5 or 6 and seats 3 & 4 but we skip any locked records! 65
  • 66.
    LOCK NOWAIT START TRANSACTION; SELECTseat_no FROM seats JOIN seat_rows USING ( row_no ) WHERE seat_no IN (3,4) AND seat_rows.row_no IN (12) AND booked = 'NO' FOR UPDATE OF seats SKIP LOCKED FOR SHARE OF seat_rows NOWAIT; Without NOWAIT, this query would have waited for innodb_lock_wait_timeout (default: 50) seconds while attempting to acquire the shared lock on seat_rows. With NOWAIT an error is immediately returned ERROR 3572 (HY000): Do not wait for lock. 66
  • 67.
    Other Fast Ways ResourceGroups Optimizer hints Partitioning Multi Value Indexes 67
  • 68.
    Resource groups –setting and using CREATE RESOURCE GROUP Batch TYPE = USER VCPU = 2-3 -- assumes a system with at least 4 CPUs THREAD_PRIORITY = 10; SET RESOURCE GROUP Batch; or INSERT /*+ RESOURCE_GROUP(Batch) */ INTO t2 VALUES(2); 68
  • 69.
    Optimizer Hints SELECT /*+JOIN_ORDER(t1, t2) */ ... FROM t1, t2; Optimizer hints can be specified within individual statements. Because the optimizer hints apply on a per-statement basis, they provide finer control over statement execution plans than can be achieved using optimizer_switch. For example, you can enable an optimization for one table in a statement and disable the optimization for a different table. 69
  • 70.
    Partitioning ● In MySQL8.0, partitioning support is provided by the InnoDB and NDB storage engines. ● Partitioning enables you to distribute portions of individual tables across a file system according to rules which you can set largely as needed. In effect, different portions of a table are stored as separate tables in different locations. 70
  • 71.
    The Query EXPLAIN SELECTCity.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 71
  • 72.
    The Query –What we want EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 72
  • 73.
    The Query –Where we want it from EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 73
  • 74.
    The Query –How the two tables relate EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 74
  • 75.
    The Query –And any filters EXPLAIN SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G 75
  • 76.
    76 EXPLAIN SELECT City.name,Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: const possible_keys: PRIMARY key: PRIMARY key_len: 3 ref: const rows: 1 filtered: 100 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: City partitions: NULL type: ref possible_keys: CountryCode key: CountryCode key_len: 3 ref: const rows: 81 filtered: 100 Extra: NULL 2 rows in set, 1 warning (0.0013 sec)
  • 77.
    The Actual QueryPlan select `world`.`city`.`Name` AS `name`, 'United Kingdom' AS `Name` from `world`.`city` join `world`.`country` where (`world`.`city`.`CountryCode` = 'GBR') 77
  • 78.
    Optimizer substituted Country.name select `world`.`city`.`Name`AS `name`, 'United Kingdom' AS `Name` from `world`.`city` join `world`.`country` where (`world`.`city`.`CountryCode` = 'GBR') 78
  • 79.
    Optimizer substituted Country.name explainformat=tree SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=26.85 rows=81) So the optimizer ‘knows’ just to grab the ‘GBR’ records from the City table and does not need to read the Country table at all. 79
  • 80.
    Optimizer substituted Country.name ExplainANALYZE SELECT City.name, Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** EXPLAIN: -> Index lookup on City using CountryCode (CountryCode='GBR') (cost=26.85 rows=81) (actual time=0.123..0.138 rows=81 loops=1) 1 row in set (0.0009 sec) Explain ANALYZE runs the query and the estimate is pretty good! 80
  • 81.
    81 EXPLAIN SELECT City.name,Country.Name FROM City JOIN Country ON (City.CountryCode = Country.Code) WHERE Country.Code = 'GBR'G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: const possible_keys: PRIMARY key: PRIMARY key_len: 3 ref: const This is a CONSTANT rows: 1 filtered: 100 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: City partitions: NULL type: ref possible_keys: CountryCode key: CountryCode key_len: 3 ref: const rows: 81 There are 81 records that match in the index filtered: 100 Extra: NULL 2 rows in set, 1 warning (0.0013 sec)
  • 82.
  • 83.
    Some General Rules ●Look at indexing/histogramming columns on right side of WHERE clause ● Maybe index SORT BY columns (test) ● JOIN on like type and size columns ○ i.e. No VARCHAR(32) to DECIMAL matches ● Can the data be found in an index? 83
  • 84.
    Slightly More Complex SELECTCONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 84
  • 85.
    And yes theoutput of EXPLAIN can be hard to read | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | 1 | SIMPLE | rental | NULL | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | customer | NULL | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | 1 | SIMPLE | address | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | 1 | SIMPLE | inventory | NULL | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | 1 | SIMPLE | film | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 85
  • 86.
  • 87.
    Tree format isoften little better Limit: 5 row(s) -> Sort: <temporary>.title, limit input to 5 row(s) per chunk -> Stream results -> Nested loop inner join (cost=3866.17 rows=1601) -> Nested loop inner join (cost=3305.89 rows=1601) -> Nested loop inner join (cost=2745.61 rows=1601) -> Nested loop inner join (cost=2185.33 rows=1601) -> Filter: (rental.return_date is null) (cost=1625.05 rows=1601) -> Table scan on rental (cost=1625.05 rows=16008) -> Single-row index lookup on customer using PRIMARY (customer_id=rental.customer_id) (cost=0.25 rows=1) -> Single-row index lookup on address using PRIMARY (address_id=customer.address_id) (cost=0.25 rows=1) -> Single-row index lookup on inventory using PRIMARY (inventory_id=rental.inventory_id) (cost=0.25 rows=1) -> Filter: ((rental.rental_date + interval film.rental_duration day) < <cache>(curdate())) (cost=0.25 rows=1) -> Single-row index lookup on film using PRIMARY (film_id=inventory.film_id) (cost=0.25 rows=1) 87
  • 88.
    Back to theold tabular EXPLAIN | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | 1 | SIMPLE | rental | NULL | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | customer | NULL | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | 1 | SIMPLE | address | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | 1 | SIMPLE | inventory | NULL | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | 1 | SIMPLE | film | NULL | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 88
  • 89.
    Back to theold tabular EXPLAIN, cleaned up | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 89
  • 90.
    First glance –what indexes are being used? | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY | PRIMARY | 2 | sakila.inventory.film_id | 1 | 100 | Using where | 90 Anytime the type column shows all it means the entire file/table must be read, which is slow. If it does have to read all the rows is it because there are no indexes available? Or does it have to read all the table because is processing the entire table?
  • 91.
    Slightly More Complex SELECTCONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 91
  • 92.
    Quiz: Why soslow if it only has to return FIVE RECORDS?!?!?!?! SELECT CONCAT(customer.last_name,', ', customer.first_name) AS customer, address.phone, film.title FROM rental INNER JOIN customer ON rental.customer_id = customer.customer_id INNER JOIN address ON customer.address_id = address.address_id INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id INNER JOIN film ON inventory.film_id = film.film_id WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; 92
  • 93.
    Second glance –what EXTRAS are being used? | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+--------+----------------------------------------+---------+---------+---------------------------- +-------+----------+----------------------------------------------+ | rental | ALL | idx_fk_inventory_id,idx_fk_customer_id | NULL | NULL | NULL | 16008 | 10 | Using where; Using temporary; Using filesort | | customer | eq_ref | PRIMARY,idx_fk_address_id | PRIMARY | 2 | sakila.rental.customer_id | 1 | 100 | NULL | | address | eq_ref | PRIMARY | PRIMARY | 2 | sakila.customer.address_id | 1 | 100 | NULL | | inventory | eq_ref | PRIMARY,idx_fk_film_id | PRIMARY | 3 | sakila.rental.inventory_id | 1 | 100 | NULL | | film | eq_ref | PRIMARY 100 | NULL | | film | eq_ref | PRIMARY 93 The where; Using temporary; Using filesort informs that the output has to be sorted (part of the sort clause) and that a temporary table needed to be used.
  • 94.
    MySQL 8.0 TemporaryTable much faster ● Previously temporary tables were size limited and when they reached that limit: ○ The processing was halted ○ The data was copied to InnoDB ○ The processing continued with the InnoDB copy of the data ○ And the above was slow 94
  • 95.
    Is there somethingunindexed that need to be indexed? ● Or something we can use to build a histogram? ● Is there a better way to make the key? 95
  • 96.
    SHOW INDEX FROMrental ● PRIMARY ● Rental_date – rental_date, inventory_id, customer_id ● Idx_fk_inventory_id – inventory_id, cutomer_id ● Idx_fk_staff_id 96
  • 97.
    SHOW INDEX FROMfilm ● PRIMARY ● Idx_title ● Idx_fk_language_id ● Idx_fk_original_language_id 97
  • 98.
    A Functional Index?Or Generated Column? WHERE rental.return_date IS NULL AND rental_date + INTERVAL film.rental_duration DAY < CURRENT_DATE() ORDER BY title LIMIT 5; Could this part in red be reduced to a functional index/generated column? AND return_date < CURRENT_DATE() film.rental_duration + rental_date Would want to store this in the record with the rental data? 98
  • 99.
    Yes, let usadd that new column ● ALTER TABLE operations can be expensive ● How do we seed data ● Do we use a GENERATED COLUMN? Can we? ● Do we add a stub table? ○ Extra index/table dives ○ How much code do we have to change? ○ Other considerations 99
  • 100.
    Where Else ToLook for Information ● MySQL Manual ● Forums.MySQL.com ● MySQLcommunity.slack.com 100
  • 101.
    New Book ● YOUNEED THESE BOOKS!! ● Author’s others work are also outstanding and well written ● Amazing detail!! 101
  • 102.
    Great Book ● Gettingvery dated ● Make sure you get the 3rd edition ● Can be found in used book stores 102
  • 103.
    My Book ● Aguide to using JSON and MySQL ○ Programming Examples ○ Best practices ○ Twice the length of the original ○ NoSQL Document Store how-to 103
  • 104.
    “We have savedaround 40% of our costs and are able to reinvest that back into the business. And we are scaling across EMEA, and that’s basically all because of Oracle.” —Asser Smidt CEO and Cofounder, BotSupply Startups get cloud credits and a 70% discount for 2 years, global exposure via marketing, events, digital promotion, and media, plus access to mentorship, capital and Oracle’s 430,000+ customers Customers meet vetted startups in transformative spaces that help them stay ahead of their competition Oracle stays at the competitive edge of innovation with solutions that complement its technology stack Oracle for Startups - enroll at oracle.com/startup A Virtuous Cycle of Innovation, Everybody Wins.
  • 105.
    Thank you ● David.Stokes@Oracle.com ● @Stoker ● PHP Related blog https://elephantdolphin.blogspot.com/ ● MySQL Latest Blogs https://planet.mysql.com/ ● Where to ask questions https://forums.mysql.com/ ● Slack https://mysqlcommunity.slack.com slides -> slideshare.net/davestokes 105