Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Query Optimization

1,395 views

Published on

ZendCon 2014 Presentation

Published in: Software
  • Be the first to comment

MySQL Query Optimization

  1. 1. Query Optimization Morgan Tocker MySQL Community Manager October, 2014 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
  2. 2. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 3
  3. 3. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 4 Today’s Agenda Introduction World Schema IMDB Schema Advanced Topics 1 2 3 4
  4. 4. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Introduction • SQL is declarative. • You say what you want, not how to process. • Can’t sight check a query to understand how it executes. • Database management chooses the best possible way -­‐ think like a GPS navigator. 5
  5. 5. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | GPS Software 6
  6. 6. MySQL Optimizer Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 7
  7. 7. Diagnostic Commands Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • EXPLAIN (all versions) • EXPLAIN FORMAT=JSON (MySQL 5.6+) • Workbench supports visual format. • OPTIMIZER TRACE (MySQL 5.6+) 8
  8. 8. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 9 Today’s Agenda Introduction World Schema IMDB Schema Advanced Topics 1 2 3 4
  9. 9. The World Schema Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Contains cities, countries and language statistics. • Download From: • http://dev.mysql.com/doc/index-­‐other.html • Very small data set -­‐ good for learning. • Not perfect for explaining performance differences 10
  10. 10. Find Countries in Asia with Population >5M Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 11 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000G ********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 239 filtered: 4.76 Extra: Using where 1 row in set, 1 warning (0.00 sec)
  11. 11. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 12 EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "rows_examined_per_scan": 239, "rows_produced_per_join": 11, "filtered": 4.7614, "cost_info": { "read_cost": "51.52", "eval_cost": "2.28", "prefix_cost": "53.80", "data_read_per_join": "2K" }, "used_columns": [ "Code", .. "Capital", "Code2" ], "attached_condition": "((`world`.`country`.`Continent` = 'Asia') and (`world`.`country`.`Population` > 5000000))" } } } FORMAT=JSON Available since MySQL 5.6. Expanded in 5.7 to add cost information.
  12. 12. What indexes will make this query faster? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Some suggestions: • Index on (population) • Index on (continent) • Index on (population, continent) • Index on (continent, population) 13 SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000;
  13. 13. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Next Question • Given so many choices, which is the best choice? • A good GPS navigator understands traffic and finds the fastest route. • A good query optimizer does similar. 14
  14. 14. Query Optimizer Strategy Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Pick the plan that consumes the least amount of resources (CPU, IO). 15
  15. 15. Query Optimizer Strategy (cont.) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Use statistics available to identify good and bad plans. • Decisions may include: • Which indexes are good/bad. • Which order to join tables in. • If indexes should be read in a particular order to avoid sorting, or if sorting is not expensive. 16
  16. 16. Using single column indexes first…. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • p (population) • c (continent) 17
  17. 17. mysql> ALTER TABLE Country ADD INDEX p (population); Query OK, 0 rows affected (0.04 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000G ************************ 1. row ************************ Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 18 id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALL possible_keys: p key: NULL key_len: NULL ref: NULL rows: 239 filtered: 6.46 Extra: Using where 1 row in set, 1 warning (0.00 sec)
  18. 18. Why would an index not be used? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 19 mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (p) WHERE Continent = 'Asia' AND population > 5000000G ******* 1. row ******* EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "152.21" }, "table": { "table_name": "Country", "access_type": "range", "possible_keys": [ "p" ], Using MySQL 5.7+ mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000G ******* 1. row ******* EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "possible_keys": [ "p" ],
  19. 19. Modifying the query range: >50M Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 20 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000G ********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: range possible_keys: p key: p key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition; Using where 1 row in set, 1 warning (0.00 sec)
  20. 20. mysql> ALTER TABLE Country ADD INDEX c (continent); Query OK, 0 rows affected (0.04 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000G *********************** 1. row *********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 21 id: 1 select_type: SIMPLE table: Country partitions: NULL type: ref possible_keys: p,c key: c key_len: 1 ref: const rows: 51 filtered: 45.19 Extra: Using where 1 row in set, 1 warning (0.00 sec)
  21. 21. Modifying the query range: >500M mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 500000000G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 22 id: 1 select_type: SIMPLE table: Country partitions: NULL type: range possible_keys: p,c key: p key_len: 4 ref: NULL rows: 2 filtered: 21.34 Extra: Using index condition; Using where 1 row in set, 1 warning (0.00 sec)
  22. 22. Composite Indexes Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • On >50M which is better? • (population, continent) • (continent, population) 23
  23. 23. mysql> ALTER TABLE Country ADD INDEX pc (pop..on, co..nt); Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 24 id: 1 select_type: SIMPLE table: Country partitions: NULL type: ref possible_keys: p,c,pc key: c key_len: 1 ref: const rows: 51 filtered: 10.04 Extra: Using where 1 row in set, 1 warning (0.00 sec)
  24. 24. FORCE INDEX mysql> EXPLAIN SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 25 id: 1 select_type: SIMPLE table: Country partitions: NULL type: range possible_keys: pc key: pc key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition 1 row in set, 1 warning (0.00 sec)
  25. 25. mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000G ********************** 1. row ********************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "34.61" }, "table": { Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 26 .. ], "key": "pc", "used_key_parts": [ "Population" ], .. Easier to see with JSON EXPLAIN
  26. 26. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Rule of Thumb • Index on (const, range) instead of (range, const). • Applies to all databases. 27
  27. 27. mysql> ALTER TABLE Country ADD INDEX cp (co..nt, pop..on); Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 28 id: 1 select_type: SIMPLE table: Country partitions: NULL type: range possible_keys: p,c,pc,cp key: cp key_len: 5 ref: NULL rows: 11 filtered: 100.00 Extra: Using index condition 1 row in set, 1 warning (0.00 sec)
  28. 28. Where Population >N. Lower is better. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Cost Estimates 29 5M 50M 500M Table Scan 53.80 53.80 53.80 (Population) 152.21 34.61 3.81 (Continent) 28.20 28.20 28.20 (Population, 152.21 34.61 3.81 Continent) (Continent, Population) 24.83 16.41 3.81
  29. 29. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 30 Today’s Agenda Introduction World Schema IMDB Schema Advanced Topics 1 2 3 4
  30. 30. Examples Using IMDB Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Larger than world database schema. • Better to understand performance differences. • License requires data to be regenerated: • http://imdbpy.sourceforge.net/ 31
  31. 31. mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 32 id: 1 select_type: SIMPLE table: title partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 2387523 filtered: 0.50 Extra: Using where 1 row in set, 1 warning (0.00 sec) Ignore EXPLAIN time Actual run time is 2.9 seconds (Not shown)
  32. 32. mysql> ALTER TABLE title ADD INDEX (title(100)); Query OK, 0 rows affected (4.73 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 33 id: 1 select_type: SIMPLE table: title partitions: NULL type: ref possible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where 1 row in set, 1 warning (0.01 sec) From 2.9 seconds to 0.0 seconds (!!!)
  33. 33. How does LIKE work? Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 34 mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The Lion %'G ********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: range possible_keys: title key: title key_len: 102 ref: NULL rows: 109 filtered: 100.00 Extra: Using where 1 row in set, 1 warning (0.01 sec)
  34. 34. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | LIKE (cont.) 35 mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The%'G ********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: range possible_keys: title key: title key_len: 102 ref: NULL rows: 325548 filtered: 100.00 Extra: Using where 1 row in set, 1 warning (0.00 sec) VS Very Close. Estimate is that it is almost as fast to table scan. Run Time is 1.9 sec
  35. 35. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Sorting 36 mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King' ORDER BY production_yearG ********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: ref possible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where; Using filesort 1 row in set, 1 warning (0.00 sec)
  36. 36. SELECT movie_info.* FROM title INNER JOIN movie_info ON title.id=movie_info.movie_id WHERE title=‘The Lion King’G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 37 id: 1 select_type: SIMPLE table: movie_info type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1212187 Extra: NULL ********************** 2. row ********************** id: 1 select_type: SIMPLE table: title type: eq_ref possible_keys: PRIMARY,title key: PRIMARY key_len: 4 ref: imdb.movie_info.movie_id rows: 1 Extra: Using where
  37. 37. mysql> ALTER TABLE movie_info ADD INDEX mi (movie_id); Query OK, 0 rows affected (1.95 sec) Records: 0 Duplicates: 0 Warnings: 0 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 38
  38. 38. EXPLAIN SELECT movie_info.* FROM title INNER .. WHERE title='The Lion King'G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 39 id: 1 select_type: SIMPLE table: title type: ref possible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where ********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: ref possible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL
  39. 39. mysql> EXPLAIN SELECT * FROM movie_info WHERE movie_id IN (SELECT id FROM title WHERE title = 'The Lion King')G ********************** 1. row ********************** Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 40 id: 1 select_type: SIMPLE table: title type: ref possible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where ********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: ref possible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL
  40. 40. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 41 Today’s Agenda Introduction World Schema IMDB Schema Advanced Topics 1 2 3 4
  41. 41. Optimizer Hints Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • USE INDEX • FORCE INDEX • IGNORE INDEX • STRAIGHT JOIN 42
  42. 42. Optimizer Hints (cont.) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 43 Expanded in MySQL 5.6 mysql> select @@optimizer_switchG ********************** 1. row ********************** @@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union =on,index_merge_intersection=on,engine_condition_pushdown= on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,bl ock_nested_loop=on,batched_key_access=off,materialization= on,semijoin=on,loosescan=on,firstmatch=on,subquery_materia lization_cost_based=on,use_index_extensions=on,condition_f anout_filter=on 1 row in set (0.00 sec)
  43. 43. Statistics Sampling Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Some decisions based on pre-­‐computed statistics. • For example, which order to join tables. • Accuracy of statistics greatly improved in MySQL 5.6. • Feature is called “InnoDB persistent statistics”. 44
  44. 44. Optimizer Decision Visibility Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Intended for MySQL developer usage, and reporting optimizer bugs. • Yet totally practical for advanced usage :) 45 New to MySQL 5.6 SET optimizer_trace="enabled=on"; SELECT /* query here */; SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
  45. 45. Optimizer Trace Output • Includes why decisions were not made: Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 46 .. "index": "pc", "ranges": [ "50000000 < Population" ], "index_dives_for_eq_ranges": true, "rowid_ordered": false, "using_mrr": false, "index_only": false, "rows": 24, "cost": 29.81, "chosen": false, "cause": "cost" },
  46. 46. Too many indexes Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Slows down modifications due to maintenance cost. • i.e. Slower INSERT/UPDATE/DELETE • May also lead to worse decisions being made by optimizer. • Too many similar choices. • Only limited time window to evaluate suitability. • Space requirements in memory and storage. 47
  47. 47. MySQL 5.7 and beyond Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • We are working on a new cost model. • Better adapt to new hardware with different costs (i.e. SSDs have cheap random IO). • Memory buffer aware cost estimates. 48
  48. 48. Cost constants configurable per engine in 5.7 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | • Previously hard-­‐coded constants. • Better adapts to differences in hardware. 49
  49. 49. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 50

×