Your SlideShare is downloading. ×
MySQL Optimizer Overview
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

MySQL Optimizer Overview

2,159
views

Published on

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,159
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 2. MySQL Optimizer OverviewOlav SandståMySQL Optimizer Team2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 3. Program Agenda §  Introduction to MySQL optimizer §  Query transformations §  Selecting data access method §  Join optimizer §  Subquery optimizations3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 4. Query Optimizer What does the optimizer do? SELECT a, b JOIN FROM t1, t2, t3 WHERE t1.a = t2.b AND t2.b = t3.c Optimizer JOIN Table scan AND t2.d > 20 AND t2.d < 30; Range Ref t1 scan access t2 t3 Metadata: Statistics: -  Index information -  Table sizes -  Uniqueness -  Cardinality -  Nullability4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 5. MySQL Architecture SQL query Parser Preprocessor: Semantic check,name resolution Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query result Query execution Storage Engine InnoDB MyISAM5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 6. MySQL Optimizer Characteristics §  Produce a query plan that uses least resources –  IO and CPU JOIN §  Optimizes a single query Table JOIN –  No inter-query optimizations scan §  Produces left-deep linear query execution plan Table t4 JOIN scan Range Ref t3 scan access t1 t26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 7. Optimizer Overview Main phases: Negation elimination Logical Equality and constant propagation transformations Conversion of outer to inner joins Subquery transformations Evaluation of constant expressions Ref access analysis Prepare for Estimation of condition fanout cost-based Const table detection optimization Range analysis Cost-based Access method selection optimization JOIN optimizer Table condition pushdown Plan Access method adjustments refinement Sort avoidance Index condition pushdown7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 8. Logical Transformations8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 9. Logical Transformations §  Logical transformations of query conditions: –  Negation elimination –  Equality propagations –  Evaluate constant expressions –  Remove trivial conditions §  Conversion of outer to inner join §  Subquery transformations9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 10. Logical Transformations Example: SELECT * FROM t1, t2 WHERE t1.a = t2.a AND t2.a = 9 AND (NOT (t1.a > 10 OR t2.b > 3) OR (t1.b = t2.b + 7 AND t2.b = 5)); Negation elimination t1.a = t2.a AND t2.a = 9 AND (t1.a <= 10 AND t2.b <= 3 OR (t1.b = t2.b + 7 AND t2.b = 5)); Equality/const propagation t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 5 + 7 AND t2.b = 5)); Evaluate const expressions t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 12 AND t2.b = 5)); Trivial condition =TRUE removal t1.a = 9 AND t2.a = 9 AND (t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 11. Cost-based Optimization11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 12. Cost-based Query Optimization JOIN General idea: §  Assign cost to operations JOIN Table scan §  Assign cost to partial or alternative plans Range Ref t3 §  Search for plan with lowest cost scan access t1 t2 §  Cost-based optimizations: Access method Join order Subquery strategy12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 13. Optimizer Cost Model The ”cost” for executing a query §  Cost unit: read of a random data page §  Main cost factors: –  IO cost: §  #pages read from table §  #pages read from index –  CPU cost: §  Cost for evaluating query conditions §  Cost for comparing keys/records13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 14. Input to Cost Model §  IO-cost: §  Statistics: –  Estimates from storage engine –  Number of records in table based on number of pages to –  Key distribution/Cardinality: read §  Average number of records –  Both index and data pages per key value §  Schema: §  Only for indexed columns –  Length of records and keys §  Maintained by storage engine –  Uniqueness for indexes –  Number of records in an index –  Nullability range14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 15. Cost Model Example SELECT * FROM t1 WHERE a BETWEEN 20 and 30; Table scan: §  IO-cost: #pages in table §  CPU cost: #records * ROW_EVALUATE_COST Range scan (on secondary index): §  IO-cost: #pages to read from index + #records_in_range §  CPU cost: #records_in_range * ROW_EVALUATE_COST15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 16. Selecting Access Method16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 17. Selecting Access Method Finding the optimal method to read data from storage engine §  For each table, find the best access method: Main access methods: §  Table scan §  1. Check if the access method is useful §  Index scan §  2. Estimate cost of using access method §  Ref access §  3. Select the cheapest to be used §  Range scan §  Index merge §  Choice of access method is cost based §  Loose index scan17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 18. Ref Access §  Read all records with a given key value using an index: §  Examples: SELECT * FROM t1 WHERE t1.key = 7; SELECT * FROM t1, t2 WHERE t1.key = t2.key; §  “eq_ref”: –  Reading from a unique index, max one record returned §  “ref”: –  Reading from a non-unique index or a prefix of an index, multiple records returned18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 19. Ref Access Analysis §  Determine which indexes that can be used for ref access in a join SELECT City.Name AS Capital, Language FROM CountryLanguage, Country, City WHERE City.CountryCode = Country.Code AND City.ID = Country.Capital AND CountryLanguage.Country = Country.Code CountryLanguage Country Country City Code CountryCode Capital ID19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 20. Range Optimizer §  Goal: find the “minimal” ranges for each index that needs to be read §  Example: SELECT * FROM t1 WHERE (key1 > 10 AND key1 < 20) AND key2 > 30 §  Range scan using INDEX(key1): 10 20 §  Range scan using INDEX(key2): 3020 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 21. Range Optimizer, cont. §  Range optimizer selects the “useful” parts of the WHERE condition: –  Conditions comparing a column value with a constant: key > 3 key BETWEEN 4 AND 6 key IS NULL key = 4 key IN (10,12,..) key LIKE ”abc%” –  Nested AND/OR conditions are supported §  Result: list of disjoint ranges that need to be read from index: §  Cost estimate based on number of records in each range: –  Record estimate is found by asking the Storage Engine (“index dives”)21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 22. Range Access for Multi-part Index Example table with multi-part index §  Table: pk a b c §  INDEX idx(a, b, c); §  Logical storage layout of index: a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 23. Range Access for Multi-part Index, cont §  Equality on 1st index part? –  Can add condition on 2nd index part in range condition §  Example: SELECT * from t1 WHERE a IN (10,11,13) AND (b=2 OR b=4) a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c §  Resulting range scan: 2 4 2 4 2 423 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 24. Range Access for Multi-part Index, cont §  Non-Equality on 1st index part: –  Can NOT add condition on 2nd index part in range condition §  Example: SELECT * from t1 WHERE a > 10 AND a < 13 AND (b=2 OR b=4) a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c §  Resulting range scan: a >10 AND a < 1324 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 25. Index Merge §  Uses multiple indexes on the same table §  Implemented index merge strategies: –  Index Merge Union §  OR-ed conditions between different indexes –  Index Merge Intersect §  AND conditions between different indexes –  Index Merge Sort-Union §  OR-ed conditions where condition is a range25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 26. Index Merge Union §  Single index cannot handle ORed conditions on different columns §  Example: SELECT * FROM t1 WHERE a=10 OR b=10 §  Index Merge Union: INDEX(a) 10 INDEX(b) 10 Union Result: a=10 OR b=1026 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 27. Loose Index Scan §  Optimization for GROUP BY and DISTINCT: SELECT a, b FROM t1 GROUP BY a, b; SELECT DISTINCT a, b FROM t1; SELECT a, MIN(b) FROM t1 GROUP BY a; §  GROUP BY/DISTINCT must be on the prefix of the index a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 28. Join Optimizer28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 29. Join Optimizer ”Greedy search strategy” §  Goal: Given a JOIN of N tables, find the best JOIN ordering §  Strategy: –  Start with all 1-table plans t1 t2 t3 t4 –  Expand each plan with remaining tables §  Depth-first t2 t3 t4 –  If “cost of partial plan” > “cost of best plan”: t3 t4 t2 t4 t2 t3 §  “prune” plan t4 t3 t4 t2 t3 t229 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 30. JOIN Optimizer Illustrated SELECT City.Name, Language FROM Language, Country, City WHERE City.CountryCode = Country.Code AND City.ID = Country.Capital AND City.Population >= 1000000 AND Language.Country = Country.Code; start Language Country City cost=862 Country City Language City Language Country cost=1245 City Country City Language Country Language cost=26568 cost=32568 cost=62730 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 31. Complexity and Cost of Join Optimizer Join of N tables: N! possible plans to evaluate Heuristics to reduce the number of plans to evaluate: §  Use optimizer_search_depth to limit the number of tables to consider §  Pre-sort tables on size and key dependency order (Improved in MySQL 5.6) §  When adding the next table to a partial plan, add all tables that it has an equality reference to (New in MySQL 5.6)31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 32. Plan Refinements32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 33. Finalizing Query Plan Main optimizations: §  Assigning conditions to tables –  Evaluate conditions as early as possible in join order §  ORDER BY optimization: avoid sorting –  Change to different index –  Read in descending order §  Change to a cheaper access method –  Example: Use range scan instead of table scan or ref access §  Index Condition Pushdown (New in MySQL 5.6)33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 34. Index Condition Pushdown New §  Pushes conditions that can be evaluated on the MySQL server index down to storage engine Query conditions –  Works only on indexed columns §  Goal: evaluate conditions without having to access the actual record –  Reduces number of disk/block accesses Index –  Reduces CPU usage Table data Storage engine §  Both conditions on a single index and conditions on earlier tables in a JOIN can be pushed down34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 35. Index Condition Pushdown How it works Without ICP: MySQL server With ICP: Optimizer Storage Engine: Storage Engine: 1. Reads index Execution 4. 1. Reads index and 2. Reads record evaluates pushed index 3. Returns record condition 3. 2. Reads record Server: 1. 3. Returns record Index 4. Evaluates condition 2. Table data Storage engine Server: 4. Evaluates rest of condition35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 36. Index Condition Pushdown Example CREATE TABLE person ( name VARCHAR(..), height INTEGER, postalcode INTEGER, age INTEGER, INDEX (postalcode, age) ); Evaluated by server SELECT name FROM person WHERE postalcode BETWEEN 5000 AND 5500 AND age BETWEEN 21 AND 22 AND height>180; Pushed to storage engine Evaluated on index entries36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 37. Subqueries37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 38. Overview of Subquery Optimizations Subquery category: Strategy: New §  IN (SELECT …) •  Semi-Join •  Materialization •  IN ➜ EXISTS §  NOT IN (SELECT …) §  FROM (SELECT …) •  Derived table §  <CompOp> ALL/ANY (SELECT ..) •  MAX/MIN re-write §  EXISTS/other •  Execute subquery38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 39. Optimization of IN subqueries New A. Semi-join Transformation 1.  Transform IN (and =ANY) subquery to semi-join: SELECT * FROM t1 WHERE query_where AND outer_expr IN (SELECT inner_expr FROM t2 WHERE cond2) SELECT * FROM t1 SEMIJOIN t2 ON outer_expr = inner_expr WHERE query_where AND cond2 2.  Apply transformations/strategies for avoiding/removing duplicates: Duplicate Table pullout Weedout First Match Semi-join LooseScan materialization 3.  Optimize using cost-based JOIN optimizer39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 40. Optimization of IN Subqueries, cont. New B. Subquery Materialization §  Only for non-correlated subqueries §  Execute subquery once –  store result in temporary table with unique index (removes duplicates) §  Outer query does lookup in temporary table Temporary SELECT title FROM film Index table Lookup WHERE film_id IN (SELECT film_id FROM actor WHERE name=“Bullock”) Materialize40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 41. Optimization of IN Subqueries, cont. C. IN è EXISTS transformation §  Convert IN subquery to EXISTS subquery by “push-down” IN-equality to subquery: SELECT title FROM film WHERE film_id IN (SELECT film_id FROM actor WHERE name=“Bullock”) SELECT title FROM film WHERE EXISTS (SELECT 1 FROM actor WHERE name=“Bullock” AND film.film_id = actor.film_id) §  Benefit: subquery will evaluate fewer records §  Note: special handling if pushed down expressions can be NULL41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 42. Query Execution Plan42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 43. Understanding the Query Plan EXPLAIN §  Use EXPLAIN to print the final query plan: EXPLAIN SELECT * FROM t1 JOIN t2 ON t1.a = t2.a WHERE b > 10 AND c > 10; +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ | 1 | SIMPLE | t1 | range | PRIMARY,idx | idx | 4 | NULL | 12 | Using index condition | | 2 | SIMPLE | t2 | ref | key1 | key1| 4 | t1.a | 1 | NULL | +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ §  Structured EXPLAIN (New in MySQL 5.6) –  JSON format –  Includes more information about query (eg. pushed index conditions)43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 44. Understanding the Query Plan Structured Explain Example: EXPLAIN FORMAT=JSON SELECT * FROM t1 WHERE b > 10 AND c > 10 AND d > 10; EXPLAIN { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": [ ”idx" ], "key": ”idx", "key_length": "4", "rows": 12, "filtered": 100, "index_condition": "((`test`.`t1`.`b` > 10) and (`test`.`t1`.`c` > 10))", "attached_condition": "(`test`.`t1`.`d` > 10)" } } }44 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 45. Optimizer Traces "table": "`t1`", "range_analysis": { (New in MySQL 5.6) "table_scan": { "rows": 54, Understand HOW a query gets optimized }, "cost": 13.9 "best_covering_index_scan": { §  Tracing of the main steps and decisions "index": ”idx", "cost": 11.903, done by the optimizer "chosen": true }, "analyzing_range_alternatives": { "range_scan_alternatives": [ §  Example: { "index": ”idx", "ranges": [ "10 < a" SET optimizer_trace=”enabled=on”; ], SELECT * FROM t1 WHERE a > 10; "rowid_ordered": false, SELECT * FROM "using_mrr": false, "index_only": true, INFORMATION_SCHEMA.OPTIMIZER_TRACE; "rows": 12, "cost": 3.4314, "chosen": true }45 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 46. Influencing the Optimizer When the optimizer does not do what you want: §  Add indexes §  Force use of specific indexes: –  USE INDEX, FORCE INDEX, IGNORE INDEX §  Force specific join order: –  STRAIGHT_JOIN §  Adjust optimizer_switch flags: –  set optimizer_switch=“index_merge=off” §  Ask question in the MySQL optimizer forum46 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 47. Query Execution47 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 48. Nested Loops Join Execution strategy for joins SELECT City.Name, Language FROM Language, Country, City WHERE ….. QEP: Execution: Result City Range scan Ref Table range access scan Language Country Index Index City Country ref access City table scan48 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 49. Presentations by the MySQL Optimizer Team Sessions: §  Overview of New Optimizer Features in MySQL 5.6 §  Query Performance Comparison of MySQL 5.5 and MySQL 5.6 §  Powerful EXPLAIN in MySQL 5.6 BOF: §  Query Optimizations: What Is New and What Is Coming?49 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 50. Summary §  Query transformations §  Selecting data access method §  Join optimizer §  Subquery optimizations §  Plan refinements Questions?50 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 51. Graphic Section Divider51 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.