• Save
MySQL Optimizer Overview
Upcoming SlideShare
Loading in...5
×
 

MySQL Optimizer Overview

on

  • 463 views

 

Statistics

Views

Total Views
463
Views on SlideShare
440
Embed Views
23

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 23

http://www.myxplain.net 18
http://MYXPLAIN.net 5

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MySQL Optimizer Overview MySQL Optimizer Overview Presentation Transcript

  • 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • MySQL Optimizer OverviewOlav SandståMySQL Optimizer Team2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Program Agenda §  Introduction to MySQL optimizer §  Query transformations §  Selecting data access method §  Join optimizer §  Subquery optimizations3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Query Optimizer What does the optimizer do? SELECT a, b JOIN FROM t1, t2, t3 WHERE t1.a = t2.b AND t2.b = t3.c Optimizer JOIN Table scan AND t2.d > 20 AND t2.d < 30; Range Ref t1 scan access t2 t3 Metadata: Statistics: -  Index information -  Table sizes -  Uniqueness -  Cardinality -  Nullability4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • MySQL Architecture SQL query Parser Preprocessor: Semantic check,name resolution Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query result Query execution Storage Engine InnoDB MyISAM5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • MySQL Optimizer Characteristics §  Produce a query plan that uses least resources –  IO and CPU JOIN §  Optimizes a single query Table JOIN –  No inter-query optimizations scan §  Produces left-deep linear query execution plan Table t4 JOIN scan Range Ref t3 scan access t1 t26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimizer Overview Main phases: Negation elimination Logical Equality and constant propagation transformations Conversion of outer to inner joins Subquery transformations Evaluation of constant expressions Ref access analysis Prepare for Estimation of condition fanout cost-based Const table detection optimization Range analysis Cost-based Access method selection optimization JOIN optimizer Table condition pushdown Plan Access method adjustments refinement Sort avoidance Index condition pushdown7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Logical Transformations8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Logical Transformations §  Logical transformations of query conditions: –  Negation elimination –  Equality propagations –  Evaluate constant expressions –  Remove trivial conditions §  Conversion of outer to inner join §  Subquery transformations9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Logical Transformations Example: SELECT * FROM t1, t2 WHERE t1.a = t2.a AND t2.a = 9 AND (NOT (t1.a > 10 OR t2.b > 3) OR (t1.b = t2.b + 7 AND t2.b = 5)); Negation elimination t1.a = t2.a AND t2.a = 9 AND (t1.a <= 10 AND t2.b <= 3 OR (t1.b = t2.b + 7 AND t2.b = 5)); Equality/const propagation t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 5 + 7 AND t2.b = 5)); Evaluate const expressions t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 12 AND t2.b = 5)); Trivial condition =TRUE removal t1.a = 9 AND t2.a = 9 AND (t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Cost-based Optimization11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Cost-based Query Optimization JOIN General idea: §  Assign cost to operations JOIN Table scan §  Assign cost to partial or alternative plans Range Ref t3 §  Search for plan with lowest cost scan access t1 t2 §  Cost-based optimizations: Access method Join order Subquery strategy12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimizer Cost Model The ”cost” for executing a query §  Cost unit: read of a random data page §  Main cost factors: –  IO cost: §  #pages read from table §  #pages read from index –  CPU cost: §  Cost for evaluating query conditions §  Cost for comparing keys/records13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Input to Cost Model §  IO-cost: §  Statistics: –  Estimates from storage engine –  Number of records in table based on number of pages to –  Key distribution/Cardinality: read §  Average number of records –  Both index and data pages per key value §  Schema: §  Only for indexed columns –  Length of records and keys §  Maintained by storage engine –  Uniqueness for indexes –  Number of records in an index –  Nullability range14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Cost Model Example SELECT * FROM t1 WHERE a BETWEEN 20 and 30; Table scan: §  IO-cost: #pages in table §  CPU cost: #records * ROW_EVALUATE_COST Range scan (on secondary index): §  IO-cost: #pages to read from index + #records_in_range §  CPU cost: #records_in_range * ROW_EVALUATE_COST15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Selecting Access Method16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Selecting Access Method Finding the optimal method to read data from storage engine §  For each table, find the best access method: Main access methods: §  Table scan §  1. Check if the access method is useful §  Index scan §  2. Estimate cost of using access method §  Ref access §  3. Select the cheapest to be used §  Range scan §  Index merge §  Choice of access method is cost based §  Loose index scan17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Ref Access §  Read all records with a given key value using an index: §  Examples: SELECT * FROM t1 WHERE t1.key = 7; SELECT * FROM t1, t2 WHERE t1.key = t2.key; §  “eq_ref”: –  Reading from a unique index, max one record returned §  “ref”: –  Reading from a non-unique index or a prefix of an index, multiple records returned18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Ref Access Analysis §  Determine which indexes that can be used for ref access in a join SELECT City.Name AS Capital, Language FROM CountryLanguage, Country, City WHERE City.CountryCode = Country.Code AND City.ID = Country.Capital AND CountryLanguage.Country = Country.Code CountryLanguage Country Country City Code CountryCode Capital ID19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Range Optimizer §  Goal: find the “minimal” ranges for each index that needs to be read §  Example: SELECT * FROM t1 WHERE (key1 > 10 AND key1 < 20) AND key2 > 30 §  Range scan using INDEX(key1): 10 20 §  Range scan using INDEX(key2): 3020 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Range Optimizer, cont. §  Range optimizer selects the “useful” parts of the WHERE condition: –  Conditions comparing a column value with a constant: key > 3 key BETWEEN 4 AND 6 key IS NULL key = 4 key IN (10,12,..) key LIKE ”abc%” –  Nested AND/OR conditions are supported §  Result: list of disjoint ranges that need to be read from index: §  Cost estimate based on number of records in each range: –  Record estimate is found by asking the Storage Engine (“index dives”)21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Range Access for Multi-part Index Example table with multi-part index §  Table: pk a b c §  INDEX idx(a, b, c); §  Logical storage layout of index: a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Range Access for Multi-part Index, cont §  Equality on 1st index part? –  Can add condition on 2nd index part in range condition §  Example: SELECT * from t1 WHERE a IN (10,11,13) AND (b=2 OR b=4) a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c §  Resulting range scan: 2 4 2 4 2 423 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Range Access for Multi-part Index, cont §  Non-Equality on 1st index part: –  Can NOT add condition on 2nd index part in range condition §  Example: SELECT * from t1 WHERE a > 10 AND a < 13 AND (b=2 OR b=4) a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c §  Resulting range scan: a >10 AND a < 1324 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Index Merge §  Uses multiple indexes on the same table §  Implemented index merge strategies: –  Index Merge Union §  OR-ed conditions between different indexes –  Index Merge Intersect §  AND conditions between different indexes –  Index Merge Sort-Union §  OR-ed conditions where condition is a range25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Index Merge Union §  Single index cannot handle ORed conditions on different columns §  Example: SELECT * FROM t1 WHERE a=10 OR b=10 §  Index Merge Union: INDEX(a) 10 INDEX(b) 10 Union Result: a=10 OR b=1026 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Loose Index Scan §  Optimization for GROUP BY and DISTINCT: SELECT a, b FROM t1 GROUP BY a, b; SELECT DISTINCT a, b FROM t1; SELECT a, MIN(b) FROM t1 GROUP BY a; §  GROUP BY/DISTINCT must be on the prefix of the index a 10 11 12 13 b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 c27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Join Optimizer28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Join Optimizer ”Greedy search strategy” §  Goal: Given a JOIN of N tables, find the best JOIN ordering §  Strategy: –  Start with all 1-table plans t1 t2 t3 t4 –  Expand each plan with remaining tables §  Depth-first t2 t3 t4 –  If “cost of partial plan” > “cost of best plan”: t3 t4 t2 t4 t2 t3 §  “prune” plan t4 t3 t4 t2 t3 t229 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • JOIN Optimizer Illustrated SELECT City.Name, Language FROM Language, Country, City WHERE City.CountryCode = Country.Code AND City.ID = Country.Capital AND City.Population >= 1000000 AND Language.Country = Country.Code; start Language Country City cost=862 Country City Language City Language Country cost=1245 City Country City Language Country Language cost=26568 cost=32568 cost=62730 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Complexity and Cost of Join Optimizer Join of N tables: N! possible plans to evaluate Heuristics to reduce the number of plans to evaluate: §  Use optimizer_search_depth to limit the number of tables to consider §  Pre-sort tables on size and key dependency order (Improved in MySQL 5.6) §  When adding the next table to a partial plan, add all tables that it has an equality reference to (New in MySQL 5.6)31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Plan Refinements32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Finalizing Query Plan Main optimizations: §  Assigning conditions to tables –  Evaluate conditions as early as possible in join order §  ORDER BY optimization: avoid sorting –  Change to different index –  Read in descending order §  Change to a cheaper access method –  Example: Use range scan instead of table scan or ref access §  Index Condition Pushdown (New in MySQL 5.6)33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Index Condition Pushdown New §  Pushes conditions that can be evaluated on the MySQL server index down to storage engine Query conditions –  Works only on indexed columns §  Goal: evaluate conditions without having to access the actual record –  Reduces number of disk/block accesses Index –  Reduces CPU usage Table data Storage engine §  Both conditions on a single index and conditions on earlier tables in a JOIN can be pushed down34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Index Condition Pushdown How it works Without ICP: MySQL server With ICP: Optimizer Storage Engine: Storage Engine: 1. Reads index Execution 4. 1. Reads index and 2. Reads record evaluates pushed index 3. Returns record condition 3. 2. Reads record Server: 1. 3. Returns record Index 4. Evaluates condition 2. Table data Storage engine Server: 4. Evaluates rest of condition35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Index Condition Pushdown Example CREATE TABLE person ( name VARCHAR(..), height INTEGER, postalcode INTEGER, age INTEGER, INDEX (postalcode, age) ); Evaluated by server SELECT name FROM person WHERE postalcode BETWEEN 5000 AND 5500 AND age BETWEEN 21 AND 22 AND height>180; Pushed to storage engine Evaluated on index entries36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Subqueries37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Overview of Subquery Optimizations Subquery category: Strategy: New §  IN (SELECT …) •  Semi-Join •  Materialization •  IN ➜ EXISTS §  NOT IN (SELECT …) §  FROM (SELECT …) •  Derived table §  <CompOp> ALL/ANY (SELECT ..) •  MAX/MIN re-write §  EXISTS/other •  Execute subquery38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimization of IN subqueries New A. Semi-join Transformation 1.  Transform IN (and =ANY) subquery to semi-join: SELECT * FROM t1 WHERE query_where AND outer_expr IN (SELECT inner_expr FROM t2 WHERE cond2) SELECT * FROM t1 SEMIJOIN t2 ON outer_expr = inner_expr WHERE query_where AND cond2 2.  Apply transformations/strategies for avoiding/removing duplicates: Duplicate Table pullout Weedout First Match Semi-join LooseScan materialization 3.  Optimize using cost-based JOIN optimizer39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimization of IN Subqueries, cont. New B. Subquery Materialization §  Only for non-correlated subqueries §  Execute subquery once –  store result in temporary table with unique index (removes duplicates) §  Outer query does lookup in temporary table Temporary SELECT title FROM film Index table Lookup WHERE film_id IN (SELECT film_id FROM actor WHERE name=“Bullock”) Materialize40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimization of IN Subqueries, cont. C. IN è EXISTS transformation §  Convert IN subquery to EXISTS subquery by “push-down” IN-equality to subquery: SELECT title FROM film WHERE film_id IN (SELECT film_id FROM actor WHERE name=“Bullock”) SELECT title FROM film WHERE EXISTS (SELECT 1 FROM actor WHERE name=“Bullock” AND film.film_id = actor.film_id) §  Benefit: subquery will evaluate fewer records §  Note: special handling if pushed down expressions can be NULL41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Query Execution Plan42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Understanding the Query Plan EXPLAIN §  Use EXPLAIN to print the final query plan: EXPLAIN SELECT * FROM t1 JOIN t2 ON t1.a = t2.a WHERE b > 10 AND c > 10; +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ | 1 | SIMPLE | t1 | range | PRIMARY,idx | idx | 4 | NULL | 12 | Using index condition | | 2 | SIMPLE | t2 | ref | key1 | key1| 4 | t1.a | 1 | NULL | +----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+ §  Structured EXPLAIN (New in MySQL 5.6) –  JSON format –  Includes more information about query (eg. pushed index conditions)43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Understanding the Query Plan Structured Explain Example: EXPLAIN FORMAT=JSON SELECT * FROM t1 WHERE b > 10 AND c > 10 AND d > 10; EXPLAIN { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": [ ”idx" ], "key": ”idx", "key_length": "4", "rows": 12, "filtered": 100, "index_condition": "((`test`.`t1`.`b` > 10) and (`test`.`t1`.`c` > 10))", "attached_condition": "(`test`.`t1`.`d` > 10)" } } }44 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Optimizer Traces "table": "`t1`", "range_analysis": { (New in MySQL 5.6) "table_scan": { "rows": 54, Understand HOW a query gets optimized }, "cost": 13.9 "best_covering_index_scan": { §  Tracing of the main steps and decisions "index": ”idx", "cost": 11.903, done by the optimizer "chosen": true }, "analyzing_range_alternatives": { "range_scan_alternatives": [ §  Example: { "index": ”idx", "ranges": [ "10 < a" SET optimizer_trace=”enabled=on”; ], SELECT * FROM t1 WHERE a > 10; "rowid_ordered": false, SELECT * FROM "using_mrr": false, "index_only": true, INFORMATION_SCHEMA.OPTIMIZER_TRACE; "rows": 12, "cost": 3.4314, "chosen": true }45 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Influencing the Optimizer When the optimizer does not do what you want: §  Add indexes §  Force use of specific indexes: –  USE INDEX, FORCE INDEX, IGNORE INDEX §  Force specific join order: –  STRAIGHT_JOIN §  Adjust optimizer_switch flags: –  set optimizer_switch=“index_merge=off” §  Ask question in the MySQL optimizer forum46 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Query Execution47 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Nested Loops Join Execution strategy for joins SELECT City.Name, Language FROM Language, Country, City WHERE ….. QEP: Execution: Result City Range scan Ref Table range access scan Language Country Index Index City Country ref access City table scan48 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Presentations by the MySQL Optimizer Team Sessions: §  Overview of New Optimizer Features in MySQL 5.6 §  Query Performance Comparison of MySQL 5.5 and MySQL 5.6 §  Powerful EXPLAIN in MySQL 5.6 BOF: §  Query Optimizations: What Is New and What Is Coming?49 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Summary §  Query transformations §  Selecting data access method §  Join optimizer §  Subquery optimizations §  Plan refinements Questions?50 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • Graphic Section Divider51 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.