SlideShare a Scribd company logo
1
The MySQL Query Optimizer
Explained Through Optimizer Trace
Øystein Grøvlen
Senior Staff Engineer
Alibaba Cloud
MySQL Query Optimizer
SELECT a, b
FROM t1, t2, t3
WHERE t1.a = t2.b
AND t2.b = t3.c
AND t2.d > 20
AND t2.d < 30;
Table/index info
(data dictionary)
Statistics
(storage engines)
t2 t3
t1
Table
scan
Range
scan
Ref
access
JOIN
JOINQuery
Optimizer
2
MySQL Architecture
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver (Prepare):
Semantic check,name resolution
Storage Engine
InnoDB MyISAM
SQL query
Query result
3
Optimizer Trace
• EXPLAIN shows the selected plan
• Optimizer trace shows WHY the plan was selected
SET optimizer_trace= "enabled=on";
SELECT * FROM t1, t2 WHERE f1=1 AND f1=f2 AND f2>0;
SELECT trace FROM information_schema.optimizer_trace
INTO OUTFILE filename LINES TERMINATED BY ’’;
SET optimizer_trace="enabled=off";
QUERY SELECT * FROM t1, t2 WHERE f1=1 AND f1=f2 AND f2>0;
TRACE { "steps": [ { "join_preparation": { "select#": 1,… } … } …] }
MISSING_BYTES_BEYOND_MAX_MEM_SIZE 0
INSUFFICIENT_PRIVILEGES 0
How to generate it
4
Optimizer Trace
{
"steps": [
{
"join_preparation": {
"select#": 1,
"steps": [
{
"expanded_query": "/* select#1 */ select `t1`.`f1` AS `f1`,`t2`.`f2` AS `f2` from `t1` join `t2` where ((`t1`.`f1` = 1) and (`t1`.`f1` = `t2`.`f2`) and (`t2`.`f2` > 0))"
}
]
}
},
{
"join_optimization": {
"select#": 1,
"steps": [
{
"condition_processing": {
"condition": "WHERE",
"original_condition": "((`t1`.`f1` = 1) and (`t1`.`f1` = `t2`.`f2`) and (`t2`.`f2` > 0))",
"steps": [
{
…
Example
5
Optimizer Trace
JSON browser plugin
6
Browser Plugin
Collapse to see main trace objects/phases
7
Browser Plugin
Expand JSON objects
8
Prepare Phase
• Name resolving
• Map names to database objects (tables, columns, …)
• Semantic checks
• Permanent transformations:
• Conversion of outer join to inner join
• Merging of views and derived tables
• Subquery transformations
• IN to EXISTS
• IN to Semijoin (5.6)
• EXISTS to IN (8.0.16)
• Etc.
Simpler query to
optimize and
execute
Prepare for later
optimizations
9
Conversion of outer join to inner join
SELECT o_orderkey FROM orders LEFT JOIN lineitem ON o_orderkey = l_orderkey
WHERE l_discount > 0.10;
"join_preparation": {
"select#": 1,
"steps": [
{
"expanded_query": "/* select#1 */ select `orders`.`o_orderkey` AS `o_orderkey` from (`orders` left join `lineitem` on
((`orders`.`o_orderkey` = `lineitem`.`l_orderkey`))) where (`lineitem`.`l_discount` > 0.10)"
},
{
"transformations_to_nested_joins": {
"transformations": [
"outer_join_to_inner_join",
"JOIN_condition_to_WHERE",
"parenthesis_removal"
],
"expanded_query": "/* select#1 */ select `orders`.`o_orderkey` AS `o_orderkey` from `orders` join `lineitem` where
((`lineitem`.`l_discount` > 0.10) and(`orders`.`o_orderkey` = `lineitem`.`l_orderkey`))"
} …
10
Conversion from EXISTS-subquery to IN-subquery
SELECT o_orderpriority, COUNT(*) AS order_count FROM orders
WHERE EXISTS
(SELECT * FROM lineitem WHERE l_orderkey = o_orderkey AND l_commitdate < l_receiptdate)
GROUP BY o_orderpriority ORDER BY o_orderpriority;
SELECT o_orderpriority, COUNT(*) AS order_count FROM orders
WHERE o_orderkey IN
(SELECT l_orderkey FROM lineitem WHERE l_commitdate < l_receiptdate
GROUP BY o_orderpriority ORDER BY o_orderpriority;
SELECT o_orderpriority, COUNT(*) AS order_count
FROM orders SEMIJOIN lineitem ON l_orderkey = o_orderkey
WHERE l_commitdate < l_receiptdate
GROUP BY o_orderpriority ORDER BY o_orderpriority;
New in 8.0.16
11
Conversion from EXISTS-subquery to IN-subquery
"join_preparation": {
"select#": 1,
"steps": [
{
"join_preparation": {
"select#": 2,
"steps": [
{
"expanded_query": "/* select#2 */ select 1 from `lineitem` where ((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`)
and (`lineitem`.`l_commitDATE` < `lineitem`.`l_receiptDATE`))"
},
{
"transformation": {
"select#": 2,
"from": "EXISTS (SELECT)",
"to": "semijoin",
"chosen": true
}
}
]
…
Optimizer trace
12
Conversion from EXISTS-subquery to IN-subquery
Optimizer trace, cont.
13
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
Access method selection
Join order
14
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Access method selection
Join order
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
15
=TRUE
Trivial condition
removal
t1.a = 9 AND t2.a = 9 AND (t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));
t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 5 + 7 AND t2.b = 5));
Equality/const
propagation
Logical Transformations
Condition processing
t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));
Evaluate const
expressions
SELECT * FROM t1, t2 WHERE
t1.a = t2.a AND t2.a = 9 AND (NOT (t1.a > 10 OR t2.b > 3) OR (t1.b = t2.b + 7 AND t2.b = 5));
Negation
elimination
t1.a = t2.a AND t2.a = 9 AND (t1.a <= 10 AND t2.b <= 3 OR (t1.b = t2.b + 7 AND t2.b = 5));
16
Logical Transformations
"join_optimization": {
"select#": 1,
"steps": [
{
"condition_processing": {
"condition": "WHERE",
"original_condition": "((`t1`.`a` = `t2`.`a`) and (`t2`.`a` = 9) and (((`t1`.`a` <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` =(`t2`.`b` + 7)) and
(`t2`.`b` = 5))))",
"steps": [
{
"transformation": "equality_propagation",
"resulting_condition": "((((9 <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` = (5 + 7)) and multiple equal(5, `t2`.`b`))) and multiple equal(9,
`t1`.`a`, `t2`.`a`))"
},
{
"transformation": "constant_propagation",
"resulting_condition": "((((9 <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` = 12) and multiple equal(5, `t2`.`b`))) and multiple equal(9, `t1`.`a`
, `t2`.`a`))"
},
{
"transformation": "trivial_condition_removal",
"resulting_condition": "(((`t2`.`b` <= 3) or ((`t1`.`b` = 12) and multiple equal(5, `t2`.`b`))) and multiple equal(9, `t1`.`a`, `t2`.`a`))"
}
Optimizer Trace
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Access method selection
Join order
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
18
Ref Access Analysis
Determine which indexes that can be used for index lookup in a join
SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate, o_shippriority
FROM customer JOIN orders ON c_custkey = o_custkey
JOIN lineitem ON l_orderkey = o_orderkey
WHERE c_mktsegment = 'FURNITURE’
AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15'
GROUP by l_orderkey, o_orderdate, o_shippriority
ORDER by revenue desc, o_orderdate
LIMIT 10;
customer
c_custkey
orders
o_orderkey
o_custkey lineitem
l_orderkey
l_linenumber
19
Ref Access Analysis
{
"ref_optimizer_key_uses": [
{
"table": "`customer`",
"field": "c_custkey",
"equals": "`orders`.`o_custkey`",
"null_rejecting": true
},
{
"table": "`orders`",
"field": "o_orderkey",
"equals": "`lineitem`.`l_orderkey`",
"null_rejecting": false
},
{
"table": "`lineitem`",
"field": "l_orderkey",
"equals": "`orders`.`o_orderkey`",
"null_rejecting": false
},
{
"table": "`lineitem`",
"field": "l_orderkey",
"equals": "`orders`.`o_orderkey`",
"null_rejecting": false
}
]
},
Optimizer trace
20
Cost-based Query Optimization
General idea:
• Assign cost to operations
• Assign cost to partial or alternative plans
• Search for plan with lowest cost
t2 t3
t1
Table
scan
Range
scan
Ref
access
JOIN
JOIN
21
Cost-based Query Optimizations
The main cost-based optimizations:
• Index and access method:
• Table scan
• Index scan
• Range scan
• Index lookup (ref access)
• Join order
• Join buffering strategy
• Subquery strategy t2 t3
t1
Table
scan
Range
scan
Ref
access
JOIN
JOIN
22
Optimizer Cost Model
t1 Cost estimate
Row estimate
Cost Model
Cost formulas
Access
methods
Join Subquery
Cost constants
CPU IO
Metadata:
- Record and index size
- Index information
- Uniqueness
Statistics:
- Table size
- Cardinality
- Range estimates
- Histograms
Cost model
configuration
Range
scan
JOIN
23
Cost Estimates
• The cost for executing a query
• Cost unit:
• “read a random data page from disk”
• Main cost factors:
• IO cost:
• #pages read from table
• #pages read from index
• CPU cost:
• Evaluating query conditions
• Comparing keys/records
• Sorting keys
Cost MySQL
5.7
MySQL
8.0
Read a random disk page 1.0 1.0
Read a data page from memory
buffer
1.0 0.25
Evaluate query condition 0.2 0.1
Compare keys/records 0.1 0.05
Main cost constants
24
Range analysis
• Cost of table scan
• Based on number of pages to read
• Cost of index scan alternatives
• Index scan (covering)
• Index range scan
• Index merge
• Index for grouping
• Skip scan (new in 8.0)
• Loose-index scan
Compare cost of table scan to index scan
25
Range Analysis Example
Table scan:
• IO-cost: #pages in table * IO_BLOCK_READ_COST
• CPU cost: #rows * ROW_EVALUATE_COST
Range scan (on secondary index):
• IO-cost: #rows_in_range * IO_BLOCK_READ_COST
• CPU cost: #rows_in_range * ROW_EVALUATE_COST
SELECT SUM(o_totalprice) FROM orders
WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-12-31';
Table scan vs Index range scan
26
Range Analysis Example
EXPLAIN SELECT SUM(o_totalprice) FROM orders
WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-12-31';
EXPLAIN SELECT SUM(o_totalprice) FROM orders
WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-06-30';
id
select
type
table type possible keys key
key
len
rows filtered Extra
1 SIMPLE orders ALL i_o_orderdate NULL NULL 1480845 31.13 Using where
Id
select
type
table type possible keys key
key
len
rows filtered Extra
1 SIMPLE orders range i_o_orderdate i_o_orderdate 4 222102 100.00
Using index
condition
Table scan vs index range scan cnt.
27
"rows_estimation": [
{
"table": "`orders`",
"range_analysis": {
"table_scan": {
"rows": 1480845,
"cost": 151138
},
"potential_range_indexes": [
{
"index": "PRIMARY",
"usable": false,
"cause": "not_applicable"
},
{
"index": "i_o_orderdate",
"usable": true,
"key_parts": [
"o_orderDATE",
"o_orderkey"
] } ],
"setup_range_conditions": [
],
"group_index_range": {
"chosen": false,
"cause": "not_group_by_or_distinct"
},
"skip_scan_range": {
"potential_skip_scan_indexes": [
{
"index": "i_o_orderdate",
"usable": false,
"cause": "query_references_nonkey_column"
}
]
},
"analyzing_range_alternatives": {
"range_scan_alternatives": [
{
"index": "i_o_orderdate",
"ranges": [
"0x21940f <= o_orderDATE <= 0x9f950f"
],
"index_dives_for_eq_ranges": true,
"rowid_ordered": false,
"using_mrr": false,
"index_only": false,
"rows": 460938,
"cost": 161329,
"chosen": false,
"cause": "cost"
}
],
"analyzing_roworder_intersect": {
"usable": false,
"cause": "too_few_roworder_scans"
} } } }]
Range Analysis Example
Optimizer trace
28
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Join order
Access method selection
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
29
Join Optimizer
• Goal: Given a JOIN of N tables, find the best JOIN ordering
• Only considers left-deap plans
• Strategy:
• Start with all 1-table plans
• Sorted based on size and key dependency
• Expand each plan with remaining tables
• Depth-first
• If “cost of partial plan” > “cost of best plan”:
• “Prune” plan
• Heuristic pruning:
• Prune less promising partial plans
• May in rare cases miss most optimal plan (turn off with set
optimizer_prune_level = 0)
”Greedy search strategy”
t1
t2
t2
t2
t2
t3
t3
t3
t4t4
t4
t4t4
t3
t3 t2
t4t2 t3
N! possible
plans
30
Join Optimizer Illustrated
DBT3-Q3
customer lineitem orders
customer
customer
customerlineitem
lineitem lineitem
lineitem
orders
ordersorders
orders
cost=239843 cost=102033
cost=119972
cost=1869 cost=1149
start
SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate, o_shippriority
FROM customer JOIN orders ON c_custkey = o_custkey
JOIN lineitem ON l_orderkey = o_orderkey
WHERE c_mktsegment = 'FURNITURE’
AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15
...
cost=1001
Heuristic
pruning
customer
31
Join Optimizer
Optimizer trace
32
Choosing Access Path
• Ref Access (index look-up)
• Read all records with a given key value using an index
• Examples:
SELECT * FROM t1 WHERE t1.key = 7;
SELECT * FROM t1, t2 WHERE t1.key = t2.key;
• “eq_ref”:
• Reading from a unique index, max one record returned
• “ref”:
• Reading from a non-unique index or a prefix of an unique index, possibly multiple records returned
• The record estimate is based on cardinality number from index statistics
• Indexes available for ref access depends on the join prefix
• Table/index scan
• The “winner” of the earlier ”range_analysis”
• Also considers:
• Join buffering (BNL/BKA)
• Filtering effects of conditions
Ref access vs. table/index scan
33
Choosing the Access Path
"plan_prefix": [
],
"table": "`customer`",
"best_access_path": {
"considered_access_paths": [
{
"access_type": "ref",
"index": "PRIMARY",
"usable": false,
"chosen": false
},
{
"rows_to_scan": 2367,
"filtering_effect": [
{
"condition": "(`customer`.`c_mktsegment` = 'FURNITURE')",
"histogram_selectivity": 0.2023
}
],
customer
"final_filtering_effect": 0.2023,
"access_type": "scan",
"resulting_rows": 478.9,
"cost": 244.2,
"chosen": true
}
]
},
"condition_filtering_pct": 100,
"rows_for_plan": 478.9,
"cost_for_plan": 244.2,
34
Choosing the Access Path
"rest_of_plan": [
{
"plan_prefix": [
"`customer`"
],
"table": "`lineitem`",
"best_access_path": {
"considered_access_paths": [
{
"access_type": "ref",
"index": "PRIMARY",
"usable": false,
"chosen": false
},
{
"access_type": "ref",
"index": "i_l_orderkey",
"usable": false,
"chosen": false
},
customer → lineitem
{
"rows_to_scan": 2495,
"filtering_effect": [
],
"final_filtering_effect": 1,
"access_type": "scan",
"using_join_cache": true,
"buffers_needed": 1,
"resulting_rows": 2495,
"cost": 119491,
"chosen": true
}
]
},
"condition_filtering_pct": 100,
"rows_for_plan": 1.19e6,
"cost_for_plan": 119735,
35
Choosing the Access Path
"rest_of_plan": [
{
"plan_prefix": [
"`customer`",
"`lineitem`"
],
"table": "`orders`",
"best_access_path": {
"considered_access_paths": [
{
"access_type": "eq_ref",
"index": "PRIMARY",
"rows": 1,
"cost": 120108,
"chosen": true,
"cause": "clustered_pk_chosen_by_heuristics"
},
customer → lineitem → orders
{
"rows_to_scan": 2502,
"filtering_effect": [
],
"final_filtering_effect": 0.8477,
"access_type": "scan",
"using_join_cache": true,
"buffers_needed": 292,
"resulting_rows": 2121,
"cost": 2.53e8,
"chosen": false
}
]
},
"condition_filtering_pct": 5,
"rows_for_plan": 59742,
"cost_for_plan": 239843,
"chosen": true
}
] 36
Condensed Trace For the Join Optimizer
• With many tables, the trace for join optimization may be thousands of lines
• The trace for DBT3 Q8 (8 tables) is 16000 lines
• Beware: Size for optimizer trace is limited by session variable optimizer_trace_max_mem_size
• Default MySQL 5.7: 16 kB
• Default MySQL 8.0: 1 MB
• If information_schema.optimizer_trace.missing_bytes_beyond_max_mem_size > 0,
increase optimizer_trace_max_mem_size
• joinopttrace.js
• A script to present the trace in a condensed form
• Available at https://github.com/ogrovlen/opttrace
• Usage
node joinopttrace.js tracefile
• Please:
• Test it out
• Suggest/submit improvements
• Report issues
37
joinopttrace
Table AccessType:IndexName Rows/Cost TotalRows/TotalCost
--------------------------------------------------------
`customer` scan 2367/244.2 478.9/244.2
`lineitem` scan 2495/119491 1190000/119735
`orders` eq_ref:PRIMARY 1/120108 59742/239843 *** NEW BEST PLAN ***
`orders` scan 2502/101621 478.9/101865
`lineitem` ref:PRIMARY 1.0155/168.35 486.3/102033 *** NEW BEST PLAN ***
`lineitem` scan 2495/255.5 2495/255.5
`customer` scan 2367/119716 1190000/119972 PRUNED(cost)
`orders` eq_ref:PRIMARY 1/873.25 2115.1/1128.8
`customer` eq_ref:PRIMARY 1/740.27 427.92/1869 *** NEW BEST PLAN ***
`orders` scan 2502/255.7 2121/255.7
`customer` eq_ref:PRIMARY 1/742.35 429.12/998.05
`lineitem` ref:PRIMARY 1.0155/150.86 435.76/1148.9 *** NEW BEST PLAN ***
`lineitem` ref:PRIMARY 1.0155/745.63 2153.8/1001.3 PRUNED(heuristic)
Example output (DBT3-Q3)
38
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Access method selection
Join order
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
39
Assigning Query Conditions to Tables
"attaching_conditions_to_tables": {
"original_condition": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and
(`customer`.`c_mktsegment` = 'FURNITURE') and (`orders`.`o_orderDATE` < DATE'1997-04-15') and (`lineitem`.`l_shipDATE` > DATE'1997-
04-15'))",
"attached_conditions_computation": [
{
"table": "`orders`",
"rechecking_index_usage": {
"recheck_reason": "low_limit",
"limit": 10,
"row_estimate": 2121 } } ],
"attached_conditions_summary": [
{
"table": "`orders`",
"attached": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))" },
{
"table": "`customer`",
"attached": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`customer`.`c_mktsegment` = 'FURNITURE'))" },
{
"table": "`lineitem`",
"attached": "((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`lineitem`.`l_shipDATE` > DATE'1997-04-15'))" } ] }
Evaluate conditions as early as possible in join order
40
Assigning Query Conditions to Tables
"finalizing_table_conditions": [
{
"table": "`orders`",
"original_table_condition": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))",
"final_table_condition ": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))” },
{
"table": "`customer`",
"original_table_condition": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`customer`.`c_mktsegment` = 'FURNITURE'))",
"final_table_condition ": "(`customer`.`c_mktsegment` = 'FURNITURE’)” },
{
"table": "`lineitem`",
"original_table_condition": "((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`lineitem`.`l_shipDATE` > DATE'1997-04-15'))",
"final_table_condition ": "(`lineitem`.`l_shipDATE` > DATE'1997-04-15’)” }
]
Remove conditions satisfied by ref access
41
ORDER BY Optimization
• Change to a different index that provides result in sorted order
• Read in descending order
• Example:
• Join queries:
• Join order is already fixed.
• Sorting can only be avoided if ordering is provided by an index from the first table in the join order
Avoid sorting, if possible
EXPLAIN SELECT * FROM orders WHERE o_totalprice > 400000
ORDER BY o_orderdate DESC LIMIT 10;
id
select
type
table type
possible
keys
key
key
len
rows filtered Extra
1 SIMPLE orders index NULL i_o_orderdate 4 10 0.57
Using where;
Backward index scan
42
Assigning Query Conditions to Tables
{
"optimizing_distinct_group_by_order_by": {
"simplifying_order_by": {
"original_clause": "`orders`.`o_orderDATE` desc",
"items": [
{
"item": "`orders`.`o_orderDATE`"
}
],
"resulting_clause_is_simple": true,
"resulting_clause": "`orders`.`o_orderDATE` desc"
}
}
},
{
"reconsidering_access_paths_for_index_ordering": {
"clause": "ORDER BY",
"steps": [
],
"index_order_summary": {
"table": "`orders`",
"index_provides_order": true,
"order_direction": "desc",
"index": "i_o_orderdate",
"plan_changed": true,
"access_type": "index"
}
}
},
Evaluate conditions as early as possible in join order
• Unfortunately, no information on cost calculations!
43
Index Condition Pushdown
• Pushes conditions that can be evaluated on the index
down to storage engine
• Works only on indexed columns
• Goal: evaluate conditions without having to access the
actual record
• Reduces number of disk/block accesses
• Reduces CPU usage
Query
conditions
Index
Table data
Storage engine
MySQL server
44
Index Condition Pushdown
How it works
Without ICP:
Storage Engine:
1. Reads index
2. Reads record
3. Returns record
Server:
4. Evaluates condition
With ICP:
Storage Engine:
1. Reads index and
evaluates pushed index
condition
2. Reads record
3. Returns record
Server:
4. Evaluates rest of condition
Execution
Index
Table data
2.
1. 3.
4.
Storage engine
MySQL server
Optimizer
45
Index Condition Pushdown
{
"refine_plan": [
{
"table": "`part`"
},
{
"table": "`lineitem` FORCE INDEX (`i_l_partkey_plus`)",
"pushed_index_condition": "((`lineitem`.`l_quantity` >= 1.00) and (`lineitem`.`l_quantity` <= <cache>((1 + 10))) and
(`lineitem`.`l_shipmode` in ('AIR','AIR REG')))",
"table_condition_attached": "(`lineitem`.`l_shipinstruct` = 'DELIVER IN PERSON')"
}
]
},
Optimizer trace
Index: lineitem(l_partkey,l_quantity,l_shipmode)
46
Prepare Temporary Tables
{
"considering_tmp_tables": [
{
"adding_tmp_table_in_plan_at_position": 3,
"write_method": "continuously_update_group_row"
},
{
"adding_sort_to_table_in_plan_at_position": 3
}
]
}
DBT3-Q3
SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate, o_shippriority
FROM customer JOIN orders ON c_custkey = o_custkey
JOIN lineitem ON l_orderkey = o_orderkey
WHERE c_mktsegment = 'FURNITURE’
AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15'
GROUP by l_orderkey, o_orderdate, o_shippriority
ORDER by revenue desc, o_orderdate
LIMIT 10;
47
Query Optimization
Main phases
Optimizer
Logical transformations
Cost-based optimizer:
Join order and access methods
Plan refinement
Query execution
plan
Query execution
Parser
Resolver:
Semantic check,name resolution
Storage engine
InnoDB MyISAM
Prepare for cost-based
optimization
Negation elimination
Equality and constant propagation
Evaluation of constant expressions
Substitution of generated columns
Ref access analysis
Range access analysis
Estimation of condition fan out
Constant table detection
Access method selection
Join order
Table condition pushdown
Access method adjustments
Sort avoidance
Index condition pushdown
Prepare temporary tables
48
Query Execution Trace
• Information on temporary table creation
• Sort parameters and summary
• Subquery execution
• Dynamic range optimization
• May create large volumes of trace
• Not available after EXPLAIN
• Volume may be reduced by setting session variable:
SET optimizer_trace_features = “dynamic_range=off, repeated_subselect=off”;
49
Query Execution Trace
{
"join_execution": {
"select#": 1,
"steps": [
{
"temp_table_aggregate": {
"select#": 1,
"steps": [
{
"creating_tmp_table": {
"tmp_table_info": {
"in_plan_at_position": 3,
"columns": 4,
"row_length": 34,
"key_length": 13,
"unique_constraint": false,
"makes_grouped_rows": true,
"cannot_insert_duplicates": false,
"location": "TempTable"
} } } ] }
Temporary table information
50
Query Execution Trace
{
"sorting_table_in_plan_at_position": 3,
"filesort_information": [
{
"direction": "desc",
"field": "revenue"
},
{
"direction": "asc",
"field": "o_orderDATE"
}
],
"filesort_priority_queue_optimization": {
"limit": 10,
"chosen": true
},
"filesort_execution": [
],
"filesort_summary": {
"memory_available": 262144,
"key_size": 33,
"row_size": 33,
"max_rows_per_buffer": 11,
"num_rows_estimate": 452,
"num_rows_found": 442,
"num_initial_chunks_spilled_to_disk": 0,
"peak_memory_used": 451,
"sort_algorithm": "std::sort",
"unpacked_addon_fields": "using_priority_queue",
"sort_mode": "<fixed_sort_key, rowid>"
}
}
Sort information
51
A Last Tip
• Example: Present the filesort summary from the last execution:
SELECT JSON_PRETTY(trace->'$.steps[*].join_execution.steps[*].filesort_summary’)
FROM information_schema.optimizer_trace;
• Queries against information_schema.optimizer_trace will not generate a new trace, so
you can execute multiple queries on the same trace.
Use MySQL JSON functionality to process the optimizer trace
52
More information
• The optimizer trace manual: https://dev.mysql.com/doc/internals/en/optimizer-tracing.html
• Enable/disable trace of selected features of the optimizer
• Tune trace purging (by default an optimizer trace overwrites the previous trace)
• Make it more human readable (but no longer valid JSON)
• My blog: https://oysteing.blogspot.com
• For information about MySQL related work at Alibaba:
Attend the AliSQL & POLARDB track tomorrow
53
Thank You!
Please, remember to rate this session in the
Percona Live APP!
54
55

More Related Content

What's hot

MySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer GuideMySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer Guide
Morgan Tocker
 
MySQL 8.0 EXPLAIN ANALYZE
MySQL 8.0 EXPLAIN ANALYZEMySQL 8.0 EXPLAIN ANALYZE
MySQL 8.0 EXPLAIN ANALYZE
Norvald Ryeng
 
MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바
NeoClova
 
MySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software TestMySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software Test
I Goo Lee
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuning
elliando dias
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning
Alexander Rubin
 
Postgresql database administration volume 1
Postgresql database administration volume 1Postgresql database administration volume 1
Postgresql database administration volume 1
Federico Campoli
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScale
MariaDB plc
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performance
MariaDB plc
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and Engine
Abdul Manaf
 
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
NAVER D2
 
MySQL: Indexing for Better Performance
MySQL: Indexing for Better PerformanceMySQL: Indexing for Better Performance
MySQL: Indexing for Better Performance
jkeriaki
 
The PostgreSQL Query Planner
The PostgreSQL Query PlannerThe PostgreSQL Query Planner
The PostgreSQL Query Planner
Command Prompt., Inc
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
PGConf APAC
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova
 
MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
Sql query patterns, optimized
Sql query patterns, optimizedSql query patterns, optimized
Sql query patterns, optimized
Karwin Software Solutions LLC
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
Altinity Ltd
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
Karwin Software Solutions LLC
 
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
PgDay.Seoul
 

What's hot (20)

MySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer GuideMySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer Guide
 
MySQL 8.0 EXPLAIN ANALYZE
MySQL 8.0 EXPLAIN ANALYZEMySQL 8.0 EXPLAIN ANALYZE
MySQL 8.0 EXPLAIN ANALYZE
 
MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바
 
MySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software TestMySQL/MariaDB Proxy Software Test
MySQL/MariaDB Proxy Software Test
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuning
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning
 
Postgresql database administration volume 1
Postgresql database administration volume 1Postgresql database administration volume 1
Postgresql database administration volume 1
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScale
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performance
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and Engine
 
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
 
MySQL: Indexing for Better Performance
MySQL: Indexing for Better PerformanceMySQL: Indexing for Better Performance
MySQL: Indexing for Better Performance
 
The PostgreSQL Query Planner
The PostgreSQL Query PlannerThe PostgreSQL Query Planner
The PostgreSQL Query Planner
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
 
MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6
 
Sql query patterns, optimized
Sql query patterns, optimizedSql query patterns, optimized
Sql query patterns, optimized
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
 

Similar to The MySQL Query Optimizer Explained Through Optimizer Trace

SQL Performance Solutions: Refactor Mercilessly, Index Wisely
SQL Performance Solutions: Refactor Mercilessly, Index WiselySQL Performance Solutions: Refactor Mercilessly, Index Wisely
SQL Performance Solutions: Refactor Mercilessly, Index Wisely
Enkitec
 
Optimizer Trace Walkthrough
Optimizer Trace WalkthroughOptimizer Trace Walkthrough
Optimizer Trace Walkthrough
Sergey Petrunya
 
MariaDB Optimizer - further down the rabbit hole
MariaDB Optimizer - further down the rabbit holeMariaDB Optimizer - further down the rabbit hole
MariaDB Optimizer - further down the rabbit hole
Sergey Petrunya
 
Query Optimizer: further down the rabbit hole
Query Optimizer: further down the rabbit holeQuery Optimizer: further down the rabbit hole
Query Optimizer: further down the rabbit hole
MariaDB plc
 
Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007
paulguerin
 
Histograms: Pre-12c and now
Histograms: Pre-12c and nowHistograms: Pre-12c and now
Histograms: Pre-12c and now
Anju Garg
 
A few things about the Oracle optimizer - 2013
A few things about the Oracle optimizer - 2013A few things about the Oracle optimizer - 2013
A few things about the Oracle optimizer - 2013
Connor McDonald
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
Connor McDonald
 
Ground Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_planGround Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_plan
Maria Colgan
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
Clayton Groom
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internals
Alexey Ermakov
 
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project ADN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
Dataconomy Media
 
解决Ora 14098分区交换索引不匹配错误
解决Ora 14098分区交换索引不匹配错误解决Ora 14098分区交换索引不匹配错误
解决Ora 14098分区交换索引不匹配错误
maclean liu
 
Sql
SqlSql
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
EDB
 
Dramatically increase your database's performance using hierarchical and recu...
Dramatically increase your database's performance using hierarchical and recu...Dramatically increase your database's performance using hierarchical and recu...
Dramatically increase your database's performance using hierarchical and recu...
rcmoutinho
 
lesson 2.pptx
lesson 2.pptxlesson 2.pptx
lesson 2.pptx
khaledahmed316
 
Les09 Manipulating Data
Les09 Manipulating DataLes09 Manipulating Data
The Ring programming language version 1.7 book - Part 10 of 196
The Ring programming language version 1.7 book - Part 10 of 196The Ring programming language version 1.7 book - Part 10 of 196
The Ring programming language version 1.7 book - Part 10 of 196
Mahmoud Samir Fayed
 
Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance
Alexey Ermakov
 

Similar to The MySQL Query Optimizer Explained Through Optimizer Trace (20)

SQL Performance Solutions: Refactor Mercilessly, Index Wisely
SQL Performance Solutions: Refactor Mercilessly, Index WiselySQL Performance Solutions: Refactor Mercilessly, Index Wisely
SQL Performance Solutions: Refactor Mercilessly, Index Wisely
 
Optimizer Trace Walkthrough
Optimizer Trace WalkthroughOptimizer Trace Walkthrough
Optimizer Trace Walkthrough
 
MariaDB Optimizer - further down the rabbit hole
MariaDB Optimizer - further down the rabbit holeMariaDB Optimizer - further down the rabbit hole
MariaDB Optimizer - further down the rabbit hole
 
Query Optimizer: further down the rabbit hole
Query Optimizer: further down the rabbit holeQuery Optimizer: further down the rabbit hole
Query Optimizer: further down the rabbit hole
 
Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007
 
Histograms: Pre-12c and now
Histograms: Pre-12c and nowHistograms: Pre-12c and now
Histograms: Pre-12c and now
 
A few things about the Oracle optimizer - 2013
A few things about the Oracle optimizer - 2013A few things about the Oracle optimizer - 2013
A few things about the Oracle optimizer - 2013
 
How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012How to tune a query - ODTUG 2012
How to tune a query - ODTUG 2012
 
Ground Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_planGround Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_plan
 
Simplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functionsSimplifying SQL with CTE's and windowing functions
Simplifying SQL with CTE's and windowing functions
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internals
 
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project ADN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
DN 2017 | Reducing pain in data engineering | Martin Loetzsch | Project A
 
解决Ora 14098分区交换索引不匹配错误
解决Ora 14098分区交换索引不匹配错误解决Ora 14098分区交换索引不匹配错误
解决Ora 14098分区交换索引不匹配错误
 
Sql
SqlSql
Sql
 
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
Understand the Query Plan to Optimize Performance with EXPLAIN and EXPLAIN AN...
 
Dramatically increase your database's performance using hierarchical and recu...
Dramatically increase your database's performance using hierarchical and recu...Dramatically increase your database's performance using hierarchical and recu...
Dramatically increase your database's performance using hierarchical and recu...
 
lesson 2.pptx
lesson 2.pptxlesson 2.pptx
lesson 2.pptx
 
Les09 Manipulating Data
Les09 Manipulating DataLes09 Manipulating Data
Les09 Manipulating Data
 
The Ring programming language version 1.7 book - Part 10 of 196
The Ring programming language version 1.7 book - Part 10 of 196The Ring programming language version 1.7 book - Part 10 of 196
The Ring programming language version 1.7 book - Part 10 of 196
 
Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance
 

More from oysteing

POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
oysteing
 
POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
oysteing
 
POLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel QueryPOLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel Query
oysteing
 
JSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worldsJSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worlds
oysteing
 
Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0
oysteing
 
MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0
oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
Common Table Expressions (CTE) & Window Functions in MySQL 8.0
Common Table Expressions (CTE) & Window Functions in MySQL 8.0Common Table Expressions (CTE) & Window Functions in MySQL 8.0
Common Table Expressions (CTE) & Window Functions in MySQL 8.0
oysteing
 
How to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performanceHow to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performance
oysteing
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table ExpressionsMySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
oysteing
 
How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016
oysteing
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15
oysteing
 
How to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinarHow to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinar
oysteing
 

More from oysteing (15)

POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
 
POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
 
POLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel QueryPOLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel Query
 
JSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worldsJSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worlds
 
Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0
 
MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
Common Table Expressions (CTE) & Window Functions in MySQL 8.0
Common Table Expressions (CTE) & Window Functions in MySQL 8.0Common Table Expressions (CTE) & Window Functions in MySQL 8.0
Common Table Expressions (CTE) & Window Functions in MySQL 8.0
 
How to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performanceHow to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performance
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table ExpressionsMySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
 
How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15
 
How to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinarHow to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinar
 

Recently uploaded

Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
blueshagoo1
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
Vietnam Cotton & Spinning Association
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
hiju9823
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
hqfek
 
Senior Software Profiles Backend Sample - Sheet1.pdf
Senior Software Profiles  Backend Sample - Sheet1.pdfSenior Software Profiles  Backend Sample - Sheet1.pdf
Senior Software Profiles Backend Sample - Sheet1.pdf
Vineet
 
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
actyx
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
ArshadAyub49
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
nyvan3
 
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Marlon Dumas
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
keesa2
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
eudsoh
 
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdfsaps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
newdirectionconsulta
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
GeorgiiSteshenko
 
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdfOverview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
nhutnguyen355078
 
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
zoykygu
 
Q4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slideQ4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slide
mukulupadhayay1
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
eoxhsaa
 
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions?  Correspondence between Sel...Do People Really Know Their Fertility Intentions?  Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Xiao Xu
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
prijesh mathew
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
sapna sharmap11
 

Recently uploaded (20)

Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
 
Senior Software Profiles Backend Sample - Sheet1.pdf
Senior Software Profiles  Backend Sample - Sheet1.pdfSenior Software Profiles  Backend Sample - Sheet1.pdf
Senior Software Profiles Backend Sample - Sheet1.pdf
 
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
 
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
Discovering Digital Process Twins for What-if Analysis: a Process Mining Appr...
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
 
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdfsaps4hanaandsapanalyticswheretodowhat1565272000538.pdf
saps4hanaandsapanalyticswheretodowhat1565272000538.pdf
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
 
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdfOverview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
 
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
一比一原版(heriotwatt学位证书)英国赫瑞瓦特大学毕业证如何办理
 
Q4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slideQ4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slide
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
 
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions?  Correspondence between Sel...Do People Really Know Their Fertility Intentions?  Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
 

The MySQL Query Optimizer Explained Through Optimizer Trace

  • 1. 1 The MySQL Query Optimizer Explained Through Optimizer Trace Øystein Grøvlen Senior Staff Engineer Alibaba Cloud
  • 2. MySQL Query Optimizer SELECT a, b FROM t1, t2, t3 WHERE t1.a = t2.b AND t2.b = t3.c AND t2.d > 20 AND t2.d < 30; Table/index info (data dictionary) Statistics (storage engines) t2 t3 t1 Table scan Range scan Ref access JOIN JOINQuery Optimizer 2
  • 3. MySQL Architecture Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver (Prepare): Semantic check,name resolution Storage Engine InnoDB MyISAM SQL query Query result 3
  • 4. Optimizer Trace • EXPLAIN shows the selected plan • Optimizer trace shows WHY the plan was selected SET optimizer_trace= "enabled=on"; SELECT * FROM t1, t2 WHERE f1=1 AND f1=f2 AND f2>0; SELECT trace FROM information_schema.optimizer_trace INTO OUTFILE filename LINES TERMINATED BY ’’; SET optimizer_trace="enabled=off"; QUERY SELECT * FROM t1, t2 WHERE f1=1 AND f1=f2 AND f2>0; TRACE { "steps": [ { "join_preparation": { "select#": 1,… } … } …] } MISSING_BYTES_BEYOND_MAX_MEM_SIZE 0 INSUFFICIENT_PRIVILEGES 0 How to generate it 4
  • 5. Optimizer Trace { "steps": [ { "join_preparation": { "select#": 1, "steps": [ { "expanded_query": "/* select#1 */ select `t1`.`f1` AS `f1`,`t2`.`f2` AS `f2` from `t1` join `t2` where ((`t1`.`f1` = 1) and (`t1`.`f1` = `t2`.`f2`) and (`t2`.`f2` > 0))" } ] } }, { "join_optimization": { "select#": 1, "steps": [ { "condition_processing": { "condition": "WHERE", "original_condition": "((`t1`.`f1` = 1) and (`t1`.`f1` = `t2`.`f2`) and (`t2`.`f2` > 0))", "steps": [ { … Example 5
  • 7. Browser Plugin Collapse to see main trace objects/phases 7
  • 9. Prepare Phase • Name resolving • Map names to database objects (tables, columns, …) • Semantic checks • Permanent transformations: • Conversion of outer join to inner join • Merging of views and derived tables • Subquery transformations • IN to EXISTS • IN to Semijoin (5.6) • EXISTS to IN (8.0.16) • Etc. Simpler query to optimize and execute Prepare for later optimizations 9
  • 10. Conversion of outer join to inner join SELECT o_orderkey FROM orders LEFT JOIN lineitem ON o_orderkey = l_orderkey WHERE l_discount > 0.10; "join_preparation": { "select#": 1, "steps": [ { "expanded_query": "/* select#1 */ select `orders`.`o_orderkey` AS `o_orderkey` from (`orders` left join `lineitem` on ((`orders`.`o_orderkey` = `lineitem`.`l_orderkey`))) where (`lineitem`.`l_discount` > 0.10)" }, { "transformations_to_nested_joins": { "transformations": [ "outer_join_to_inner_join", "JOIN_condition_to_WHERE", "parenthesis_removal" ], "expanded_query": "/* select#1 */ select `orders`.`o_orderkey` AS `o_orderkey` from `orders` join `lineitem` where ((`lineitem`.`l_discount` > 0.10) and(`orders`.`o_orderkey` = `lineitem`.`l_orderkey`))" } … 10
  • 11. Conversion from EXISTS-subquery to IN-subquery SELECT o_orderpriority, COUNT(*) AS order_count FROM orders WHERE EXISTS (SELECT * FROM lineitem WHERE l_orderkey = o_orderkey AND l_commitdate < l_receiptdate) GROUP BY o_orderpriority ORDER BY o_orderpriority; SELECT o_orderpriority, COUNT(*) AS order_count FROM orders WHERE o_orderkey IN (SELECT l_orderkey FROM lineitem WHERE l_commitdate < l_receiptdate GROUP BY o_orderpriority ORDER BY o_orderpriority; SELECT o_orderpriority, COUNT(*) AS order_count FROM orders SEMIJOIN lineitem ON l_orderkey = o_orderkey WHERE l_commitdate < l_receiptdate GROUP BY o_orderpriority ORDER BY o_orderpriority; New in 8.0.16 11
  • 12. Conversion from EXISTS-subquery to IN-subquery "join_preparation": { "select#": 1, "steps": [ { "join_preparation": { "select#": 2, "steps": [ { "expanded_query": "/* select#2 */ select 1 from `lineitem` where ((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`lineitem`.`l_commitDATE` < `lineitem`.`l_receiptDATE`))" }, { "transformation": { "select#": 2, "from": "EXISTS (SELECT)", "to": "semijoin", "chosen": true } } ] … Optimizer trace 12
  • 13. Conversion from EXISTS-subquery to IN-subquery Optimizer trace, cont. 13
  • 14. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables Access method selection Join order 14
  • 15. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Access method selection Join order Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables 15
  • 16. =TRUE Trivial condition removal t1.a = 9 AND t2.a = 9 AND (t2.b <= 3 OR (t1.b = 12 AND t2.b = 5)); t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 5 + 7 AND t2.b = 5)); Equality/const propagation Logical Transformations Condition processing t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 12 AND t2.b = 5)); Evaluate const expressions SELECT * FROM t1, t2 WHERE t1.a = t2.a AND t2.a = 9 AND (NOT (t1.a > 10 OR t2.b > 3) OR (t1.b = t2.b + 7 AND t2.b = 5)); Negation elimination t1.a = t2.a AND t2.a = 9 AND (t1.a <= 10 AND t2.b <= 3 OR (t1.b = t2.b + 7 AND t2.b = 5)); 16
  • 17. Logical Transformations "join_optimization": { "select#": 1, "steps": [ { "condition_processing": { "condition": "WHERE", "original_condition": "((`t1`.`a` = `t2`.`a`) and (`t2`.`a` = 9) and (((`t1`.`a` <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` =(`t2`.`b` + 7)) and (`t2`.`b` = 5))))", "steps": [ { "transformation": "equality_propagation", "resulting_condition": "((((9 <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` = (5 + 7)) and multiple equal(5, `t2`.`b`))) and multiple equal(9, `t1`.`a`, `t2`.`a`))" }, { "transformation": "constant_propagation", "resulting_condition": "((((9 <= 10) and (`t2`.`b` <= 3)) or ((`t1`.`b` = 12) and multiple equal(5, `t2`.`b`))) and multiple equal(9, `t1`.`a` , `t2`.`a`))" }, { "transformation": "trivial_condition_removal", "resulting_condition": "(((`t2`.`b` <= 3) or ((`t1`.`b` = 12) and multiple equal(5, `t2`.`b`))) and multiple equal(9, `t1`.`a`, `t2`.`a`))" } Optimizer Trace
  • 18. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Access method selection Join order Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables 18
  • 19. Ref Access Analysis Determine which indexes that can be used for index lookup in a join SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer JOIN orders ON c_custkey = o_custkey JOIN lineitem ON l_orderkey = o_orderkey WHERE c_mktsegment = 'FURNITURE’ AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15' GROUP by l_orderkey, o_orderdate, o_shippriority ORDER by revenue desc, o_orderdate LIMIT 10; customer c_custkey orders o_orderkey o_custkey lineitem l_orderkey l_linenumber 19
  • 20. Ref Access Analysis { "ref_optimizer_key_uses": [ { "table": "`customer`", "field": "c_custkey", "equals": "`orders`.`o_custkey`", "null_rejecting": true }, { "table": "`orders`", "field": "o_orderkey", "equals": "`lineitem`.`l_orderkey`", "null_rejecting": false }, { "table": "`lineitem`", "field": "l_orderkey", "equals": "`orders`.`o_orderkey`", "null_rejecting": false }, { "table": "`lineitem`", "field": "l_orderkey", "equals": "`orders`.`o_orderkey`", "null_rejecting": false } ] }, Optimizer trace 20
  • 21. Cost-based Query Optimization General idea: • Assign cost to operations • Assign cost to partial or alternative plans • Search for plan with lowest cost t2 t3 t1 Table scan Range scan Ref access JOIN JOIN 21
  • 22. Cost-based Query Optimizations The main cost-based optimizations: • Index and access method: • Table scan • Index scan • Range scan • Index lookup (ref access) • Join order • Join buffering strategy • Subquery strategy t2 t3 t1 Table scan Range scan Ref access JOIN JOIN 22
  • 23. Optimizer Cost Model t1 Cost estimate Row estimate Cost Model Cost formulas Access methods Join Subquery Cost constants CPU IO Metadata: - Record and index size - Index information - Uniqueness Statistics: - Table size - Cardinality - Range estimates - Histograms Cost model configuration Range scan JOIN 23
  • 24. Cost Estimates • The cost for executing a query • Cost unit: • “read a random data page from disk” • Main cost factors: • IO cost: • #pages read from table • #pages read from index • CPU cost: • Evaluating query conditions • Comparing keys/records • Sorting keys Cost MySQL 5.7 MySQL 8.0 Read a random disk page 1.0 1.0 Read a data page from memory buffer 1.0 0.25 Evaluate query condition 0.2 0.1 Compare keys/records 0.1 0.05 Main cost constants 24
  • 25. Range analysis • Cost of table scan • Based on number of pages to read • Cost of index scan alternatives • Index scan (covering) • Index range scan • Index merge • Index for grouping • Skip scan (new in 8.0) • Loose-index scan Compare cost of table scan to index scan 25
  • 26. Range Analysis Example Table scan: • IO-cost: #pages in table * IO_BLOCK_READ_COST • CPU cost: #rows * ROW_EVALUATE_COST Range scan (on secondary index): • IO-cost: #rows_in_range * IO_BLOCK_READ_COST • CPU cost: #rows_in_range * ROW_EVALUATE_COST SELECT SUM(o_totalprice) FROM orders WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-12-31'; Table scan vs Index range scan 26
  • 27. Range Analysis Example EXPLAIN SELECT SUM(o_totalprice) FROM orders WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-12-31'; EXPLAIN SELECT SUM(o_totalprice) FROM orders WHERE o_orderdate BETWEEN '1994-01-01' AND '1994-06-30'; id select type table type possible keys key key len rows filtered Extra 1 SIMPLE orders ALL i_o_orderdate NULL NULL 1480845 31.13 Using where Id select type table type possible keys key key len rows filtered Extra 1 SIMPLE orders range i_o_orderdate i_o_orderdate 4 222102 100.00 Using index condition Table scan vs index range scan cnt. 27
  • 28. "rows_estimation": [ { "table": "`orders`", "range_analysis": { "table_scan": { "rows": 1480845, "cost": 151138 }, "potential_range_indexes": [ { "index": "PRIMARY", "usable": false, "cause": "not_applicable" }, { "index": "i_o_orderdate", "usable": true, "key_parts": [ "o_orderDATE", "o_orderkey" ] } ], "setup_range_conditions": [ ], "group_index_range": { "chosen": false, "cause": "not_group_by_or_distinct" }, "skip_scan_range": { "potential_skip_scan_indexes": [ { "index": "i_o_orderdate", "usable": false, "cause": "query_references_nonkey_column" } ] }, "analyzing_range_alternatives": { "range_scan_alternatives": [ { "index": "i_o_orderdate", "ranges": [ "0x21940f <= o_orderDATE <= 0x9f950f" ], "index_dives_for_eq_ranges": true, "rowid_ordered": false, "using_mrr": false, "index_only": false, "rows": 460938, "cost": 161329, "chosen": false, "cause": "cost" } ], "analyzing_roworder_intersect": { "usable": false, "cause": "too_few_roworder_scans" } } } }] Range Analysis Example Optimizer trace 28
  • 29. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Join order Access method selection Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables 29
  • 30. Join Optimizer • Goal: Given a JOIN of N tables, find the best JOIN ordering • Only considers left-deap plans • Strategy: • Start with all 1-table plans • Sorted based on size and key dependency • Expand each plan with remaining tables • Depth-first • If “cost of partial plan” > “cost of best plan”: • “Prune” plan • Heuristic pruning: • Prune less promising partial plans • May in rare cases miss most optimal plan (turn off with set optimizer_prune_level = 0) ”Greedy search strategy” t1 t2 t2 t2 t2 t3 t3 t3 t4t4 t4 t4t4 t3 t3 t2 t4t2 t3 N! possible plans 30
  • 31. Join Optimizer Illustrated DBT3-Q3 customer lineitem orders customer customer customerlineitem lineitem lineitem lineitem orders ordersorders orders cost=239843 cost=102033 cost=119972 cost=1869 cost=1149 start SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer JOIN orders ON c_custkey = o_custkey JOIN lineitem ON l_orderkey = o_orderkey WHERE c_mktsegment = 'FURNITURE’ AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15 ... cost=1001 Heuristic pruning customer 31
  • 33. Choosing Access Path • Ref Access (index look-up) • Read all records with a given key value using an index • Examples: SELECT * FROM t1 WHERE t1.key = 7; SELECT * FROM t1, t2 WHERE t1.key = t2.key; • “eq_ref”: • Reading from a unique index, max one record returned • “ref”: • Reading from a non-unique index or a prefix of an unique index, possibly multiple records returned • The record estimate is based on cardinality number from index statistics • Indexes available for ref access depends on the join prefix • Table/index scan • The “winner” of the earlier ”range_analysis” • Also considers: • Join buffering (BNL/BKA) • Filtering effects of conditions Ref access vs. table/index scan 33
  • 34. Choosing the Access Path "plan_prefix": [ ], "table": "`customer`", "best_access_path": { "considered_access_paths": [ { "access_type": "ref", "index": "PRIMARY", "usable": false, "chosen": false }, { "rows_to_scan": 2367, "filtering_effect": [ { "condition": "(`customer`.`c_mktsegment` = 'FURNITURE')", "histogram_selectivity": 0.2023 } ], customer "final_filtering_effect": 0.2023, "access_type": "scan", "resulting_rows": 478.9, "cost": 244.2, "chosen": true } ] }, "condition_filtering_pct": 100, "rows_for_plan": 478.9, "cost_for_plan": 244.2, 34
  • 35. Choosing the Access Path "rest_of_plan": [ { "plan_prefix": [ "`customer`" ], "table": "`lineitem`", "best_access_path": { "considered_access_paths": [ { "access_type": "ref", "index": "PRIMARY", "usable": false, "chosen": false }, { "access_type": "ref", "index": "i_l_orderkey", "usable": false, "chosen": false }, customer → lineitem { "rows_to_scan": 2495, "filtering_effect": [ ], "final_filtering_effect": 1, "access_type": "scan", "using_join_cache": true, "buffers_needed": 1, "resulting_rows": 2495, "cost": 119491, "chosen": true } ] }, "condition_filtering_pct": 100, "rows_for_plan": 1.19e6, "cost_for_plan": 119735, 35
  • 36. Choosing the Access Path "rest_of_plan": [ { "plan_prefix": [ "`customer`", "`lineitem`" ], "table": "`orders`", "best_access_path": { "considered_access_paths": [ { "access_type": "eq_ref", "index": "PRIMARY", "rows": 1, "cost": 120108, "chosen": true, "cause": "clustered_pk_chosen_by_heuristics" }, customer → lineitem → orders { "rows_to_scan": 2502, "filtering_effect": [ ], "final_filtering_effect": 0.8477, "access_type": "scan", "using_join_cache": true, "buffers_needed": 292, "resulting_rows": 2121, "cost": 2.53e8, "chosen": false } ] }, "condition_filtering_pct": 5, "rows_for_plan": 59742, "cost_for_plan": 239843, "chosen": true } ] 36
  • 37. Condensed Trace For the Join Optimizer • With many tables, the trace for join optimization may be thousands of lines • The trace for DBT3 Q8 (8 tables) is 16000 lines • Beware: Size for optimizer trace is limited by session variable optimizer_trace_max_mem_size • Default MySQL 5.7: 16 kB • Default MySQL 8.0: 1 MB • If information_schema.optimizer_trace.missing_bytes_beyond_max_mem_size > 0, increase optimizer_trace_max_mem_size • joinopttrace.js • A script to present the trace in a condensed form • Available at https://github.com/ogrovlen/opttrace • Usage node joinopttrace.js tracefile • Please: • Test it out • Suggest/submit improvements • Report issues 37
  • 38. joinopttrace Table AccessType:IndexName Rows/Cost TotalRows/TotalCost -------------------------------------------------------- `customer` scan 2367/244.2 478.9/244.2 `lineitem` scan 2495/119491 1190000/119735 `orders` eq_ref:PRIMARY 1/120108 59742/239843 *** NEW BEST PLAN *** `orders` scan 2502/101621 478.9/101865 `lineitem` ref:PRIMARY 1.0155/168.35 486.3/102033 *** NEW BEST PLAN *** `lineitem` scan 2495/255.5 2495/255.5 `customer` scan 2367/119716 1190000/119972 PRUNED(cost) `orders` eq_ref:PRIMARY 1/873.25 2115.1/1128.8 `customer` eq_ref:PRIMARY 1/740.27 427.92/1869 *** NEW BEST PLAN *** `orders` scan 2502/255.7 2121/255.7 `customer` eq_ref:PRIMARY 1/742.35 429.12/998.05 `lineitem` ref:PRIMARY 1.0155/150.86 435.76/1148.9 *** NEW BEST PLAN *** `lineitem` ref:PRIMARY 1.0155/745.63 2153.8/1001.3 PRUNED(heuristic) Example output (DBT3-Q3) 38
  • 39. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Access method selection Join order Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables 39
  • 40. Assigning Query Conditions to Tables "attaching_conditions_to_tables": { "original_condition": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`customer`.`c_mktsegment` = 'FURNITURE') and (`orders`.`o_orderDATE` < DATE'1997-04-15') and (`lineitem`.`l_shipDATE` > DATE'1997- 04-15'))", "attached_conditions_computation": [ { "table": "`orders`", "rechecking_index_usage": { "recheck_reason": "low_limit", "limit": 10, "row_estimate": 2121 } } ], "attached_conditions_summary": [ { "table": "`orders`", "attached": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))" }, { "table": "`customer`", "attached": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`customer`.`c_mktsegment` = 'FURNITURE'))" }, { "table": "`lineitem`", "attached": "((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`lineitem`.`l_shipDATE` > DATE'1997-04-15'))" } ] } Evaluate conditions as early as possible in join order 40
  • 41. Assigning Query Conditions to Tables "finalizing_table_conditions": [ { "table": "`orders`", "original_table_condition": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))", "final_table_condition ": "((`orders`.`o_orderDATE` < DATE'1997-04-15') and (`orders`.`o_custkey` is not null))” }, { "table": "`customer`", "original_table_condition": "((`customer`.`c_custkey` = `orders`.`o_custkey`) and (`customer`.`c_mktsegment` = 'FURNITURE'))", "final_table_condition ": "(`customer`.`c_mktsegment` = 'FURNITURE’)” }, { "table": "`lineitem`", "original_table_condition": "((`lineitem`.`l_orderkey` = `orders`.`o_orderkey`) and (`lineitem`.`l_shipDATE` > DATE'1997-04-15'))", "final_table_condition ": "(`lineitem`.`l_shipDATE` > DATE'1997-04-15’)” } ] Remove conditions satisfied by ref access 41
  • 42. ORDER BY Optimization • Change to a different index that provides result in sorted order • Read in descending order • Example: • Join queries: • Join order is already fixed. • Sorting can only be avoided if ordering is provided by an index from the first table in the join order Avoid sorting, if possible EXPLAIN SELECT * FROM orders WHERE o_totalprice > 400000 ORDER BY o_orderdate DESC LIMIT 10; id select type table type possible keys key key len rows filtered Extra 1 SIMPLE orders index NULL i_o_orderdate 4 10 0.57 Using where; Backward index scan 42
  • 43. Assigning Query Conditions to Tables { "optimizing_distinct_group_by_order_by": { "simplifying_order_by": { "original_clause": "`orders`.`o_orderDATE` desc", "items": [ { "item": "`orders`.`o_orderDATE`" } ], "resulting_clause_is_simple": true, "resulting_clause": "`orders`.`o_orderDATE` desc" } } }, { "reconsidering_access_paths_for_index_ordering": { "clause": "ORDER BY", "steps": [ ], "index_order_summary": { "table": "`orders`", "index_provides_order": true, "order_direction": "desc", "index": "i_o_orderdate", "plan_changed": true, "access_type": "index" } } }, Evaluate conditions as early as possible in join order • Unfortunately, no information on cost calculations! 43
  • 44. Index Condition Pushdown • Pushes conditions that can be evaluated on the index down to storage engine • Works only on indexed columns • Goal: evaluate conditions without having to access the actual record • Reduces number of disk/block accesses • Reduces CPU usage Query conditions Index Table data Storage engine MySQL server 44
  • 45. Index Condition Pushdown How it works Without ICP: Storage Engine: 1. Reads index 2. Reads record 3. Returns record Server: 4. Evaluates condition With ICP: Storage Engine: 1. Reads index and evaluates pushed index condition 2. Reads record 3. Returns record Server: 4. Evaluates rest of condition Execution Index Table data 2. 1. 3. 4. Storage engine MySQL server Optimizer 45
  • 46. Index Condition Pushdown { "refine_plan": [ { "table": "`part`" }, { "table": "`lineitem` FORCE INDEX (`i_l_partkey_plus`)", "pushed_index_condition": "((`lineitem`.`l_quantity` >= 1.00) and (`lineitem`.`l_quantity` <= <cache>((1 + 10))) and (`lineitem`.`l_shipmode` in ('AIR','AIR REG')))", "table_condition_attached": "(`lineitem`.`l_shipinstruct` = 'DELIVER IN PERSON')" } ] }, Optimizer trace Index: lineitem(l_partkey,l_quantity,l_shipmode) 46
  • 47. Prepare Temporary Tables { "considering_tmp_tables": [ { "adding_tmp_table_in_plan_at_position": 3, "write_method": "continuously_update_group_row" }, { "adding_sort_to_table_in_plan_at_position": 3 } ] } DBT3-Q3 SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer JOIN orders ON c_custkey = o_custkey JOIN lineitem ON l_orderkey = o_orderkey WHERE c_mktsegment = 'FURNITURE’ AND o_orderdate < '1997-04-15' AND l_shipdate > '1997-04-15' GROUP by l_orderkey, o_orderdate, o_shippriority ORDER by revenue desc, o_orderdate LIMIT 10; 47
  • 48. Query Optimization Main phases Optimizer Logical transformations Cost-based optimizer: Join order and access methods Plan refinement Query execution plan Query execution Parser Resolver: Semantic check,name resolution Storage engine InnoDB MyISAM Prepare for cost-based optimization Negation elimination Equality and constant propagation Evaluation of constant expressions Substitution of generated columns Ref access analysis Range access analysis Estimation of condition fan out Constant table detection Access method selection Join order Table condition pushdown Access method adjustments Sort avoidance Index condition pushdown Prepare temporary tables 48
  • 49. Query Execution Trace • Information on temporary table creation • Sort parameters and summary • Subquery execution • Dynamic range optimization • May create large volumes of trace • Not available after EXPLAIN • Volume may be reduced by setting session variable: SET optimizer_trace_features = “dynamic_range=off, repeated_subselect=off”; 49
  • 50. Query Execution Trace { "join_execution": { "select#": 1, "steps": [ { "temp_table_aggregate": { "select#": 1, "steps": [ { "creating_tmp_table": { "tmp_table_info": { "in_plan_at_position": 3, "columns": 4, "row_length": 34, "key_length": 13, "unique_constraint": false, "makes_grouped_rows": true, "cannot_insert_duplicates": false, "location": "TempTable" } } } ] } Temporary table information 50
  • 51. Query Execution Trace { "sorting_table_in_plan_at_position": 3, "filesort_information": [ { "direction": "desc", "field": "revenue" }, { "direction": "asc", "field": "o_orderDATE" } ], "filesort_priority_queue_optimization": { "limit": 10, "chosen": true }, "filesort_execution": [ ], "filesort_summary": { "memory_available": 262144, "key_size": 33, "row_size": 33, "max_rows_per_buffer": 11, "num_rows_estimate": 452, "num_rows_found": 442, "num_initial_chunks_spilled_to_disk": 0, "peak_memory_used": 451, "sort_algorithm": "std::sort", "unpacked_addon_fields": "using_priority_queue", "sort_mode": "<fixed_sort_key, rowid>" } } Sort information 51
  • 52. A Last Tip • Example: Present the filesort summary from the last execution: SELECT JSON_PRETTY(trace->'$.steps[*].join_execution.steps[*].filesort_summary’) FROM information_schema.optimizer_trace; • Queries against information_schema.optimizer_trace will not generate a new trace, so you can execute multiple queries on the same trace. Use MySQL JSON functionality to process the optimizer trace 52
  • 53. More information • The optimizer trace manual: https://dev.mysql.com/doc/internals/en/optimizer-tracing.html • Enable/disable trace of selected features of the optimizer • Tune trace purging (by default an optimizer trace overwrites the previous trace) • Make it more human readable (but no longer valid JSON) • My blog: https://oysteing.blogspot.com • For information about MySQL related work at Alibaba: Attend the AliSQL & POLARDB track tomorrow 53
  • 54. Thank You! Please, remember to rate this session in the Percona Live APP! 54
  • 55. 55