The document presents a new approach for troubleshooting optimizer performance in MariaDB 10.1, detailing a three-step workflow that identifies poorly performing queries, examines optimizer behavior, and analyzes execution statistics. It introduces an 'analyze' command that offers execution insights similar to PostgreSQL’s EXPLAIN ANALYZE and facilitates deeper analysis through improved query planning. Furthermore, it compares observed and expected row counts and filtered values to identify potential inefficiencies in query execution.

1. A new way to do optimizer troubleshooting in MariaDB 10.1
ANALYZE for executable statements
Sergei Petrunia, MariaDB
Percona Live Santa Clara
April 2015

2. 2
Background
Optimizer troubleshooting
workflow
2

3. 3
Step #1: Find badly-performing queries
Ways to find slow queries
• Slow query log
• PERFORMANCE_SCHEMA
• tcpdump + pt_query_digest
• (logs from your app)
3
# User@Host: root[root] @ localhost []
# Thread_id: 3 Schema: dbt3sf1 QC_hit: No
# Query_time: 7.891693 Lock_time: 0.000359 Rows_sent: 1 Rows_examined: 1500000
# Rows_affected: 0
# Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No
# Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No
SET timestamp=1428947722;
select sum(o_totalprice) from orders;

4. 4
Step #2: Determine the problem is in optimizer
• Slow query log (or P_S) has *some* *clues*
4
# User@Host: root[root] @ localhost []
# Thread_id: 3 Schema: dbt3sf1 QC_hit: No
# Query_time: 7.891693 Lock_time: 0.000359 Rows_sent: 1 Rows_examined: 1500000
# Rows_affected: 0
# Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No
# Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No
SET timestamp=1428947722;
select sum(o_totalprice) from orders;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
...
| Sending data | 7.704266 |
• SHOW PROFILE data (or P_S).
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
...
| Copying to tmp table | 4.002318 |

5. 5
5
+----+-----------+--------+----+-------------+-------+-------+-----------------+-------+-----------+
|id |select_type|table |type|possible_keys|key |key_len|ref |rows |Extra |
+----+-----------+--------+----+-------------+-------+-------+-----------------+-------+-----------+
| 1|SIMPLE |orders |ALL |PRIMARY,... |NULL |NULL |NULL |1507320|Using where|
| 1|SIMPLE |lineitem|ref |PRIMARY,... |PRIMARY|4 |orders.o_orderkey| 1|Using where|
+----+-----------+--------+----+-------------+-------+-------+-----------------+-------+-----------+
• Sometimes problem is apparent
• Sometimes not
– Query plan vs reality?
– Where the time was spent?.
Step #3: Analyze optimizer behavior
• EXPLAIN shows the query *plan*

6. 6
6
Existing solution: global counters
• Slow query log: Rows_examined
• Handler_XXX status variables
• Userstat:
SHOW (TABLE|INDEX)_STATISTICS
• PERFORMANCE_SCHEMA:
table_io_waits_summary_by_table
+--+-----------+--------+------+-------------+---------+-------+-------------------+----+----------------------------+
|id|select_type|table |type |possible_keys|key |key_len|ref |rows|Extra |
+--+-----------+--------+------+-------------+---------+-------+-------------------+----+----------------------------+
|1 |PRIMARY |orders |const |PRIMARY |PRIMARY |4 |const |1 |Using index |
|1 |PRIMARY |lineitem|ref |PRIMARY,i_...|PRIMARY |4 |const |2 |Using where; Start temporary|
|1 |PRIMARY |lineitem|ref |PRIMARY,i_...|i_suppkey|5 |lineitem.l_partkey |14 |Using index |
|1 |PRIMARY |orders |eq_ref|PRIMARY,i_...|PRIMARY |4 |lineitem.l_orderkey|1 |Using where |
|1 |PRIMARY |customer|eq_ref|PRIMARY |PRIMARY |4 |orders.o_custkey |1 |End temporary |
+--+-----------+--------+------+-------------+---------+-------+-------------------+----+----------------------------+
Counter
• All query-global
• Or server-global
• => Analysis can be
very difficult.

7. 7
A solution
ANALYZE command
7
Similar to
• PostgreSQL's EXPLAIN ANALYZE
• Oracle's V$SQL_PLAN_STATISTICS.

8. 8
ANALYZE command
EXPLAIN
• Optimize the query
8
ANALYZE
• Optimize the query
• Run the query
– Collect execution statistics
– Discard query output
• Return EXPLAIN output
– With statistics.
• Return EXPLAIN output

9. 9
(Tabular) EXPLAIN vs ANALYZE
9
explain
select count(*) from orders
where year(o_orderdate)=1995G
************** 1. row ****************
id: 1
select_type: SIMPLE
table: orders
type: index
possible_keys: NULL
key: i_o_orderdate
key_len: 4
ref: NULL
rows: 1507320
Extra: Using where; Using index
analyze
select count(*) from orders
where year(o_orderdate)=1995G
************** 1. row ****************
id: 1
select_type: SIMPLE
table: orders
type: index
possible_keys: NULL
key: i_o_orderdate
key_len: 4
ref: NULL
rows: 1507320
r_rows: 1500000.00
filtered: 100.00
r_filtered: 15.24
Extra: Using where; Using index

10. 10
ANALYZE columns: r_*
ANALYZE columns start with r_
• r_rows is observed # rows
– r_rows≈ rows, ok
• r_filtered is observed selectivity
– r_filtered < filtered, bad(?).
analyze
select count(*) from orders
where year(o_orderdate)=1995G
************** 1. row ****************
id: 1
select_type: SIMPLE
table: orders
type: index
possible_keys: NULL
key: i_o_orderdate
key_len: 4
ref: NULL
rows: 1507320
r_rows: 1500000.00
filtered: 100.00
r_filtered: 15.24
Extra: Using where; Using index

11. 11
Trying with better indexing
• r_rows is observed # rows
– r_rows = 0.5 * rows
– 2x difference is typical for
innodb
• r_filtered is observed selectivity
– r_filtered=filtered
– r_filtered=100%, best possible.
analyze
select count(*) from orders
where
o_orderdate between '1995-01-01' and
'1995-12-31'
************** 1. row ****************
id: 1
select_type: SIMPLE
table: orders
type: range
possible_keys: i_o_orderdate
key: i_o_orderdate
key_len: 4
ref: NULL
rows: 424048
r_rows: 228637.00
filtered: 100.00
r_filtered: 100.00
Extra: Using where; Using index

12. 12
ANALYZE example #2
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
analyze select *
from lineitem, orders
where o_orderkey=l_orderkey and
o_orderdate between '1990-01-01' and '1998-12-06' and
l_extendedprice > 1000000
• orders:
rows≈r_rows, ok.
r_filtered =0.5*filtered, ok.
• lineitem:
r_rows = 2* rows, ok.
r_filtered=0.00 (it's “less than 0.01%”) - inefficiency.

13. 13
Interpreting r_rows (1)
• ALL/index
– r_rows≈rows_in_table
●
except for index+limit or EXIST subqueries
• range/index_merge
– ~2x difference common for InnoDB
– Bigger difference indicates a problem
●
IGNORE INDEX?
13
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+

14. 14
Interpreting r_rows (2)
ref/eq_ref
• Bigger discrepancies are normal
• Too big discrepancy (10x or more) requires investigation
– No index statistics: rows=1, r_rows >> rows? → ANALYZE TABLE
– Lots of NULL values → innodb_stats_method
– Skewed value distribution (mega-orders)
– Different value domains (orders w/o lineitem's?)
14
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
→ IGNORE INDEX?

15. 15
Interpreting r_filtered (1)
15
WHERE tbl.key1='foo' AND tbl.key2='bar' AND tbl.non_key_col<'baz'
• filtered is selectivity of “Using where”
• Optimizer uses filtered=100% by default
– filtered=50% is a guess
• r_filtered is “observed selectivity”
– r_filtered < filtered typically.
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
index access Using where, filtered

16. 16
Interpreting r_filtered (2)
16
WHERE tbl.key1='foo' AND tbl.key2='bar' AND tbl.non_key_col<'baz'
• r_filtered << filtered means selective conditions are “not used”
– Add indexes/make conditions usable
– (since MariaDB 10.0) add histgorams.
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1504278|1500000| 50.00 | 100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|2 |4.00 | 100.00 | 0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+-------+--------+----------+-----------+
index access Using where, filtered

17. 17
Conclusions so far
17
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1498194|1500000.00| 50.00 |100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|1 |4.00 | 100.00 |0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+
• Tabular ANALYZE has r_* columns
– r_rows, r_filtered
• These show data from execution
– Can check query plan vs reality
●
Wrong statistics
●
Query plan inefficiencies

18. 18
FORMAT=JSON
18

19. 19
EXPLAIN FORMAT=JSON
MySQL 5.6 introduced EXPLAIN FORMAT=JSON
• Good! It shows more info (http://s.petrunia.net/blog/?p=83)
• But it has bugs
Bug#69567: EXPLAIN FORMAT=JSON lists subquery in optimized_away_subqueries, but it is run
Bug#69795: EXPLAIN FORMAT=JSON doesn't show Using filesort for UNION
Bug#74462: EXPLAIN FORMAT=JSON produces ordering_operation when no ordering takes place
Bug#74661: EXPLAIN FORMAT=JSON says two temptables are used, execution shows just one
Bug#74744: EXPLAIN FORMAT=JSON produces duplicates_removal where there is none
[no bug#]: EXPLAIN FORMAT=JSON shows the same subquery as two different subqueries
…
• And we were not happy with output
– Even MySQL Workbench choked on it (http://s.petrunia.net/blog/?p=93)
– “JSON format” != “print tabular EXPLAIN in JSON”
19
INSERT:EXPLAINFORMAT=JSON

20. 20
EXPLAIN FORMAT=JSON in MariaDB 10.1
Improved over MySQL 5.6
• Attached conditions printout is more readable
– No ridiculous overquoting
– Subqueries are not printed in full
• JSON pretty-printer is smarter
• Index Merge output is JSON-ish, shows used_key_parts
• Range checked for each record output is JSON-ish, shows more info
• “Full scan on NULL key” prints JSON, not “index map: 0xNNN”
• Query plans for “Using Join buffer” show more details
• …
• !Alas, some ORDER/GROUP BY problems remain*
20
INSERT:EXPLAINFORMAT=JSON

21. 21
ANALYZE FORMAT=JSON
21
• Works like ANALYZE
• Produces EXPLAIN FORMAT=JSON like output
– with more data.
EXPLAIN
FORMAT=JSON
+ ANALYZE = ANALYZE FORMAT=JSON

22. 22
ANALYZE FORMAT=JSON basics
• Consider an example
22
analyze select *
from
lineitem, orders
where
o_orderkey=l_orderkey and
o_orderdate between '1990-01-01' and '1998-12-06' and
l_extendedprice > 1000000
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+
|id|select_type|table |type|possible_keys|key |key_len|ref |rows |r_rows |filtered|r_filtered|Extra |
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+
|1 |SIMPLE |orders |ALL |PRIMARY,i_...|NULL |NULL |NULL |1498194|1500000.00| 50.00 |100.00 |Using where|
|1 |SIMPLE |lineitem|ref |PRIMARY,i_...|PRIMARY|4 |orders.o_orderkey|1 |4.00 | 100.00 |0.00 |Using where|
+--+-----------+--------+----+-------------+-------+-------+-----------------+-------+----------+--------+----------+-----------+

23. 23
ANALYZE FORMAT=JSON basics
23
{
"query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 191747,
"table": {
"table_name": "orders",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "i_o_orderdate"],
"r_loops": 1,
"rows": 1498194,
"r_rows": 1.5e6,
"r_total_time_ms": 14261,
"filtered": 50,
"r_filtered": 100,
"attached_condition": "(orders.o_orderDATE between
1990-01-01 and 1998-12-06)"
},
},
"table": {
"table_name": "lineitem",
"access_type": "ref",
"possible_keys": ["PRIMARY", "i_l_orderkey",
"i_l_orderkey_quantity"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["l_orderkey"],
"ref": ["dbt3sf1.orders.o_orderkey"],
"r_loops": 1500000,
"rows": 1,
"r_rows": 4.0008,
"r_total_time_ms": 170456,
"filtered": 100,
"r_filtered": 0,
"attached_condition": "(lineitem.l_extendedprice > 1000000)"
}
}
}

24. 24
ANALYZE FORMAT=JSON basics
All ANALYZE fields start with r_
• Each table has
– r_loops
– r_total_time_ms ←!
• Checking
orders.r_total_time_ms=14261
lineitem.r_total_time_ms=170456
• Aha!
24
{
"query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 191747,
"table": {
"table_name": "orders",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "i_o_orderdate"],
"r_loops": 1,
"rows": 1498194,
"r_rows": 1.5e6,
"r_total_time_ms": 14261,
"filtered": 50,
"r_filtered": 100,
"attached_condition": "(orders.o_orderDATE between
1990-01-01 and 1998-12-06)"
},

25. 25
ANALYZE and subqueries
25
+--+---------------+--------+------+-------------+-------+-------+------------------+------+-----------+
|id|select_type |table |type |possible_keys|key |key_len|ref |rows |Extra |
+--+---------------+--------+------+-------------+-------+-------+------------------+------+-----------+
|1 |PRIMARY |customer|ALL |NULL |NULL |NULL |NULL |150081|Using where|
|2 |DEPENDENT SUBQ.|orders |eq_ref|PRIMARY |PRIMARY|4 |customer.c_custkey|1 | |
+--+---------------+--------+------+-------------+-------+-------+------------------+------+-----------+
select *
from customer
where (select max(o_totalprice)
from orders
where o_orderkey=c_custkey) > 500000;
Example: customers who have big orders

26. 26
26
ANALYZE: {
"query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 11214,
"table": {
"table_name": "customer",
"access_type": "ALL",
"r_loops": 1,
"rows": 150081,
"r_rows": 150000,
"r_total_time_ms": 1181.2,
"filtered": 100,
"r_filtered": 0,
"attached_condition": "((subquery#2) > 500000)"
},
"subqueries": [
{
"subqueries": [
{
"query_block": {
"select_id": 2,
"r_loops": 150000,
"r_total_time_ms": 9658.6,
"table": {
"table_name": "orders",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["o_orderkey"],
"ref": ["dbt3sf1.customer.c_custkey"],
"r_loops": 150000,
"rows": 1,
"r_rows": 0.25,
"r_total_time_ms": 8497.7,
"filtered": 100,
"r_filtered": 100
}
}
}
]
}
}
ANALYZE subq

27. 27
ANALYZE and subqueries summary
27
• query_block.r_loops
number of times the subquery executed
• query_block.r_total_time_ms
– total time spent
– includes tables, children subqueries
• Again: can instantly see the most expensive subquery.

28. 28
ANALYZE and join buffer
28
• Join buffer optimization
– Reads rows into buffer, then sorts
– EXPLAIN somewhat misleading
– @@join_buffer_size?
+--+-----------+-----+----+-------------+----+-------+----+----+-----------------------------------------------+
|id|select_type|table|type|possible_keys|key |key_len|ref |rows|Extra |
+--+-----------+-----+----+-------------+----+-------+----+----+-----------------------------------------------+
|1 |SIMPLE |t2 |ALL |NULL |NULL|NULL |NULL|820 |Using where |
|1 |SIMPLE |t1 |ALL |NULL |NULL|NULL |NULL|889 |Using where; Using join buffer (flat, BNL join)|
+--+-----------+-----+----+-------------+----+-------+----+----+-----------------------------------------------+
select * from t1, t2 where t1.col1<100 and t2.col1<100 and t1.col2=t2.col2

29. 29
ANALYZE and join buffer (2)
29
"query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 3.5363,
"table": {
"table_name": "t2",
"access_type": "ALL",
"r_loops": 1,
"rows": 820,
"r_rows": 1000,
"r_total_time_ms": 0.8818,
"filtered": 100,
"r_filtered": 10,
"attached_condition": "(t2.col1 < 100)"
},
"block-nl-join": {
"table": {
"table_name": "t1",
"access_type": "ALL",
"r_loops": 1,
"rows": 889,
"r_rows": 1000,
"r_total_time_ms": 0.875,
"filtered": 100,
"r_filtered": 10,
"attached_condition": "(t1.col1 < 100)"
},
"buffer_type": "flat",
"buffer_size": "128Kb",
"join_type": "BNL",
"attached_condition": "(t1.col2 = t2.col2)",
}

30. 30
ORDER/GROUP BY optimization
30
• “Late” choice if/how do sorting/grouping
– Different execution paths for EXPLAIN and SELECT
– They do not match :-)
• A lot of problems:
Bug#69795: EXPLAIN FORMAT=JSON doesn't show Using filesort for UNION
Bug#74462: EXPLAIN FORMAT=JSON produces ordering_operation when no ordering takes
place
Bug#74661: EXPLAIN FORMAT=JSON says two temptables are used, execution shows just one
Bug#74744: EXPLAIN FORMAT=JSON produces duplicates_removal where there is none
Bug#76679: EXPLAIN incorrectly shows Distinct for tables using join buffer
…?
• MySQL 5.6: filesort/priority_queue continues the pattern
– Not visible in EXPLAIN.

31. 31
ORDER/GROUP BY optimization
31
ANALYZE FORMAT=JSON
• Tracks how the query executed
– Whether sorting was done (and at which stage)
– Whether join result was buffered in a temp.table
– Whether duplicate removal was done
• => It's a way to know how what really was executed.

32. 32
ANALYZE and filesort: example #1
32
• Consider an example: raise priority for 10 earliest orders
update orders
set o_shippriority=o_shippriority+1
where
o_clerk='Clerk#000000001'
order by
o_shipDATE
limit 10;
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
|id|select_type|table |type |possible_keys |key |key_len|ref |rows|Extra |
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
|1 |SIMPLE |orders|range|i_o_order_clerk_date|i_o_order_clerk_date|16 |NULL|1466|Using where; Using filesort|
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
• Let's run ANALYZE
– (CAUTION: ANALYZE UPDATE will do the updates!)

33. 33
ANALYZE and filesort: example #1
33
• r_limit
• r_used_priority_queue
• r_output_rows
• ...
"table": {
"update": 1,
"table_name": "orders",
"access_type": "range",
"possible_keys": ["i_o_order_clerk_date"],
"key": "i_o_order_clerk_date",
"key_length": "16",
"used_key_parts": ["o_clerk"],
"rows": 1466,
"r_rows": 1467,
"r_filtered": 100,
"r_total_time_ms": 107.12,
"attached_condition": "(orders.o_clerk = 'Clerk#00001')"
}
}
}
}
ANALYZE: {
"query_block": {
"select_id": 1,
"r_total_time_ms": 109.02,
"filesort": {
"r_loops": 1,
"r_limit": 10,
"r_used_priority_queue": true,
"r_output_rows": 10,
"r_total_time_ms": 46.875,
"table": {

34. 34
ANALYZE and filesort: example #2
34
Now, delete these orders
delete from orders
where
o_clerk='Clerk#000000001'
order by
o_shipDATE
limit 10;
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
|id|select_type|table |type |possible_keys |key |key_len|ref |rows|Extra |
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
|1 |SIMPLE |orders|range|i_o_order_clerk_date|i_o_order_clerk_date|16 |NULL|1466|Using where; Using filesort|
+--+-----------+------+-----+--------------------+--------------------+-------+----+----+---------------------------+
EXPLAIN is the same as in UPDATE

35. 35
ANALYZE and filesort: example #2
35
• DELETE doesnt' pass
LIMIT to filesort :-(.
ANALYZE: {
"query_block": {
"select_id": 1,
"r_total_time_ms": 11.265,
"filesort": {
"r_loops": 1,
"r_limit": "none",
"r_used_priority_queue": false,
"r_output_rows": 1494,
"r_total_time_ms": 10.228,
"r_buffer_size": "2048Kb",
"table": {
"table": {
"delete": 1,
"table_name": "orders",
"access_type": "range",
"possible_keys": ["i_o_order_clerk_date"],
"key": "i_o_order_clerk_date",
"key_length": "16",
"used_key_parts": ["o_clerk"],
"rows": 1493,
"r_rows": 1494,
"r_filtered": 100,
"r_total_time_ms": 9.7133,
"attached_condition": "(orders2.o_clerk = 'Clerk#00001')"
}
}
}
}
delete from orders where o_clerk='Clerk#00001' order by o_shipDATE limit 10
Wow :-(

36. 36
ANALYZE and “range checked for each record”
36
• Optimization for non-equality joins
• Example:
orders with nearby shipdate and nearby order date
select * from orders A, orders B
where
A.o_clerk='Clerk#000000001' and
B.o_orderdate between DATE_SUB(A.o_orderdate, interval 1 day) and
DATE_ADD(A.o_orderdate, interval 1 day)
and
B.o_shipdate between DATE_SUB(A.o_shipdate, interval 1 day) and
DATE_ADD(A.o_shipdate, interval 1 day)

37. 37
ANALYZE and “range checked for each record”
37
select * from orders A, orders B
where
A.o_clerk='Clerk#000000001' and
B.o_orderdate between DATE_SUB(A.o_orderdate, interval 1 day) and
DATE_ADD(A.o_orderdate, interval 1 day)
and
B.o_shipdate between DATE_SUB(A.o_shipdate, interval 1 day) and
DATE_ADD(A.o_shipdate, interval 1 day)
+--+-----------+-----+----+------------------------+----------+-------+-...
|id|select_type|table|type|possible_keys |key |key_len|
+--+-----------+-----+----+------------------------+----------+-------+-...
|1 |SIMPLE |A |ref |i_o_order_clerk_date |i_o_clerk |16 |
|1 |SIMPLE |B |ALL |i_o_orderdate,o_shipDATE|NULL |NULL |
+--+-----------+-----+----+------------------------+----------+-------+-...
..-+-----+-------+-----------------------------------------------+
|ref |rows |Extra |
..-+-----+-------+-----------------------------------------------+
|const|1466 |Using index condition |
|NULL |1499649|Range checked for each record (index map: 0x22)|
..-+-----+-------+-----------------------------------------------+

38. 38
ANALYZE and “range checked for each record”
38
ANALYZE: {
"query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 5769,
"table": {
"table_name": "A",
"access_type": "ref",
"possible_keys": ["i_o_order_clerk_date"],
"key": "i_o_order_clerk_date",
"key_length": "16",
"used_key_parts": ["o_clerk"],
"ref": ["const"],
"r_loops": 1,
"rows": 1466,
"r_rows": 1467,
"r_total_time_ms": 3.6642,
"filtered": 100,
"r_filtered": 100,
"index_condition": "(A.o_clerk = 'Clerk#00001')"
},
"range-checked-for-each-record": {
"keys": ["i_o_orderdate", "o_shipDATE"],
"r_keys": {
"full_scan": 0,
"index_merge": 0,
"range": {
"i_o_orderdate": 1467,
"o_shipDATE": 0
}
},
"table": {
"table_name": "B",
"access_type": "ALL",
"possible_keys": ["i_o_orderdate", "o_shipDATE"],
"r_loops": 1467,
"rows": 1499649,
"r_rows": 1871.2,
"r_total_time_ms": 3649.9,
"filtered": 100,
"r_filtered": 100
}
}
}
}.

39. 39
Final bits
39
• Target version: MariaDB 10.1
• Current status: BETA
– Good enough for joins
– Will add the missing bits.
• log_slow_verbosity=explain prints ANALYZE.

40. 40
Conclusions
40
• MariaDB 10.1 adds new commands
– ANALYZE statement
– ANALYZE FORMAT=JSON statement
• Show details about query execution
• Help in diagnosing the optimizer.

41. 41
Thanks
41
Q & A