Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

Jaime Crespo
Jaime CrespoSr Database Administrator at the Wikimedia Foundation
••
DBAHire.com Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Jaime Crespo Percona Live London 2014 -3 Nov 2014- dbahire.com/pluk14 1
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Agenda - 3 hours 2 1. Introduction 8. Subquery Optimization 2. Break (VM installation) 9. Fulltext Search 3. General Optimizer Improvements 10. GIS Improvements 4. Computed/Virtual Columns 11. Break 5. Query Rewrite Plugins 12. Query Profiling 6. SQL Mode Changes 13. Results and Conclusions 7. JOIN Optimization 14. Q&A
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks INTRODUCTION © 2014 DBAHIRE.COM 3
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks This is Me Fighting Bad Query Performance © 2014 DBAHIRE.COM • MySQL Consultant at DBAHire.com • Used to work for Oracle (MySQL), Percona • Loves MySQL query optimization and HA 4
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Versions • Oracle MySQL 5.6 (GA) almost 2 years old • MariaDB 10 over 7 months old • Several lab/alpha releases of MySQL 5.7 and MariaDB 10.1 with great new features – We will focus only on SQL-related ones – We will not perform formal benchmarks or comparisons 5
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New Query Optimization Features • Index Condition Pushdown • Subquery Optimizations (materialization and semijoin) • IN-to-EXISTS • JOIN-to-WHERE • Multi-Range Read • Batched Key Access • Persistent InnoDB Statistics • EXPLAIN FORMAT=JSON • EXPLAIN INSERT/UPDATE/DELETE • Hash Join • New cost-based optimizer • Optimizer Trace • Filesort with a short LIMIT © 2014 DBAHIRE.COM • EXISTS-to-IN • EXPLAIN FOR CONNECTION • Faster eq_ref • Extended secondary keys • Engine-Independent statistics • Improved Spatial Indexes (GIS) • Histograms • Virtual columns • PERFORMANCE_SCHEMA improvements • Forced strict SQL mode; functional dependencies in GROUP BY • Query rewrite plugin • Configurable optimizer_search_depth 6
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Already Mentioned Some of Those Topics © 2014 DBAHIRE.COM One Year Ago • Check my presentation here: http://www.slideshare.net/jynus/query-optimization56 7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Example Application • OSM Database is licensed under the Open DataBase License and is Copyright OpenStreetMap Contributors 8
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GENERAL OPTIMIZER IMPROVEMENTS © 2014 DBAHIRE.COM 9
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM EXPLAIN FOR CONNECTION (SHOW EXPLAIN FOR) mysql-5.7.5 (osm)> SHOW PROCESSLIST; +----+------+-----------+------+---------+------+-----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+-----------+------------------+ | 18 | root | localhost | osm | Query | 1867 | updating | UPDATE nodes n | | | | | | | | reference | JOIN users u | | | | | | | | tables | ON n.user=u.name | | | | | | | | | SET n.user=u.id | | 19 | root | localhost | NULL | Query | 0 | starting | SHOW PROCESSLIST | +----+------+-----------+------+---------+------+-----------+------------------+ 2 rows in set (0.00 sec) mysql-5.7.5 (osm)> EXPLAIN FOR CONNECTION 18; +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | id | select | table | parti- | type | possible | key | key | ref | rows | filte- | Extra | | | _type | | tions | | _keys | | _len | | | red | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | 1 | SIMPLE | u | NULL | index | name | name | 258 | NULL | 11271 | 100.00 | Using where; | | | | | | | | | | | | | Using index | | 1 | UPDATE | n | NULL | ref | user | user | 258 | osm.u. | 280 | 100.00 | NULL | | | | | | | | | name | | | | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ 2 rows in set (0.00 sec) 10 Use “SHOW EXPLAIN FOR #” with MariaDB 10
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.7.5 Introduced an New Cost- © 2014 DBAHIRE.COM Based Model for Its Optimizer mysql-5.5.40 (osm) > SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (1.37 sec) 11 mysql-5.5.40 (osm) > SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (0.86 sec) This condition is very selective, but there is no index available
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Why does it take the wrong table order? mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: node_tags type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 839031 filtered: 100.00 Extra: *************************** 2. row **************************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.node_tags.node_id rows: 1 filtered: 100.00 Extra: Using where 2 rows in set, 1 warning (0.00 sec) 12
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Unindexed Columns Are not Accounted mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865521 filtered: 100.00 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 1 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.00 sec) 13 The optimizer assumes that all rows will be returned
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM What’s new in 5.7? mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2773853 filtered: 11.11 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: NULL 2 rows in set, 1 warning (0.00 sec) 14 It is actually 0.002%, but it is good enough
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM If things go wrong -our Friend optimizer_switch mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 15
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Improved EXPLAIN FORMAT=JSON "nested_loop": [ © 2014 DBAHIRE.COM { "table": { "table_name": "nodes", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 2773853, "rows_produced_per_join": 308175, "filtered": 11.11, "cost_info": { "read_cost": "512783.58", "eval_cost": "61635.01", "prefix_cost": "574418.60", "data_read_per_join": "21M" }, 16 Cost information is now included Engine statistics are now floats for improved precision
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Optimizer Cost Tuning 17 This point depends on the hardware and buffer status
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Configurable Costs mysql-5.7.5 ((none))> SELECT * FROM mysql.server_cost; +------------------------------+------------+---------------------+---------+ | cost_name | cost_value | last_update | comment | +------------------------------+------------+---------------------+---------+ | disk_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | disk_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | key_compare_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | row_evaluate_cost | NULL | 2014-10-21 23:03:30 | NULL | +------------------------------+------------+---------------------+---------+ 6 rows in set (0.00 sec) mysql-5.7.5 ((none))> SELECT * FROM mysql.engine_cost; +-------------+-------------+--------------------+------------+---------------------+---------+ | engine_name | device_type | cost_name | cost_value | last_update | comment | +-------------+-------------+--------------------+------------+---------------------+---------+ | default | 0 | io_block_read_cost | NULL | 2014-10-21 23:03:30 | NULL | +-------------+-------------+--------------------+------------+---------------------+---------+ 1 row in set (0.00 sec) 18 The unit is “read of a random data page”
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Changing Costs Example mysql-5.7.5 (osm) > EXPLAIN FORMAT=JSON SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'G "cost_info": { "query_cost": "22567.80" 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > UPDATE mysql.server_cost SET cost_value = 10; mysql-5.7.5 (osm) > FLUSH OPTIMIZER_COSTS; <session restart> mysql-5.7.5 (osm) > EXPLAIN ...G "cost_info": { "query_cost": "661371.00" 1 row in set, 1 warning (0.00 sec) 19 More info on usage in the manual: http://dev.mysql.com/doc/refman/5.7/en/cost-model.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Proposed Features • Technology-dependent costs (HD, SSD, memory) – Some of them could be auto-detected • Buffer “hotness”-aware statistics • Statistics for non-indexed columns/histograms 20 For more on cost optimizer changes and future, read Olav Sandstå’s slides: http://www.slideshare.net/olavsa/mysql-optimizer- cost-model
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Engine-Independent Statistics/Histograms © 2014 DBAHIRE.COM in MariaDB 10 mariadb-10.0.14 (osm) > SET histogram_size = 255; Query OK, 0 rows affected (0.00 sec) mariadb-10.0.14 (osm) > SET use_stat_tables = 2; Query OK, 0 rows affected (0.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE node_tags; +---------------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+---------+----------+-----------------------------------------+ | osm.node_tags | analyze | status | Engine-independent statistics collected | | osm.node_tags | analyze | status | OK | +---------------+---------+----------+-----------------------------------------+ 2 rows in set (3.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE nodes; +-----------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +-----------+---------+----------+-----------------------------------------+ | osm.nodes | analyze | status | Engine-independent statistics collected | | osm.nodes | analyze | status | OK | +-----------+---------+----------+-----------------------------------------+ 2 rows in set (32.52 sec) mariadb-10.0.14 (osm) > SET optimizer_use_condition_selectivity = 3; (or 4 for histograms) Query OK, 0 rows affected (0.00 sec) 21
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Stats! mariadb-10.0.14 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865312 filtered: 0.39 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.01 sec) 22 Much better estimate
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Other Changes/Bugs Fixed • UNION ALL does not create temporary tables, returns tables faster • (a, b) IN ((1, 2), (2, 3)) can use index ranges • EXPLAIN EXTENDED is now the default behavior • I.C. Pushdown support for partitioned tables • IGNORE clause meaning has been standardized between sentence types • Increased default for optimizer_search_depth 23
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks COMPUTED/VIRTUAL COLUMNS © 2014 DBAHIRE.COM 24
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Syntax ALTER TABLE nodes ADD COLUMN lon DECIMAL (10, 7) as (longitude/10000000) VIRTUAL, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) VIRTUAL; • They can be used to simplify SELECTS, calculating values on the fly •Non accessed rows are not calculated 25 VIRTUAL is optional (default)
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Virtual Columns Performance mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (1.10 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude > 515000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.32 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.79 sec) © 2014 DBAHIRE.COM 26
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Explanation mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | SIMPLE | nodes | NULL | ALL | NULL | NULL | NULL | NULL | 2602012 | 33.33 | Using where | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude > 515000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | range | latitude | latitude | 4 | NULL | 1301006 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | index | NULL | latitude | 8 | NULL | 2602012 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > ALTER TABLE nodes add index(lon); ERROR 1951 (HY000): Key/Index cannot be defined on a non-stored virtual column. 27
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stored Columns mysql-5.7.5 (osm) > ALTER TABLE nodes CHANGE lat lat DECIMAL (9, 7) as (latitude/10000000) STORED; ERROR 1954 (HY000): 'Changing the STORED status' is not yet supported for virtual columns. mysql-5.7.5 (osm) > ALTER TABLE nodes DROP COLUMN lat, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) STORED; Query OK, 2865312 rows affected (4 min 51.05 sec) Records: 2865312 Duplicates: 0 Warnings: 0 28 MariaDB uses the 'PERMANENT' keyword
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Query That Cannot Be Indexed mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.57 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: index possible_keys: NULL key: nodes_timestamp_idx key_len: 5 ref: NULL rows: 2576970 filtered: 100.00 Extra: Using where; Using index 1 row in set, 1 warning (0.00 sec) 29
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Add an Indexed Stored Computed © 2014 DBAHIRE.COM Column mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN `month` tinyint UNSIGNED AS (MONTH(timestamp)) STORED, ADD INDEX(`month`); Query OK, 2865312 rows affected (7 min 9.12 sec) Records: 2865312 Duplicates: 0 Warnings: 0 30
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Functional Index Implementation mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE month = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.05 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT month, count(*) FROM nodes WHERE month = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: month key: month key_len: 2 ref: const rows: 775360 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) 31
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Why Not Just Use Triggers? CREATE TABLE `nodes_trigger` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned DEFAULT NULL, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TRIGGER nodes_trigger_BI BEFORE INSERT ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TRIGGER nodes_trigger_BU BEFORE UPDATE ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TABLE `nodes_computed` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned AS (MONTH(timestamp)) STORED, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 32
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Performance? $ ./sandboxes/msb_5_7_5/my sqldump osm nodes > nodes.sql $ time sed -e 's/nodes/nodes_trigger/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m21.528s user 0m3.073s sys 0m0.463s $ time sed -e 's/nodes/nodes_computed/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m24.458s user 0m3.079s sys 0m0.475s mysql-5.7.5 (osm) > INSERT INTO nodes_trigger(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (1 min 6.84 sec) Records: 2865312 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > INSERT INTO nodes_computed(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (54.75 sec) Records: 2865312 Duplicates: 0 Warnings: 0 33 No significative improvements seen within a session
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Consistency? mysql-5.7.5 (osm) > UPDATE nodes_trigger SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected (1.71 sec) Rows matched: 286169 Changed: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_trigger WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.45 sec) mysql-5.7.5 (osm) > UPDATE nodes_computed SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected, 65535 warnings (1.31 sec) Rows matched: 286177 Changed: 0 Warnings: 286177 mysql-5.7.5 (osm) > SHOW WARNINGS; +---------+------+-----------------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------------------------------------------+ | Warning | 1953 | The value specified for virtual column 'month' in table 'nodes_computed' ignored. | mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_computed WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.57 sec) 34 Maybe an error should be produced in strict mode, as MariaDB does?
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Features and Limitations • Virtual “non-stored” columns cannot be indexed • “Stored columns” can be PRIMARY, UNIQUE, FOREIGN and MULTI keys • It cannot contain subqueries or other tables or rows references • It cannot contain used-defined stored functions • It can contain server and user-defined variables • It can be computed based on other virtual columns 35
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY REWRITE PLUGINS © 2014 DBAHIRE.COM 36
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New APIs for Query Rewriting • One for pre-parsing rewriting • Another for post-parsing rewriting © 2014 DBAHIRE.COM 37
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Post- Plugin Installation $ ./sandboxes/msb_5_7_5/use -u root < 5.7.5/lib/plugin/ install_rewriter_plugin.sql – It is mostly broken in the lab build (for my tests) – Uninstall it before restart, or MySQL will segfault © 2014 DBAHIRE.COM 38 More on this: http://mysqlserverteam.com/the-query-rewrite-plugins/
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Query Rewriting Setup mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules (pattern, replacement) VALUES ('SELECT ?', 'SELECT ? + 1'); Query OK, 1 row affected (0.01 sec) mysql-5.7.5 ((none)) > CALL query_rewrite.flush_rewrite_rules(); Query OK, 1 row affected (0.00 sec) mysql-5.7.5 ((none)) > SELECT 1; +-------+ | 1 + 1 | +-------+ | 2 | +-------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 ((none)) > SHOW WARNINGS; +-------+------+-------------------------------------------------------------------+ | Level | Code | Message | +-------+------+-------------------------------------------------------------------+ | Note | 1105 | Query 'SELECT 1' rewritten to 'SELECT 1 + 1' by plugin: Rewriter. | +-------+------+-------------------------------------------------------------------+ 1 row in set (0.00 sec) 39
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Considerations • It cannot correct malformed queries- pattern and replacement must be syntactically correct • Useful for query optimizations for 3rd party applications • User is notified that a rewrite happened with a note-level message • Low overhead (5%); specially for untouched queries • You can do stupid things like: mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules( pattern, replacement ) VALUES ( 'SELECT 1', 'DROP TABLE test.test' ); Query OK, 1 row affected (0.01 sec) 40
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SQL MODE CHANGES © 2014 DBAHIRE.COM 41
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Default SQL Mode Changes • MySQL 5.5 and earlier – '' • MySQL 5.6 (from 5.6.6): – 'NO_ENGINE_SUBSTITUTION' is the default – NO_ENGINE_SUBSTITUTION and STRICT_TRANS_TABLES were suggested in upstream default my.cnf • MySQL 5.7 (from 5.7.5): – 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_EN GINE_SUBSTITUTION' is the new default 42 More on this: http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stricter Defaults mysql> CREATE TABLE `test` ( `id` int(11) NOT NULL, `type` enum('movie','album','videogame') NOT NULL, `date` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB; mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); 43 What happens?
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.5/5.6 with default settings mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); Query OK, 1 row affected, 3 warnings (0.00 sec) mysql> SHOW WARNINGS; +---------+------+-----------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------+ | Warning | 1364 | Field 'id' doesn't have a default value | | Warning | 1265 | Data truncated for column 'type' at row 1 | | Warning | 1264 | Out of range value for column 'date' at row 1 | +---------+------+-----------------------------------------------+ 3 rows in set (0.00 sec) mysql> SELECT * FROM test; +----+------+---------------------+ | id | type | date | +----+------+---------------------+ | 0 | | 0000-00-00 00:00:00 | +----+------+---------------------+ 1 row in set (0.00 sec) © 2014 DBAHIRE.COM 44
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); ERROR 1265 (01000): Data truncated for column 'type' at row 1 mysql> INSERT INTO test (type, date) VALUES ('videogame', -1); ERROR 1292 (22007): Incorrect datetime value: '-1' for column 'date' at row 1 mysql> INSERT INTO test (type, date) VALUES ('movie', now()); ERROR 1364 (HY000): Field 'id' doesn't have a default value mysql> INSERT INTO test (id, type, date) VALUES (1, 'videogame', now()); Query OK, 1 row affected (0.01 sec) 45
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GROUP BY Behavior Changes mysql-5.6.21 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; +-----------+----------+------------+ | way_id | count(*) | node_id | +-----------+----------+------------+ | 155339744 | 1187 | 1558095871 | | 243986064 | 1156 | 2604713337 | | 87136668 | 1128 | 1013304944 | | 148812873 | 852 | 1618837453 | | 149200774 | 835 | 34921158 | | 183618216 | 826 | 1940223096 | | 273858696 | 824 | 1267549776 | | 261584374 | 770 | 2669122104 | | 227880171 | 704 | 2240011804 | | 193564006 | 684 | 1808872763 | +-----------+----------+------------+ 10 rows in set (1.24 sec) 46 Undeterministic behavior
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks With ONLY_FULL_GROUP_BY (default in 5.7) mysql-5.7.5 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'osm.way_nodes.node_id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by 47
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Problem with ONLY_FULL_GROUP_BY in Functional dependency © 2014 DBAHIRE.COM MySQL <= 5.6 mysql-5.6.21 (osm) > SET SQL_mode='ONLY_FULL_GROUP_BY'; Query OK, 0 rows affected (0.00 sec) mysql-5.6.21 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): 'osm.u.display_name' isn't in GROUP BY 48 More on this: http://rpbouman.blogspot.com/2014/09/mysql-575-group-by-respects-functional.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks 5.7 Aims for SQL99 Compliance mysql-5.7.5 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; +---------+---------------+--------------+ | user id | user name | # changesets | +---------+---------------+--------------+ | 31257 | Ed Avis | 4333 | | 508 | Welshie | 2696 | | 1016290 | Amaroussi | 2351 | | 352985 | ecatmur | 1450 | | 142807 | SDavies | 1342 | | 736 | Steve Chilton | 1182 | | 346 | Tom Chance | 1175 | | 38784 | Tom Morris | 1165 | | 88718 | UrbanRambler | 1151 | | 1611 | Harry Wood | 1091 | +---------+---------------+--------------+ 10 rows in set (0.09 sec) © 2014 DBAHIRE.COM 49
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Backward Compatibility • Some ORMs and frameworks change the default SQL Mode: – Ruby on Rails 4+ sets STRICT_ALL_TABLES – Drupal 7+ sets TRADITIONAL • Other applications do not work in standard-compliance modes: – Wordpress does not work in strict mode: http:// www.xaprb.com/blog/2013/03/15/wordpress-and-mysqls-strict-mode/ – Cacti does not work with ONLY_FULL_GROUP_BY,NO_ZERO_DATE 50
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Deprecated Modes • ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE are deprecated and do nothing – Use STRICT_TRANS_TABLES or STRICT_ALL_TABLES, which include those modes and produce an error 51
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks JOIN OPTIMIZATIONS © 2014 DBAHIRE.COM 52
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM (Block) Nested Loop Join • Until MySQL 5.5, there was only one algorithm to execute a JOIN: 53 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 name London Eye
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM “New” JOIN Methods • MySQL 5.6 added: – Batch Key Access* • MariaDB has since 5.3: – Batch Key Access* – Hash Joins – Slightly modified versions of the above (“incremental” buffers for joining 3 or more tables) 54 *I already mentioned BKA last year
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Batch Key Access 55 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Buffer sorted in physical/ PK order
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Hash Joins • Only work for equi-joins 56 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Hash table for faster access to rows
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Configuration • BKA requires changes of default optimizer configuration: mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr=on'; mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr_cost_based=off'; mysql-5.7.5 (osm) > SET optimizer_switch= 'batch_key_access=on'; 57
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MariaDB configuration mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_incremental=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_hashed=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_bka=on'; - Enabled by default mariadb-10.0.14 (osm) > SET join_cache_level = 3 (for hash joins) mariadb-10.0.14 (osm) > SET join_cache_level = 5 (for BKA) - Also, configure join_buffer_size appropriately. - Hash joins, like BKA, are highly dependent on disk-bound DBs to be effective due to the extra overhead 58
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Nested Loop Join (cold buffers, buffer_pool=100MB) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 19 Extra: 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (32.86 sec) 59
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Hash Join (cold buffers, buffer_pool=100M, © 2014 DBAHIRE.COM join_buffer=4M) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: hash_ALL possible_keys: changeset_id key: #hash#changeset_id key_len: 8 ref: osm.changesets.id rows: 2781732 Extra: Using join buffer (flat, BNLH join) 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (6.66 sec) 60
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SUBQUERY OPTIMIZATIONS © 2014 DBAHIRE.COM 61
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Delayed Materialization of Derived Tables • EXPLAIN performance improved (it does not execute subqueries) • Derived tables can be indexed for improved performance © 2014 DBAHIRE.COM 62
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example with a Derived Table mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2865312 | | | 1 | PRIMARY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | n.changeset_id | 1 | Using index | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ 3 rows in set (1.42 sec) mysql-5.6.21 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | 1 | PRIMARY | changesets | index | PRIMARY | PRIMARY | 8 | NULL | 70917 | Using index | | 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 8 | osm.changesets.id | 40 | NULL | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2853846 | Using where | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ 3 rows in set (0.00 sec) © 2014 DBAHIRE.COM 63 No subquery execution Auto-generated index
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Common 5.5 Performance Problem mysql-5.5.40 (osm) > EXPLAIN SELECT * FROM nodes © 2014 DBAHIRE.COM WHERE nodes.changeset_id IN ( SELECT changesets.id FROM changesets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | 1 | PRIMARY | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | | 2 | DEPENDENT SUBQUERY | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 2 | DEPENDENT SUBQUERY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | func | 1 | Using where | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ 3 rows in set (0.00 sec) mysql-5.5.40 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (2.60 sec) 64 This means that the subquery is executed almost 3 million times
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Semijoin Optimization • The only way to execute certain IN subqueries was to execute them with poor strategy – This forced to rewrite certain queries into JOINS or scalar subqueries, when possible • There are now several additional strategies/ optimizations: – Convert to a JOIN – Materialization (including index creation) – FirstMatch – LooseScan – Duplicate Weedout 65
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks The Previous Query is Not a Problem in © 2014 DBAHIRE.COM 5.6/5.7/MariaDB 5.3+ mysql-5.6.21 (osm) > EXPLAIN SELECT * FROM nodes WHERE nodes.changeset_id IN ( SELECT changesets.id FROM change sets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | 1 | SIMPLE | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 1 | SIMPLE | changesets | ALL | PRIMARY | NULL | NULL | NULL | 70917 | Using where | | 1 | SIMPLE | nodes | ref | changeset_id | changeset_id | 8 | osm.changesets.id | 21 | NULL | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ 3 rows in set (0.00 sec) mysql-5.6.21 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (0.02 sec) 66 Executed as a regular JOIN
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM First Match Strategy mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM changesets WHERE id IN (SELECT changeset_id FROM nodes)G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: changesets partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 70917 filtered: 100.00 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 33 filtered: 100.00 Extra: Using index; FirstMatch(changesets) 67 It is converting the ref into an eq_ref
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Enabling and disabling materialization/ © 2014 DBAHIRE.COM semijoin mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 68
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks FULLTEXT SEARCH © 2014 DBAHIRE.COM 69
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Fuzzy Search of “gloucester/Gloucester’s/ © 2014 DBAHIRE.COM etc” • “Typical” way to solve this: mysql-5.7.5 (osm) > SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'; 425 rows in set (0.46 sec) 70 Too slow
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Let’s Add an Index mysql-5.7.5 (osm) > ALTER TABLE way_tags ADD INDEX(v); Query OK, 0 rows affected (6.44 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.38 sec) • Still slow, why? mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE v like '%gloucester%'; +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | way_tags | NULL | index | NULL | v_2 | 767 | NULL | 1333338 | 11.11 | Using where; Using index | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ 1 row in set, 1 warning (0.01 sec) 71
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Fulltext Index mysql-5.7.5 (osm) > ALTER TABLE way_tags add FULLTEXT index(v); Query OK, 0 rows affected (3.20 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE MATCH(v) AGAINST ('+gloucester*' IN BOOLEAN MODE); +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | 1 | SIMPLE | way_tags | NULL | fulltext | v | v | 0 | const | 1 | 100.00 | Using where; Ft_hints: no_ranking | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ 1 row in set, 1 warning (0.00 sec) 72
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM way_tags_myisam WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | way_tags_myisam | fulltext | v | v | 0 | | 1 | Using where | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec) mysql-5.5.40 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 0 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 425 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 16 rows in set (0.00 sec) © 2014 DBAHIRE.COM 73
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM way_tags WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 2 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 0 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) © 2014 DBAHIRE.COM 74 It’s counting directly from the FULLTEXT index
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Open Issues and Limitations • No postfix support (wildcards) • Simple Ranking (and different from MyISAM) • No stemming support • Some multi-language limitations © 2014 DBAHIRE.COM 75 More on FULLTEXT InnoDB support: http://www.drdobbs.com/database/full-text- search-with-innodb/231902587
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GIS IMPROVEMENTS © 2014 DBAHIRE.COM 76
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Find the Closest Starbucks mysql-5.7.5 (osm) > SET @lat:=51.49353; SET @lon:=-0.18340; mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.20 sec) 77 You are here
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM This Query is Slow mysql-5.7.5 (osm) > EXPLAIN SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/ 10000000 as latitude, sqrt(pow((latitude/10000000- @lat)*111257.67, 2) + pow((longitude/10000000- @lon)*69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1G ********************* 1. row ********************* id: 1 select_type: SIMPLE table: n_t1 partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 832040 filtered: 0.00 Extra: Using where; Using temporary; Using file sort ********************* 2. row ********************* id: 1 select_type: SIMPLE table: n_t2 partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 3 filtered: 1.41 Extra: Using where ********************* 3. row ********************* id: 1 select_type: SIMPLE table: n partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 1 filtered: 100.00 Extra: NULL 3 rows in set, 1 warning (0.00 sec) 78
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Can We Optimize it? • We could add a bounding box: mysql-5.7.5 (osm) > ALTER TABLE nodes add index(latitude, longitude); mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and n.latitude BETWEEN ((@lat - 1000/111257.67) * 10000000) AND ((@lat + 1000/111257.67) * 10000000) and n.longitude BETWEEN ((@lon - 1000/69450.32) * 10000000) AND ((@lon + 1000/69450.32) * 10000000) ORDER BY `distance in metres` ASC LIMIT 1; 79 This is not a square, only an approximation
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Create an Index… and Force It mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | 1 | SIMPLE | n_t1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 832040 | 0.00 | Using where; | | | | | | | | | | | | | Using temporary; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n | NULL | ref | PRIMARY, | PRIMARY | 8 | osm.n_t1.node_id | 1 | 5.00 | Using where | | | | | | | latitude | | | | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n_t1.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | 1 | SIMPLE | n | NULL | range | latitude | latitude | 8 | NULL | 493666 | 11.11 | Using where; | | | | | | | | | | | | | Using index; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.09 sec) © 2014 DBAHIRE.COM 80 Still many rows are examined Most of the gain comes from the covering index, not the filtering MySQL ignores the newly created index, why? Performance improvement is not great
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Multiple Range Scans Cannot Be Optimized with BTREE Indexes • We need quadtrees or R-TREE Indexes for indexing in multiple dimensions – The later are implemented in MySQL with the name “SPATIAL indexes”, as they only apply to GIS types • Spatial indexing is available for the first time for InnoDB tables on MySQL 5.7.5 81
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Creating a Spatial Index mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN coord GEOMETRY NOT NULL; Query OK, 0 rows affected (21.80 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > UPDATE nodes SET coord = point(longitude/1000000, latitude/10000000); Query OK, 2865312 rows affected (34.66 sec) Rows matched: 2865312 Changed: 2865312 Warnings: 0 mysql-5.7.5 (osm) > ALTER TABLE nodes add SPATIAL index(coord); Query OK, 0 rows affected (1 min 50.00 sec) Records: 0 Duplicates: 0 Warnings: 0 82 This is new in 5.7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM New Query mysql> SET @area := envelope(linestring(POINT(@lon - 500/69450.32, @lat - 500/111257.67), POINT(@lon + 500/69450.32, @lat + 500/111257.67))); mysql> SELECT n.node_id, x(n.coord) as longitude, y(n.coord) as latitude, st_distance(POINT(@lon, @lat), coord) as distance FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and st_within(coord, @area) ORDER BY st_distance(POINT(@lon, @lat), coord) ASC LIMIT 1; 83 We can use any shape we want thanks to 5.6 improvements
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Performance mysql-5.7.5 (osm) > SELECT ...; +-----------+------------+------------+-----------------------+ | node_id | longitude | latitude | distance | +-----------+------------+------------+-----------------------+ | 699693936 | -0.1822879 | 51.4944808 | 0.0014631428672541478 | +-----------+------------+------------+-----------------------+ 1 row in set (0.02 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | id | select | table | parti | type | possible | key | key | ref | rows | filtered | Extra | | | _type | | tions | | _keys | | _len | | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | 1 | SIMPLE | n | NULL | range | PRIMARY | coord | 34 | NULL | 2 | 100.00 | Using where; | | | | | | | ,coord | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+---------------------------+ 3 rows in set, 1 warning (0.00 sec) 84 This field is almost useless
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Filtering Not using the index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 1 | | Handler_read_key | 1914 | | Handler_read_last | 0 | | Handler_read_next | 1954 | | Handler_read_prev | 0 | | Handler_read_rnd | 1 | | Handler_read_rnd_next | 833426 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 1 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the BTREE index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 274 | | Handler_read_last | 0 | | Handler_read_next | 246540 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the SPATIAL index: +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 522 | | Handler_read_last | 0 | | Handler_read_next | 5254 | | Handler_read_prev | 0 | | Handler_read_rnd | 259 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) 85 mysql-5.7.5 (osm) > SHOW STATUS LIKE 'Hand%';
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Geohash Functions • mysql-5.7.5 (osm) > SELECT ST_GeoHash(@lon, @lat, 10); +----------------------------+ | ST_GeoHash(@lon, @lat, 10) | +----------------------------+ | gcpugy47w3 | +----------------------------+ 1 row in set (0.00 sec) • Useful to index coordinates with a BTREE – It could be specially useful combined with indexed STORED columns (emulating quadtrees) 86 More on Geohashing: http://mysqlserverteam.com/geohash-functions/
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions mysql-5.7.5 (osm) > SELECT nm.v, ST_AsGeoJson(n.coord) FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' AND n_t.v='attraction' AND nm.k='name'; +-----------------------------+--------------------------------------------------------+ | v | ST_AsGeoJson(n.coord) | +-----------------------------+--------------------------------------------------------+ | BedZED | {"type":"Point","coordinates":[-0.1560632,51.3821745]} | | Blewcoat School | {"type":"Point","coordinates":[-0.1360484,51.4983179]} | | Camden / Buck Street Market | {"type":"Point","coordinates":[-0.143193,51.5400398]} | | Camden Lock Village | {"type":"Point","coordinates":[-0.1447181,51.5416552]} | | Wimbledon Windmill | {"type":"Point","coordinates":[-0.2315468,51.4376583]} | . . . 87
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions (cont.) $ ./sandboxes/msb_5_7_5/use osm -B -e "SET SESSION group_concat_max_len = 10000; SELECT CONCAT('{"type":"FeatureCollection", "features":[ ', GROUP_CONCAT(CONCAT('{"type":"Feature", "geometry":', ST_AsGeoJson(n.coord), ',"properties":{"name":"',nm.v,'"}}')), ' ]}') FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' and n_t.v='attraction' AND nm.k='name'" http://geojsonlint.com/ 88
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Open Issues • The SRID can be set and retrieved, but all operations are done in squared euclidean coordinates: 89 mysql-5.7.5 (osm) > SET @p1 := GeomFromText('POINT(-1 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p2 := GeomFromText('POINT(0 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p3 := GeomFromText('POINT(-1 52)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SELECT srid(@p1); +-----------+ | srid(@p1) | +-----------+ | 4326 | +-----------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p2); +-----------------------+ | st_distance(@p1, @p2) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p3); +-----------------------+ | st_distance(@p1, @p3) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec)
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY PROFILING © 2014 DBAHIRE.COM 90
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Which Queries Should I Optimize First? • My two favorite methods: – pt-query-digest – PERFORMANCE_SCHEMA • I prefer pt-query-digest for long-term reports, PERFORMANCE_SCHEMA for more real-time evaluation and fine-tuning – Also, PERFORMANCE_SCHEMA was not “ready” until © 2014 DBAHIRE.COM MySQL 5.6 91
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest • It is a 3rd party tool written in Perl, originally created by Baron Schwartz • It requires activation of the slow log: – SET GLOBAL slow_query_log = 1; – SET long_query_time = 0; • In Percona Server and MariaDB it provides extra information: – SHOW GLOBAL VARIABLES like 'log_slow_verbosity'; 92 Be careful with extra IO and latency!
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest Execution $ pt-query-digest ~/sandboxes/msb_5_7_5/data/localhost-slow.log # 190ms user time, 30ms system time, 25.45M rss, 219.82M vsz # Current date: Sat Nov 1 17:26:58 2014 # Hostname: localhost.localdomain # Files: /home/dbahire/sandboxes/msb_5_7_5/data/localhost-slow.log # Overall: 21 total, 7 unique, 0 QPS, 0x concurrency _____________________ # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 7s 6us 482ms 324ms 455ms 201ms 433ms # Lock time 18ms 0 11ms 840us 224us 2ms 113us # Rows sent 7.08k 0 1.22k 345.43 420.77 300.48 420.77 # Rows examine 19.22M 0 1.28M 937.14k 1.26M 581.94k 1.26M # Query size 1.10k 11 67 53.81 65.89 18.23 65.89 # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x5060CBA490BE371E 6.4236 94.4% 14 0.4588 0.00 SELECT way_tags 93
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks pt-query-digest Execution (cont.) # Query 1: 0 QPS, 0x concurrency, ID 0x5060CBA490BE371E at byte 8795 _____ # This item is included in the report because it matches --limit. # Scores: V/M = 0.00 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 66 14 # Exec time 94 6s 435ms 482ms 459ms 477ms 15ms 455ms # Lock time 10 2ms 0 227us 137us 204us 54us 113us # Rows sent 87 6.21k 0 1.22k 453.86 420.77 242.38 420.77 # Rows examine 93 17.94M 1.28M 1.28M 1.28M 1.28M 0 1.28M # Query size 80 912 59 67 65.14 65.89 2.70 65.89 # String: # Databases osm # Hosts localhost # Time 2014-11-01... (1/7%), 2014-11-01... (1/7%)... 12 more # Users msandbox # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms # 100ms ################################################################ # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `osm` LIKE 'way_tags'G # SHOW CREATE TABLE `osm`.`way_tags`G # EXPLAIN /*!50100 PARTITIONS*/ SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'G © 2014 DBAHIRE.COM 94
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM PERFORMANCE_SCHEMA • Monitoring schema enabled by default since MySQL 5.6 – performance_schema = 1 (it is not dynamic) • Deprecates the old query profiling • It is way more user-friendly when combined with the SYS schema/ps_helper (a set of views and stored procedures created by Mark Leith) 95
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Installation of the SYS Schema $ git clone https://github.com/MarkLeith/mysql-sys.git Cloning into 'mysql-sys'... remote: Counting objects: 926, done. remote: Compressing objects: 100% (73/73), done. remote: Total 926 (delta 35), reused 6 (delta 2) Receiving objects: 100% (926/926), 452.19 KiB | 225.00 KiB/s, done. Resolving deltas: 100% (584/584), done. $ cd mysql-sys/ $ ~/sandboxes/msb_5_7_5/use < sys_57.sql © 2014 DBAHIRE.COM 96
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Discovering Unused © 2014 DBAHIRE.COM Indexes mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_nodes | current_nodes_tile_idx | | osm | current_relations | current_relations_timestamp_idx | +---------------+-------------------+---------------------------------+ 5 rows in set (0.04 sec) mysql-5.7.5 (osm) > SELECT * FROM current_nodes WHERE tile = 100; ... mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_relations | current_relations_timestamp_idx | | osm | current_relations | changeset_id | +---------------+-------------------+---------------------------------+ 5 rows in set (0.03 sec) 97 With enough activity, it can help us clean up our schema
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Slow Queries (ordered by © 2014 DBAHIRE.COM server time) ********* 1. row ********* query: SELECT `way_id` AS `id` , `v` FROM `way_tags` WHERE `v` LIKE ? db: osm full_scan: * exec_count: 15 err_count: 0 warn_count: 0 total_latency: 7.83 s max_latency: 1.33 s avg_latency: 521.84 ms lock_latency: 17.94 ms rows_sent: 6779 rows_sent_avg: 452 rows_examined: 20152155 rows_examined_avg: 1343477 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 21f90695b1ebf20a5f4d4c1e5e860f58 first_seen: 2014-11-01 17:04:51 last_seen: 2014-11-01 17:05:22 98 mysql-5.7.5 (osm) > SELECT * FROM sys.statement_analysis LIMIT 10G
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables mysql-5.7.5 (osm) > SELECT * FROM sys.statements_with_temp_tables WHERE db = 'osm' LIMIT 10G *************************** 1. row *************************** query: SELECT ? AS TYPE , `node_id` A ... gs` WHERE `k` = ? AND `v` = ? db: osm exec_count: 11 total_latency: 7.57 s memory_tmp_tables: 11 disk_tmp_tables: 0 avg_tmp_tables_per_query: 1 tmp_tables_to_disk_pct: 0 first_seen: 2014-11-01 17:33:55 last_seen: 2014-11-01 17:34:45 digest: 5e6e82799b7c7c0e5c57cfe63eb98d5d 99
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > SELECT DIGEST_TEXT FROM performance_schema.events_statements_summary_by_digest WHERE digest = '5e6e82799b7c7c0e5c57cfe63eb98d5d'G *************************** 1. row *************************** DIGEST_TEXT: SELECT ? AS TYPE , `node_id` AS `id` FROM `node_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `way_id` AS `id` FROM `way_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `relation_id` AS `id` FROM `relation_tags` WHERE `k` = ? AND `v` = ? 1 row in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | | NULL | UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ 4 rows in set, 1 warning (0.01 sec) 100
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ 3 rows in set, 1 warning (0.04 sec) 101 No Temporary tables now in 5.7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks CONCLUSIONS © 2014 DBAHIRE.COM 102
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM 5.7/10.1 Still in Alpha • New features usually involve new bugs – I only suffered a single crashing bug – Some of the bleeding edge features are not yet optimized for performance and usability • The greatest problem is finding documentation, as some features are only documented on the developer’s blogs – It will probably catch up slowly • It is great to have the lab builds, though, for testing 103
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 New Features • MySQL 5.6 seemed Percona Server-inspired • MySQL 5.7 seems MariaDB-inspired – Competition is always good for consumer 104
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Many Optimizer Advantages Have to Be © 2014 DBAHIRE.COM Manually Enabled • Modifying on a per-query basis: SET optimizer_switch='batched_key_access=on'; SET join_cache_level=8; # for MariaDB in order to take advantage of them make the features useless unless you are fine-tuning – I expect that to change in the future 105
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Herby Declare MyISAM as Dead • All major MyISAM-only features are now on MySQL 5.7 – FULLTEXT – GIS – Transportable tables • There are still reasons to use MyISAM – MyISAM is still required for the mysql © 2014 DBAHIRE.COM schema and non-durable temporary tables (WIP) 106
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Benchmarks • Do not trust first party benchmarks – In fact, do not trust 3rd party benchmarks either • Only care about the performance of your application running on your hardware 107
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Q&A 108
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Knowing More about Query Optimization • Tomorrow: Bill Karwin’s session “Query Patterns, Optimized” • “Optimization, Administration and High Availability with MySQL 5.6” course by DBAHire.com on 15th December (Spain) – More dates coming soon © 2014 DBAHIRE.COM 109
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Talk to me During the Percona Live • Talk to me about: – MySQL/MariaDB/Percona – OpenStreetMap – Game of Thrones and I will offer you 1 hour of free MySQL © 2014 DBAHIRE.COM online consulting by myself* 110 *Subjected to availability
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Thank You for Attending! • Special thanks to -in ORDER BY rand(): Bill Karwin, Morgan Tocker, Baron Schwartz, Giuseppe Maxia, Ewen Fortune, Henrik Ingo, Tim “do-not- confuse-with-Mark” Callaghan, David Hildebrandt, Cédric Peintre, Kortney Runyan, Miguel Ángel Nieto, Valerii Kravchuk, Aleksandr Kuzminsky and everyone at the Percona and Oracle-MySQL teams and the Percona Live London organization, committee and sponsors • Do not forget, after the session finishes, to please login with your Percona Live account and go to http:// www.percona.com/live/london-2014/sessions/query-optimization- mysql-57-and-mariadb-10-even-newer-tricks to “Rate This Session” for a chance to win great prizes! 111
1 of 111

Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

••
Download to read offline
Technology

Tutorial delivered at Percona Live London 2014, where we explore new features and techniques for faster queries with MySQL 5.6 and 5.7 and MariaDB 10, including the newest options in MySQL 5.7.5 and MariaDB 10.1. Download here the virtual machine with the example database: http://dbahire.com/pluk14 Update: WordPress has a workaround for STRICT mode: https://core.trac.wordpress.org/ticket/26847

Jaime Crespo
Jaime CrespoSr Database Administrator at the Wikimedia Foundation

Recommended

Galera cluster for high availability by
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability Mydbops
2.2K views•28 slides
InnoDB Performance Optimisation by
InnoDB Performance OptimisationInnoDB Performance Optimisation
InnoDB Performance OptimisationMydbops
2.2K views•32 slides
Maxscale 소개 1.1.1 by
Maxscale 소개 1.1.1Maxscale 소개 1.1.1
Maxscale 소개 1.1.1NeoClova
1.1K views•49 slides
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013 by
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
20.3K views•65 slides
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale by
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleThe Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
13K views•21 slides
Optimizing MariaDB for maximum performance by
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceMariaDB plc
2.1K views•35 slides

More Related Content

What's hot

MySQL8.0_performance_schema.pptx by
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxNeoClova
405 views•48 slides
MariaDB 마이그레이션 - 네오클로바 by
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바NeoClova
886 views•49 slides
MySQL Administrator 2021 - 네오클로바 by
MySQL Administrator 2021 - 네오클로바MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바NeoClova
1.5K views•238 slides
Explain that explain by
Explain that explainExplain that explain
Explain that explainFabrizio Parrella
5.1K views•43 slides
MySQL Fabricでぼっこぼこにされたはなし by
MySQL FabricでぼっこぼこにされたはなしMySQL Fabricでぼっこぼこにされたはなし
MySQL Fabricでぼっこぼこにされたはなしyoku0825
8.3K views•85 slides
MySQL_MariaDB-성능개선-202201.pptx by
MySQL_MariaDB-성능개선-202201.pptxMySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptxNeoClova
631 views•71 slides

What's hot(20)

MySQL8.0_performance_schema.pptx by NeoClova
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova•405 views
MariaDB 마이그레이션 - 네오클로바 by NeoClova
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바
NeoClova•886 views
MySQL Administrator 2021 - 네오클로바 by NeoClova
MySQL Administrator 2021 - 네오클로바MySQL Administrator 2021 - 네오클로바
MySQL Administrator 2021 - 네오클로바
NeoClova•1.5K views
Explain that explain by Fabrizio Parrella
Explain that explainExplain that explain
Explain that explain
Fabrizio Parrella•5.1K views
MySQL Fabricでぼっこぼこにされたはなし by yoku0825
MySQL FabricでぼっこぼこにされたはなしMySQL Fabricでぼっこぼこにされたはなし
MySQL Fabricでぼっこぼこにされたはなし
yoku0825•8.3K views
MySQL_MariaDB-성능개선-202201.pptx by NeoClova
MySQL_MariaDB-성능개선-202201.pptxMySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptx
NeoClova•631 views
How to Manage Scale-Out Environments with MariaDB MaxScale by MariaDB plc
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScale
MariaDB plc•1.2K views
MySQL GTID Concepts, Implementation and troubleshooting by Mydbops
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting
Mydbops•2.5K views
The Full MySQL and MariaDB Parallel Replication Tutorial by Jean-François Gagné
The Full MySQL and MariaDB Parallel Replication TutorialThe Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication Tutorial
Jean-François Gagné•3.3K views
PL22 - Backup and Restore Performance.pptx by Vinicius M Grippa
PL22 - Backup and Restore Performance.pptxPL22 - Backup and Restore Performance.pptx
PL22 - Backup and Restore Performance.pptx
Vinicius M Grippa•272 views
MySQL_MariaDB로의_전환_기술요소-202212.pptx by NeoClova
MySQL_MariaDB로의_전환_기술요소-202212.pptxMySQL_MariaDB로의_전환_기술요소-202212.pptx
MySQL_MariaDB로의_전환_기술요소-202212.pptx
NeoClova•169 views
How Booking.com avoids and deals with replication lag by Jean-François Gagné
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
Jean-François Gagné•5.4K views
ProxySQL for MySQL by Mydbops
ProxySQL for MySQLProxySQL for MySQL
ProxySQL for MySQL
Mydbops•4.3K views
Using all of the high availability options in MariaDB by MariaDB plc
Using all of the high availability options in MariaDBUsing all of the high availability options in MariaDB
Using all of the high availability options in MariaDB
MariaDB plc•2.9K views
Planning for Disaster Recovery (DR) with Galera Cluster by Codership Oy - Creators of Galera Cluster
Planning for Disaster Recovery (DR) with Galera ClusterPlanning for Disaster Recovery (DR) with Galera Cluster
Planning for Disaster Recovery (DR) with Galera Cluster
Codership Oy - Creators of Galera Cluster•744 views
Highly efficient backups with percona xtrabackup by Nilnandan Joshi
Highly efficient backups with percona xtrabackupHighly efficient backups with percona xtrabackup
Highly efficient backups with percona xtrabackup
Nilnandan Joshi•4.7K views
Faster, better, stronger: The new InnoDB by MariaDB plc
Faster, better, stronger: The new InnoDBFaster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDB
MariaDB plc•651 views
High performance and high availability proxies for MySQL by Mydbops
High performance and high availability proxies for MySQLHigh performance and high availability proxies for MySQL
High performance and high availability proxies for MySQL
Mydbops•1.7K views
MariaDB Galera Cluster presentation by Francisco Gonçalves
MariaDB Galera Cluster presentationMariaDB Galera Cluster presentation
MariaDB Galera Cluster presentation
Francisco Gonçalves•4.3K views
MariaDB 10.11 key features overview for DBAs by Federico Razzoli
MariaDB 10.11 key features overview for DBAsMariaDB 10.11 key features overview for DBAs
MariaDB 10.11 key features overview for DBAs
Federico Razzoli•145 views

Viewers also liked

MySQL Best Practices - OTN LAD Tour by
MySQL Best Practices - OTN LAD TourMySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourRonald Bradford
9.5K views•75 slides
MySQL Server Defaults by
MySQL Server DefaultsMySQL Server Defaults
MySQL Server DefaultsMorgan Tocker
2.3K views•27 slides
Hbase源码初探 by
Hbase源码初探Hbase源码初探
Hbase源码初探zhaolinjnu
2.5K views•29 slides
SQL Outer Joins for Fun and Profit by
SQL Outer Joins for Fun and ProfitSQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and ProfitKarwin Software Solutions LLC
18.8K views•43 slides
What you wanted to know about MySQL, but could not find using inernal instrum... by
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
2.2K views•70 slides
Load Data Fast! by
Load Data Fast!Load Data Fast!
Load Data Fast!Karwin Software Solutions LLC
10.2K views•53 slides

Viewers also liked(20)

MySQL Best Practices - OTN LAD Tour by Ronald Bradford
MySQL Best Practices - OTN LAD TourMySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD Tour
Ronald Bradford•9.5K views
MySQL Server Defaults by Morgan Tocker
MySQL Server DefaultsMySQL Server Defaults
MySQL Server Defaults
Morgan Tocker•2.3K views
Hbase源码初探 by zhaolinjnu
Hbase源码初探Hbase源码初探
Hbase源码初探
zhaolinjnu•2.5K views
SQL Outer Joins for Fun and Profit by Karwin Software Solutions LLC
SQL Outer Joins for Fun and ProfitSQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and Profit
Karwin Software Solutions LLC•18.8K views
What you wanted to know about MySQL, but could not find using inernal instrum... by Sveta Smirnova
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...
Sveta Smirnova•2.2K views
Load Data Fast! by Karwin Software Solutions LLC
Load Data Fast!Load Data Fast!
Load Data Fast!
Karwin Software Solutions LLC•10.2K views
MySQL InnoDB Cluster - A complete High Availability solution for MySQL by Olivier DASINI
MySQL InnoDB Cluster - A complete High Availability solution for MySQLMySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI•7.4K views
Mysql参数-GDB by zhaolinjnu
Mysql参数-GDBMysql参数-GDB
Mysql参数-GDB
zhaolinjnu•898 views
MySQL Replication Performance Tuning for Fun and Profit! by Vitor Oliveira
MySQL Replication Performance Tuning for Fun and Profit!MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!
Vitor Oliveira•2.9K views
MySQL High Availability with Group Replication by Nuno Carvalho
MySQL High Availability with Group ReplicationMySQL High Availability with Group Replication
MySQL High Availability with Group Replication
Nuno Carvalho•3.1K views
Capturing, Analyzing and Optimizing MySQL by Ronald Bradford
Capturing, Analyzing and Optimizing MySQLCapturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQL
Ronald Bradford•5K views
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore by Sujatha Sivakumar
MySQL InnoDB Cluster and Group Replication - OSI 2017 BangaloreMySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
Sujatha Sivakumar•593 views
MySQL High Availability and Disaster Recovery with Continuent, a VMware company by Continuent
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyMySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
Continuent•1.8K views
Using Apache Spark and MySQL for Data Analysis by Sveta Smirnova
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data Analysis
Sveta Smirnova•2.8K views
MySQL InnoDB 源码实现分析(一) by frogd
MySQL InnoDB 源码实现分析(一)MySQL InnoDB 源码实现分析(一)
MySQL InnoDB 源码实现分析(一)
frogd•7.6K views
淘宝数据库架构演进历程 by zhaolinjnu
淘宝数据库架构演进历程淘宝数据库架构演进历程
淘宝数据库架构演进历程
zhaolinjnu•5K views
Inno db internals innodb file formats and source code structure by zhaolinjnu
Inno db internals innodb file formats and source code structureInno db internals innodb file formats and source code structure
Inno db internals innodb file formats and source code structure
zhaolinjnu•3.7K views
Requirements the Last Bottleneck by Karwin Software Solutions LLC
Requirements the Last BottleneckRequirements the Last Bottleneck
Requirements the Last Bottleneck
Karwin Software Solutions LLC•2.4K views
Extensible Data Modeling by Karwin Software Solutions LLC
Extensible Data ModelingExtensible Data Modeling
Extensible Data Modeling
Karwin Software Solutions LLC•41.3K views
MySQL aio by zhaolinjnu
MySQL aioMySQL aio
MySQL aio
zhaolinjnu•1.8K views

Similar to Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko by
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoData Con LA
152 views•34 slides
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020) by
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
275 views•28 slides
Confoo 2021 -- MySQL New Features by
Confoo 2021 -- MySQL New FeaturesConfoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New FeaturesDave Stokes
217 views•52 slides
Common schema my sql uc 2012 by
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012Roland Bouman
519 views•35 slides
Common schema my sql uc 2012 by
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012Roland Bouman
600 views•35 slides
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen by
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenNETWAYS
29 views•38 slides

Similar to Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks(20)

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko by Data Con LA
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
Data Con LA•152 views
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020) by Valeriy Kravchuk
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
Valeriy Kravchuk•275 views
Confoo 2021 -- MySQL New Features by Dave Stokes
Confoo 2021 -- MySQL New FeaturesConfoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New Features
Dave Stokes•217 views
Common schema my sql uc 2012 by Roland Bouman
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012
Roland Bouman•519 views
Common schema my sql uc 2012 by Roland Bouman
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012
Roland Bouman•600 views
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen by NETWAYS
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
NETWAYS•29 views
What's New In MySQL 5.6 by Abdul Manaf
What's New In MySQL 5.6What's New In MySQL 5.6
What's New In MySQL 5.6
Abdul Manaf•144 views
Memcached Functions For My Sql Seemless Caching In My Sql by MySQLConference
Memcached Functions For My Sql Seemless Caching In My SqlMemcached Functions For My Sql Seemless Caching In My Sql
Memcached Functions For My Sql Seemless Caching In My Sql
MySQLConference•4K views
MySQL Best Practices - OTN by Ronald Bradford
MySQL Best Practices - OTNMySQL Best Practices - OTN
MySQL Best Practices - OTN
Ronald Bradford•2.4K views
MySQLinsanity by Stanley Huang
MySQLinsanityMySQLinsanity
MySQLinsanity
Stanley Huang•470 views
My sql 5.7-upcoming-changes-v2 by Morgan Tocker
My sql 5.7-upcoming-changes-v2My sql 5.7-upcoming-changes-v2
My sql 5.7-upcoming-changes-v2
Morgan Tocker•1.1K views
Optimizer overviewoow2014 by Mysql User Camp
Optimizer overviewoow2014Optimizer overviewoow2014
Optimizer overviewoow2014
Mysql User Camp•410 views
Performance schema and sys schema by Mark Leith
Performance schema and sys schemaPerformance schema and sys schema
Performance schema and sys schema
Mark Leith•12.6K views
DB Floripa - ProxySQL para MySQL by Marcelo Altmann
DB Floripa - ProxySQL para MySQLDB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQL
Marcelo Altmann•170 views
MariaDB 5.5 and what comes next - Percona Live NYC 2012 by Colin Charles
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
Colin Charles•1.3K views
MySQL Performance Schema in 20 Minutes by Sveta Smirnova
MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
Sveta Smirnova•1.5K views
Query Optimization with MySQL 5.6: Old and New Tricks by MYXPLAIN
Query Optimization with MySQL 5.6: Old and New TricksQuery Optimization with MySQL 5.6: Old and New Tricks
Query Optimization with MySQL 5.6: Old and New Tricks
MYXPLAIN•2.1K views
ProxySQL para mysql by Marcelo Altmann
ProxySQL para mysqlProxySQL para mysql
ProxySQL para mysql
Marcelo Altmann•1.3K views
Advanced Query Optimizer Tuning and Analysis by MYXPLAIN
Advanced Query Optimizer Tuning and AnalysisAdvanced Query Optimizer Tuning and Analysis
Advanced Query Optimizer Tuning and Analysis
MYXPLAIN•1.1K views
Perf Tuning Short by Ligaya Turmelle
Perf Tuning ShortPerf Tuning Short
Perf Tuning Short
Ligaya Turmelle•17K views

More from Jaime Crespo

Haciendo copias de seguridad de todo el conocimiento humano con python y soft... by
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Jaime Crespo
240 views•24 slides
Backing up Wikipedia Databases by
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia DatabasesJaime Crespo
408 views•52 slides
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics by
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsJaime Crespo
1.6K views•202 slides
Wikipedia Burgos devfest 2017 by
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Jaime Crespo
470 views•21 slides
Scaling and hardware provisioning for databases (lessons learned at wikipedia) by
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Jaime Crespo
296 views•55 slides
MySQL Schema Design in Practice by
MySQL Schema Design in PracticeMySQL Schema Design in Practice
MySQL Schema Design in PracticeJaime Crespo
1.6K views•118 slides

More from Jaime Crespo(12)

Haciendo copias de seguridad de todo el conocimiento humano con python y soft... by Jaime Crespo
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Jaime Crespo•240 views
Backing up Wikipedia Databases by Jaime Crespo
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia Databases
Jaime Crespo•408 views
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics by Jaime Crespo
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Jaime Crespo•1.6K views
Wikipedia Burgos devfest 2017 by Jaime Crespo
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017
Jaime Crespo•470 views
Scaling and hardware provisioning for databases (lessons learned at wikipedia) by Jaime Crespo
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Jaime Crespo•296 views
MySQL Schema Design in Practice by Jaime Crespo
MySQL Schema Design in PracticeMySQL Schema Design in Practice
MySQL Schema Design in Practice
Jaime Crespo•1.6K views
MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation by Jaime Crespo
MySQL at Wikipedia: How we do relational data at the Wikimedia FoundationMySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
Jaime Crespo•2.6K views
Query optimization: from 0 to 10 (and up to 5.7) by Jaime Crespo
Query optimization: from 0 to 10 (and up to 5.7)Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)
Jaime Crespo•3K views
Why MySQL by Jaime Crespo
Why MySQLWhy MySQL
Why MySQL
Jaime Crespo•1.5K views
MySQL GUI Administration by Jaime Crespo
MySQL GUI AdministrationMySQL GUI Administration
MySQL GUI Administration
Jaime Crespo•1.2K views
Software libre para gestión de proyectos by Jaime Crespo
Software libre para gestión de proyectosSoftware libre para gestión de proyectos
Software libre para gestión de proyectos
Jaime Crespo•1.2K views
warptalk: Experiencia de usuario y usabilidad by Jaime Crespo
warptalk: Experiencia de usuario y usabilidadwarptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidad
Jaime Crespo•513 views

Recently uploaded

Transcript: Redefining the book supply chain: A glimpse into the future - Tec... by
Transcript: Redefining the book supply chain: A glimpse into the future - Tec...Transcript: Redefining the book supply chain: A glimpse into the future - Tec...
Transcript: Redefining the book supply chain: A glimpse into the future - Tec...BookNet Canada
41 views•16 slides
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P... by
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...ShapeBlue
196 views•62 slides
State of the Union - Rohit Yadav - Apache CloudStack by
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStackShapeBlue
303 views•53 slides
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue by
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlueShapeBlue
152 views•23 slides
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...ShapeBlue
162 views•25 slides
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueShapeBlue
207 views•54 slides

Recently uploaded(20)

Transcript: Redefining the book supply chain: A glimpse into the future - Tec... by BookNet Canada
Transcript: Redefining the book supply chain: A glimpse into the future - Tec...Transcript: Redefining the book supply chain: A glimpse into the future - Tec...
Transcript: Redefining the book supply chain: A glimpse into the future - Tec...
BookNet Canada•41 views
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P... by ShapeBlue
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
ShapeBlue•196 views
State of the Union - Rohit Yadav - Apache CloudStack by ShapeBlue
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStack
ShapeBlue•303 views
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue by ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
ShapeBlue•152 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue•162 views
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue•207 views
Business Analyst Series 2023 - Week 4 Session 8 by DianaGray10
Business Analyst Series 2023 - Week 4 Session 8Business Analyst Series 2023 - Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray10•145 views
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by ShapeBlue
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
ShapeBlue•208 views
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023 by BookNet Canada
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023Redefining the book supply chain: A glimpse into the future - Tech Forum 2023
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023
BookNet Canada•44 views
Ransomware is Knocking your Door_Final.pdf by Security Bootcamp
Ransomware is Knocking your Door_Final.pdfRansomware is Knocking your Door_Final.pdf
Ransomware is Knocking your Door_Final.pdf
Security Bootcamp•98 views
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc•176 views
CryptoBotsAI by chandureddyvadala199
CryptoBotsAICryptoBotsAI
CryptoBotsAI
chandureddyvadala199•42 views
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... by Moses Kemibaro
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Moses Kemibaro•35 views
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue by ShapeBlue
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlueCloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue
ShapeBlue•137 views
NTGapps NTG LowCode Platform by Mustafa Kuğu
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform
Mustafa Kuğu•437 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue•225 views
Future of AR - Facebook Presentation by Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty•65 views
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... by ShapeBlue
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
ShapeBlue•141 views
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue•178 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE•84 views

Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

  • 1. DBAHire.com Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Jaime Crespo Percona Live London 2014 -3 Nov 2014- dbahire.com/pluk14 1
  • 2. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Agenda - 3 hours 2 1. Introduction 8. Subquery Optimization 2. Break (VM installation) 9. Fulltext Search 3. General Optimizer Improvements 10. GIS Improvements 4. Computed/Virtual Columns 11. Break 5. Query Rewrite Plugins 12. Query Profiling 6. SQL Mode Changes 13. Results and Conclusions 7. JOIN Optimization 14. Q&A
  • 3. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks INTRODUCTION © 2014 DBAHIRE.COM 3
  • 4. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks This is Me Fighting Bad Query Performance © 2014 DBAHIRE.COM • MySQL Consultant at DBAHire.com • Used to work for Oracle (MySQL), Percona • Loves MySQL query optimization and HA 4
  • 5. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Versions • Oracle MySQL 5.6 (GA) almost 2 years old • MariaDB 10 over 7 months old • Several lab/alpha releases of MySQL 5.7 and MariaDB 10.1 with great new features – We will focus only on SQL-related ones – We will not perform formal benchmarks or comparisons 5
  • 6. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New Query Optimization Features • Index Condition Pushdown • Subquery Optimizations (materialization and semijoin) • IN-to-EXISTS • JOIN-to-WHERE • Multi-Range Read • Batched Key Access • Persistent InnoDB Statistics • EXPLAIN FORMAT=JSON • EXPLAIN INSERT/UPDATE/DELETE • Hash Join • New cost-based optimizer • Optimizer Trace • Filesort with a short LIMIT © 2014 DBAHIRE.COM • EXISTS-to-IN • EXPLAIN FOR CONNECTION • Faster eq_ref • Extended secondary keys • Engine-Independent statistics • Improved Spatial Indexes (GIS) • Histograms • Virtual columns • PERFORMANCE_SCHEMA improvements • Forced strict SQL mode; functional dependencies in GROUP BY • Query rewrite plugin • Configurable optimizer_search_depth 6
  • 7. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Already Mentioned Some of Those Topics © 2014 DBAHIRE.COM One Year Ago • Check my presentation here: http://www.slideshare.net/jynus/query-optimization56 7
  • 8. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Example Application • OSM Database is licensed under the Open DataBase License and is Copyright OpenStreetMap Contributors 8
  • 9. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GENERAL OPTIMIZER IMPROVEMENTS © 2014 DBAHIRE.COM 9
  • 10. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM EXPLAIN FOR CONNECTION (SHOW EXPLAIN FOR) mysql-5.7.5 (osm)> SHOW PROCESSLIST; +----+------+-----------+------+---------+------+-----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+-----------+------------------+ | 18 | root | localhost | osm | Query | 1867 | updating | UPDATE nodes n | | | | | | | | reference | JOIN users u | | | | | | | | tables | ON n.user=u.name | | | | | | | | | SET n.user=u.id | | 19 | root | localhost | NULL | Query | 0 | starting | SHOW PROCESSLIST | +----+------+-----------+------+---------+------+-----------+------------------+ 2 rows in set (0.00 sec) mysql-5.7.5 (osm)> EXPLAIN FOR CONNECTION 18; +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | id | select | table | parti- | type | possible | key | key | ref | rows | filte- | Extra | | | _type | | tions | | _keys | | _len | | | red | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | 1 | SIMPLE | u | NULL | index | name | name | 258 | NULL | 11271 | 100.00 | Using where; | | | | | | | | | | | | | Using index | | 1 | UPDATE | n | NULL | ref | user | user | 258 | osm.u. | 280 | 100.00 | NULL | | | | | | | | | name | | | | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ 2 rows in set (0.00 sec) 10 Use “SHOW EXPLAIN FOR #” with MariaDB 10
  • 11. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.7.5 Introduced an New Cost- © 2014 DBAHIRE.COM Based Model for Its Optimizer mysql-5.5.40 (osm) > SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (1.37 sec) 11 mysql-5.5.40 (osm) > SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (0.86 sec) This condition is very selective, but there is no index available
  • 12. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Why does it take the wrong table order? mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: node_tags type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 839031 filtered: 100.00 Extra: *************************** 2. row **************************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.node_tags.node_id rows: 1 filtered: 100.00 Extra: Using where 2 rows in set, 1 warning (0.00 sec) 12
  • 13. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Unindexed Columns Are not Accounted mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865521 filtered: 100.00 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 1 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.00 sec) 13 The optimizer assumes that all rows will be returned
  • 14. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM What’s new in 5.7? mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2773853 filtered: 11.11 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: NULL 2 rows in set, 1 warning (0.00 sec) 14 It is actually 0.002%, but it is good enough
  • 15. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM If things go wrong -our Friend optimizer_switch mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 15
  • 16. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Improved EXPLAIN FORMAT=JSON "nested_loop": [ © 2014 DBAHIRE.COM { "table": { "table_name": "nodes", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 2773853, "rows_produced_per_join": 308175, "filtered": 11.11, "cost_info": { "read_cost": "512783.58", "eval_cost": "61635.01", "prefix_cost": "574418.60", "data_read_per_join": "21M" }, 16 Cost information is now included Engine statistics are now floats for improved precision
  • 17. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Optimizer Cost Tuning 17 This point depends on the hardware and buffer status
  • 18. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Configurable Costs mysql-5.7.5 ((none))> SELECT * FROM mysql.server_cost; +------------------------------+------------+---------------------+---------+ | cost_name | cost_value | last_update | comment | +------------------------------+------------+---------------------+---------+ | disk_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | disk_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | key_compare_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | row_evaluate_cost | NULL | 2014-10-21 23:03:30 | NULL | +------------------------------+------------+---------------------+---------+ 6 rows in set (0.00 sec) mysql-5.7.5 ((none))> SELECT * FROM mysql.engine_cost; +-------------+-------------+--------------------+------------+---------------------+---------+ | engine_name | device_type | cost_name | cost_value | last_update | comment | +-------------+-------------+--------------------+------------+---------------------+---------+ | default | 0 | io_block_read_cost | NULL | 2014-10-21 23:03:30 | NULL | +-------------+-------------+--------------------+------------+---------------------+---------+ 1 row in set (0.00 sec) 18 The unit is “read of a random data page”
  • 19. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Changing Costs Example mysql-5.7.5 (osm) > EXPLAIN FORMAT=JSON SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'G "cost_info": { "query_cost": "22567.80" 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > UPDATE mysql.server_cost SET cost_value = 10; mysql-5.7.5 (osm) > FLUSH OPTIMIZER_COSTS; <session restart> mysql-5.7.5 (osm) > EXPLAIN ...G "cost_info": { "query_cost": "661371.00" 1 row in set, 1 warning (0.00 sec) 19 More info on usage in the manual: http://dev.mysql.com/doc/refman/5.7/en/cost-model.html
  • 20. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Proposed Features • Technology-dependent costs (HD, SSD, memory) – Some of them could be auto-detected • Buffer “hotness”-aware statistics • Statistics for non-indexed columns/histograms 20 For more on cost optimizer changes and future, read Olav Sandstå’s slides: http://www.slideshare.net/olavsa/mysql-optimizer- cost-model
  • 21. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Engine-Independent Statistics/Histograms © 2014 DBAHIRE.COM in MariaDB 10 mariadb-10.0.14 (osm) > SET histogram_size = 255; Query OK, 0 rows affected (0.00 sec) mariadb-10.0.14 (osm) > SET use_stat_tables = 2; Query OK, 0 rows affected (0.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE node_tags; +---------------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+---------+----------+-----------------------------------------+ | osm.node_tags | analyze | status | Engine-independent statistics collected | | osm.node_tags | analyze | status | OK | +---------------+---------+----------+-----------------------------------------+ 2 rows in set (3.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE nodes; +-----------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +-----------+---------+----------+-----------------------------------------+ | osm.nodes | analyze | status | Engine-independent statistics collected | | osm.nodes | analyze | status | OK | +-----------+---------+----------+-----------------------------------------+ 2 rows in set (32.52 sec) mariadb-10.0.14 (osm) > SET optimizer_use_condition_selectivity = 3; (or 4 for histograms) Query OK, 0 rows affected (0.00 sec) 21
  • 22. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Stats! mariadb-10.0.14 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865312 filtered: 0.39 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.01 sec) 22 Much better estimate
  • 23. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Other Changes/Bugs Fixed • UNION ALL does not create temporary tables, returns tables faster • (a, b) IN ((1, 2), (2, 3)) can use index ranges • EXPLAIN EXTENDED is now the default behavior • I.C. Pushdown support for partitioned tables • IGNORE clause meaning has been standardized between sentence types • Increased default for optimizer_search_depth 23
  • 24. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks COMPUTED/VIRTUAL COLUMNS © 2014 DBAHIRE.COM 24
  • 25. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Syntax ALTER TABLE nodes ADD COLUMN lon DECIMAL (10, 7) as (longitude/10000000) VIRTUAL, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) VIRTUAL; • They can be used to simplify SELECTS, calculating values on the fly •Non accessed rows are not calculated 25 VIRTUAL is optional (default)
  • 26. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Virtual Columns Performance mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (1.10 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude > 515000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.32 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.79 sec) © 2014 DBAHIRE.COM 26
  • 27. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Explanation mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | SIMPLE | nodes | NULL | ALL | NULL | NULL | NULL | NULL | 2602012 | 33.33 | Using where | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude > 515000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | range | latitude | latitude | 4 | NULL | 1301006 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | index | NULL | latitude | 8 | NULL | 2602012 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > ALTER TABLE nodes add index(lon); ERROR 1951 (HY000): Key/Index cannot be defined on a non-stored virtual column. 27
  • 28. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stored Columns mysql-5.7.5 (osm) > ALTER TABLE nodes CHANGE lat lat DECIMAL (9, 7) as (latitude/10000000) STORED; ERROR 1954 (HY000): 'Changing the STORED status' is not yet supported for virtual columns. mysql-5.7.5 (osm) > ALTER TABLE nodes DROP COLUMN lat, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) STORED; Query OK, 2865312 rows affected (4 min 51.05 sec) Records: 2865312 Duplicates: 0 Warnings: 0 28 MariaDB uses the 'PERMANENT' keyword
  • 29. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Query That Cannot Be Indexed mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.57 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: index possible_keys: NULL key: nodes_timestamp_idx key_len: 5 ref: NULL rows: 2576970 filtered: 100.00 Extra: Using where; Using index 1 row in set, 1 warning (0.00 sec) 29
  • 30. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Add an Indexed Stored Computed © 2014 DBAHIRE.COM Column mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN `month` tinyint UNSIGNED AS (MONTH(timestamp)) STORED, ADD INDEX(`month`); Query OK, 2865312 rows affected (7 min 9.12 sec) Records: 2865312 Duplicates: 0 Warnings: 0 30
  • 31. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Functional Index Implementation mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE month = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.05 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT month, count(*) FROM nodes WHERE month = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: month key: month key_len: 2 ref: const rows: 775360 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) 31
  • 32. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Why Not Just Use Triggers? CREATE TABLE `nodes_trigger` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned DEFAULT NULL, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TRIGGER nodes_trigger_BI BEFORE INSERT ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TRIGGER nodes_trigger_BU BEFORE UPDATE ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TABLE `nodes_computed` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned AS (MONTH(timestamp)) STORED, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 32
  • 33. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Performance? $ ./sandboxes/msb_5_7_5/my sqldump osm nodes > nodes.sql $ time sed -e 's/nodes/nodes_trigger/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m21.528s user 0m3.073s sys 0m0.463s $ time sed -e 's/nodes/nodes_computed/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m24.458s user 0m3.079s sys 0m0.475s mysql-5.7.5 (osm) > INSERT INTO nodes_trigger(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (1 min 6.84 sec) Records: 2865312 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > INSERT INTO nodes_computed(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (54.75 sec) Records: 2865312 Duplicates: 0 Warnings: 0 33 No significative improvements seen within a session
  • 34. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Consistency? mysql-5.7.5 (osm) > UPDATE nodes_trigger SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected (1.71 sec) Rows matched: 286169 Changed: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_trigger WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.45 sec) mysql-5.7.5 (osm) > UPDATE nodes_computed SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected, 65535 warnings (1.31 sec) Rows matched: 286177 Changed: 0 Warnings: 286177 mysql-5.7.5 (osm) > SHOW WARNINGS; +---------+------+-----------------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------------------------------------------+ | Warning | 1953 | The value specified for virtual column 'month' in table 'nodes_computed' ignored. | mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_computed WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.57 sec) 34 Maybe an error should be produced in strict mode, as MariaDB does?
  • 35. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Features and Limitations • Virtual “non-stored” columns cannot be indexed • “Stored columns” can be PRIMARY, UNIQUE, FOREIGN and MULTI keys • It cannot contain subqueries or other tables or rows references • It cannot contain used-defined stored functions • It can contain server and user-defined variables • It can be computed based on other virtual columns 35
  • 36. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY REWRITE PLUGINS © 2014 DBAHIRE.COM 36
  • 37. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New APIs for Query Rewriting • One for pre-parsing rewriting • Another for post-parsing rewriting © 2014 DBAHIRE.COM 37
  • 38. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Post- Plugin Installation $ ./sandboxes/msb_5_7_5/use -u root < 5.7.5/lib/plugin/ install_rewriter_plugin.sql – It is mostly broken in the lab build (for my tests) – Uninstall it before restart, or MySQL will segfault © 2014 DBAHIRE.COM 38 More on this: http://mysqlserverteam.com/the-query-rewrite-plugins/
  • 39. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Query Rewriting Setup mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules (pattern, replacement) VALUES ('SELECT ?', 'SELECT ? + 1'); Query OK, 1 row affected (0.01 sec) mysql-5.7.5 ((none)) > CALL query_rewrite.flush_rewrite_rules(); Query OK, 1 row affected (0.00 sec) mysql-5.7.5 ((none)) > SELECT 1; +-------+ | 1 + 1 | +-------+ | 2 | +-------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 ((none)) > SHOW WARNINGS; +-------+------+-------------------------------------------------------------------+ | Level | Code | Message | +-------+------+-------------------------------------------------------------------+ | Note | 1105 | Query 'SELECT 1' rewritten to 'SELECT 1 + 1' by plugin: Rewriter. | +-------+------+-------------------------------------------------------------------+ 1 row in set (0.00 sec) 39
  • 40. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Considerations • It cannot correct malformed queries- pattern and replacement must be syntactically correct • Useful for query optimizations for 3rd party applications • User is notified that a rewrite happened with a note-level message • Low overhead (5%); specially for untouched queries • You can do stupid things like: mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules( pattern, replacement ) VALUES ( 'SELECT 1', 'DROP TABLE test.test' ); Query OK, 1 row affected (0.01 sec) 40
  • 41. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SQL MODE CHANGES © 2014 DBAHIRE.COM 41
  • 42. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Default SQL Mode Changes • MySQL 5.5 and earlier – '' • MySQL 5.6 (from 5.6.6): – 'NO_ENGINE_SUBSTITUTION' is the default – NO_ENGINE_SUBSTITUTION and STRICT_TRANS_TABLES were suggested in upstream default my.cnf • MySQL 5.7 (from 5.7.5): – 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_EN GINE_SUBSTITUTION' is the new default 42 More on this: http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html
  • 43. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stricter Defaults mysql> CREATE TABLE `test` ( `id` int(11) NOT NULL, `type` enum('movie','album','videogame') NOT NULL, `date` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB; mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); 43 What happens?
  • 44. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.5/5.6 with default settings mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); Query OK, 1 row affected, 3 warnings (0.00 sec) mysql> SHOW WARNINGS; +---------+------+-----------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------+ | Warning | 1364 | Field 'id' doesn't have a default value | | Warning | 1265 | Data truncated for column 'type' at row 1 | | Warning | 1264 | Out of range value for column 'date' at row 1 | +---------+------+-----------------------------------------------+ 3 rows in set (0.00 sec) mysql> SELECT * FROM test; +----+------+---------------------+ | id | type | date | +----+------+---------------------+ | 0 | | 0000-00-00 00:00:00 | +----+------+---------------------+ 1 row in set (0.00 sec) © 2014 DBAHIRE.COM 44
  • 45. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); ERROR 1265 (01000): Data truncated for column 'type' at row 1 mysql> INSERT INTO test (type, date) VALUES ('videogame', -1); ERROR 1292 (22007): Incorrect datetime value: '-1' for column 'date' at row 1 mysql> INSERT INTO test (type, date) VALUES ('movie', now()); ERROR 1364 (HY000): Field 'id' doesn't have a default value mysql> INSERT INTO test (id, type, date) VALUES (1, 'videogame', now()); Query OK, 1 row affected (0.01 sec) 45
  • 46. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GROUP BY Behavior Changes mysql-5.6.21 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; +-----------+----------+------------+ | way_id | count(*) | node_id | +-----------+----------+------------+ | 155339744 | 1187 | 1558095871 | | 243986064 | 1156 | 2604713337 | | 87136668 | 1128 | 1013304944 | | 148812873 | 852 | 1618837453 | | 149200774 | 835 | 34921158 | | 183618216 | 826 | 1940223096 | | 273858696 | 824 | 1267549776 | | 261584374 | 770 | 2669122104 | | 227880171 | 704 | 2240011804 | | 193564006 | 684 | 1808872763 | +-----------+----------+------------+ 10 rows in set (1.24 sec) 46 Undeterministic behavior
  • 47. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks With ONLY_FULL_GROUP_BY (default in 5.7) mysql-5.7.5 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'osm.way_nodes.node_id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by 47
  • 48. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Problem with ONLY_FULL_GROUP_BY in Functional dependency © 2014 DBAHIRE.COM MySQL <= 5.6 mysql-5.6.21 (osm) > SET SQL_mode='ONLY_FULL_GROUP_BY'; Query OK, 0 rows affected (0.00 sec) mysql-5.6.21 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): 'osm.u.display_name' isn't in GROUP BY 48 More on this: http://rpbouman.blogspot.com/2014/09/mysql-575-group-by-respects-functional.html
  • 49. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks 5.7 Aims for SQL99 Compliance mysql-5.7.5 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; +---------+---------------+--------------+ | user id | user name | # changesets | +---------+---------------+--------------+ | 31257 | Ed Avis | 4333 | | 508 | Welshie | 2696 | | 1016290 | Amaroussi | 2351 | | 352985 | ecatmur | 1450 | | 142807 | SDavies | 1342 | | 736 | Steve Chilton | 1182 | | 346 | Tom Chance | 1175 | | 38784 | Tom Morris | 1165 | | 88718 | UrbanRambler | 1151 | | 1611 | Harry Wood | 1091 | +---------+---------------+--------------+ 10 rows in set (0.09 sec) © 2014 DBAHIRE.COM 49
  • 50. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Backward Compatibility • Some ORMs and frameworks change the default SQL Mode: – Ruby on Rails 4+ sets STRICT_ALL_TABLES – Drupal 7+ sets TRADITIONAL • Other applications do not work in standard-compliance modes: – Wordpress does not work in strict mode: http:// www.xaprb.com/blog/2013/03/15/wordpress-and-mysqls-strict-mode/ – Cacti does not work with ONLY_FULL_GROUP_BY,NO_ZERO_DATE 50
  • 51. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Deprecated Modes • ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE are deprecated and do nothing – Use STRICT_TRANS_TABLES or STRICT_ALL_TABLES, which include those modes and produce an error 51
  • 52. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks JOIN OPTIMIZATIONS © 2014 DBAHIRE.COM 52
  • 53. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM (Block) Nested Loop Join • Until MySQL 5.5, there was only one algorithm to execute a JOIN: 53 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 name London Eye
  • 54. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM “New” JOIN Methods • MySQL 5.6 added: – Batch Key Access* • MariaDB has since 5.3: – Batch Key Access* – Hash Joins – Slightly modified versions of the above (“incremental” buffers for joining 3 or more tables) 54 *I already mentioned BKA last year
  • 55. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Batch Key Access 55 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Buffer sorted in physical/ PK order
  • 56. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Hash Joins • Only work for equi-joins 56 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Hash table for faster access to rows
  • 57. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Configuration • BKA requires changes of default optimizer configuration: mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr=on'; mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr_cost_based=off'; mysql-5.7.5 (osm) > SET optimizer_switch= 'batch_key_access=on'; 57
  • 58. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MariaDB configuration mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_incremental=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_hashed=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_bka=on'; - Enabled by default mariadb-10.0.14 (osm) > SET join_cache_level = 3 (for hash joins) mariadb-10.0.14 (osm) > SET join_cache_level = 5 (for BKA) - Also, configure join_buffer_size appropriately. - Hash joins, like BKA, are highly dependent on disk-bound DBs to be effective due to the extra overhead 58
  • 59. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Nested Loop Join (cold buffers, buffer_pool=100MB) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 19 Extra: 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (32.86 sec) 59
  • 60. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Hash Join (cold buffers, buffer_pool=100M, © 2014 DBAHIRE.COM join_buffer=4M) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: hash_ALL possible_keys: changeset_id key: #hash#changeset_id key_len: 8 ref: osm.changesets.id rows: 2781732 Extra: Using join buffer (flat, BNLH join) 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (6.66 sec) 60
  • 61. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SUBQUERY OPTIMIZATIONS © 2014 DBAHIRE.COM 61
  • 62. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Delayed Materialization of Derived Tables • EXPLAIN performance improved (it does not execute subqueries) • Derived tables can be indexed for improved performance © 2014 DBAHIRE.COM 62
  • 63. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example with a Derived Table mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2865312 | | | 1 | PRIMARY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | n.changeset_id | 1 | Using index | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ 3 rows in set (1.42 sec) mysql-5.6.21 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | 1 | PRIMARY | changesets | index | PRIMARY | PRIMARY | 8 | NULL | 70917 | Using index | | 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 8 | osm.changesets.id | 40 | NULL | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2853846 | Using where | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ 3 rows in set (0.00 sec) © 2014 DBAHIRE.COM 63 No subquery execution Auto-generated index
  • 64. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Common 5.5 Performance Problem mysql-5.5.40 (osm) > EXPLAIN SELECT * FROM nodes © 2014 DBAHIRE.COM WHERE nodes.changeset_id IN ( SELECT changesets.id FROM changesets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | 1 | PRIMARY | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | | 2 | DEPENDENT SUBQUERY | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 2 | DEPENDENT SUBQUERY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | func | 1 | Using where | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ 3 rows in set (0.00 sec) mysql-5.5.40 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (2.60 sec) 64 This means that the subquery is executed almost 3 million times
  • 65. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Semijoin Optimization • The only way to execute certain IN subqueries was to execute them with poor strategy – This forced to rewrite certain queries into JOINS or scalar subqueries, when possible • There are now several additional strategies/ optimizations: – Convert to a JOIN – Materialization (including index creation) – FirstMatch – LooseScan – Duplicate Weedout 65
  • 66. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks The Previous Query is Not a Problem in © 2014 DBAHIRE.COM 5.6/5.7/MariaDB 5.3+ mysql-5.6.21 (osm) > EXPLAIN SELECT * FROM nodes WHERE nodes.changeset_id IN ( SELECT changesets.id FROM change sets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | 1 | SIMPLE | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 1 | SIMPLE | changesets | ALL | PRIMARY | NULL | NULL | NULL | 70917 | Using where | | 1 | SIMPLE | nodes | ref | changeset_id | changeset_id | 8 | osm.changesets.id | 21 | NULL | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ 3 rows in set (0.00 sec) mysql-5.6.21 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (0.02 sec) 66 Executed as a regular JOIN
  • 67. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM First Match Strategy mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM changesets WHERE id IN (SELECT changeset_id FROM nodes)G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: changesets partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 70917 filtered: 100.00 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 33 filtered: 100.00 Extra: Using index; FirstMatch(changesets) 67 It is converting the ref into an eq_ref
  • 68. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Enabling and disabling materialization/ © 2014 DBAHIRE.COM semijoin mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 68
  • 69. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks FULLTEXT SEARCH © 2014 DBAHIRE.COM 69
  • 70. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Fuzzy Search of “gloucester/Gloucester’s/ © 2014 DBAHIRE.COM etc” • “Typical” way to solve this: mysql-5.7.5 (osm) > SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'; 425 rows in set (0.46 sec) 70 Too slow
  • 71. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Let’s Add an Index mysql-5.7.5 (osm) > ALTER TABLE way_tags ADD INDEX(v); Query OK, 0 rows affected (6.44 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.38 sec) • Still slow, why? mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE v like '%gloucester%'; +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | way_tags | NULL | index | NULL | v_2 | 767 | NULL | 1333338 | 11.11 | Using where; Using index | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ 1 row in set, 1 warning (0.01 sec) 71
  • 72. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Fulltext Index mysql-5.7.5 (osm) > ALTER TABLE way_tags add FULLTEXT index(v); Query OK, 0 rows affected (3.20 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE MATCH(v) AGAINST ('+gloucester*' IN BOOLEAN MODE); +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | 1 | SIMPLE | way_tags | NULL | fulltext | v | v | 0 | const | 1 | 100.00 | Using where; Ft_hints: no_ranking | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ 1 row in set, 1 warning (0.00 sec) 72
  • 73. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM way_tags_myisam WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | way_tags_myisam | fulltext | v | v | 0 | | 1 | Using where | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec) mysql-5.5.40 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 0 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 425 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 16 rows in set (0.00 sec) © 2014 DBAHIRE.COM 73
  • 74. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM way_tags WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 2 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 0 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) © 2014 DBAHIRE.COM 74 It’s counting directly from the FULLTEXT index
  • 75. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Open Issues and Limitations • No postfix support (wildcards) • Simple Ranking (and different from MyISAM) • No stemming support • Some multi-language limitations © 2014 DBAHIRE.COM 75 More on FULLTEXT InnoDB support: http://www.drdobbs.com/database/full-text- search-with-innodb/231902587
  • 76. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GIS IMPROVEMENTS © 2014 DBAHIRE.COM 76
  • 77. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Find the Closest Starbucks mysql-5.7.5 (osm) > SET @lat:=51.49353; SET @lon:=-0.18340; mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.20 sec) 77 You are here
  • 78. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM This Query is Slow mysql-5.7.5 (osm) > EXPLAIN SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/ 10000000 as latitude, sqrt(pow((latitude/10000000- @lat)*111257.67, 2) + pow((longitude/10000000- @lon)*69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1G ********************* 1. row ********************* id: 1 select_type: SIMPLE table: n_t1 partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 832040 filtered: 0.00 Extra: Using where; Using temporary; Using file sort ********************* 2. row ********************* id: 1 select_type: SIMPLE table: n_t2 partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 3 filtered: 1.41 Extra: Using where ********************* 3. row ********************* id: 1 select_type: SIMPLE table: n partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 1 filtered: 100.00 Extra: NULL 3 rows in set, 1 warning (0.00 sec) 78
  • 79. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Can We Optimize it? • We could add a bounding box: mysql-5.7.5 (osm) > ALTER TABLE nodes add index(latitude, longitude); mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and n.latitude BETWEEN ((@lat - 1000/111257.67) * 10000000) AND ((@lat + 1000/111257.67) * 10000000) and n.longitude BETWEEN ((@lon - 1000/69450.32) * 10000000) AND ((@lon + 1000/69450.32) * 10000000) ORDER BY `distance in metres` ASC LIMIT 1; 79 This is not a square, only an approximation
  • 80. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Create an Index… and Force It mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | 1 | SIMPLE | n_t1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 832040 | 0.00 | Using where; | | | | | | | | | | | | | Using temporary; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n | NULL | ref | PRIMARY, | PRIMARY | 8 | osm.n_t1.node_id | 1 | 5.00 | Using where | | | | | | | latitude | | | | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n_t1.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | 1 | SIMPLE | n | NULL | range | latitude | latitude | 8 | NULL | 493666 | 11.11 | Using where; | | | | | | | | | | | | | Using index; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.09 sec) © 2014 DBAHIRE.COM 80 Still many rows are examined Most of the gain comes from the covering index, not the filtering MySQL ignores the newly created index, why? Performance improvement is not great
  • 81. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Multiple Range Scans Cannot Be Optimized with BTREE Indexes • We need quadtrees or R-TREE Indexes for indexing in multiple dimensions – The later are implemented in MySQL with the name “SPATIAL indexes”, as they only apply to GIS types • Spatial indexing is available for the first time for InnoDB tables on MySQL 5.7.5 81
  • 82. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Creating a Spatial Index mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN coord GEOMETRY NOT NULL; Query OK, 0 rows affected (21.80 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > UPDATE nodes SET coord = point(longitude/1000000, latitude/10000000); Query OK, 2865312 rows affected (34.66 sec) Rows matched: 2865312 Changed: 2865312 Warnings: 0 mysql-5.7.5 (osm) > ALTER TABLE nodes add SPATIAL index(coord); Query OK, 0 rows affected (1 min 50.00 sec) Records: 0 Duplicates: 0 Warnings: 0 82 This is new in 5.7
  • 83. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM New Query mysql> SET @area := envelope(linestring(POINT(@lon - 500/69450.32, @lat - 500/111257.67), POINT(@lon + 500/69450.32, @lat + 500/111257.67))); mysql> SELECT n.node_id, x(n.coord) as longitude, y(n.coord) as latitude, st_distance(POINT(@lon, @lat), coord) as distance FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and st_within(coord, @area) ORDER BY st_distance(POINT(@lon, @lat), coord) ASC LIMIT 1; 83 We can use any shape we want thanks to 5.6 improvements
  • 84. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Performance mysql-5.7.5 (osm) > SELECT ...; +-----------+------------+------------+-----------------------+ | node_id | longitude | latitude | distance | +-----------+------------+------------+-----------------------+ | 699693936 | -0.1822879 | 51.4944808 | 0.0014631428672541478 | +-----------+------------+------------+-----------------------+ 1 row in set (0.02 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | id | select | table | parti | type | possible | key | key | ref | rows | filtered | Extra | | | _type | | tions | | _keys | | _len | | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | 1 | SIMPLE | n | NULL | range | PRIMARY | coord | 34 | NULL | 2 | 100.00 | Using where; | | | | | | | ,coord | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+---------------------------+ 3 rows in set, 1 warning (0.00 sec) 84 This field is almost useless
  • 85. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Filtering Not using the index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 1 | | Handler_read_key | 1914 | | Handler_read_last | 0 | | Handler_read_next | 1954 | | Handler_read_prev | 0 | | Handler_read_rnd | 1 | | Handler_read_rnd_next | 833426 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 1 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the BTREE index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 274 | | Handler_read_last | 0 | | Handler_read_next | 246540 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the SPATIAL index: +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 522 | | Handler_read_last | 0 | | Handler_read_next | 5254 | | Handler_read_prev | 0 | | Handler_read_rnd | 259 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) 85 mysql-5.7.5 (osm) > SHOW STATUS LIKE 'Hand%';
  • 86. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Geohash Functions • mysql-5.7.5 (osm) > SELECT ST_GeoHash(@lon, @lat, 10); +----------------------------+ | ST_GeoHash(@lon, @lat, 10) | +----------------------------+ | gcpugy47w3 | +----------------------------+ 1 row in set (0.00 sec) • Useful to index coordinates with a BTREE – It could be specially useful combined with indexed STORED columns (emulating quadtrees) 86 More on Geohashing: http://mysqlserverteam.com/geohash-functions/
  • 87. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions mysql-5.7.5 (osm) > SELECT nm.v, ST_AsGeoJson(n.coord) FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' AND n_t.v='attraction' AND nm.k='name'; +-----------------------------+--------------------------------------------------------+ | v | ST_AsGeoJson(n.coord) | +-----------------------------+--------------------------------------------------------+ | BedZED | {"type":"Point","coordinates":[-0.1560632,51.3821745]} | | Blewcoat School | {"type":"Point","coordinates":[-0.1360484,51.4983179]} | | Camden / Buck Street Market | {"type":"Point","coordinates":[-0.143193,51.5400398]} | | Camden Lock Village | {"type":"Point","coordinates":[-0.1447181,51.5416552]} | | Wimbledon Windmill | {"type":"Point","coordinates":[-0.2315468,51.4376583]} | . . . 87
  • 88. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions (cont.) $ ./sandboxes/msb_5_7_5/use osm -B -e "SET SESSION group_concat_max_len = 10000; SELECT CONCAT('{"type":"FeatureCollection", "features":[ ', GROUP_CONCAT(CONCAT('{"type":"Feature", "geometry":', ST_AsGeoJson(n.coord), ',"properties":{"name":"',nm.v,'"}}')), ' ]}') FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' and n_t.v='attraction' AND nm.k='name'" http://geojsonlint.com/ 88
  • 89. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Open Issues • The SRID can be set and retrieved, but all operations are done in squared euclidean coordinates: 89 mysql-5.7.5 (osm) > SET @p1 := GeomFromText('POINT(-1 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p2 := GeomFromText('POINT(0 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p3 := GeomFromText('POINT(-1 52)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SELECT srid(@p1); +-----------+ | srid(@p1) | +-----------+ | 4326 | +-----------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p2); +-----------------------+ | st_distance(@p1, @p2) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p3); +-----------------------+ | st_distance(@p1, @p3) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec)
  • 90. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY PROFILING © 2014 DBAHIRE.COM 90
  • 91. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Which Queries Should I Optimize First? • My two favorite methods: – pt-query-digest – PERFORMANCE_SCHEMA • I prefer pt-query-digest for long-term reports, PERFORMANCE_SCHEMA for more real-time evaluation and fine-tuning – Also, PERFORMANCE_SCHEMA was not “ready” until © 2014 DBAHIRE.COM MySQL 5.6 91
  • 92. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest • It is a 3rd party tool written in Perl, originally created by Baron Schwartz • It requires activation of the slow log: – SET GLOBAL slow_query_log = 1; – SET long_query_time = 0; • In Percona Server and MariaDB it provides extra information: – SHOW GLOBAL VARIABLES like 'log_slow_verbosity'; 92 Be careful with extra IO and latency!
  • 93. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest Execution $ pt-query-digest ~/sandboxes/msb_5_7_5/data/localhost-slow.log # 190ms user time, 30ms system time, 25.45M rss, 219.82M vsz # Current date: Sat Nov 1 17:26:58 2014 # Hostname: localhost.localdomain # Files: /home/dbahire/sandboxes/msb_5_7_5/data/localhost-slow.log # Overall: 21 total, 7 unique, 0 QPS, 0x concurrency _____________________ # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 7s 6us 482ms 324ms 455ms 201ms 433ms # Lock time 18ms 0 11ms 840us 224us 2ms 113us # Rows sent 7.08k 0 1.22k 345.43 420.77 300.48 420.77 # Rows examine 19.22M 0 1.28M 937.14k 1.26M 581.94k 1.26M # Query size 1.10k 11 67 53.81 65.89 18.23 65.89 # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x5060CBA490BE371E 6.4236 94.4% 14 0.4588 0.00 SELECT way_tags 93
  • 94. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks pt-query-digest Execution (cont.) # Query 1: 0 QPS, 0x concurrency, ID 0x5060CBA490BE371E at byte 8795 _____ # This item is included in the report because it matches --limit. # Scores: V/M = 0.00 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 66 14 # Exec time 94 6s 435ms 482ms 459ms 477ms 15ms 455ms # Lock time 10 2ms 0 227us 137us 204us 54us 113us # Rows sent 87 6.21k 0 1.22k 453.86 420.77 242.38 420.77 # Rows examine 93 17.94M 1.28M 1.28M 1.28M 1.28M 0 1.28M # Query size 80 912 59 67 65.14 65.89 2.70 65.89 # String: # Databases osm # Hosts localhost # Time 2014-11-01... (1/7%), 2014-11-01... (1/7%)... 12 more # Users msandbox # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms # 100ms ################################################################ # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `osm` LIKE 'way_tags'G # SHOW CREATE TABLE `osm`.`way_tags`G # EXPLAIN /*!50100 PARTITIONS*/ SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'G © 2014 DBAHIRE.COM 94
  • 95. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM PERFORMANCE_SCHEMA • Monitoring schema enabled by default since MySQL 5.6 – performance_schema = 1 (it is not dynamic) • Deprecates the old query profiling • It is way more user-friendly when combined with the SYS schema/ps_helper (a set of views and stored procedures created by Mark Leith) 95
  • 96. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Installation of the SYS Schema $ git clone https://github.com/MarkLeith/mysql-sys.git Cloning into 'mysql-sys'... remote: Counting objects: 926, done. remote: Compressing objects: 100% (73/73), done. remote: Total 926 (delta 35), reused 6 (delta 2) Receiving objects: 100% (926/926), 452.19 KiB | 225.00 KiB/s, done. Resolving deltas: 100% (584/584), done. $ cd mysql-sys/ $ ~/sandboxes/msb_5_7_5/use < sys_57.sql © 2014 DBAHIRE.COM 96
  • 97. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Discovering Unused © 2014 DBAHIRE.COM Indexes mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_nodes | current_nodes_tile_idx | | osm | current_relations | current_relations_timestamp_idx | +---------------+-------------------+---------------------------------+ 5 rows in set (0.04 sec) mysql-5.7.5 (osm) > SELECT * FROM current_nodes WHERE tile = 100; ... mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_relations | current_relations_timestamp_idx | | osm | current_relations | changeset_id | +---------------+-------------------+---------------------------------+ 5 rows in set (0.03 sec) 97 With enough activity, it can help us clean up our schema
  • 98. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Slow Queries (ordered by © 2014 DBAHIRE.COM server time) ********* 1. row ********* query: SELECT `way_id` AS `id` , `v` FROM `way_tags` WHERE `v` LIKE ? db: osm full_scan: * exec_count: 15 err_count: 0 warn_count: 0 total_latency: 7.83 s max_latency: 1.33 s avg_latency: 521.84 ms lock_latency: 17.94 ms rows_sent: 6779 rows_sent_avg: 452 rows_examined: 20152155 rows_examined_avg: 1343477 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 21f90695b1ebf20a5f4d4c1e5e860f58 first_seen: 2014-11-01 17:04:51 last_seen: 2014-11-01 17:05:22 98 mysql-5.7.5 (osm) > SELECT * FROM sys.statement_analysis LIMIT 10G
  • 99. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables mysql-5.7.5 (osm) > SELECT * FROM sys.statements_with_temp_tables WHERE db = 'osm' LIMIT 10G *************************** 1. row *************************** query: SELECT ? AS TYPE , `node_id` A ... gs` WHERE `k` = ? AND `v` = ? db: osm exec_count: 11 total_latency: 7.57 s memory_tmp_tables: 11 disk_tmp_tables: 0 avg_tmp_tables_per_query: 1 tmp_tables_to_disk_pct: 0 first_seen: 2014-11-01 17:33:55 last_seen: 2014-11-01 17:34:45 digest: 5e6e82799b7c7c0e5c57cfe63eb98d5d 99
  • 100. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > SELECT DIGEST_TEXT FROM performance_schema.events_statements_summary_by_digest WHERE digest = '5e6e82799b7c7c0e5c57cfe63eb98d5d'G *************************** 1. row *************************** DIGEST_TEXT: SELECT ? AS TYPE , `node_id` AS `id` FROM `node_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `way_id` AS `id` FROM `way_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `relation_id` AS `id` FROM `relation_tags` WHERE `k` = ? AND `v` = ? 1 row in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | | NULL | UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ 4 rows in set, 1 warning (0.01 sec) 100
  • 101. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ 3 rows in set, 1 warning (0.04 sec) 101 No Temporary tables now in 5.7
  • 102. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks CONCLUSIONS © 2014 DBAHIRE.COM 102
  • 103. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM 5.7/10.1 Still in Alpha • New features usually involve new bugs – I only suffered a single crashing bug – Some of the bleeding edge features are not yet optimized for performance and usability • The greatest problem is finding documentation, as some features are only documented on the developer’s blogs – It will probably catch up slowly • It is great to have the lab builds, though, for testing 103
  • 104. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 New Features • MySQL 5.6 seemed Percona Server-inspired • MySQL 5.7 seems MariaDB-inspired – Competition is always good for consumer 104
  • 105. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Many Optimizer Advantages Have to Be © 2014 DBAHIRE.COM Manually Enabled • Modifying on a per-query basis: SET optimizer_switch='batched_key_access=on'; SET join_cache_level=8; # for MariaDB in order to take advantage of them make the features useless unless you are fine-tuning – I expect that to change in the future 105
  • 106. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Herby Declare MyISAM as Dead • All major MyISAM-only features are now on MySQL 5.7 – FULLTEXT – GIS – Transportable tables • There are still reasons to use MyISAM – MyISAM is still required for the mysql © 2014 DBAHIRE.COM schema and non-durable temporary tables (WIP) 106
  • 107. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Benchmarks • Do not trust first party benchmarks – In fact, do not trust 3rd party benchmarks either • Only care about the performance of your application running on your hardware 107
  • 108. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Q&A 108
  • 109. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Knowing More about Query Optimization • Tomorrow: Bill Karwin’s session “Query Patterns, Optimized” • “Optimization, Administration and High Availability with MySQL 5.6” course by DBAHire.com on 15th December (Spain) – More dates coming soon © 2014 DBAHIRE.COM 109
  • 110. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Talk to me During the Percona Live • Talk to me about: – MySQL/MariaDB/Percona – OpenStreetMap – Game of Thrones and I will offer you 1 hour of free MySQL © 2014 DBAHIRE.COM online consulting by myself* 110 *Subjected to availability
  • 111. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Thank You for Attending! • Special thanks to -in ORDER BY rand(): Bill Karwin, Morgan Tocker, Baron Schwartz, Giuseppe Maxia, Ewen Fortune, Henrik Ingo, Tim “do-not- confuse-with-Mark” Callaghan, David Hildebrandt, Cédric Peintre, Kortney Runyan, Miguel Ángel Nieto, Valerii Kravchuk, Aleksandr Kuzminsky and everyone at the Percona and Oracle-MySQL teams and the Percona Live London organization, committee and sponsors • Do not forget, after the session finishes, to please login with your Percona Live account and go to http:// www.percona.com/live/london-2014/sessions/query-optimization- mysql-57-and-mariadb-10-even-newer-tricks to “Rate This Session” for a chance to win great prizes! 111