MySQL 8.0: Common Table Expressions

209 views

Published on

Presentation at Percona Live Amsterdam, October 2016

Published in: Data & Analytics
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
209
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

MySQL 8.0: Common Table Expressions

  1. 1. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | MySQL 8.0: Common Table Expressions Øystein Grøvlen – Senior Principal Software Engineer MySQL Optimizer Team, Oracle
  2. 2. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 2
  3. 3. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 3
  4. 4. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression (MySQL 8.0.0 labs release) • A derived table is a subquery in the FROM clause SELECT … FROM (subquery) AS derived, t1 ... • Common Table Expression (CTE) is just like a derived table, but its declaration is put before the query block instead of in FROM clause WITH derived AS (subquery) SELECT … FROM derived, t1 ... • A CTE may precede SELECT/UPDATE/DELETE including sub-queries WITH derived AS (subquery) DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived); 4 Alternative to derived table
  5. 5. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | CTE WITH cte_name [( <list of column names> )] AS ( SELECT ... # Definition ) [, <any number of other CTE definitions> ] <SELECT/UPDATE/DELETE statement> 5 Syntax
  6. 6. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression (CTE) WITH qn AS (SELECT a FROM t1) SELECT * from qn; WITH qn AS (SELECT a+2 AS a, b FROM t1) UPDATE t1, qn SET t1.a=qn.a + 10 WHERE t1.a - qn.a = 0; WITH qn(a, b) AS (SELECT a+2, b FROM t2) DELETE t1 FROM t1, qn WHERE t1.a - qn.a = 0; INSERT INTO t2 WITH qn AS (SELECT 10*a AS a FROM t1) SELECT * from qn; SELECT * FROM t1 WHERE t1.a IN (WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1) SELECT a + 0 FROM cte); 6 Examples
  7. 7. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression versus Derived Table Better readability Can be referenced multiple times Can refer to other CTEs Improved performance 7
  8. 8. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Better readability • Derived table: SELECT … FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON … • CTE: WITH dt AS (SELECT ... FROM ...) SELECT ... FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ... 8
  9. 9. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Can be referenced multiple times • Derived table can not be referenced twice: SELECT ... FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1 JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a; • CTE can: WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a; 9
  10. 10. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Can refer to other CTEs • Derived tables can not refer to other derived tables: SELECT … FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 … ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist • CTEs can refer other CTEs: WITH d1 AS (SELECT … FROM …), d2 AS (SELECT … FROM d1 …) SELECT FROM d1, d2 … 10
  11. 11. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Chained CTEs WITH cte1(txt) AS (SELECT "This "), cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1), cte3(txt) AS (SELECT "nice query" UNION SELECT "query that rocks" UNION SELECT "query"), cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3) SELECT MAX(txt), MIN(txt) FROM cte4; +----------------------------+----------------------+ | MAX(txt) | MIN(txt) | +----------------------------+----------------------+ | This is a query that rocks | This is a nice query | +----------------------------+----------------------+ 1 row in set (0,00 sec) 11 Neat, but not very useful example
  12. 12. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Better performance • Derived table: – For derived tables that are materialized, two identical derived tables will be materialized. Performance problem (more space, more time, longer locks) – Similar with view references • CTE: – Will be materialized once, regardless of how many references 12
  13. 13. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT3 Query 15 CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey; SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey; Top Supplier Query
  14. 14. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Materialization of view DBT-3 Query 15 ORDER JOIN Supplier (eq_ref) GROUP Lineitem (range) Materialize GROUP Lineitem (range) Materialize SELECT Subquery Revenue0 Revenue0
  15. 15. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 EXPLAIN CTE id select type table type possible keys key rows filtered Extra 1 PRIMARY <derived3> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort 1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL 3 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary 2 SUBQUERY <derived4> ALL NULL NULL 4801074 100.00 NULL 4 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
  16. 16. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | CTE Materialization WITH revenue0 (supplier_no, total_revenue) AS (SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey) SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey; DBT3 Query 15: Top Supplier Query
  17. 17. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15, CTE EXPLAIN CTE id select type table type possible keys key rows filtered Extra 1 PRIMARY <derived2> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort 1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL 3 SUBQUERY <derived2> ALL NULL NULL 4801074 100.00 NULL 2 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
  18. 18. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 Materialization of CTE ORDER JOIN Supplier (eq_ref) GROUP Lineitem (range) Materialize SELECT Subquery Revenue0 FROM
  19. 19. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 Confidential – Oracle Internal/Restricted/Highly Restricted 19 Query Performance 0 2 4 6 8 10 12 14 16 18 View CTE QueryExecutionTime(seconds)
  20. 20. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Non-recursive CTE • Find best and worst month: WITH sales_by_month(month,total) AS # first CTE: one row per month, with amount sold on all days of month (SELECT MONTH(day_of_sale), SUM(amount) FROM sales_days WHERE YEAR(day_of_sale)=2015 GROUP BY MONTH(day_of_sale)), best_month(month, total, award) AS # second CTE: best month (SELECT month, total, "best" FROM sales_by_month WHERE total=(SELECT MAX(total) FROM sales_by_month)), worst_month(month, total, award) AS # 3rd CTE: worst month (SELECT month, total, "worst" FROM sales_by_month WHERE total=(SELECT MIN(total) FROM sales_by_month)) # Now show best and worst: SELECT * FROM best_month UNION All SELECT * FROM worst_month; 20 A more useful example
  21. 21. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Non-recursive CTE • Result: +-------+-------+-------+ | month | total | award | +-------+-------+-------+ | 1 | 300 | best | | 3 | 11 | worst | +-------+-------+-------+ 21 A more useful example
  22. 22. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 22
  23. 23. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • A recursive CTE refers to itself in a subquery • The “seed” SELECT is executed once to create the initial data subset, the recursive SELECT is repeatedly executed to return subsets of data until the complete result set is obtained. • Recursion stops when an iteration does not generate any new rows • Useful to dig in hierarchies (parent/child, part/subpart) 23 WITH RECURSIVE cte AS ( SELECT ... FROM table_name /* "seed" SELECT */ UNION ALL SELECT ... FROM cte, table_name) /* "recursive" SELECT */ SELECT ... FROM cte;
  24. 24. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE 24 A simple example Print 1 to 10 : WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; a 1 2 3 4 5 6 7 8 9 10
  25. 25. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE 25 INSERT Insert 1 to 10 : INSERT INTO numbers WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; SELECT * FROM numbers; a 1 2 3 4 5 6 7 8 9 10
  26. 26. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Date sequence 26 Missing dates SELECT orderdate, SUM(totalprice) sales FROM orders GROUP BY orderdate ORDER BY orderdate; +------------+-----------+ | orderdate | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-07 | 185991.79 | +------------+-----------+
  27. 27. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Date sequence 27 All dates WITH RECURSIVE dates(date) AS ( SELECT '2016-09-01' UNION ALL SELECT DATE_ADD(date, INTERVAL 1 DAY) FROM dates WHERE date < '2016-09-07‘ ) SELECT dates.date, COALESCE(SUM(totalprice), 0) sales FROM dates LEFT JOIN orders ON dates.date = orders.orderdate GROUP BY dates.date ORDER BY dates.date; +------------+-----------+ | date | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-02 | 0.00 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-06 | 0.00 | | 2016-09-07 | 185991.79 | +------------+-----------+
  28. 28. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | n un unp 1 1 1 1 2 1 2 3 2 3 4 3 5 5 5 8 6 8 13 7 13 21 8 21 34 9 34 55 10 55 89 Fibonacci Numbers 28 WITH RECURSIVE qn(n, un, unp1) AS ( SELECT 1, 1 , 1 UNION ALL SELECT 1+n, unp1, un+unp1 FROM qn WHERE n<10) SELECT * FROM qn;
  29. 29. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 29 Employee database CREATE TABLE employees ( id INT PRIMARY KEY, name VARCHAR(100), manager_id INT, FOREIGN KEY (manager_id) REFERENCES employees(id) ); INSERT INTO employees VALUES (333, "Yasmina", NULL), # CEO (198, "John", 333), # John reports to 333 (692, "Tarek", 333), (29, "Pedro", 198), (4610, "Sarah", 29), (72, "Pierre", 29), (123, "Adil", 692);
  30. 30. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 30 List reporting chain WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 692 Tarek 333,692 29 Pedro 333,198,29 123 Adil 333,692,123 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72
  31. 31. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 31 List descendants WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 29 Pedro 333,198,29 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72 692 Tarek 333,692 123 Adil 333,692,123
  32. 32. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • WITH RECURSIVE cte_name [list of column names ] AS ( SELECT ... <-- specifies initial set UNION ALL SELECT ... <-- specifies initial set UNION ALL ... SELECT ... <-- specifies how to derive new rows UNION ALL SELECT ... <-- specifies how to derive new rows ... ) [, any number of other CTE definitions ] 32 Complete syntax
  33. 33. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • A recursive SELECT must not contain – GROUP BY – Aggregate functions (like SUM) – ORDER BY – LIMIT – DISTINCT • A recursive SELECT must reference the CTE only once and only in its FROM clause, not in any subquery • A recursive SELECT can not be the inner table of an outer join Confidential – Oracle Internal/Restricted/Highly Restricted 33 Restrictions
  34. 34. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Want to try it out? • Goto labs.mysql.com • Select MySQL Server 8.0.0 Optimizer • Available as – Linux binaries – Source code • Warning! – For testing purposes only! – NOT FIT FOR PRODUCTION. 34
  35. 35. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Other New Features in 8.0.0 Optimizer Labs Release • MERGE() and NO_MERGE() hints WITH cte AS ( ... ) SELECT /*+ NO_MERGE(cte) */ ... FROM cte ... • Column name aliases for derived tables SELECT ... FROM (SELECT ...) dt(a, b, c) ... • Descending index • Join order hints SELECT /*+ JOIN_ORDER(t1, t2) */ * FROM t1 JOIN t2 ... • JSON aggregation function – JSON_ARRAYAGG – JSON_OBJECTAGG Confidential – Oracle Internal/Restricted/Highly Restricted 35
  36. 36. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Want to learn more? • MySQL Server Team blog – http://mysqlserverteam.com/ – http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in- mysql-ctes/ – More to come • My blog: – http://oysteing.blogspot.com/ • MySQL forum – Optimizer & Parser: http://forums.mysql.com/list.php?115

×