MySQL 8.0: Common Table Expressions

982 views

Published on

Presentation at pre-FOSDEM'17 MySQL Day

Published in: Software
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
982
On SlideShare
0
From Embeds
0
Number of Embeds
513
Actions
Shares
0
Downloads
12
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

MySQL 8.0: Common Table Expressions

  1. 1. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | MySQL 8.0: Common Table Expressions Øystein Grøvlen – Senior Principal Software Engineer MySQL Optimizer Team, Oracle
  2. 2. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 2
  3. 3. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 3
  4. 4. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression (MySQL 8.0.0 labs release) • A derived table is a subquery in the FROM clause SELECT … FROM (subquery) AS derived, t1 ... • Common Table Expression (CTE) is just like a derived table, but its declaration is put before the query block instead of in FROM clause WITH derived AS (subquery) SELECT … FROM derived, t1 ... • A CTE may precede SELECT/UPDATE/DELETE including sub-queries WITH derived AS (subquery) DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived); 4 Alternative to derived table
  5. 5. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | CTE WITH cte_name [( <list of column names> )] AS ( SELECT ... # Definition ) [, <any number of other CTE definitions> ] <SELECT/UPDATE/DELETE statement> 5 Syntax
  6. 6. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression (CTE) WITH qn AS (SELECT a FROM t1) SELECT * from qn; WITH qn AS (SELECT a+2 AS a, b FROM t1) UPDATE t1, qn SET t1.a=qn.a + 10 WHERE t1.a - qn.a = 0; WITH qn(a, b) AS (SELECT a+2, b FROM t2) DELETE t1 FROM t1, qn WHERE t1.a - qn.a = 0; INSERT INTO t2 WITH qn AS (SELECT 10*a AS a FROM t1) SELECT * from qn; SELECT * FROM t1 WHERE t1.a IN (WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1) SELECT a + 0 FROM cte); 6 Examples
  7. 7. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Common Table Expression versus Derived Table Better readability Can be referenced multiple times Can refer to other CTEs Improved performance 7
  8. 8. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Better readability • Derived table: SELECT … FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON … • CTE: WITH dt AS (SELECT ... FROM ...) SELECT ... FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ... 8
  9. 9. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Can be referenced multiple times • Derived table can not be referenced twice: SELECT ... FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1 JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a; • CTE can: WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a; 9
  10. 10. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Can refer to other CTEs • Derived tables can not refer to other derived tables: SELECT … FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 … ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist • CTEs can refer other CTEs: WITH d1 AS (SELECT … FROM …), d2 AS (SELECT … FROM d1 …) SELECT FROM d1, d2 … 10
  11. 11. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Chained CTEs WITH cte1(txt) AS (SELECT "This "), cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1), cte3(txt) AS (SELECT "nice query" UNION SELECT "query that rocks" UNION SELECT "query"), cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3) SELECT MAX(txt), MIN(txt) FROM cte4; +----------------------------+----------------------+ | MAX(txt) | MIN(txt) | +----------------------------+----------------------+ | This is a query that rocks | This is a nice query | +----------------------------+----------------------+ 1 row in set (0,00 sec) 11 Neat, but not very useful example
  12. 12. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Better performance • Derived table: – For derived tables that are materialized, two identical derived tables will be materialized. Performance problem (more space, more time, longer locks) – Similar with view references • CTE: – Will be materialized once, regardless of how many references 12
  13. 13. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT3 Query 15 CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey; SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey; Top Supplier Query
  14. 14. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Materialization of view DBT-3 Query 15 ORDER JOIN Supplier (eq_ref) GROUP Lineitem (range) Materialize GROUP Lineitem (range) Materialize SELECT Subquery Revenue0 Revenue0
  15. 15. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 EXPLAIN CTE id select type table type possible keys key rows filtered Extra 1 PRIMARY <derived3> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort 1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL 3 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary 2 SUBQUERY <derived4> ALL NULL NULL 4801074 100.00 NULL 4 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
  16. 16. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | CTE Materialization WITH revenue0 (supplier_no, total_revenue) AS (SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey) SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey; DBT3 Query 15: Top Supplier Query
  17. 17. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15, CTE EXPLAIN CTE id select type table type possible keys key rows filtered Extra 1 PRIMARY <derived2> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort 1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL 3 SUBQUERY <derived2> ALL NULL NULL 4801074 100.00 NULL 2 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
  18. 18. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 Materialization of CTE ORDER JOIN Supplier (eq_ref) GROUP Lineitem (range) Materialize SELECT Subquery Revenue0 FROM
  19. 19. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | DBT-3 Query 15 Confidential – Oracle Internal/Restricted/Highly Restricted 19 Query Performance 0 2 4 6 8 10 12 14 16 18 View CTE QueryExecutionTime(seconds)
  20. 20. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Non-recursive CTE • Find best and worst month: WITH sales_by_month(month,total) AS # first CTE: one row per month, with amount sold on all days of month (SELECT MONTH(day_of_sale), SUM(amount) FROM sales_days WHERE YEAR(day_of_sale)=2015 GROUP BY MONTH(day_of_sale)), best_month(month, total, award) AS # second CTE: best month (SELECT month, total, "best" FROM sales_by_month WHERE total=(SELECT MAX(total) FROM sales_by_month)), worst_month(month, total, award) AS # 3rd CTE: worst month (SELECT month, total, "worst" FROM sales_by_month WHERE total=(SELECT MIN(total) FROM sales_by_month)) # Now show best and worst: SELECT * FROM best_month UNION All SELECT * FROM worst_month; 20 A more useful example
  21. 21. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Non-recursive CTE • Result: +-------+-------+-------+ | month | total | award | +-------+-------+-------+ | 1 | 300 | best | | 3 | 11 | worst | +-------+-------+-------+ 21 A more useful example
  22. 22. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Program Agenda Non-recursive common table expressions Recursive common table expressions 1 2 22
  23. 23. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • A recursive CTE refers to itself in a subquery • The “seed” SELECT is executed once to create the initial data subset, the recursive SELECT is repeatedly executed to return subsets of data until the complete result set is obtained. • Recursion stops when an iteration does not generate any new rows • Useful to dig in hierarchies (parent/child, part/subpart) 23 WITH RECURSIVE cte AS ( SELECT ... FROM table_name /* "seed" SELECT */ UNION [DISTINCT|ALL] SELECT ... FROM cte, table_name) /* "recursive" SELECT */ SELECT ... FROM cte;
  24. 24. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE 24 A simple example Print 1 to 10 : WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; a 1 2 3 4 5 6 7 8 9 10
  25. 25. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE 25 INSERT Insert 1 to 10 : INSERT INTO numbers WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; SELECT * FROM numbers; a 1 2 3 4 5 6 7 8 9 10
  26. 26. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Date sequence 26 Missing dates SELECT orderdate, SUM(totalprice) sales FROM orders GROUP BY orderdate ORDER BY orderdate; +------------+-----------+ | orderdate | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-07 | 185991.79 | +------------+-----------+
  27. 27. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Date sequence 27 All dates WITH RECURSIVE dates(date) AS ( SELECT '2016-09-01' UNION ALL SELECT DATE_ADD(date, INTERVAL 1 DAY) FROM dates WHERE date < '2016-09-07‘ ) SELECT dates.date, COALESCE(SUM(totalprice), 0) sales FROM dates LEFT JOIN orders ON dates.date = orders.orderdate GROUP BY dates.date ORDER BY dates.date; +------------+-----------+ | date | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-02 | 0.00 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-06 | 0.00 | | 2016-09-07 | 185991.79 | +------------+-----------+
  28. 28. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | n un unp 1 1 1 1 2 1 2 3 2 3 4 3 5 5 5 8 6 8 13 7 13 21 8 21 34 9 34 55 10 55 89 Fibonacci Numbers 28 WITH RECURSIVE qn(n, un, unp1) AS ( SELECT 1, 1 , 1 UNION ALL SELECT 1+n, unp1, un+unp1 FROM qn WHERE n<10) SELECT * FROM qn;
  29. 29. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 29 Employee database CREATE TABLE employees ( id INT PRIMARY KEY, name VARCHAR(100), manager_id INT, FOREIGN KEY (manager_id) REFERENCES employees(id) ); INSERT INTO employees VALUES (333, "Yasmina", NULL), # CEO (198, "John", 333), # John reports to 333 (692, "Tarek", 333), (29, "Pedro", 198), (4610, "Sarah", 29), (72, "Pierre", 29), (123, "Adil", 692);
  30. 30. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 30 List reporting chain WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 692 Tarek 333,692 29 Pedro 333,198,29 123 Adil 333,692,123 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72
  31. 31. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Hierarchy Traversal 31 List descendants WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 29 Pedro 333,198,29 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72 692 Tarek 333,692 123 Adil 333,692,123
  32. 32. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • WITH RECURSIVE cte_name [list of column names ] AS ( SELECT ... <-- specifies initial set UNION [DISTINCT|ALL] SELECT ... <-- specifies initial set UNION [DISTINCT|ALL] ... SELECT ... <-- specifies how to derive new rows UNION [DISTINCT|ALL] SELECT ... <-- specifies how to derive new rows ... ) [, any number of other CTE definitions ] 32 Complete syntax
  33. 33. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Recursive CTE • A recursive SELECT must not contain – GROUP BY – Aggregate functions (like SUM) – ORDER BY – LIMIT – DISTINCT • A recursive SELECT must reference the CTE only once and only in its FROM clause, not in any subquery • A recursive SELECT can not be the inner table of an outer join Confidential – Oracle Internal/Restricted/Highly Restricted 33 Restrictions
  34. 34. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Want to try it out? • Goto labs.mysql.com • Select MySQL Server 8.0.0 Optimizer • Available as – Linux binaries – Source code • Warning! – For testing purposes only! – NOT FIT FOR PRODUCTION. 34
  35. 35. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Want to learn more? • MySQL Server Team blog: http://mysqlserverteam.com/ – http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in- mysql-ctes/ – http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in- mysql-ctes-part-two-how-to-generate-series/ – http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in- mysql-ctes-part-three-hierarchies/ • My blog: http://oysteing.blogspot.com/ • MySQL Optimizer & Parser forum: http://forums.mysql.com/list.php?115
  36. 36. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 36

×