SlideShare a Scribd company logo
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Common Table Expressions (CTE) &
Window Functions in MySQL 8.0
Øystein Grøvlen
Senior Principal Software Engineer
MySQL Optimizer Team, Oracle
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Common table expression
Window functions
Program Agenda
3
1
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Common table expression
Window functions
Program Agenda
4
1
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Common Table Expression
• A derived table is a subquery in the FROM clause
SELECT … FROM (subquery) AS derived, t1 ...
• Common Table Expression (CTE) is just like a derived table, but its
declaration is put before the query block instead of in FROM clause
WITH derived AS (subquery)
SELECT … FROM derived, t1 ...
• A CTE may precede SELECT/UPDATE/DELETE including sub-queries
WITH derived AS (subquery)
DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived);
5
Alternative to derived table
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Common Table Expression (CTE)
WITH cte_name [( <list of column names> )] AS (
SELECT ... # Definition
)
[, <any number of other CTE definitions> ]
<SELECT/UPDATE/DELETE statement>
6
WITH qn AS (SELECT a FROM t1)
SELECT * from qn;
INSERT INTO t2
WITH qn AS (SELECT 10*a AS a FROM t1)
SELECT * from qn;
SELECT * FROM t1 WHERE t1.a IN
(WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1)
SELECT a + 0 FROM cte);
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Common Table Expression versus Derived Table
Better readability
Can be referenced multiple times
Can refer to other CTEs
Improved performance
7
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Better readability
• Derived table:
SELECT …
FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON …
• CTE:
WITH dt AS (SELECT ... FROM ...)
SELECT ...
FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ...
8
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Can be referenced multiple times
• Derived table can not be referenced twice:
SELECT ...
FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1
JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a;
• CTE can:
WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b)
SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a;
9
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Can refer to other CTEs
• Derived tables can not refer to other derived tables:
SELECT …
FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 …
ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist
• CTEs can refer other CTEs:
WITH d1 AS (SELECT … FROM …),
d2 AS (SELECT … FROM d1 …)
SELECT
FROM d1, d2 …
10
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Chained CTEs
WITH cte1(txt) AS (SELECT "This "),
cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1),
cte3(txt) AS (SELECT "nice query" UNION
SELECT "query that rocks" UNION
SELECT "query"),
cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3)
SELECT MAX(txt), MIN(txt) FROM cte4;
+----------------------------+----------------------+
| MAX(txt) | MIN(txt) |
+----------------------------+----------------------+
| This is a query that rocks | This is a nice query |
+----------------------------+----------------------+
1 row in set (0,00 sec)
11
Neat, but not very useful example
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Better performance
• Derived table:
– For derived tables that are materialized, two identical derived tables will be
materialized. Performance problem (more space, more time, longer locks)
– Similar with view references
• CTE:
– Will be materialized once, regardless of how many references
12
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
DBT3 Query 15 Top Supplier Query
Using view
CREATE VIEW revenue0 (supplier_no, total_revenue)
AS SELECT l_suppkey,
SUM(l_extendedprice * (1-l_discount))
FROM lineitem
WHERE l_shipdate >= '1996-07-01'
AND l_shipdate < DATE_ADD('1996-07-01‘,
INTERVAL '90' day)
GROUP BY l_suppkey;
Using CTE
WITH revenue0 (supplier_no, total_revenue)
AS (SELECT l_suppkey, SUM(l_extendedprice *
(1-l_discount))
FROM lineitem
WHERE l_shipdate >= '1996-07-01'
AND l_shipdate < DATE_ADD('1996-07-01‘,
INTERVAL '90' day)
GROUP BY l_suppkey)
SELECT s_suppkey, s_name, s_address, s_phone, total_revenue
FROM supplier, revenue0
WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0)
ORDER BY s_suppkey;
rewrite
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
DBT-3 Query 15
0
2
4
6
8
10
12
14
16
18
View CTE
QueryExecutionTime(seconds)
Confidential – Oracle Internal/Restricted/Highly Restricted 14
Query Performance
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Recursive CTE
• A recursive CTE refers to itself in a subquery
• The “seed” SELECT is executed once to create the initial data subset, the
recursive SELECT is repeatedly executed to return subsets of data until the
complete result set is obtained.
• Recursion stops when an iteration does not generate any new rows
• Useful to dig in hierarchies (parent/child, part/subpart)
15
WITH RECURSIVE cte AS
( SELECT ... FROM table_name /* "seed" SELECT */
UNION [DISTINCT|ALL]
SELECT ... FROM cte, table_name) /* "recursive" SELECT */
SELECT ... FROM cte;
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Recursive CTE
16
A simple example
Print 1 to 10 :
WITH RECURSIVE qn AS
( SELECT 1 AS a
UNION ALL
SELECT 1+a FROM qn WHERE a<10
)
SELECT * FROM qn;
a
1
2
3
4
5
6
7
8
9
10
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Recursive CTE
17
INSERT
Insert 1 to 10 :
INSERT INTO numbers
WITH RECURSIVE qn AS
( SELECT 1 AS a
UNION ALL
SELECT 1+a FROM qn WHERE a<10
)
SELECT * FROM qn;
SELECT * FROM numbers;
a
1
2
3
4
5
6
7
8
9
10
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Date sequence
18
Missing dates
SELECT orderdate,
SUM(totalprice) sales
FROM orders
GROUP BY orderdate
ORDER BY orderdate;
+------------+-----------+
| orderdate | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-07 | 185991.79 |
+------------+-----------+
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Date sequence
19
All dates
WITH RECURSIVE dates(date) AS
( SELECT '2016-09-01'
UNION ALL
SELECT DATE_ADD(date, INTERVAL 1 DAY)
FROM dates
WHERE date < '2016-09-07‘ )
SELECT dates.date,
COALESCE(SUM(totalprice), 0) sales
FROM dates LEFT JOIN orders
ON dates.date = orders.orderdate
GROUP BY dates.date
ORDER BY dates.date;
+------------+-----------+
| date | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-02 | 0.00 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-06 | 0.00 |
| 2016-09-07 | 185991.79 |
+------------+-----------+
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Hierarchy Traversal
20
Employee database
CREATE TABLE employees (
id INT PRIMARY KEY,
name VARCHAR(100),
manager_id INT,
FOREIGN KEY (manager_id)
REFERENCES employees(id) );
INSERT INTO employees VALUES
(333, "Yasmina", NULL), # CEO
(198, "John", 333), # John reports to 333
(692, "Tarek", 333),
(29, "Pedro", 198),
(4610, "Sarah", 29),
(72, "Pierre", 29),
(123, "Adil", 692);
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Hierarchy Traversal
21
List reporting chain
WITH RECURSIVE
emp_ext (id, name, path) AS (
SELECT id, name, CAST(id AS CHAR(200))
FROM employees
WHERE manager_id IS NULL
UNION ALL
SELECT s.id, s.name,
CONCAT(m.path, ",", s.id)
FROM emp_ext m JOIN employees s
ON m.id=s.manager_id )
SELECT * FROM emp_ext ORDER BY path;
id name path
333 Yasmina 333
198 John 333,198
692 Tarek 333,692
29 Pedro 333,198,29
123 Adil 333,692,123
4610 Sarah 333,198,29,4610
72 Pierre 333,198,29,72
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Hierarchy Traversal
22
List reporting chain
WITH RECURSIVE
emp_ext (id, name, path) AS (
SELECT id, name, CAST(id AS CHAR(200))
FROM employees
WHERE manager_id IS NULL
UNION ALL
SELECT s.id, s.name,
CONCAT(m.path, ",", s.id)
FROM emp_ext m JOIN employees s
ON m.id=s.manager_id )
SELECT * FROM emp_ext ORDER BY path;
id name path
333 Yasmina 333
198 John 333,198
29 Pedro 333,198,29
4610 Sarah 333,198,29,4610
72 Pierre 333,198,29,72
692 Tarek 333,692
123 Adil 333,692,123
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Program Agenda
23
1
2
Non recursive common table expression
Window functions
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Window functions: what are they?
• A window function performs a calculation across a set of rows that are
related to the current row, similar to an aggregate function.
• But unlike aggregate functions, a window function does not cause rows to
become grouped into a single output row.
• Window functions can access values of other rows “in the vicinity” of the
current row
24
Aggreation function Window function
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Window function example
25
PARTITION == disjoint
set of rows in result set
name dept_id salary dept_total
Newt NULL 75000 75000
Dag 10 NULL 370000
Ed 10 100000 370000
Fred 10 60000 370000
Jon 10 60000 370000
Michael 10 70000 370000
Newt 10 80000 370000
Lebedev 20 65000 130000
Pete 20 65000 130000
Jeff 30 300000 370000
Will 30 70000 370000
Sum up total salary for each department:
SELECT name, dept_id, salary,
SUM(salary) OVER (PARTITION BY
dept_id) AS dept_total
FROM employee
ORDER BY dept_id, name;
The OVER keyword
signals a window function
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
With GROUP BY
26
SELECT name, dept_id, salary,
SUM(salary) AS dept_total
FROM employee GROUP BY dept_id
ORDER BY dept_id, name;
ERROR 1055 (42000): Expression #1 of SELECT list is not in GROUP BY clause and
contains nonaggregated column 'mysql.employee.name' which is not functionally dependent
on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
With GROUP BY
27
SELECT /* name, */ dept_id, /* salary,*/
SUM(salary) AS dept_total
FROM employee GROUP BY dept_id
ORDER BY dept_id /*, name */;
dept_id dept_total
NULL 75000
10 370000
20 130000
30 370000
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
name dept_id salary total
Newt NULL 75000 75000
Dag 10 NULL NULL
Ed 10 100000 100000
Fred 10 60000 160000
Jon 10 60000 220000
Michael 10 70000 190000
Newt 10 80000 210000
Lebedev 20 65000 65000
Pete 20 65000 130000
Jeff 30 300000 300000
Will 30 70000 370000
28
Window function example, with frame
ORDER BY
name within
each partition
moving window frame:
SUM(salary) ...
ROWS 2 PRECEDING
a frame is a subset of a
partition
SELECT name, dept_id, salary,
SUM(salary)
OVER (PARTITION BY dept_id
ORDER BY name
ROWS 2 PRECEDING) total
FROM employee
ORDER BY dept_id, name;
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 29
Window function example, with frame
SELECT name, dept_id, salary,
SUM(salary)
OVER (PARTITION BY dept_id
ORDER BY name
ROWS 2 PRECEDING) total
FROM employee
ORDER BY dept_id, name;
name dept_id salary total
Newt NULL 75000 75000
Dag 10 NULL NULL
Ed 10 100000 100000
Fred 10 60000 160000
Jon 10 60000 220000
Michael 10 70000 190000
Newt 10 80000 210000
Lebedev 20 65000 65000
Pete 20 65000 130000
Jeff 30 300000 300000
Will 30 70000 370000
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
SELECT name, dept_id, salary,
AVG(salary) OVER w AS `avg`,
salary - AVG(salary) OVER w AS diff
FROM employee
WINDOW w AS (PARTITION BY dept_id)
ORDER BY diff DESC;
Window function example
30
name dept_id salary average diff
Jeff 30 300000 185000 115000
Ed 10 100000 74000 26000
Newt 10 80000 74000 6000
Newt NULL 75000 75000 0
Pete 20 65000 65000 0
Lebedev 20 65000 65000 0
Michael 10 70000 74000 -4000
Jon 10 60000 74000 -14000
Fred 10 60000 74000 -14000
Will 30 70000 185000 -115000
Dag 10 NULL 74000 NULL
• i.e. find the employees with the largest
difference between their wage and that of
the department average
• Note: explicit window definition of “w”
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Implicit and explicit windows
• Windows can be implicit and unnamed:
COUNT(*) OVER (PARTITION BY dept_ID)
• Windows can be defined and named via the windows clause:
SELECT COUNT(*) OVER w FROM t
WINDOW w as (PARTITION BY dept_id)
• Allows sharing of windows between several window functions
• Avoids redundant windowing steps since more functions can be evaluated
in the same step
31
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Types of window functions
• Aggregates
– COUNT, SUM, AVG, MAX, MIN + more to come
• Ranking
– RANK, DENSE_RANK, PERCENT_RANK,
– CUME_DIST, ROW_NUMBER
• Analytical
– NTILE, LEAD, LAG
– NTH, FIRST_VALUE, LAST_VALUE
Blue ones use frames, all obey partitions
32
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 33
Syntax for window specification
window specification ::=
[ existing window name ]
[PARTITION BY expr-1, ... ]
[ORDER BY expr-1, ... [DESC] ]
[ frame clause ]
frame clause ::= { ROWS | RANGE } { start | between }
start ::= { CURRENT ROW | UNBOUNDED PRECEDING | n PRECEDING}
between ::= BETWEEN bound-1 AND bound-2
bound ::= start | UNBOUNDED FOLLOWING | n FOLLOWING
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 34
Frame clause bound
partition
CURRENT ROW
UNBOUNDED PRECEDING
UNBOUNDED FOLLOWING
n PRECEDING
m FOLLOWING
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 35
RANGE frame example
SELECT date, amount,
SUM(amount) OVER w AS `sum` FROM payments
WINDOW w AS
(ORDER BY date
RANGE BETWEEN INTERVAL 1 WEEK PRECEDING
AND CURRENT ROW)
ORDER BY date;
Current row's date is the 10th, so first row in range is the 3rd .
Frame cardinality is 4 due to peer in next row.
For Jan 5, the frame cardinality is 5, and sum is 900.50.
date amount sum
2017-01-01 100.50 300.50
2017-01-01 200.00 300.50
2017-01-02 200.00 500.50
2017-01-03 200.00 700.50
2017-01-05 200.00 900.50
2017-01-10 200.00 700.00
2017-01-10 100.00 700.00
2017-01-11 200.00 700.00
Find the sum of payments within the last 8 days
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
When are they evaluated?
• After GROUP BY/ HAVING
• Before final ORDER BY, DISTINCT, LIMIT
• You can have several window functions and several different windows
• To filter on window function’s value, use a subquery, e.g.
36
SELECT *
FROM (
SELECT SUM(salary) OVER (PARTITION BY dept_id) `sum`
FROM employee
) AS s
WHERE `sum` < 100000;
sum
75000
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
• Tmp table between each windowing step
• (in-mem if result set can fit †)
• Streamable wfs vs buffered
• Depends on wf and frame
• Buffered: re-read rows
• O(rows * frame size)
• Move frame for SUM 1 row
• Optimization: Invert by subtraction,
add new row.
37
Logical flow
JOIN
GROUP
BY
WINDOW
1
WINDOW
n
ORDER BY/
DISTINCT/
LIMIT
Input goes into a tmp table
Sort for
PARTITION BY
and
ORDER BY
† cf. variables tmp_table_size, max_heap_table_size
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Streamable evaluation
38
SELECT name, dept_id, salary,
SUM(salary) OVER (PARTITION BY
dept_id ORDER BY name
ROWS BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW) AS `sum`
FROM employee;
name dept_id salary sum
Newt NULL 75000 75000
Dag 10 NULL NULL
Ed 10 100000 100000
Fred 10 60000 160000
Jon 10 60000 220000
Michael 10 70000 290000
Newt 10 80000 370000
Lebedev 20 65000 65000
Pete 20 65000 130000
Jeff 30 300000 300000
Will 30 70000 370000
Just accumulate as we see rows
Accumulate the salary in each department as sum
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 39
Non-streamable evaluation
SELECT name, dept_id, salary,
SUM(salary) OVER (PARTITION BY
dept_id ORDER BY name
ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW) AS `sum`
FROM employee;
name dept_id salary sum
Newt NULL 75000 75000
Dag 10 NULL NULL
Ed 10 100000 100000
Fred 10 60000 160000
Jon 10 60000 220000
Michael 10 70000 190000
Newt 10 80000 210000
Lebedev 20 65000 65000
Pete 20 65000 130000
Jeff 30 300000 300000
Will 30 70000 370000
Sum two preceding rows and the current row
When evaluating Michael, subtract Ed's
contribution, add Michael
or just evaluate entire frame over again
(non-optimized). In both cases we need
re-visit rows.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Explain, last query
40
EXPLAIN FORMAT=JSON
SELECT name, dept_id, salary,
SUM(salary) OVER (PARTITION BY
dept_id ORDER BY name
ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW) AS `sum`
FROM employee;
:
"windows": [
{
"name": "<unnamed window>",
"using_filesort": true,
"frame_buffer": {
"using_temporary_table": true,
"optimized_frame_evaluation": true
},
"functions": [ "sum" ]
}
],
:
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
RANK
41
SELECT name, dept_id AS dept, salary,
RANK() OVER w AS `rank`
FROM employee
WINDOW w AS (PARTITION BY dept_id
ORDER BY salary DESC);
name dept_id salary rank
Newt NULL 75000 1
Ed 10 100000 1
Newt 10 80000 2
Fred 10 70000 3
Michael 10 70000 3
Jon 10 60000 5
Dag 10 NULL 6
Pete 20 65000 1
Lebedev 20 65000 1
Jeff 30 300000 1
Will 30 70000 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
DENSE_RANK
42
SELECT name, dept_id AS dept, salary,
RANK() OVER w AS `rank`,
DENSE_RANK() OVER w AS dense
FROM employee
WINDOW w AS (PARTITION BY dept_id
ORDER BY salary DESC);
name dept_id salary rank dense
Newt NULL 75000 1 1
Ed 10 100000 1 1
Newt 10 80000 2 2
Fred 10 70000 3 3
Michael 10 70000 3 3
Jon 10 60000 5 4
Dag 10 NULL 6 5
Pete 20 65000 1 1
Lebedev 20 65000 1 1
Jeff 30 300000 1 1
Will 30 70000 2 2
DENSE_RANK doesn't skip
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
ROW_NUMBER
43
SELECT name, dept_id AS dept, salary,
RANK() OVER w AS `rank`,
DENSE_RANK() OVER w AS dense,
ROW_NUMBER() OVER w AS `rowno`
FROM employee
WINDOW w AS (PARTITION BY dept_id
ORDER BY salary DESC);
name dept_id salary rank dense rowno
Newt NULL 75000 1 1 1
Ed 10 100000 1 1 1
Newt 10 80000 2 2 2
Fred 10 70000 3 3 3
Michael 10 70000 3 3 4
Jon 10 60000 5 4 5
Dag 10 NULL 6 5 6
Pete 20 65000 1 1 1
Lebedev 20 65000 1 1 2
Jeff 30 300000 1 1 1
Will 30 70000 2 2 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Implicit and explicit windows
A window definition can inherit from another
window definition in its specification, adding
detail, no override
44
SELECT name, dept_id,
COUNT(*) OVER w1 AS cnt1,
COUNT(*) OVER w2 AS cnt2
FROM employee
WINDOW w1 AS (PARTITION BY dept_id),
w2 AS (w1 ORDER BY name)
ORDER BY dept_id, name;
name dept_id cnt1 cnt2
Newt NULL 1 1
Dag 10 6 1
Ed 10 6 2
Fred 10 6 3
Jon 10 6 4
Michael 10 6 5
Newt 10 6 6
Levedev 20 2 1
Pete 20 2 2
Jeff 30 2 1
Will 30 2 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Want to learn more?
• MySQL Server Team blog: http://mysqlserverteam.com/
• MySQL Optimizer & Parser forum: http://forums.mysql.com/list.php?115
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Want to learn more?
• Thursday 12:50pm: Recursive Query Throwdown in MySQL 8 (Bill Karwin)
• MySQL Server Team blog
– http://mysqlserverteam.com/
• My blog:
– http://oysteing.blogspot.com/
• MySQL forums:
– Optimizer & Parser: http://forums.mysql.com/list.php?115
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 47
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
LEAD, LAG
48
Returns value evaluated at the row that is offset rows after/before the current row within
the partition; if there is no such row, instead return default (which must be of the same
type as value).
Both offset and default are evaluated with respect to the current row. If omitted, offset
defaults to 1 and default to null
lead or lag function ::= { LEAD | LAG } ( expr [ , offset [ , default expression>] ] )
[ RESPECT NULLS ]
Note: “IGNORE NULLS” not supported, RESPECT NULLS is default but can be specified.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
FIRST_VALUE, LAST_VALUE, NTH_VALUE
49
Returns value evaluated at the first, last, nth in the frame of the current row within the
partition; if there is no nth row (frame is too small), the NTH_VALUE returns NULL.
first or last value ::= { FIRST_VALUE | LAST_VALUE } ( expr )
[ RESPECT NULLS ]
nth_value ::= NTH_VALUE ( expr, nth-row ) [FROM FIRST]
[ RESPECT NULLS ]
Note: “IGNORE NULLS” is not supported, RESPECT NULLS is used but can be specified.
Note: For NTH_VALUE, “FROM LAST” is not supported, FROM FIRST is used but can be
specified
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
FIRST_VALUE “in frame”
50
SELECT name, dept_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
FIRST_VALUE(salary) OVER w AS `first`
FROM employee
WINDOW w AS (PARTITION BY dept_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW)
name dept_id salary sum first
Newt NULL 75000 75000 75000
Dag 10 NULL NULL NULL
Ed 10 100000 100000 NULL
Fred 10 60000 160000 NULL
Jon 10 60000 220000 100000
Michael 10 70000 190000 60000
Newt 10 80000 210000 60000
Lebedev 20 65000 65000 65000
Pete 20 65000 130000 65000
Jeff 30 300000 300000 30000
Will 30 70000 370000 30000
Current row: Jon
FIRST_VALUE in frame is: Ed
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
LAST_VALUE “in frame”
51
SELECT name, dept_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
FIRST_VALUE(salary) OVER w AS `first`,
LAST_VALUE(salary) OVER w AS `last`
FROM employee
WINDOW w AS (
PARTITION BY dept_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW)
name dept_id salary sum first last
Newt NULL 75000 75000 75000 75000
Dag 10 NULL NULL NULL NULL
Ed 10 100000 100000 NULL 100000
Fred 10 60000 160000 NULL 60000
Jon 10 60000 220000 100000 60000
Michael 10 70000 190000 60000 70000
Newt 10 80000 210000 60000 80000
Lebedev 20 65000 65000 65000 65000
Pete 20 65000 130000 65000 65000
Jeff 30 300000 300000 30000 300000
Will 30 70000 370000 30000 70000
Current row: Jon
LAST_VALUE in frame is: Jon
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
NTH_VALUE “in frame”
52
SELECT name, dept_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
NTH_VALUE(salary, 2) OVER w AS `nth`
FROM employee
WINDOW w AS (PARTITION BY dept_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW)
name dept_id salary sum nth
Newt NULL 75000 75000 NULL
Dag 10 NULL NULL NULL
Ed 10 100000 100000 100000
Fred 10 60000 160000 100000
Jon 10 60000 220000 60000
Michael 10 70000 190000 60000
Newt 10 80000 210000 70000
Lebedev 20 65000 65000 NULL
Pete 20 65000 130000 65000
Jeff 30 300000 300000 NULL
Will 30 70000 370000 70000
Current row: Jon
NTH_VALUE(...,2) in frame is: Fred
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 53
Logical flow
JOIN
GROUP
BY
WINDOW
1
WINDOW
n
ORDER BY/
DISTINCT/
LIMIT
Row
addressable
buffer
in-mem:
overflows to disk
Permits re-reading rows when frame moves

More Related Content

What's hot

Mysql index
Mysql indexMysql index
Mysql indexYuan Yao
 
MySQL Multi-Source Replication for PL2016
MySQL Multi-Source Replication for PL2016MySQL Multi-Source Replication for PL2016
MySQL Multi-Source Replication for PL2016Wagner Bianchi
 
MySQL User Group NL - MySQL 8
MySQL User Group NL - MySQL 8MySQL User Group NL - MySQL 8
MySQL User Group NL - MySQL 8
Frederic Descamps
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning
Alexander Rubin
 
MySQL Shell for DBAs
MySQL Shell for DBAsMySQL Shell for DBAs
MySQL Shell for DBAs
Frederic Descamps
 
MySQL InnoDB Cluster and Group Replication in a Nutshell: hands-on tutorial
MySQL InnoDB Cluster and Group Replication in a Nutshell:  hands-on tutorialMySQL InnoDB Cluster and Group Replication in a Nutshell:  hands-on tutorial
MySQL InnoDB Cluster and Group Replication in a Nutshell: hands-on tutorial
Frederic Descamps
 
MySQL Router REST API
MySQL Router REST APIMySQL Router REST API
MySQL Router REST API
Frederic Descamps
 
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
Frederic Descamps
 
Upgrade from MySQL 5.7 to MySQL 8.0
Upgrade from MySQL 5.7 to MySQL 8.0Upgrade from MySQL 5.7 to MySQL 8.0
Upgrade from MySQL 5.7 to MySQL 8.0
Olivier DASINI
 
MySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & OperationsMySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & Operations
Frederic Descamps
 
DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic e...
DataOpsbarcelona 2019:  Deep dive into MySQL Group Replication... the magic e...DataOpsbarcelona 2019:  Deep dive into MySQL Group Replication... the magic e...
DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic e...
Frederic Descamps
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova
 
Open Source 101 2022 - MySQL Indexes and Histograms
Open Source 101 2022 - MySQL Indexes and HistogramsOpen Source 101 2022 - MySQL Indexes and Histograms
Open Source 101 2022 - MySQL Indexes and Histograms
Frederic Descamps
 
MySQL Group Replication: Handling Network Glitches - Best Practices
MySQL Group Replication: Handling Network Glitches - Best PracticesMySQL Group Replication: Handling Network Glitches - Best Practices
MySQL Group Replication: Handling Network Glitches - Best Practices
Frederic Descamps
 
10 Creating Triggers
10 Creating Triggers10 Creating Triggers
10 Creating Triggers
rehaniltifat
 
Less06 networking
Less06 networkingLess06 networking
Less06 networking
Amit Bhalla
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index TuningManikanda kumar
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTID
Mydbops
 
Indexes in postgres
Indexes in postgresIndexes in postgres
Indexes in postgres
Louise Grandjonc
 
MySQL SYSスキーマのご紹介
MySQL SYSスキーマのご紹介MySQL SYSスキーマのご紹介
MySQL SYSスキーマのご紹介
Shinya Sugiyama
 

What's hot (20)

Mysql index
Mysql indexMysql index
Mysql index
 
MySQL Multi-Source Replication for PL2016
MySQL Multi-Source Replication for PL2016MySQL Multi-Source Replication for PL2016
MySQL Multi-Source Replication for PL2016
 
MySQL User Group NL - MySQL 8
MySQL User Group NL - MySQL 8MySQL User Group NL - MySQL 8
MySQL User Group NL - MySQL 8
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning
 
MySQL Shell for DBAs
MySQL Shell for DBAsMySQL Shell for DBAs
MySQL Shell for DBAs
 
MySQL InnoDB Cluster and Group Replication in a Nutshell: hands-on tutorial
MySQL InnoDB Cluster and Group Replication in a Nutshell:  hands-on tutorialMySQL InnoDB Cluster and Group Replication in a Nutshell:  hands-on tutorial
MySQL InnoDB Cluster and Group Replication in a Nutshell: hands-on tutorial
 
MySQL Router REST API
MySQL Router REST APIMySQL Router REST API
MySQL Router REST API
 
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
Another MySQL HA Solution for ProxySQL Users, Easy and All Integrated: MySQL ...
 
Upgrade from MySQL 5.7 to MySQL 8.0
Upgrade from MySQL 5.7 to MySQL 8.0Upgrade from MySQL 5.7 to MySQL 8.0
Upgrade from MySQL 5.7 to MySQL 8.0
 
MySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & OperationsMySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & Operations
 
DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic e...
DataOpsbarcelona 2019:  Deep dive into MySQL Group Replication... the magic e...DataOpsbarcelona 2019:  Deep dive into MySQL Group Replication... the magic e...
DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic e...
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
 
Open Source 101 2022 - MySQL Indexes and Histograms
Open Source 101 2022 - MySQL Indexes and HistogramsOpen Source 101 2022 - MySQL Indexes and Histograms
Open Source 101 2022 - MySQL Indexes and Histograms
 
MySQL Group Replication: Handling Network Glitches - Best Practices
MySQL Group Replication: Handling Network Glitches - Best PracticesMySQL Group Replication: Handling Network Glitches - Best Practices
MySQL Group Replication: Handling Network Glitches - Best Practices
 
10 Creating Triggers
10 Creating Triggers10 Creating Triggers
10 Creating Triggers
 
Less06 networking
Less06 networkingLess06 networking
Less06 networking
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index Tuning
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTID
 
Indexes in postgres
Indexes in postgresIndexes in postgres
Indexes in postgres
 
MySQL SYSスキーマのご紹介
MySQL SYSスキーマのご紹介MySQL SYSスキーマのご紹介
MySQL SYSスキーマのご紹介
 

Similar to Common Table Expressions (CTE) & Window Functions in MySQL 8.0

MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0
oysteing
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table ExpressionsMySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
oysteing
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
oysteing
 
Ctes percona live_2017
Ctes percona live_2017Ctes percona live_2017
Ctes percona live_2017
Guilhem Bichot
 
MySQL Optimizer: What's New in 8.0
MySQL Optimizer: What's New in 8.0MySQL Optimizer: What's New in 8.0
MySQL Optimizer: What's New in 8.0
Manyi Lu
 
More SQL in MySQL 8.0
More SQL in MySQL 8.0More SQL in MySQL 8.0
More SQL in MySQL 8.0
Norvald Ryeng
 
MySQL Optimizer Overview
MySQL Optimizer OverviewMySQL Optimizer Overview
MySQL Optimizer Overview
Olav Sandstå
 
Melbourne Groundbreakers Tour - Upgrading without risk
Melbourne Groundbreakers Tour - Upgrading without riskMelbourne Groundbreakers Tour - Upgrading without risk
Melbourne Groundbreakers Tour - Upgrading without risk
Connor McDonald
 
Sangam 18 - The New Optimizer in Oracle 12c
Sangam 18 - The New Optimizer in Oracle 12cSangam 18 - The New Optimizer in Oracle 12c
Sangam 18 - The New Optimizer in Oracle 12c
Connor McDonald
 
20180420 hk-the powerofmysql8
20180420 hk-the powerofmysql820180420 hk-the powerofmysql8
20180420 hk-the powerofmysql8
Ivan Ma
 
18(ish) Things You'll Love About Oracle Database 18c
18(ish) Things You'll Love About Oracle Database 18c18(ish) Things You'll Love About Oracle Database 18c
18(ish) Things You'll Love About Oracle Database 18c
Chris Saxon
 
Developers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman OracleDevelopers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman Oracle
mCloud
 
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
mCloud
 
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c OptimizerWellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Connor McDonald
 
MySQL Optimizer Overview
MySQL Optimizer OverviewMySQL Optimizer Overview
MySQL Optimizer Overview
Olav Sandstå
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
Altinity Ltd
 
Partition and conquer large data in PostgreSQL 10
Partition and conquer large data in PostgreSQL 10Partition and conquer large data in PostgreSQL 10
Partition and conquer large data in PostgreSQL 10
Ashutosh Bapat
 
CS4200 2019 | Lecture 5 | Transformation by Term Rewriting
CS4200 2019 | Lecture 5 | Transformation by Term RewritingCS4200 2019 | Lecture 5 | Transformation by Term Rewriting
CS4200 2019 | Lecture 5 | Transformation by Term Rewriting
Eelco Visser
 
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Altinity Ltd
 
Pumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency AnalysisPumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency Analysis
University of Illinois,Chicago
 

Similar to Common Table Expressions (CTE) & Window Functions in MySQL 8.0 (20)

MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table ExpressionsMySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
 
MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
 
Ctes percona live_2017
Ctes percona live_2017Ctes percona live_2017
Ctes percona live_2017
 
MySQL Optimizer: What's New in 8.0
MySQL Optimizer: What's New in 8.0MySQL Optimizer: What's New in 8.0
MySQL Optimizer: What's New in 8.0
 
More SQL in MySQL 8.0
More SQL in MySQL 8.0More SQL in MySQL 8.0
More SQL in MySQL 8.0
 
MySQL Optimizer Overview
MySQL Optimizer OverviewMySQL Optimizer Overview
MySQL Optimizer Overview
 
Melbourne Groundbreakers Tour - Upgrading without risk
Melbourne Groundbreakers Tour - Upgrading without riskMelbourne Groundbreakers Tour - Upgrading without risk
Melbourne Groundbreakers Tour - Upgrading without risk
 
Sangam 18 - The New Optimizer in Oracle 12c
Sangam 18 - The New Optimizer in Oracle 12cSangam 18 - The New Optimizer in Oracle 12c
Sangam 18 - The New Optimizer in Oracle 12c
 
20180420 hk-the powerofmysql8
20180420 hk-the powerofmysql820180420 hk-the powerofmysql8
20180420 hk-the powerofmysql8
 
18(ish) Things You'll Love About Oracle Database 18c
18(ish) Things You'll Love About Oracle Database 18c18(ish) Things You'll Love About Oracle Database 18c
18(ish) Things You'll Love About Oracle Database 18c
 
Developers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman OracleDevelopers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman Oracle
 
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
 
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c OptimizerWellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
 
MySQL Optimizer Overview
MySQL Optimizer OverviewMySQL Optimizer Overview
MySQL Optimizer Overview
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
 
Partition and conquer large data in PostgreSQL 10
Partition and conquer large data in PostgreSQL 10Partition and conquer large data in PostgreSQL 10
Partition and conquer large data in PostgreSQL 10
 
CS4200 2019 | Lecture 5 | Transformation by Term Rewriting
CS4200 2019 | Lecture 5 | Transformation by Term RewritingCS4200 2019 | Lecture 5 | Transformation by Term Rewriting
CS4200 2019 | Lecture 5 | Transformation by Term Rewriting
 
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
Webinar slides: MORE secrets of ClickHouse Query Performance. By Robert Hodge...
 
Pumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency AnalysisPumps, Compressors and Turbine Fault Frequency Analysis
Pumps, Compressors and Turbine Fault Frequency Analysis
 

More from oysteing

POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
oysteing
 
The MySQL Query Optimizer Explained Through Optimizer Trace
The MySQL Query Optimizer Explained Through Optimizer TraceThe MySQL Query Optimizer Explained Through Optimizer Trace
The MySQL Query Optimizer Explained Through Optimizer Trace
oysteing
 
POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
oysteing
 
POLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel QueryPOLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel Query
oysteing
 
JSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worldsJSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worlds
oysteing
 
Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0
oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
How to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performanceHow to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performance
oysteing
 
Using Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query PerformanceUsing Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query Performance
oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016
oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15
oysteing
 
How to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinarHow to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinar
oysteing
 

More from oysteing (14)

POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
 
The MySQL Query Optimizer Explained Through Optimizer Trace
The MySQL Query Optimizer Explained Through Optimizer TraceThe MySQL Query Optimizer Explained Through Optimizer Trace
The MySQL Query Optimizer Explained Through Optimizer Trace
 
POLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloudPOLARDB: A database architecture for the cloud
POLARDB: A database architecture for the cloud
 
POLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel QueryPOLARDB for MySQL - Parallel Query
POLARDB for MySQL - Parallel Query
 
JSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worldsJSON_TABLE -- The best of both worlds
JSON_TABLE -- The best of both worlds
 
Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0Histogram Support in MySQL 8.0
Histogram Support in MySQL 8.0
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
How to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performanceHow to analyze and tune sql queries for better performance
How to analyze and tune sql queries for better performance
 
Using Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query PerformanceUsing Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query Performance
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016How to analyze and tune sql queries for better performance vts2016
How to analyze and tune sql queries for better performance vts2016
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15
 
How to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinarHow to analyze and tune sql queries for better performance webinar
How to analyze and tune sql queries for better performance webinar
 

Recently uploaded

GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
wottaspaceseo
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
rickgrimesss22
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
AMB-Review
 
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptxText-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
ShamsuddeenMuhammadA
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
XfilesPro
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Globus
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
Shane Coughlan
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
Juraj Vysvader
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Top 7 Unique WhatsApp API Benefits | Saudi Arabia
Top 7 Unique WhatsApp API Benefits | Saudi ArabiaTop 7 Unique WhatsApp API Benefits | Saudi Arabia
Top 7 Unique WhatsApp API Benefits | Saudi Arabia
Yara Milbes
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
Fermin Galan
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
Large Language Models and the End of Programming
Large Language Models and the End of ProgrammingLarge Language Models and the End of Programming
Large Language Models and the End of Programming
Matt Welsh
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Adele Miller
 
Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"
Donna Lenk
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 

Recently uploaded (20)

GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
 
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptxText-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Top 7 Unique WhatsApp API Benefits | Saudi Arabia
Top 7 Unique WhatsApp API Benefits | Saudi ArabiaTop 7 Unique WhatsApp API Benefits | Saudi Arabia
Top 7 Unique WhatsApp API Benefits | Saudi Arabia
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
Large Language Models and the End of Programming
Large Language Models and the End of ProgrammingLarge Language Models and the End of Programming
Large Language Models and the End of Programming
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
 
Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 

Common Table Expressions (CTE) & Window Functions in MySQL 8.0

  • 1. Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Common Table Expressions (CTE) & Window Functions in MySQL 8.0 Øystein Grøvlen Senior Principal Software Engineer MySQL Optimizer Team, Oracle Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
  • 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  • 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Common table expression Window functions Program Agenda 3 1 2
  • 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Common table expression Window functions Program Agenda 4 1 2
  • 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Common Table Expression • A derived table is a subquery in the FROM clause SELECT … FROM (subquery) AS derived, t1 ... • Common Table Expression (CTE) is just like a derived table, but its declaration is put before the query block instead of in FROM clause WITH derived AS (subquery) SELECT … FROM derived, t1 ... • A CTE may precede SELECT/UPDATE/DELETE including sub-queries WITH derived AS (subquery) DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived); 5 Alternative to derived table
  • 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Common Table Expression (CTE) WITH cte_name [( <list of column names> )] AS ( SELECT ... # Definition ) [, <any number of other CTE definitions> ] <SELECT/UPDATE/DELETE statement> 6 WITH qn AS (SELECT a FROM t1) SELECT * from qn; INSERT INTO t2 WITH qn AS (SELECT 10*a AS a FROM t1) SELECT * from qn; SELECT * FROM t1 WHERE t1.a IN (WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1) SELECT a + 0 FROM cte);
  • 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Common Table Expression versus Derived Table Better readability Can be referenced multiple times Can refer to other CTEs Improved performance 7
  • 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Better readability • Derived table: SELECT … FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON … • CTE: WITH dt AS (SELECT ... FROM ...) SELECT ... FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ... 8
  • 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Can be referenced multiple times • Derived table can not be referenced twice: SELECT ... FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1 JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a; • CTE can: WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a; 9
  • 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Can refer to other CTEs • Derived tables can not refer to other derived tables: SELECT … FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 … ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist • CTEs can refer other CTEs: WITH d1 AS (SELECT … FROM …), d2 AS (SELECT … FROM d1 …) SELECT FROM d1, d2 … 10
  • 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Chained CTEs WITH cte1(txt) AS (SELECT "This "), cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1), cte3(txt) AS (SELECT "nice query" UNION SELECT "query that rocks" UNION SELECT "query"), cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3) SELECT MAX(txt), MIN(txt) FROM cte4; +----------------------------+----------------------+ | MAX(txt) | MIN(txt) | +----------------------------+----------------------+ | This is a query that rocks | This is a nice query | +----------------------------+----------------------+ 1 row in set (0,00 sec) 11 Neat, but not very useful example
  • 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Better performance • Derived table: – For derived tables that are materialized, two identical derived tables will be materialized. Performance problem (more space, more time, longer locks) – Similar with view references • CTE: – Will be materialized once, regardless of how many references 12
  • 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. DBT3 Query 15 Top Supplier Query Using view CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey; Using CTE WITH revenue0 (supplier_no, total_revenue) AS (SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey) SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey; rewrite
  • 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. DBT-3 Query 15 0 2 4 6 8 10 12 14 16 18 View CTE QueryExecutionTime(seconds) Confidential – Oracle Internal/Restricted/Highly Restricted 14 Query Performance
  • 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Recursive CTE • A recursive CTE refers to itself in a subquery • The “seed” SELECT is executed once to create the initial data subset, the recursive SELECT is repeatedly executed to return subsets of data until the complete result set is obtained. • Recursion stops when an iteration does not generate any new rows • Useful to dig in hierarchies (parent/child, part/subpart) 15 WITH RECURSIVE cte AS ( SELECT ... FROM table_name /* "seed" SELECT */ UNION [DISTINCT|ALL] SELECT ... FROM cte, table_name) /* "recursive" SELECT */ SELECT ... FROM cte;
  • 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Recursive CTE 16 A simple example Print 1 to 10 : WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; a 1 2 3 4 5 6 7 8 9 10
  • 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Recursive CTE 17 INSERT Insert 1 to 10 : INSERT INTO numbers WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn; SELECT * FROM numbers; a 1 2 3 4 5 6 7 8 9 10
  • 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Date sequence 18 Missing dates SELECT orderdate, SUM(totalprice) sales FROM orders GROUP BY orderdate ORDER BY orderdate; +------------+-----------+ | orderdate | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-07 | 185991.79 | +------------+-----------+
  • 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Date sequence 19 All dates WITH RECURSIVE dates(date) AS ( SELECT '2016-09-01' UNION ALL SELECT DATE_ADD(date, INTERVAL 1 DAY) FROM dates WHERE date < '2016-09-07‘ ) SELECT dates.date, COALESCE(SUM(totalprice), 0) sales FROM dates LEFT JOIN orders ON dates.date = orders.orderdate GROUP BY dates.date ORDER BY dates.date; +------------+-----------+ | date | sales | +------------+-----------+ | 2016-09-01 | 43129.83 | | 2016-09-02 | 0.00 | | 2016-09-03 | 218347.61 | | 2016-09-04 | 142568.40 | | 2016-09-05 | 299244.83 | | 2016-09-06 | 0.00 | | 2016-09-07 | 185991.79 | +------------+-----------+
  • 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Hierarchy Traversal 20 Employee database CREATE TABLE employees ( id INT PRIMARY KEY, name VARCHAR(100), manager_id INT, FOREIGN KEY (manager_id) REFERENCES employees(id) ); INSERT INTO employees VALUES (333, "Yasmina", NULL), # CEO (198, "John", 333), # John reports to 333 (692, "Tarek", 333), (29, "Pedro", 198), (4610, "Sarah", 29), (72, "Pierre", 29), (123, "Adil", 692);
  • 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Hierarchy Traversal 21 List reporting chain WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 692 Tarek 333,692 29 Pedro 333,198,29 123 Adil 333,692,123 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72
  • 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Hierarchy Traversal 22 List reporting chain WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path; id name path 333 Yasmina 333 198 John 333,198 29 Pedro 333,198,29 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72 692 Tarek 333,692 123 Adil 333,692,123
  • 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Program Agenda 23 1 2 Non recursive common table expression Window functions
  • 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Window functions: what are they? • A window function performs a calculation across a set of rows that are related to the current row, similar to an aggregate function. • But unlike aggregate functions, a window function does not cause rows to become grouped into a single output row. • Window functions can access values of other rows “in the vicinity” of the current row 24 Aggreation function Window function
  • 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Window function example 25 PARTITION == disjoint set of rows in result set name dept_id salary dept_total Newt NULL 75000 75000 Dag 10 NULL 370000 Ed 10 100000 370000 Fred 10 60000 370000 Jon 10 60000 370000 Michael 10 70000 370000 Newt 10 80000 370000 Lebedev 20 65000 130000 Pete 20 65000 130000 Jeff 30 300000 370000 Will 30 70000 370000 Sum up total salary for each department: SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id) AS dept_total FROM employee ORDER BY dept_id, name; The OVER keyword signals a window function
  • 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. With GROUP BY 26 SELECT name, dept_id, salary, SUM(salary) AS dept_total FROM employee GROUP BY dept_id ORDER BY dept_id, name; ERROR 1055 (42000): Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'mysql.employee.name' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
  • 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. With GROUP BY 27 SELECT /* name, */ dept_id, /* salary,*/ SUM(salary) AS dept_total FROM employee GROUP BY dept_id ORDER BY dept_id /*, name */; dept_id dept_total NULL 75000 10 370000 20 130000 30 370000
  • 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. name dept_id salary total Newt NULL 75000 75000 Dag 10 NULL NULL Ed 10 100000 100000 Fred 10 60000 160000 Jon 10 60000 220000 Michael 10 70000 190000 Newt 10 80000 210000 Lebedev 20 65000 65000 Pete 20 65000 130000 Jeff 30 300000 300000 Will 30 70000 370000 28 Window function example, with frame ORDER BY name within each partition moving window frame: SUM(salary) ... ROWS 2 PRECEDING a frame is a subset of a partition SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id ORDER BY name ROWS 2 PRECEDING) total FROM employee ORDER BY dept_id, name;
  • 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 29 Window function example, with frame SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id ORDER BY name ROWS 2 PRECEDING) total FROM employee ORDER BY dept_id, name; name dept_id salary total Newt NULL 75000 75000 Dag 10 NULL NULL Ed 10 100000 100000 Fred 10 60000 160000 Jon 10 60000 220000 Michael 10 70000 190000 Newt 10 80000 210000 Lebedev 20 65000 65000 Pete 20 65000 130000 Jeff 30 300000 300000 Will 30 70000 370000
  • 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. SELECT name, dept_id, salary, AVG(salary) OVER w AS `avg`, salary - AVG(salary) OVER w AS diff FROM employee WINDOW w AS (PARTITION BY dept_id) ORDER BY diff DESC; Window function example 30 name dept_id salary average diff Jeff 30 300000 185000 115000 Ed 10 100000 74000 26000 Newt 10 80000 74000 6000 Newt NULL 75000 75000 0 Pete 20 65000 65000 0 Lebedev 20 65000 65000 0 Michael 10 70000 74000 -4000 Jon 10 60000 74000 -14000 Fred 10 60000 74000 -14000 Will 30 70000 185000 -115000 Dag 10 NULL 74000 NULL • i.e. find the employees with the largest difference between their wage and that of the department average • Note: explicit window definition of “w”
  • 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Implicit and explicit windows • Windows can be implicit and unnamed: COUNT(*) OVER (PARTITION BY dept_ID) • Windows can be defined and named via the windows clause: SELECT COUNT(*) OVER w FROM t WINDOW w as (PARTITION BY dept_id) • Allows sharing of windows between several window functions • Avoids redundant windowing steps since more functions can be evaluated in the same step 31
  • 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Types of window functions • Aggregates – COUNT, SUM, AVG, MAX, MIN + more to come • Ranking – RANK, DENSE_RANK, PERCENT_RANK, – CUME_DIST, ROW_NUMBER • Analytical – NTILE, LEAD, LAG – NTH, FIRST_VALUE, LAST_VALUE Blue ones use frames, all obey partitions 32
  • 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 33 Syntax for window specification window specification ::= [ existing window name ] [PARTITION BY expr-1, ... ] [ORDER BY expr-1, ... [DESC] ] [ frame clause ] frame clause ::= { ROWS | RANGE } { start | between } start ::= { CURRENT ROW | UNBOUNDED PRECEDING | n PRECEDING} between ::= BETWEEN bound-1 AND bound-2 bound ::= start | UNBOUNDED FOLLOWING | n FOLLOWING
  • 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 34 Frame clause bound partition CURRENT ROW UNBOUNDED PRECEDING UNBOUNDED FOLLOWING n PRECEDING m FOLLOWING
  • 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 35 RANGE frame example SELECT date, amount, SUM(amount) OVER w AS `sum` FROM payments WINDOW w AS (ORDER BY date RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND CURRENT ROW) ORDER BY date; Current row's date is the 10th, so first row in range is the 3rd . Frame cardinality is 4 due to peer in next row. For Jan 5, the frame cardinality is 5, and sum is 900.50. date amount sum 2017-01-01 100.50 300.50 2017-01-01 200.00 300.50 2017-01-02 200.00 500.50 2017-01-03 200.00 700.50 2017-01-05 200.00 900.50 2017-01-10 200.00 700.00 2017-01-10 100.00 700.00 2017-01-11 200.00 700.00 Find the sum of payments within the last 8 days
  • 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. When are they evaluated? • After GROUP BY/ HAVING • Before final ORDER BY, DISTINCT, LIMIT • You can have several window functions and several different windows • To filter on window function’s value, use a subquery, e.g. 36 SELECT * FROM ( SELECT SUM(salary) OVER (PARTITION BY dept_id) `sum` FROM employee ) AS s WHERE `sum` < 100000; sum 75000
  • 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. • Tmp table between each windowing step • (in-mem if result set can fit †) • Streamable wfs vs buffered • Depends on wf and frame • Buffered: re-read rows • O(rows * frame size) • Move frame for SUM 1 row • Optimization: Invert by subtraction, add new row. 37 Logical flow JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Input goes into a tmp table Sort for PARTITION BY and ORDER BY † cf. variables tmp_table_size, max_heap_table_size
  • 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Streamable evaluation 38 SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `sum` FROM employee; name dept_id salary sum Newt NULL 75000 75000 Dag 10 NULL NULL Ed 10 100000 100000 Fred 10 60000 160000 Jon 10 60000 220000 Michael 10 70000 290000 Newt 10 80000 370000 Lebedev 20 65000 65000 Pete 20 65000 130000 Jeff 30 300000 300000 Will 30 70000 370000 Just accumulate as we see rows Accumulate the salary in each department as sum
  • 39. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 39 Non-streamable evaluation SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum` FROM employee; name dept_id salary sum Newt NULL 75000 75000 Dag 10 NULL NULL Ed 10 100000 100000 Fred 10 60000 160000 Jon 10 60000 220000 Michael 10 70000 190000 Newt 10 80000 210000 Lebedev 20 65000 65000 Pete 20 65000 130000 Jeff 30 300000 300000 Will 30 70000 370000 Sum two preceding rows and the current row When evaluating Michael, subtract Ed's contribution, add Michael or just evaluate entire frame over again (non-optimized). In both cases we need re-visit rows.
  • 40. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Explain, last query 40 EXPLAIN FORMAT=JSON SELECT name, dept_id, salary, SUM(salary) OVER (PARTITION BY dept_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum` FROM employee; : "windows": [ { "name": "<unnamed window>", "using_filesort": true, "frame_buffer": { "using_temporary_table": true, "optimized_frame_evaluation": true }, "functions": [ "sum" ] } ], :
  • 41. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. RANK 41 SELECT name, dept_id AS dept, salary, RANK() OVER w AS `rank` FROM employee WINDOW w AS (PARTITION BY dept_id ORDER BY salary DESC); name dept_id salary rank Newt NULL 75000 1 Ed 10 100000 1 Newt 10 80000 2 Fred 10 70000 3 Michael 10 70000 3 Jon 10 60000 5 Dag 10 NULL 6 Pete 20 65000 1 Lebedev 20 65000 1 Jeff 30 300000 1 Will 30 70000 2
  • 42. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. DENSE_RANK 42 SELECT name, dept_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense FROM employee WINDOW w AS (PARTITION BY dept_id ORDER BY salary DESC); name dept_id salary rank dense Newt NULL 75000 1 1 Ed 10 100000 1 1 Newt 10 80000 2 2 Fred 10 70000 3 3 Michael 10 70000 3 3 Jon 10 60000 5 4 Dag 10 NULL 6 5 Pete 20 65000 1 1 Lebedev 20 65000 1 1 Jeff 30 300000 1 1 Will 30 70000 2 2 DENSE_RANK doesn't skip
  • 43. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. ROW_NUMBER 43 SELECT name, dept_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense, ROW_NUMBER() OVER w AS `rowno` FROM employee WINDOW w AS (PARTITION BY dept_id ORDER BY salary DESC); name dept_id salary rank dense rowno Newt NULL 75000 1 1 1 Ed 10 100000 1 1 1 Newt 10 80000 2 2 2 Fred 10 70000 3 3 3 Michael 10 70000 3 3 4 Jon 10 60000 5 4 5 Dag 10 NULL 6 5 6 Pete 20 65000 1 1 1 Lebedev 20 65000 1 1 2 Jeff 30 300000 1 1 1 Will 30 70000 2 2 2
  • 44. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Implicit and explicit windows A window definition can inherit from another window definition in its specification, adding detail, no override 44 SELECT name, dept_id, COUNT(*) OVER w1 AS cnt1, COUNT(*) OVER w2 AS cnt2 FROM employee WINDOW w1 AS (PARTITION BY dept_id), w2 AS (w1 ORDER BY name) ORDER BY dept_id, name; name dept_id cnt1 cnt2 Newt NULL 1 1 Dag 10 6 1 Ed 10 6 2 Fred 10 6 3 Jon 10 6 4 Michael 10 6 5 Newt 10 6 6 Levedev 20 2 1 Pete 20 2 2 Jeff 30 2 1 Will 30 2 2
  • 45. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Want to learn more? • MySQL Server Team blog: http://mysqlserverteam.com/ • MySQL Optimizer & Parser forum: http://forums.mysql.com/list.php?115
  • 46. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Want to learn more? • Thursday 12:50pm: Recursive Query Throwdown in MySQL 8 (Bill Karwin) • MySQL Server Team blog – http://mysqlserverteam.com/ • My blog: – http://oysteing.blogspot.com/ • MySQL forums: – Optimizer & Parser: http://forums.mysql.com/list.php?115
  • 47. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 47
  • 48. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. LEAD, LAG 48 Returns value evaluated at the row that is offset rows after/before the current row within the partition; if there is no such row, instead return default (which must be of the same type as value). Both offset and default are evaluated with respect to the current row. If omitted, offset defaults to 1 and default to null lead or lag function ::= { LEAD | LAG } ( expr [ , offset [ , default expression>] ] ) [ RESPECT NULLS ] Note: “IGNORE NULLS” not supported, RESPECT NULLS is default but can be specified.
  • 49. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. FIRST_VALUE, LAST_VALUE, NTH_VALUE 49 Returns value evaluated at the first, last, nth in the frame of the current row within the partition; if there is no nth row (frame is too small), the NTH_VALUE returns NULL. first or last value ::= { FIRST_VALUE | LAST_VALUE } ( expr ) [ RESPECT NULLS ] nth_value ::= NTH_VALUE ( expr, nth-row ) [FROM FIRST] [ RESPECT NULLS ] Note: “IGNORE NULLS” is not supported, RESPECT NULLS is used but can be specified. Note: For NTH_VALUE, “FROM LAST” is not supported, FROM FIRST is used but can be specified
  • 50. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. FIRST_VALUE “in frame” 50 SELECT name, dept_id AS dept, salary, SUM(salary) OVER w AS `sum`, FIRST_VALUE(salary) OVER w AS `first` FROM employee WINDOW w AS (PARTITION BY dept_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) name dept_id salary sum first Newt NULL 75000 75000 75000 Dag 10 NULL NULL NULL Ed 10 100000 100000 NULL Fred 10 60000 160000 NULL Jon 10 60000 220000 100000 Michael 10 70000 190000 60000 Newt 10 80000 210000 60000 Lebedev 20 65000 65000 65000 Pete 20 65000 130000 65000 Jeff 30 300000 300000 30000 Will 30 70000 370000 30000 Current row: Jon FIRST_VALUE in frame is: Ed
  • 51. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. LAST_VALUE “in frame” 51 SELECT name, dept_id AS dept, salary, SUM(salary) OVER w AS `sum`, FIRST_VALUE(salary) OVER w AS `first`, LAST_VALUE(salary) OVER w AS `last` FROM employee WINDOW w AS ( PARTITION BY dept_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) name dept_id salary sum first last Newt NULL 75000 75000 75000 75000 Dag 10 NULL NULL NULL NULL Ed 10 100000 100000 NULL 100000 Fred 10 60000 160000 NULL 60000 Jon 10 60000 220000 100000 60000 Michael 10 70000 190000 60000 70000 Newt 10 80000 210000 60000 80000 Lebedev 20 65000 65000 65000 65000 Pete 20 65000 130000 65000 65000 Jeff 30 300000 300000 30000 300000 Will 30 70000 370000 30000 70000 Current row: Jon LAST_VALUE in frame is: Jon
  • 52. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. NTH_VALUE “in frame” 52 SELECT name, dept_id AS dept, salary, SUM(salary) OVER w AS `sum`, NTH_VALUE(salary, 2) OVER w AS `nth` FROM employee WINDOW w AS (PARTITION BY dept_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) name dept_id salary sum nth Newt NULL 75000 75000 NULL Dag 10 NULL NULL NULL Ed 10 100000 100000 100000 Fred 10 60000 160000 100000 Jon 10 60000 220000 60000 Michael 10 70000 190000 60000 Newt 10 80000 210000 70000 Lebedev 20 65000 65000 NULL Pete 20 65000 130000 65000 Jeff 30 300000 300000 NULL Will 30 70000 370000 70000 Current row: Jon NTH_VALUE(...,2) in frame is: Fred
  • 53. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 53 Logical flow JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Row addressable buffer in-mem: overflows to disk Permits re-reading rows when frame moves