Modern SQL in PostgreSQL

285,829 views
293,158 views

Published on

SQL has gone out of fashion lately—partly due to the NoSQL movement, but mostly because SQL is often still used like 20 years ago. As a matter of fact, the SQL standard continued to evolve during the past decades resulting in the current release of 2011. In this session, we will go through the most important additions since the widely known SQL-92, explain how they work and how PostgreSQL supports and extends them. We will cover common table expressions and window functions in detail and have a very short look at the temporal features of SQL:2011 and the related features of PostgreSQL.

Published in: Software, Technology
6 Comments
375 Likes
Statistics
Notes
No Downloads
Views
Total views
285,829
On SlideShare
0
From Embeds
0
Number of Embeds
145,376
Actions
Shares
0
Downloads
1,183
Comments
6
Likes
375
Embeds 0
No embeds

No notes for slide

Modern SQL in PostgreSQL

  1. 1. Still using Windows 3.1? So why stick to SQL-92? @ModernSQL in PostgreSQL @MarkusWinand
  2. 2. SQL:1999
  3. 3. LATERAL
  4. 4. Inline views can't refer to outside the view: SELECT(*( ((FROM(t1( ((JOIN((SELECT(*( ((((((((((FROM(t2( (((((((((WHERE(t2.x(=(t1.x( ((((((()(inline_view( ((((ON((inline_view.x(=(t1.x) LATERAL Before SQL:1999 Invalid
  5. 5. Inline views can't refer to outside the view: SELECT(*( ((FROM(t1( ((JOIN((SELECT(*( ((((((((((FROM(t2( (((((((((WHERE(t2.x(=(t1.x( ((((((()(inline_view( ((((ON((inline_view.x(=(t1.x) LATERAL Before SQL:1999 Belongs there
  6. 6. SQL:99 LATERAL views can: SELECT(*( ((FROM(t1( ((JOIN(LATERAL((SELECT(*( ((((((((((((((((((FROM(t2( (((((((((((((((((WHERE(t2.x(=(t1.x( ((((((((((((((()(inline_view( ((((ON((true) LATERAL Since SQL:1999 Valid due to LATERAL keyword Useless, but still required except
 for CROSS join
  7. 7. But WHY?
  8. 8. Join table functions: SELECT(t1.id,(tf.*( ((FROM(t1( ((JOIN(LATERAL(table_function(t1.id)7tf( 7777ON7(true)7 Note: This is PostgreSQL specific. LATERAL is 
 even optional here.(
 LATERAL and table functions
  9. 9. Apply LIMIT per row from previous table: SELECT(top_products.*( ((FROM(categories(c( ((JOIN(LATERAL((SELECT(*( ((((((((((((((((((FROM(products(p( (((((((((((((((((WHERE(p.cat(=(c.cat( 77777777777777777ORDER7BY7p.rank7DESC7 77777777777777777LIMIT737 ((((((((((((((()(top_products LATERAL and Top-N per Group
  10. 10. Get the 10 most recent news for subscribed topics: SELECT(n.*( ((FROM(news(n( ((JOIN(subscriptions(s( ((((ON((n.topic(=(s.topic)( (WHERE(s.user(=(?( (ORDER(BY(n.created(DESC( (LIMIT(10 LATERAL and Multi-Source Top-N
  11. 11. LATERAL and Multi-Source Top-N Sort/Reduce Join everything Limit((time=236 707(rows=10)( V>(Sort((time=236 707(rows=10)( (((Sort(Method:(topVN(heapsort(Mem:(30kB( (((V>(Hash(Join((time=233 800(rows=905 029)( ((((((V>(Seq(Scan(on(subscriptions(s
 ((((((((((time=369(rows=80)( ((((((V>(Hash((time=104 986(rows=10^7)( (((((((((V>(Seq(Scan(on(news(n
 (((((((((((((time=91 218(rows=10^7)( Planning(time:(0.294(ms( Execution(time:(236 707.261(ms
  12. 12. LATERAL and Multi-Source Top-N Sort/Reduce Join everything Limit((time=236 707(rows=10)( V>(Sort((time=236 707(rows=10)( (((Sort(Method:(topVN(heapsort(Mem:(30kB( (((V>(Hash(Join((time=233 800(rows=905 029)( ((((((V>(Seq(Scan(on(subscriptions(s
 ((((((((((time=369(rows=80)( ((((((V>(Hash((time=104 986(rows=10^7)( (((((((((V>(Seq(Scan(on(news(n
 (((((((((((((time=91 218(rows=10^7)( Planning(time:(0.294(ms( Execution(time:(236 707.261(ms Why producing 900k rows... ...when there are only 80 subscriptions?
  13. 13. LATERAL and Multi-Source Top-N Sort/Reduce Join everything Limit((time=236 707(rows=10)( V>(Sort((time=236 707(rows=10)( (((Sort(Method:(topVN(heapsort(Mem:(30kB( (((V>(Hash(Join((time=233 800(rows=905 029)( ((((((V>(Seq(Scan(on(subscriptions(s
 ((((((((((time=369(rows=80)( ((((((V>(Hash((time=104 986(rows=10^7)( (((((((((V>(Seq(Scan(on(news(n
 (((((((((((((time=91 218(rows=10^7)( Planning(time:(0.294(ms( Execution(time:(236 707.261(ms Only the 10 most recent per subscription, you need.
  14. 14. SELECT(n.*( ((FROM(subscriptions(s( ((JOIN(LATERAL((SELECT(*( ((((((((((((((((((FROM(news(n( (((((((((((((((((WHERE(n.topic(=(s.topic( (((((((((((((((((ORDER7BY7n.created7DESC7 77777777777777777LIMIT710( ((((((((((((((()(top_news(ON((true)( (WHERE(s.user_id(=(?( (ORDER(BY(n.created(DESC( (LIMIT(10 LATERAL and Multi-Source Top-N
  15. 15. Limit((time=2.488(rows=10)( V>(Sort((time=2.487(rows=10)( (((V>(Nested(Loop((time=2.339(rows=800)( ((((((V>(Index(Only(Scan(using(pk(on(s(((( ((((((((((time=0.042(rows=80)( ((((((V>(Limit( ((((((((((time=0.027(rows=10(loops=80)( (((((((((V>(Index(Scan(Backward( ((((((((((((using(news_topic_ts_id(on(n(((( Planning(time:(0.161(ms( Execution(time:(2.519(ms LATERAL and Multi-Source Top-N About 100 000 times faster Limited to 10 times # of subscriptions
  16. 16. LATERAL in an Nutshell LATERAL is the "for each" loop of SQL LATERAL plays well with outer joins( LATERAL is great for Top-N subqueries LATERAL can join table functions (unnest!)
  17. 17. LATERAL Availability (SQL:1999) 1999 2001 2003 2005 2007 2009 2011 2013 9.1 DB2 LUW MySQL 11gR1[0] 12c Oracle 9.3 PostgreSQL 2005[1] SQL Server SQLite [0] Undocumented. Requires setting trace event 22829. [1] LATERAL is not supported as of SQL Server 2014 but [CROSS|OUTER] APPLY can be used for the same effect.
  18. 18. WITH (Common Table Expressions)
  19. 19. WITH Before SQL:99 Nested queries are hard to read: SELECT(…( ((FROM((SELECT(…( ((((((((((FROM(t1( ((((((((((JOIN((SELECT(…(FROM(…( ((((((((((((((()(a(ON((…)( ((((((()(b( ((JOIN((SELECT(…(FROM(…( ((((((()(c(ON((…)
  20. 20. Understand this first WITH Before SQL:99 Nested queries are hard to read: SELECT(…( ((FROM((SELECT(…( ((((((((((FROM(t1( ((((((((((JOIN((SELECT(…(FROM(…( ((((((((((((((()(a(ON((…)( ((((((()(b( ((JOIN((SELECT(…(FROM(…( ((((((()(c(ON((…)
  21. 21. WITH Before SQL:99 Then this... Nested queries are hard to read: SELECT(…( ((FROM((SELECT(…( ((((((((((FROM(t1( ((((((((((JOIN((SELECT(…(FROM(…( ((((((((((((((()(a(ON((…)( ((((((()(b( ((JOIN((SELECT(…(FROM(…( ((((((()(c(ON((…)
  22. 22. WITH Before SQL:99 Then this... Nested queries are hard to read: SELECT(…( ((FROM((SELECT(…( ((((((((((FROM(t1( ((((((((((JOIN((SELECT(…(FROM(…( ((((((((((((((()(a(ON((…)( ((((((()(b( ((JOIN((SELECT(…(FROM(…( ((((((()(c(ON((…)
  23. 23. WITH Before SQL:99 Finally the first line makes sense Nested queries are hard to read: SELECT(…( ((FROM((SELECT(…( ((((((((((FROM(t1( ((((((((((JOIN((SELECT(…(FROM(…( ((((((((((((((()(a(ON((…)( ((((((()(b( ((JOIN((SELECT(…(FROM(…( ((((((()(c(ON((…)
  24. 24. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( WITH Since SQL:99
  25. 25. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( Keyword WITH Since SQL:99
  26. 26. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( WITH Since SQL:99 Name of CTE and (here optional) column names
  27. 27. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( WITH Since SQL:99 Definition
  28. 28. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( WITH Since SQL:99 Introduces another CTE Don't repeat WITH
  29. 29. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( WITH Since SQL:99 May refer to
 previous CTEs
  30. 30. WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( (c((…)( AS((SELECT(…(FROM(…)( SELECT(…( ((FROM(b(JOIN(c(ON((…) WITH Since SQL:99 Third CTE
  31. 31. WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( (c((…)( AS((SELECT(…(FROM(…)( SELECT(…( ((FROM(b(JOIN(c(ON((…) WITH Since SQL:99 No comma!
  32. 32. WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( (c((…)( AS((SELECT(…(FROM(…)( SELECT(…( ((FROM(b(JOIN(c(ON((…) WITH Since SQL:99 Main query
  33. 33. CTEs are statement-scoped views: WITH( (a((c1,(c2,(c3)( AS((SELECT(c1,(c2,(c3(FROM(…),( (b((c4,(…)( AS((SELECT(c4,(…( ((((((FROM(t1( ((((((JOIN(a( ((((((((ON((…)( (((),( (c((…)( AS((SELECT(…(FROM(…)( SELECT(…( ((FROM(b(JOIN(c(ON((…) WITH Since SQL:99 Read top down
  34. 34. WITH in an Nutshell WITH are the "private methods" of SQL WITH views can be referred to multiple times WITH allows chaining instead of nesting WITH is allowed where SELECT is allowed INSERT(INTO(tbl
 WITH(...(SELECT(...
  35. 35. CTE(Scan(on(cte( ((rows=6370)( (Filter:(topic(=(1( (CTE(cte( (V>(Seq(Scan(on(news( (((((rows=10000001) WITH PostgreSQL Particularities In PostgreSQL WITH views are more like materialized views:
 
 WITH(cte(AS
 (SELECT(*
 (((FROM(news)
 SELECT(*(
 ((FROM(cte
 (WHERE(topic=1
  36. 36. CTE(Scan(on(cte( ((rows=6370)( (Filter:(topic(=(1( (CTE(cte( (V>(Seq(Scan(on(news( (((((rows=10000001) WITH PostgreSQL Particularities In PostgreSQL WITH views are more like materialized views:
 
 WITH(cte(AS
 (SELECT(*
 (((FROM(news)
 SELECT(*(
 ((FROM(cte
 (WHERE(topic=1 CTE doesn't know about the outer filter
  37. 37. Normal views and inline-views support "predicate pushdown":
 
 SELECT(*
 ((FROM((
 ((SELECT(*
 ((((FROM(news
 (()(n
 WHERE(topic=1; Bitmap(Heap(Scan( on(news((rows=6370)( V>Bitmap(Index(Scan( ((on(idx((rows=6370)( ((Cond:(topic=1 WITH PostgreSQL Particularities
  38. 38. PostgreSQL 9.1+ allows INSERT, UPDATE and DELETE within WITH:
 
 WITH(deleted_rows(AS((
 (((DELETE(FROM(source_tbl
 (((RETURNING(*
 )
 INSERT(INTO(destination_tbl
 SELECT(*(FROM(deleted_rows; WITH PostgreSQL Particularities
  39. 39. WITH Availability (SQL:99) 1999 2001 2003 2005 2007 2009 2011 2013 7 DB2 LUW MySQL 9iR2 Oracle 8.4 PostgreSQL 2005[0] SQL Server 3.8.3[1] SQLite [0] Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT. [1] Only for top-level SELECT statements
  40. 40. WITH(RECURSIVE (Common Table Expressions)
  41. 41. WITH RECURSIVE Before SQL:99
  42. 42. (This page is intentionally left blank) WITH RECURSIVE Before SQL:99
  43. 43. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99
  44. 44. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 Keyword
  45. 45. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 Column list mandatory here
  46. 46. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 Executed first
  47. 47. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 Result sent there
  48. 48. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 Result visible twice
  49. 49. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) Once it becomes part of
 the final
 result
  50. 50. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) Second
 leg of UNION 
 is executed
  51. 51. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) Result sent there again
  52. 52. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows)
  53. 53. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) It's a loop!
  54. 54. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) It's a loop!
  55. 55. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) It's a loop!
  56. 56. Recursive common table expressions may refer to themselves in the second leg of a UNION([ALL]: WITH(RECURSIVE(cte((n)( ((AS((SELECT(1( (((((((UNION(ALL( ((((((SELECT(n+1( ((((((((FROM(cte( (((((((WHERE(n(<(3)( SELECT(*(FROM(cte; WITH RECURSIVE Since SQL:99 (n(( VVV( (1( (2( (3( (3(rows) n=3
 doesn't match Loop terminates
  57. 57. WITH RECURSIVE Use Cases • Row generators (previous example)
 (generate_series is proprietary) • Processing graphs
 (don't forget the cycle detection!) • Generally said: Loops that... ‣ ... pass data to the next iteration ‣ ... need a "dynamic" abort condition
  58. 58. WITH RECURSIVE in a Nutshell WITH(RECURSIVE is the while of SQL WITH(RECURSIVE "supports" infinite loops
 (SQL Server requires setting MAXRECURSION(0)( Except PostgreSQL, databases generally don't require the RECURSIVE keyword.
 SQL Server & Oracle don’t even know the keyword, but allow recursive CTEs anyway.
  59. 59. WITH RECURSIVE Availability 1999 2001 2003 2005 2007 2009 2011 2013 7 DB2 LUW MySQL[0] 11gR2 Oracle 8.4 PostgreSQL 2005[1] SQL Server 3.8.3[2] SQLite [0] Feature request #16244 from 2006-01-06 [1] Default limit of 100 iterations. Use OPTION (MAXRECURSION 0) to disable [2] Only for top-level SELECT statements
  60. 60. SQL:2003
  61. 61. FILTER
  62. 62. Pivot table: Years on the Y asis, Month on X axis: SELECT-YEAR,-- SUM(CASE7WHEN7MONTH7=717 777777777THEN7sales7ELSE707END)7JAN,- SUM(CASE-WHEN-MONTH-=-2- ---------THEN-sales-ELSE-0-END)-FEB,…- --FROM-sale_data- -GROUP-BY-YEAR FILTER Before SQL:2003
  63. 63. SQL:2003 has FILTER: SELECT-YEAR,- SUM(sales)7FILTER7(WHERE7MONTH7=71)7JAN,7 SUM(sales)-FILTER-(WHERE-MONTH-=-2)-FEB,- …- --FROM-sale_data- -GROUP-BY-YEAR; FILTER Since SQL:2003
  64. 64. FILTER Availability (SQL:2003) 1999 2001 2003 2005 2007 2009 2011 2013 DB2 LUW MySQL Oracle 9.4 PostgreSQL SQL Server SQLite
  65. 65. OVER and PARTITION(BY
  66. 66. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003
  67. 67. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003
  68. 68. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003
  69. 69. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003 WITH intermezzo
  70. 70. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003 WITH intermezzo
  71. 71. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003 WITH intermezzo
  72. 72. Show percentage of department salary: WITH(total_salary_by_department( ((AS((SELECT(dep,(SUM(salary)(total( ((((((((FROM(emp( (((((((GROUP(BY(dep)( SELECT(dep,(emp_id,(salary,( (((((((salary/ts.total*100("%(of(dep"( ((FROM(emp( ((JOIN(total_salary_by_department(ts( ((((ON((emp.dep(=(ts.dep)( (WHERE(emp.dep(=(? OVER Before SQL:2003 WITH intermezzo
  73. 73. GROUP(BY(= DISTINCT( + Aggregates OVER Before SQL:2003
  74. 74. Build aggregates without GROUP(BY: SELECT(dep,(emp_id,(salary,( (((((((salary/SUM(salary)7 77777777777777OVER(PARTITION7BY7dep)( (((((((((((((*(100("%(of(dep"( ((FROM(emp( OVER Since SQL:2003
  75. 75. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  76. 76. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  77. 77. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  78. 78. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  79. 79. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  80. 80. OVER How It Works dep salary 1 1000 6000 22 1000 6000 22 1000 6000 333 1000 6000 333 1000 6000 333 1000 6000 SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(()( ((FROM(emp;
  81. 81. SELECT(dep,(( (((((((salary,( (((((((SUM(salary)( (((((((OVER(PARTITION(BY(dep)( ((FROM(emp; dep salary ts 1 1000 1000 22 1000 2000 22 1000 2000 333 1000 3000 333 1000 3000 333 1000 3000 OVER How It Works
  82. 82. T(dep,(( ((salary,( ((SUM(salary)( ((OVER(PARTITION(BY(dep)( M(emp; dep salary ts 1 1000 1000 22 1000 2000 22 1000 2000 333 1000 3000 333 1000 3000 333 1000 3000 OVER How It Works
  83. 83. T(dep,(( ((salary,( ((SUM(salary)( ((OVER(PARTITION(BY(dep)( M(emp; dep salary ts 1 1000 1000 22 1000 2000 22 1000 2000 333 1000 3000 333 1000 3000 333 1000 3000 OVER How It Works
  84. 84. T(dep,(( ((salary,( ((SUM(salary)( ((OVER(PARTITION(BY(dep)( M(emp; dep salary ts 1 1000 1000 22 1000 2000 22 1000 2000 333 1000 3000 333 1000 3000 333 1000 3000 OVER How It Works
  85. 85. T(dep,(( ((salary,( ((SUM(salary)( ((OVER(PARTITION(BY(dep)( M(emp; dep salary ts 1 1000 1000 22 1000 2000 22 1000 2000 333 1000 3000 333 1000 3000 333 1000 3000 OVER How It Works
  86. 86. OVER in a Nutshell OVER may follow any aggregate function( OVER defines which rows are visible at each row
 (it does not limit the result in any way)( OVER() makes all rows visible at every row( OVER(PARTITION(BY x) segregates like GROUP(BY
  87. 87. OVER and ORDER(BY
  88. 88. OVER Before SQL:2003 Calculating a running total: SELECT(txid,(value,( ((((((((SELECT(SUM(value)( ((((((((((FROM(transactions(tx2( (((((((((WHERE(tx2.acnt(=(tx1.acnt( (((((((((((AND(tx2.txid(<=(tx1.txid)(bal( ((FROM(transactions(tx1( (WHERE(acnt(=(?( (ORDER(BY(txid
  89. 89. OVER Before SQL:2003 Calculating a running total: SELECT(txid,(value,( ((((((((SELECT(SUM(value)( ((((((((((FROM(transactions(tx2( (((((((((WHERE(tx2.acnt(=(tx1.acnt( (((((((((((AND(tx2.txid(<=(tx1.txid)(bal( ((FROM(transactions(tx1( (WHERE(acnt(=(?( (ORDER(BY(txid
  90. 90. OVER Before SQL:2003 Calculating a running total: SELECT(txid,(value,( ((((((((SELECT(SUM(value)( ((((((((((FROM(transactions(tx2( (((((((((WHERE(tx2.acnt(=(tx1.acnt( (((((((((((AND(tx2.txid(<=(tx1.txid)(bal( ((FROM(transactions(tx1( (WHERE(acnt(=(?( (ORDER(BY(txid
  91. 91. OVER Before SQL:2003 Before SQL:2003 running totals were awkward: ‣ Requires a scalar sub-select or self-join ‣ Poor maintainability (repetitive clauses) ‣ Poor performance
 The only real answer was: Do it in the application
  92. 92. With SQL:2003 you can narrow the window: SELECT(txid,(value,( (((((((SUM(value)( (((((((OVER(ORDER7BY7txid7 777777777777ROWS7 777777777777BETWEEN7UNBOUNDED7PRECEDING7 7777777777777777AND7CURRENT7ROW)(bal( ((FROM(transactions(tx1( (WHERE(acnt(=(?( (ORDER(BY(txid OVER Since SQL:2003
  93. 93. With OVER((ORDER(BY(…) a new type of functions makes sense: ‣ ROW_NUMBER( ‣ Ranking functions:(
 RANK,(DENSE_RANK,(PERCENT_RANK,
 CUME_DIST OVER Since SQL:2003
  94. 94. OVER Availability (SQL:2003) 1999 2001 2003 2005 2007 2009 2011 2013 7 DB2 LUW MySQL[0] 8i Oracle 8.4 PostgreSQL 2005 SQL Server SQLite [0] Feature request #35893 from 2008-04-08
  95. 95. WITHIN-GROUP
  96. 96. WITHIN GROUP Before SQL:2003 Getting the median: SELECT-d1.val- --FROM-data-d1- --JOIN-data-d2- ----ON-(d1.val-<-d2.val- -------OR-(d1.val=d2.val-AND-d1.id<d2.id))- -GROUP-BY-d1.val- HAVING-count(*)-=-- -------(SELECT-FLOOR(COUNT(*)/2)- ----------FROM-data)
  97. 97. WITHIN GROUP Since SQL:2003 SQL:2003 introduced ordered-set functions... SELECT-PERCENTILE_DISC(0.5) -------WITHIN-GROUP-(ORDER-BY-val) Median Which value?
  98. 98. WITHIN GROUP Since SQL:2003 SQL:2003 introduced ordered-set functions... SELECT-PERCENTILE_DISC(0.5) -------WITHIN-GROUP-(ORDER-BY-val) --FROM-data
  99. 99. WITHIN GROUP Availability 1999 2001 2003 2005 2007 2009 2011 2013 DB2 LUW MySQL 9iR1 Oracle 9.4 PostgreSQL 2012[0] SQL Server SQLite [0] Only as window function (OVER required). Feature request 728969 closed as "won't fix"
  100. 100. SQL:2008
  101. 101. OVER
  102. 102. Calculate the difference to a previous row: WITH(numbered_data(AS((( (SELECT(*,( ((((((((ROW_NUMBER()(OVER(ORDER(BY(x)(rn( (((FROM(data)(( SELECT(cur.*,(cur.balanceVprev.balance( ((FROM((((((numbered_data(cur( ((LEFT(JOIN(numbered_data(prev( ((((ON((cur.rn(=(prev.rnV1) OVER Before SQL:2008
  103. 103. Calculate the difference to a previous row: WITH(numbered_data(AS((( (SELECT(*,( ((((((((ROW_NUMBER()(OVER(ORDER(BY(x)(rn( (((FROM(data)(( SELECT(cur.*,(cur.balanceVprev.balance( ((FROM((((((numbered_data(cur( ((LEFT(JOIN(numbered_data(prev( ((((ON((cur.rn(=(prev.rnV1) OVER Before SQL:2008
  104. 104. SQL:2008 can access other rows directly:
 SELECT(*,(balance(V(LAG(balance)
 ((((((((((((((((((((OVER(ORDER(BY(x)
 ((FROM(data
 OVER Since SQL:2008
  105. 105. SQL:2008 can access other rows directly:
 SELECT(*,(balance(V(LAG(balance)
 ((((((((((((((((((((OVER(ORDER(BY(x)
 ((FROM(data
 Available functions:
 (LEAD(/(LAG
 (FIRST_VALUE(/(LAST_VALUE
 (NTH_VALUE(col,(n)(FROM(FIRST/LAST
 (((((((((((((((((((RESPECT/IGNORE(NULLS OVER Since SQL:2008
  106. 106. OVER Availability (SQL:2008) 1999 2001 2003 2005 2007 2009 2011 2013 9.5[0] DB2 LUW MySQL 8i[1] 11gR2 Oracle 8.4[2] PostgreSQL 2012[3] SQL Server SQLite [0] No NTH_VALUE as of DB2 LUW 10.5 [1] No NTH_VALUE and IGNORE NULLS until Oracle release 11gR2 [2] No support for IGNORE NULLS and FROM LAST as of PostgreSQL 9.4 [3] No NTH_VALUE as of SQL Server 2014
  107. 107. FETCH(FIRST
  108. 108. Limit the number of selected rows: SELECT(*( ((FROM((SELECT(*,( ((((((((ROW_NUMBER()(OVER(ORDER(BY(x)(rn( (((FROM(data)(numbered_data( (WHERE(rn(<=10 FETCH FIRST Before SQL:2008
  109. 109. Limit the number of selected rows: SELECT(*( ((FROM((SELECT(*,( ((((((((ROW_NUMBER()(OVER(ORDER(BY(x)(rn( (((FROM(data)(numbered_data( (WHERE(rn(<=10 FETCH FIRST Before SQL:2008 Dammit! Let's take
 LIMIT
 (or TOP)
  110. 110. SQL:2008 has FETCH(FIRST(n(ROWS(ONLY: SELECT(*( ((FROM(data( (ORDER(BY(x( (FETCH(FIRST(10(ROWS(ONLY FETCH FIRST Since SQL:2008
  111. 111. FETCH FIRST Availability 1999 2001 2003 2005 2007 2009 2011 2013 7 DB2 LUW 3.19.3[0] MySQL 12c Oracle 6.5[1] 8.4 PostgreSQL 7.0[2] 2012 SQL Server 2.1.0[3] SQLite [0] Earliest mention of LIMIT. Probably inherited from mSQL [1] Functionality available using LIMIT [2] SELECT TOP n ... SQL Server 2000 also supports expressions and bind parameters [3] Functionality available using LIMIT
  112. 112. SQL:2011
  113. 113. OFFSET
  114. 114. Skip 10 rows, then deliver only the next 10: SELECT(*( ((FROM((SELECT(*,( ((((((((ROW_NUMBER()(OVER(ORDER(BY(x)(rn( (((FROM(data( ((FETCH(FIRST(20(ROWS(ONLY( )(numbered_data( (WHERE(rn(>(10 OFFSET Before SQL:2011
  115. 115. SQL:2011 introduced OFFSET, unfortunately: SELECT(*( ((FROM(data( (ORDER(BY(x( OFFSET7107ROWS7 FETCH(NEXT(10(ROWS(ONLY OFFSET Since SQL:2011
  116. 116. OFFSET is EVIL OFFSET http://use-the-index-luke.com/no-offset
  117. 117. OFFSET Availability (SQL:2011) 1999 2001 2003 2005 2007 2009 2011 2013 9.7[0] DB2 LUW 3.20.3[1] 4.0.6[2] MySQL 12c Oracle 6.5 PostgreSQL 2012 SQL Server 2.1.0 SQLite [0] Requires enabling the MySQL compatibility vector: db2set DB2_COMPATIBILITY_VECTOR=MYS [1] LIMIT [offset,] limit: "With this it's easy to do a poor man's next page/previous page WWW application." [2] The release notes say "Added PostgreSQL compatible LIMIT syntax"
  118. 118. AS(OF
  119. 119. INSERT( UPDATE( DELETE( are DESTRUCTIVE AS OF Before SQL:2011
  120. 120. Tables can be system versioned: CREATE(TABLE(t((...,( (start_ts(TIMESTAMP(9)(GENERATED
 ((((((((((ALWAYS(AS(ROW(START,( (end_ts(((TIMESTAMP(9)(GENERATED
 ((((((((((ALWAYS(AS(ROW(END,( (PERIOD(FOR(SYSTEM(TIME((start_ts,(end_ts)( )(WITH(SYSTEM(VERSIONING AS OF Since SQL:2011
  121. 121. ID Data start_ts end_ts 1 X 10:00:00 UPDATE(...(SET(DATA(=('Y'(... ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 DELETE(...(WHERE(ID(=(1 INSERT(...((ID,(DATA)(VALUES((1,('X') AS OF Since SQL:2011
  122. 122. ID Data start_ts end_ts 1 X 10:00:00 UPDATE(...(SET(DATA(=('Y'(... ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 DELETE(...(WHERE(ID(=(1 INSERT(...((ID,(DATA)(VALUES((1,('X') AS OF Since SQL:2011
  123. 123. ID Data start_ts end_ts 1 X 10:00:00 UPDATE(...(SET(DATA(=('Y'(... ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 DELETE(...(WHERE(ID(=(1 INSERT(...((ID,(DATA)(VALUES((1,('X') AS OF Since SQL:2011
  124. 124. ID Data start_ts end_ts 1 X 10:00:00 UPDATE(...(SET(DATA(=('Y'(... ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 DELETE(...(WHERE(ID(=(1 ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 12:00:00 AS OF Since SQL:2011
  125. 125. Although multiple versions exist, only the “current” one is visible per default. After 12:00:00, SELECT(*(FROM(t doesn’t return anything anymore. AS OF Since SQL:2011 ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 12:00:00
  126. 126. AS OF Since SQL:2011 ID Data start_ts end_ts 1 X 10:00:00 11:00:00 1 Y 11:00:00 12:00:00 With AS(OF you can query anything you like: SELECT(*( ((FROM(t(FOR(SYSTEM_TIME(AS(OF( (((((((((TIMESTAMP('2015V04V02(10:30:00' ID Data start_ts end_ts 1 X 10:00:00 11:00:00
  127. 127. SYSTEM_TIME AS OF Availability 1999 2001 2003 2005 2007 2009 2011 2013 10.1[0] DB2 LUW MySQL 10gR1[1] Oracle PostgreSQL SQL Server[2] SQLite [0] Third column required (tx id), history table required. [1] Functionality available using Flashback [2] "Temporal Databases: Track historical changes" mentioned in SQL Server 2016 datasheet
  128. 128. SYSTEM_TIME AS OF Availability 1999 2001 2003 2005 2007 2009 2011 2013 10.1[0] DB2 LUW MySQL 10gR1[1] Oracle PostgreSQL SQL Server[2] SQLite [0] Third column required (tx id), history table required. [1] Functionality available using Flashback [2] "Temporal Databases: Track historical changes" mentioned in SQL Server 2016 datasheet temporal features announced for SQL Server 2016
  129. 129. WITHOUT(OVERLAPS
  130. 130. Prior SQL:2011 it was not possible to define constraints that avoid overlapping periods. Workarounds are possible, but no fun: CREATE(TRIGGER WITHOUT OVERLAPS Before SQL:2011 id begin end 1 8:00 9:00 1 9:00 11:00 1 10:00 12:00
  131. 131. SQL:2011 introduced temporal and bi-temporal features —e.g., for constraints:
 
 PRIMARY(KEY((id,(period7WITHOUT7OVERLAPS) PostgreSQL 9.2 introduced range types and "exclusive constraints" which can accomplish the same effect:
 
 EXCLUDE(USING(gist
 (((((((((id(WITH(=,(period7WITH7&&) WITHOUT OVERLAPS Since SQL:2011
  132. 132. SQL:2011 goes way further. Please read these papers to get the idea: Temporal features in SQL:2011 http://cs.ulb.ac.be/public/_media/teaching/infoh415/tempfeaturessql2011.pdf What's new in SQL:2011? http://www.sigmod.org/publications/sigmod-record/1203/pdfs/10.industry.zemke.pdf Temporal/Bi-Temporal SQL
  133. 133. WITHOUT OVERLAPS Availability 1999 2001 2003 2005 2007 2009 2011 2013 10.1[0] DB2 LUW MySQL Oracle[1] 9.2[2] PostgreSQL SQL Server SQLite [0] Minor differences: PERIOD without FOR; period name must be BUSINESS_TIME [1] Oracle 12c has partial temporal support, but no direct equivalent of WITHOUT OVERLAPS [2] Functionality available using EXCLUDE constraints
  134. 134. About @MarkusWinand Tuning developers for high SQL performance Training & tuning: http://winand.at/ Author of: http://sql-performance-explained.com/ Geeky blog: http://use-the-index-luke.com

×