Sql analytic queries tips

SQL Analytic Queries ...
Tips & Tricks
Mostly in PostgreSQL

What are we going to talk about?
- Some less (or more) know facts about SQL
- Revision history (just most important parts)
- Quickly go through SQL Basics, since we all know those, right
- Range of SQL Advanced topics with comparison and parallels of real-world
situations and applications
- Conclusion, discussion and QA

Some less (or more) know facts about SQL ...
- SQL (Structured Query Language) is STANDARDIZED
internationally!
- By ISO (International Organization for Standardization) committee.
- All existing implementations follow same standards:
Oracle, MSSQL, MySQL, IBM DB2 PostgresSQL, etc, etc ...
- Revisions of standards so far (last 30 years):
SQL-86, SQL-89, SQL-92, SQL:1999 (SQL3), SQL:2003, SQL:2008,
SQL:2011, SQL:2016

Today, after many revisions, SQL is:
- Turing complete
- Computationally Universal
- Calculation Engine
* Turing complete means that can be used to write any algorithm or “any
software”.
* In other words - it can do “anything”.

Today, SQL is also:
- Only ever successful 4th generation general-purpose
programming language in existence (known to mankind)
- Python, Java, C# and all others - are still 3rd generation languages ...
- 4th gen language - abstracts (or hides) unimportant details from user:
hardware, algorithms, processes, threads, etc...
* take a deep breath and let that sit for a while ...

SQL is also:
- Declarative
- You just tell or declare to machine what you want.
- Let the machine to figure out for you how.
* That’s how Oracle got its name
- Let’s you focus on your business logic and your problem and what
is really really important to you …

Revision history - SQL-92
SQL-92 - most important parts
- DATE, TIME, TIMESTAMP, INTERVAL, BIT string, VARCHAR strings
- UNION JOIN, NATURAL JOIN
- Conditional expressions with CASE (upgraded in SQL:2008)
- ALTER and DROP, CHECK constraint
- INFORMATION_SCHEMA tables
- Temporary tables; CREATE TEMP TABLE
- CAST (expr AS type), Scroll Cursors…
- Two extensions, published after standard:
- SQL/CLI (Call Level Interface) - 1995
- SQL/PSM (stored procedures) - 1996
* PostgresSQL 11 (released 2016-10-08) - finally implements stored procedures, standardized in 1996

SQL:1999 (SQL3) - most important parts
- Boolean type, user defined types
- Common Table Expressions (CTE), WITH clause, RECURSIVE queries
- Grouping sets, Group By ROLLUP, Group By CUBE
- Role-based Access Control - CREATE ROLE
- UNNEST keyword
Revision history - SQL:1999 (SQL3)

SQL:2003 - most important parts
- XML features and functions
- Window functions (ROW_NUMBER OVER, RANK OVER…)
- Auto-generated values (default values)
- Sequence generators, IDENTITY columns
Revision history - SQL:2003

SQL:2008 (ISO/IEC 9075:2008) - most important parts
- TRUNCATE TABLE
- CASE WHEN ELSE
- TRIGGERS (INSTEAD OF)
- Partitioned JOINS
- XQuery, pattern matching ...
Revision history - SQL:2008 (ISO/IEC 9075:2008)

- Support for TEMPORAL databases:
- Time period tables PERIOD FOR
- Temporal primary keys and temporal referential integrity
- System versioned tables (AS OF SYSTEM_TIME, and VERSIONS BETWEEN SYSTEM
TIME)
- Allows working with “historic” data
* MSSQL2016, Oracle 12c, MariaDB v10.3 fully implements, IBM DB2 v10 uses alternative syntax.
* PostgreSQL requires installation of the temporal_tables extension

- JSON functions and full support
- Row pattern recognition, matching a row sequence against a regular expression patterns
- Date and time formatting and parsing functions
- LISTAGG - function to transform values to row
- Functions without return time (polymorphic functions)

1. Basics - EVERYTHING is a set (or table)
-- this is a table:
my_table;
-- this is another table:
select * from my_table;
-- this is again table (with hardcoded values):
values ('first'), ('second'), ('third');
-- yep, you've guess it, another table (or set if you like):
select * from (
values ('first'), ('second'), ('third')
) t;
-- we can give name to our table as we like:
select * from (
values (1, 'first'), (2, 'second'), (3, 'third')
) as t (id, description);
-- we can use pre-defined functions as tables, this one will return series:
select i from generate_series(1,10) as t (i)

1. Basics - execution order
/***
Queries are always executed in following
order:
1. CTE - Common table expressions
2. FROM and JOINS
3. WHERE
4. GROUP BY
5. HAVING
6. [Window functions]
7. SELECT
8. ORDER BY
9. LIMIT
***/
CTE
WHERE
HAVING [Window func.]
FROM, JOIN
GROUP BY
SELECT
ORDER BY
LIMIT

2. TEMP TABLES
-- temp table lives during and it is limited visible to connection:
create temp table temp_test1 (id int, t text);
-- only I can see you, no other connection know that you exist
select * from temp_test1;
-- they can be created on fly (and usually are) from another table or query using "into":
select *
into temp temp_test2 from (
values (1, 'first'), (2, 'second'), (3, 'third')
) as t (id, description);
-- let's see:
select * from temp_test2;

2. TEMP TABLES
Expensive query
(joins, filters)
INTO TEMP
table
Counts and statistics
data from TEMP
Sort and page
from TEMP
Return multiple
result sets
single connection
- Used a lot for optimizations (avoid repeating expensive operations by using temp tables - caching)
- Note that hardware is abstracted, we don’t know is it on disk or in memory, that’s not the point
- Typical, common usage - paging and sorting from large tables with expensive joins, with calculation of
counts and statistics.

3. CTE - Common Table Expressions (WITH queries)
-- we can use common table expressions for same purpose as temp tables:
with my_cte as (
)
select * from my_cte;
-- we can combine multiple CTE's, Postgres will optimize every CTE individually:
with my_cte1 as (
),
my_cte2 as (
),
my_cte3 as (
)
select * from my_cte1
union --intersect
select * from my_cte2
union
select * from my_cte3;

3. CTE - Common Table Expressions (WITH queries) - RECURSION
-- CTE can be used for recursive queries:
with recursive t(i) as (
values (1) -- recursion seed
union all
select i + 1 from t where i < 10 --call
)
select i from t;
-- Typically, used for efficient processing of tree structures, example:
create temp table employees (id serial, name varchar, manager_id int);
insert into employees (name, manager_id)
values ('Michael North', NULL), ('Megan Berry', 1), ('Sarah Berry', 2),
('Zoe Black', 1), ('Tim James', 2), ('Bella Tucker', 2), ('Ryan Metcalfe',
2), ('Max Mills', 2), ('Benjamin Glover', 3) ,('Carolyn Henderson', 4);
select * from employees;
-- Returns ALL subordinates of the manager with the id 2:
with recursive subordinates AS (
select id, manager_id, name from employees where id = 2
union
select e.id, e.manager_id, e.name
from employees e
inner join subordinates s on e.manager_id = s.id
)
select * from subordinates;

4. UNNEST and AGGREGATE
-- any array can be unnest-ed to row values:
select unnest(array[1, 2, 3]);
-- any row values can aggregated back to array
select array_agg(i)
from (
values (1), (2), (3)
) t(i);
-- any row values can aggregated back to json array
select json_agg(i)
from (
values (1), (2), (3)
) t(i);
-- from row values to array and back to row values
select unnest(array_agg(i))
from (
values (1), (2), (3)
) t(i);

5. Subqueries
-- First ten dates in january with extracted day numbers
select cast(d as date), extract(day from d) as i
from generate_series(cast('2018-01-01' as date), cast('2018-01-10' as date), '1 days') as d(d); --ISO type cast
-- First ten dates in february with extracted day numbers
select d::date, extract(day from d) as i
from generate_series('2018-02-01'::date, '2018-02-10'::date, '1 days') as d(d); -- Postgres cast (using ::)
-- Any table expression anywhere can be replaced by another query which is also table expression:
-- So we can join previous queries as SUBQUERIES:
select first_month.i, first_month.d as first_month, second_month.d as second_month
from (
from generate_series(cast('2018-01-01' as date), cast('2018-01-10' as date), '1 days') as d(d)
) first_month inner join (
) second_month on first_month.i = second_month.i;

5. Subqueries
-- subquery can be literary everywhere, but, sometimes needs to be limited to single value:
select cast(d as date),
(
select cast(d as date)
from generate_series(cast('2018-02-01' as date), cast('2018-02-10' as date), '1 days') as sub(d)
where extract(day from sub) = extract(day from d)
limit 1
) as february
from generate_series(cast('2018-02-01' as date), cast('2018-02-10' as date), '1 days') as d(d);
-- or it can multiple values in single row to be filtered in where clause:
select cast(d as date)
where extract(day from d) in (
select extract(day from sub)
from generate_series(cast('2018-02-01' as date), cast('2018-02-10' as date), '1 days') as sub(d)
)
-- How efficient are these queries ??? What we actually want our machine to do?
-- Let see what execution plan has to say ...

6. LATERAL joins
-- What if want to reference one subquery from another?
-- This doesn't work, we cannot reference joined subquery from outer table:
select by_day.d as date, counts_day.count
from (
) by_day inner join (
select count(*) as count, extract(day from d) as i
from generate_series(cast('2018-01-01' as date), cast('2018-01-10' as date), '1 hours') as d(d)
where extract(day from d) = by_day.i
group by extract(day from d)
) counts_day on by_day.i = counts_day.i;

6. LATERAL joins
-- To achieve this, we must use LATERAL join:
from (
) by_day inner join lateral (
select count(*) as count, extract(day from d) as i
group by extract(day from d)
) counts_day on by_day.i = counts_day.i;

6. LATERAL joins
-- Now, we can simplify even further this query:
from (
) by_day inner join lateral (
select count(*) as count
) counts_day on true;

7. DISTINCT ON
create temp table sales (brand varchar, segment varchar, quantity int);
insert into sales values ('ABC', 'Premium', 100), ('ABC', 'Basic', 200), ('XYZ', 'Premium', 100), ('XYZ', 'Basic', 300);
select * from sales;
-- brands with highest quantities:
select brand, max(quantity)
from sales
group by brand;
-- what are segments of brands with highest quantities? This is NOT allowed:
select brand, max(quantity), segment
from sales
group by brand;
-- we must use select distinct on:
select distinct on (brand) brand, quantity, segment
from sales
order by brand, quantity desc;

8. OLAP: GROUPING, GROUPING SETS, CUBE, ROLLUP
create temp table sales (brand varchar, segment varchar, quantity int);
insert into sales values ('ABC', 'Premium', 100), ('ABC', 'Basic', 200), ('XYZ', 'Premium', 100), ('XYZ', 'Basic', 300);
-- sum quantities by brand and segment:
select brand, segment, sum(quantity) from sales group by brand, segment;
-- sum quantities by brand only:
select brand, sum(quantity) from sales group by brand;
-- sum quantities by segment only:
select segment, sum(quantity) from sales group by segment;
-- sum all quantities:
select sum(quantity) from sales;
-- we can union of all of these queries but this is long an extremely un-efficient:
select brand, segment, sum(quantity) from sales group by brand, segment
union all
select brand, null as segment, sum(quantity) from sales group by brand
union all
select null as brand, segment, sum(quantity) from sales group by segment
union all
select null as brand, null as segment, sum(quantity) from sales;

-- unless we use grouping sets to get all sums by all categories
-- this is many times more efficient instead of separate queries with union
-- and lot shorter and easier to read:
select
brand, segment, sum(quantity)
from
sales
group by grouping sets (
(brand, segment),
(brand),
(segment),
()
)
order by
brand nulls last, segment nulls last;

-- generate ALL possible grouping combinations:
CUBE(c1,c2,c3)
-- results in:
GROUPING SETS (
(c1,c2,c3),
(c1,c2),
(c1,c3),
(c2,c3),
(c1),
(c2),
(c3),
()
)
-- previous example:
select brand, segment, sum(quantity)
from sales
group by cube (brand, segment);

-- generate grouping combinations by assuming hierarchy c1 > c2 > c3
ROLLUP(c1,c2,c3)
-- results in:
GROUPING SETS (
(c1, c2, c3)
(c1, c2)
(c1)
()
)
-- previous example:
from sales
group by rollup (brand, segment);
-- results in:
from sales
group by grouping sets (
(brand, segment),
(brand),
()
);

9. OLAP: WINDOW FUNCTIONS
create temp table employee (id serial, department varchar, salary int);
insert into employee (department, salary)
values
('develop', 5200), ('develop', 4200), ('develop', 4500), ('develop', 6000), ('develop', 5200),
('personnel', 3500), ('personnel', 3900),
('sales', 4800), ('sales', 5000), ('sales', 4800);
-- average salaries by department will return less rows because it is grouped by
select department, avg(salary)
from employee
group by department;
-- but not if we use aggregate function over partition (window) - this returns ALL records:
select department, salary, avg(salary) over (partition by department)
from employee;

9. OLAP: WINDOW FUNCTIONS
-- syntax:
window_function(arg1, arg2,..) OVER (PARTITION BY expression ORDER BY expression)
-- return all employees, no grouping
select
department, salary,
-- average salary:
avg(salary) over (partition by department),
-- employee order number within department (window):
row_number() over (partition by department order by id),
-- rank of employee salary within department (window):
rank() over (partition by department order by salary)
from employee;

BONUS: Mandelbrot set fractal
WITH RECURSIVE
x(i)
AS (
VALUES(0)
UNION ALL
SELECT i + 1 FROM x WHERE i < 101
),
Z(Ix, Iy, Cx, Cy, X, Y, I)
AS (
SELECT Ix, Iy, X::FLOAT, Y::FLOAT, X::FLOAT, Y::FLOAT, 0
FROM
(SELECT -2.2 + 0.031 * i, i FROM x) AS xgen(x,ix)
CROSS JOIN
(SELECT -1.5 + 0.031 * i, i FROM x) AS ygen(y,iy)
UNION ALL
SELECT Ix, Iy, Cx, Cy, X * X - Y * Y + Cx AS X, Y * X * 2 + Cy, I + 1
FROM Z
WHERE X * X + Y * Y < 16.0
AND I < 27
),
Zt (Ix, Iy, I) AS (
SELECT Ix, Iy, MAX(I) AS I
FROM Z
GROUP BY Iy, Ix
ORDER BY Iy, Ix
)
SELECT array_to_string(
array_agg(
SUBSTRING(
' .,,,-----++++%%%%@@@@#### ',
GREATEST(I,1),
1
)
),''
)
FROM Zt GROUP BY Iy ORDER BY Iy;

Conclusion and final words
- SQL is “mysterious machine”. Even after 15 years can pull some new surprises.
- Practice is the key. You need to practice, practice and get some more practice.
- Payoffs are huge: Application performances can be improve dramatically with significantly less
code.
- It can reduce amount of code and significantly improve system maintainability many, many times.
- It can be intimidating to some. Percentage of keywords in code is much higher, levels of
assembler code or cobol code.
- Don't be intimidated, it will pay off in the end. Any day gone without learn anything new is wasted
day.

Sql analytic queries tips

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Sql analytic queries tips

Similar to Sql analytic queries tips (20)

Recently uploaded

Recently uploaded (20)

Sql analytic queries tips