This webinar will give an overview of CREATE STATISTICS in PostgreSQL. This command allows the database to collect multi-column statistics, helping the optimizer understand dependencies between columns, produce more accurate estimates, and better query plans. The following key topics will be covered during the webinar: - Why CREATE STATISTICS may be needed at all - How the command works - Which cases CREATE STATISTICS already addresses - What improvements are in the queue for future PostgreSQL versions (either already committed to PostgreSQL 13 or beyond)

1. CREATE STATISTICS
Peter Eisentraut
Webinar März 2021

2. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
2
Agenda
• Überblick Planer und Schätzungen
• Schätzungen mit korrelierten Spalten
• CREATE STATISTICS zur Rettung!
• funktionale Abhängigkeiten
• ndistinct
• MCV-Listen
• Ausblick

3. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
3
• PostgreSQL hat einen kostenbasierten Planer.
• Kosten = Kardinalitätsschätzung × Plankostenfaktor
• schlechte Schätzungen, schlechter Plan
Worum geht es?

4. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
4
-- http://download.geonames.org/export/zip/DE.zip
CREATE TABLE orte (
staat text,
plz char(5),
ort text,
land text,
regbez text,
kreis text,
...
);
copy orte from 'DE.txt'
Testdaten

5. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
5
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin';
QUERY PLAN
-------------------------------------------------------------------------
Seq Scan on orte (cost=0.00..467.98 rows=182 width=85)
(actual rows=182 loops=1)
Filter: (ort = 'Berlin'::text)
Rows Removed by Filter: 16296
Planning Time: 0.680 ms
Execution Time: 2.744 ms
EXPLAIN

6. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
6
SELECT reltuples, relpages FROM pg_class WHERE relname = 'orte';
reltuples | relpages
-----------+----------
16478 | 262
reltuples, relpages

7. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
7
SELECT * FROM pg_stats WHERE tablename = 'orte' AND attname = 'ort';
------------------+-----------------------------------------------
schemaname | public
tablename | orte
attname | ort
...
most_common_vals | {Berlin,Hamburg,München,Köln,...}
most_common_freqs | {0.0110450,0.00612938,0.0044908,0.0027309,...}
...
pg_stats

8. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
8
EXPLAIN (ANALYZE, TIMING off) SELECT * FROM orte WHERE ort = 'Berlin';
QUERY PLAN
-------------------------------------------------------------------------
Seq Scan on orte (cost=0.00..467.98 rows=182 width=85)
(actual rows=182 loops=1)
reltuples | 16478
most_common_vals | {Berlin,...}
most_common_freqs | {0.0110450,...}
16478 * 0.0110450 = 181.99951
Berechnung der Zeilenschätzung

9. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
9
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin' AND land = 'Berlin';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=2 width=85)
(actual rows=182 loops=1)
Filter: ((ort = 'Berlin'::text) AND (land = 'Berlin'::text))
Rows Removed by Filter: 16296
Planning Time: 0.127 ms
Execution Time: 2.190 ms
Unterschätzung

10. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
10
P(A ∩ B) = P(A) × P(B)
Wahrscheinlichkeit mehrerer Ereignisse

11. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
11
SELECT * FROM orte WHERE ort = 'Berlin' AND land = 'Berlin';
P(ort = 'Berlin' & land = 'Berlin')
= P(ort = 'Berlin') * P(land = 'Berlin')
= 0.01105 * 0.01177
= 0.00013
16478 * 0.00013 = 2.14214
Berechnung der Zeilenschätzung

12. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
12
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin' AND land <> 'Berlin';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=180 width=85)
(actual rows=0 loops=1)
Filter: ((land <> 'Berlin'::text) AND (ort = 'Berlin'::text))
Rows Removed by Filter: 16478
Planning Time: 0.060 ms
Execution Time: 2.163 ms
Überschätzung

13. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
13
• Es wird angenommen, dass Spaltenwerte unabhängig voneinander sind.
• falsche Schätzungen
• Unter- und Überschätzungen
• Konsequenzen:
• schlechte Scan-Wahl (Seq Scan vs. Index Scan)
• schlechte Join-Wahl (Nested Loop)
Korrelierte Spalten

14. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
14
Index Scan using orders_city_idx on orders
(cost=0.28..185.10 rows=90 width=36)
(actual rows=12248237 loops=1)
Seq Scan using orders
(cost=0.13..129385.10 rows=12248237 width=36)
(actual rows=90 loops=1)
Schlechte Scan-Wahl

15. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
15
-> Nested Loop (... rows=90 ...) (... rows=12248237 ...)
-> Nested Loop (... rows=90 ...) (... rows=12248237 ...)
-> Index Scan using orders_city_idx on orders
(cost=0.28..185.10 rows=90 width=36)
(actual rows=12248237 loops=1)
...
-> Index Scan ... (... loops=12248237)
-> Index Scan ... (... loops=12248237)
-> Index Scan ... (... loops=12248237)
Schlechte Join-Wahl

16. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
16
Funktionale Abhängigkeiten (WHERE)

17. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
17
• Wert in Spalte A bestimmt Wert in Spalte B
• Beispiel: Primärschlüssel bestimmt alles
• plz -> {ort, kreis, land}
• 89346 -> {Bibertal, Landkreis Günzburg, Bayern}
• andere Abhängigkeiten:
• Ort -> Kreis
• Kreis -> Land
• Land -> Staat
Funktionale Abhängigkeiten

18. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
18
CREATE STATISTICS s1 (dependencies)
ON ort, kreis, land FROM orte;
-- 3 8 4
ANALYZE orte;
SELECT dependencies FROM pg_stats_ext WHERE statistics_name = 's1';
dependencies
--------------------------------------------------------------
{"3 => 4": 0.918073, "3 => 8": 0.895254, "4 => 3": 0.000061,
"4 => 8": 0.035866, "8 => 3": 0.053647, "8 => 4": 1.000000,
"3, 4 => 8": 0.965105, "3, 8 => 4": 1.000000,
"4, 8 => 3": 0.053647}
CREATE STATISTICS

19. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
19
ort -> land: d = 0.918073
P(ort = 'Berlin' & land = 'Berlin')
= P(ort = 'Berlin') * [d + (1-d) * P(land = 'Berlin')]
= 0.01105 * [0.918073 + (1-0.918073) * 0.01177]
= 0.01016
16478 * 0.01016 = 167.4
Berechnung der Zeilenschätzung

20. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
20
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin' AND land = 'Berlin';
QUERY PLAN
-------------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=167 width=85)
(actual rows=182 loops=1)
Filter: ((ort = 'Berlin'::text) AND (land = 'Berlin'::text))
Rows Removed by Filter: 16296
Planning Time: 0.362 ms
Execution Time: 2.167 ms
Unterschätzung: berichtigt

21. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
21
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin' AND land <> 'Berlin';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=180 width=85)
(actual rows=0 loops=1)
Filter: ((land <> 'Berlin'::text) AND (ort = 'Berlin'::text))
Rows Removed by Filter: 16478
Planning Time: 0.077 ms
Execution Time: 2.356 ms
Funktionale Abhängigkeiten funktionieren nur mit "=".
Überschätzung Nr. 1: nicht berichtigt :-(

22. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
22
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE ort = 'Berlin' AND land = 'Bayern';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=169 width=85)
(actual rows=0 loops=1)
Filter: ((ort = 'Berlin'::text) AND (land = 'Bayern'::text))
Rows Removed by Filter: 16478
Planning Time: 0.082 ms
Execution Time: 2.204 ms
Die Anfrage muss die funktionalen Abhängigkeiten "respektieren".
Überschätzung Nr. 2: nicht berichtigt :-(

23. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
23
ndistinct (GROUP BY)

24. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
24
EXPLAIN (ANALYZE, TIMING off)
SELECT count(*) FROM places GROUP BY community;
QUERY PLAN
-------------------------------------------------------------------------
HashAggregate (cost=56576.56..56580.14 rows=358 width=27)
(actual rows=359 loops=1)
Group Key: community
-> Seq Scan on places (cost=0.00..47731.71 rows=1768971 width=19)
(actual rows=1768982 loops=1)
Planning Time: 0.043 ms
Execution Time: 361.663 ms
EXPLAIN

25. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
25
SELECT attname, n_distinct FROM pg_stats WHERE tablename = 'places';
attname | n_distinct
-------------+-------------
community | 358
country | 1
county | 92
place | 6544
postcode | -1
state | 5
... |
pg_stats

26. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
26
EXPLAIN (ANALYZE, TIMING off)
SELECT count(*) FROM places GROUP BY state, county, community;
QUERY PLAN
-----------------------------------------------------------------------
HashAggregate (cost=211361.53..240648.50 rows=164680 width=46)
(actual rows=420 loops=1)
Group Key: state, county, community
-> Seq Scan on places (cost=0.00..47731.71 rows=1768971 width=38)
(actual rows=1768982 loops=1)
Planning Time: 0.079 ms
Execution Time: 484.140 ms
Falsche Schätzung

27. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
27
ndistinct(state, county, community)
=
ndistinct(state) × ndistinct(county) × ndistinct(community)
5 × 92 × 358 = 164680
Zeilenschätzung

28. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
28
CREATE STATISTICS s2 (ndistinct)
ON state, county, community
-- 4 6 8
FROM places;
ANALYZE places;
SELECT n_distinct FROM pg_stats_ext WHERE statistics_name = 's2';
n_distinct
--------------------------------------------------------
{"4, 6": 93, "4, 8": 362, "6, 8": 384, "4, 6, 8": 385}
CREATE STATISTICS

29. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
29
EXPLAIN (ANALYZE, TIMING off)
SELECT count(*) FROM places GROUP BY state, county, community;
QUERY PLAN
-----------------------------------------------------------------------
HashAggregate (cost=65422.20..65426.05 rows=385 width=47)
(actual rows=420 loops=1)
Group Key: state, county, community
-> Seq Scan on places (cost=0.00..47732.10 rows=1769010 width=39)
(actual rows=1768982 loops=1)
Planning Time: 0.214 ms
Execution Time: 482.752 ms
Schätzung berichtigt

30. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
30
• Das alte Verhalten war defensiv.
• unzuverlässige Schätzungen mit mehreren Spalten
• HashAggregate kann nicht auf Festplatte auslagern (bis PG12) (OOM)
• statt Crash lieber Sort+GroupAggregate (langsam)
• ndistinct-Koeffizienten
• machen Schätzungen verlässlicher
• weniger Gefahr von OOM
• für große Tabellen + GROUP BY mehrere Spalten
ndistinct

31. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
31
MCV-Listen

32. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
32
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE kreis = 'Landkreis Günzburg' AND land = 'Bayern';
QUERY PLAN
----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=5 width=85)
(actual rows=34 loops=1)
Filter: ((kreis = 'Landkreis Günzburg'::text) AND (land =
'Bayern'::text))
Rows Removed by Filter: 16444
Planning Time: 0.117 ms
Execution Time: 2.692 ms
Unterschätzung

33. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
33
CREATE STATISTICS s3 (mcv) ON kreis, land FROM orte;
SET default_statistics_target = 10000;
ANALYZE orte;
SELECT most_common_vals, most_common_freqs FROM pg_stats_ext WHERE
statistics_name = 's3';
most_common_vals | {{Hamburg,"Hamburg, Freie und Hansestadt"},
{Rheinland-Pfalz,"Eifelkreis Bitburg-Prüm"},...
most_common_freqs | {0.02403204272363151,0.014200752518509527,...
CREATE STATISTICS

34. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
34
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM orte WHERE kreis = 'Landkreis Günzburg' AND land = 'Bayern';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on orte (cost=0.00..509.17 rows=34 width=85)
(actual rows=34 loops=1)
Filter: ((kreis = 'Landkreis Günzburg'::text) AND (land =
'Bayern'::text))
Rows Removed by Filter: 16444
Planning Time: 0.518 ms
Execution Time: 2.286 ms
Unterschätzung berichtigt

35. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
35
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM places WHERE place <> 'London' AND county = 'Greater
London';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on places (cost=0.00..56577.15 rows=157198 width=97)
(actual rows=1782 loops=1)
Filter: ((place <> 'London'::text) AND (county = 'Greater
London'::text))
Rows Removed by Filter: 1767200
Planning Time: 0.284 ms
Execution Time: 201.311 ms
Überschätzung Nr. 1

36. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
36
EXPLAIN (ANALYZE, TIMING off)
SELECT * FROM places WHERE place <> 'London' AND county = 'Greater
London';
QUERY PLAN
-----------------------------------------------------------------------
Seq Scan on places (cost=0.00..56576.73 rows=39789 width=96)
(actual rows=1782 loops=1)
Filter: ((place <> 'London'::text) AND (county = 'Greater
London'::text))
Rows Removed by Filter: 1767200
Planning Time: 3.533 ms
Execution Time: 228.332 ms
Überschätzung Nr. 1 mit MCV

37. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
37
• mehr Arten von Statistiken
• Histogramme?
• Statistiken für Ausdrücke
• Verwendung für Joins
• Verwendung von MCV-Listen
• Statistiken über mehrere Tabellen
Ausblick

38. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.
38
Zusammenfassung
• CREATE STATISTICS
• als Ergänzung zu CREATE INDEX
• dependencies, ndistinct, mcv
• Verbesserungen in jeder Release