The PostgreSQL Query Planner

Atri Sharma
Whats this talk based on?
●

Research done by hackers like Robert Haas in
the past, some mailing list readings,
discussion...
Who's this talk for?
●

Users of postgres, who want to know how
query planner's heuristics can affect their
queries.
What are we going to discuss?
●

●
●

A small discussion about the Postgres query
planner.
Areas where problems arise in q...
A little about the PostgreSQL Query
Planner
●

Quoting Bruce Momjian:
'The optimizer is the "brain" of the database,
inter...
Where can the problems arise in the
query planner
●

Row Count Estimates

●

DBA Ideas

●

Planner's Blocking points (What...
Row Count Stats
●

●

●

The backend essentially holds MCVs (Most
common values) and a histogram of the
distribution of th...
User Settings
●

PostgreSQL depends quite some on the page
values (seq_page_cost, random_page_cost)
and others. Setting th...
Planner Not-Done-Yets
●

Some queries are slow in execution because
their optimization depends on a feature which
is not y...
Selectivity Estimate Issues
●
●

●

Arbitrary estimates give a bad value
Cross table correlations arent taken into
conside...
Settings Issues
●

●

DBAs can use some settings to tune the query
planner.
Those settings can be harmful if set to
incorr...
Planner Cost Evaluation Parameters
●

seq_page_cost = 1.0

●

random_page_cost = 4.0

●

cpu_tuple_cost = 0.01

●

cpu_ind...
Planner Cost Evaluation Parameters
Continued
●

●

random_page_cost abstracts the cost of
reading a random page and seq_pa...
effective_cache_size
●

●

Sets the planner's assumption about the
effective size of the disk cache that is
available to a...
Cases
●

●

Some examples of situations where the
planner decisions can lead to a bad plan.
Some cases are not strictly re...
WITH
●

One of the classic cases.

●

WITH is an optimization barrier.
Why does WITH face performance
issues?
●

●

We have to understand how WITH is
processed by the planner.
The query planner...
Solution
●

Use JOINS instead.
Solution
●

Use JOINS instead.
Feature Alert!
●

The current implementation approach for
GROUPING SETS uses CTE based
architecture. Hence, it suffers fro...
Materialized Views
●

Joining a materialized view back to its parent
table is expensive, since the planner does not
know a...
Solution
●

Use intermediate tables to store the
intermediate results.
Aggregates
●

A function call like Agg(distinct) does not allow
the planner to use a HashAgg node, hence
leading to slower...
Solution
●

Use subqueries instead.
EXISTS and NOT EXISTS
●

Using EXISTS and NOT EXISTS in your
queries is a good idea and will generally lead
to better plan...
Reason
●

EXISTS and NOT EXISTS allow the planner to
use semi-joins and anti-joins, respectively.
These are much efficient...
Prepared Statements
●

●

Prepared statements are constantly a point of
query performance lags.
They perform much better n...
What is bad with prepared
statements?
●

●

Pre 9.2, user had a choice of using prepared
statements and using generic plan...
Pre 9.2
●

Prepared statements are good for workloads
where the selectivity of the queries is in a
small range. Widely var...
Post 9.2
●

The point to look at post 9.2 is if the planner is
overlooking a better alternate plan and
choosing a sub opti...
Example
●

●

●

select * from sometable where id = any ($1);
where $1 was a small array.
The generic plan for this is of ...
Example continued
●

●

It estimated a value of size of the array for the
generic plan which was much higher than the
actu...
Take away lesson
●

●

You need to be wary when using prepared
statements, pre 9.2 or post 9.2.
Prepared statements are mu...
Join Reordering
●

●
●

●

The planner may reorder the joins present in a
query based on some heuristics that it thinks
wi...
Gotcha!
●
●

●

We have a catch here, though.
Setting join_collapse_limit can lead to
suboptimal plans, but faster plannin...
Genetic Query Optimizer
●

●

The genetic query optimizer, or GEQO, is
used when the number of tables involved in a
join e...
Genetic Query Optimizer Continued
●

GEQO uses genetic algorithms to generate
query plans and evaluate their fitness,
gene...
GEQO Gotcha
●

GEQO is useful for generating plans when the
number of tables is high.

●

It is also useful for generating...
Note about join_collapse_limit and
geqo_threshold
●

Since the default value for geqo_threshold is
higher than default val...
Lack of planner heuristics
●

●

●

Refer to 'Planner Conceptual Error when
Joining a Subquery -- Outer Query Condition
no...
So, queries like...
●

SELECT *
FROM parent
INNER JOIN child ON child.parent_id = parent.id
LEFT JOIN (SELECT parent.id, parent.bin, COUNT(*) ...
Should be changed to...
●

SELECT *
FROM parent
INNER JOIN child ON child.parent_id = parent.id
LEFT JOIN (SELECT parent.id, parent.bin, COUNT(*) ...
NOT IN Vs. NOT EXISTS
●

SELECT ProductID, ProductName
FROM Northwind..Products p
WHERE p.ProductID NOT IN (
SELECT Produc...
NOT IN Vs. NOT EXISTS Continued
●

SELECT ProductID, ProductName
FROM Northwind..Products p
WHERE NOT EXISTS (
SELECT 1
FR...
What the planner does for NOT IN
●

●

NOT IN has two possible plans, a hashed
subplan (which requires a hashtable of the
...
What the planner does for NOT
EXISTS
●

●

●

The query planner generates an anti-join for
NOT EXISTS if the planner can s...
Comparison of performance
●

●

NOT IN can be faster when the subquery
result is small since that result set can be
hashed...
NOT IN Gotcha
●

●

NOT IN has some pretty well known issues
with NULL values. If a column has NULL
values, NOT IN might l...
So, what should we use?
●

●

The points discussed till now lead to the
conclusion that we should use NOT IN for
smaller d...
General Advice 1
●

Use tables to store intermediate results and
process over them wherever possible. This
allows for avoi...
General Advice 2
●

Whenever the planner's accuracy about plan
costs seems to be reducing, run ANALYZE.
This shall update ...
General Advice 3
●

When the accuracy of planner's costs doesnt
seem right, you can consider increasing the
statistics tar...
Future developments to the query
planner
●
●

●
●

The query planner still lacks some features.
Understanding of foreign k...
Case Study – Adding foreign keys
understanding to planner
●
●

In context of removal of inner joins.
The main issue identi...
Thank You!
Upcoming SlideShare
Loading in …5
×

Talk pg conf eu 2013

840 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
840
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Talk pg conf eu 2013

  1. 1. The PostgreSQL Query Planner Atri Sharma
  2. 2. Whats this talk based on? ● Research done by hackers like Robert Haas in the past, some mailing list readings, discussions with smart people etc...
  3. 3. Who's this talk for? ● Users of postgres, who want to know how query planner's heuristics can affect their queries.
  4. 4. What are we going to discuss? ● ● ● A small discussion about the Postgres query planner. Areas where problems arise in query planner. Some cases where the planner has some nifty points. Some of the cases may be well known. However, they are worthy of discussion here if only to see how the planner processes them internally.
  5. 5. A little about the PostgreSQL Query Planner ● Quoting Bruce Momjian: 'The optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution.'
  6. 6. Where can the problems arise in the query planner ● Row Count Estimates ● DBA Ideas ● Planner's Blocking points (What the planner cant do at this point of time)
  7. 7. Row Count Stats ● ● ● The backend essentially holds MCVs (Most common values) and a histogram of the distribution of the rest of the values. The number of MCVs stored is equal to a value known as statistics target. Incorrect or outdated counts can lead to bad selectivity estimates, leading to not so good plans
  8. 8. User Settings ● PostgreSQL depends quite some on the page values (seq_page_cost, random_page_cost) and others. Setting them to random values will cause the query planner to choose an inefficient plan over an efficient alternate plan.
  9. 9. Planner Not-Done-Yets ● Some queries are slow in execution because their optimization depends on a feature which is not yet present in the query planner.
  10. 10. Selectivity Estimate Issues ● ● ● Arbitrary estimates give a bad value Cross table correlations arent taken into consideration. Planner doesnt understand expressions like (a + 0 = a)
  11. 11. Settings Issues ● ● DBAs can use some settings to tune the query planner. Those settings can be harmful if set to incorrect values.
  12. 12. Planner Cost Evaluation Parameters ● seq_page_cost = 1.0 ● random_page_cost = 4.0 ● cpu_tuple_cost = 0.01 ● cpu_index_tuple_cost = 0.005 ● cpu_operator_cost = 0.0025 ● These parameters are used by the planner to estimate the costs of plans.
  13. 13. Planner Cost Evaluation Parameters Continued ● ● random_page_cost abstracts the cost of reading a random page and seq_page_cost abstacts the same for a sequential read. In real world scenarios, setting the random_page_cost to a lower value can help get better plans, since modern day disks have reduced the gap between random page read cost and sequential page read costs.
  14. 14. effective_cache_size ● ● Sets the planner's assumption about the effective size of the disk cache that is available to a single query. Should be set to 75% of the RAM.
  15. 15. Cases ● ● Some examples of situations where the planner decisions can lead to a bad plan. Some cases are not strictly related to the query planner, but are fun in general!
  16. 16. WITH ● One of the classic cases. ● WITH is an optimization barrier.
  17. 17. Why does WITH face performance issues? ● ● We have to understand how WITH is processed by the planner. The query planner intentionally runs the WITH part first. It divides the query into separate plans, which doesnt match up with the pull architecture of the executor.
  18. 18. Solution ● Use JOINS instead.
  19. 19. Solution ● Use JOINS instead.
  20. 20. Feature Alert! ● The current implementation approach for GROUPING SETS uses CTE based architecture. Hence, it suffers from the same problem of divided query plans. If this problem is solved, we could have an efficient implementation of GROUPING SETS in postgres, which would be very nice.
  21. 21. Materialized Views ● Joining a materialized view back to its parent table is expensive, since the planner does not know about any correlation between the two and the two tables are dependent on each other.
  22. 22. Solution ● Use intermediate tables to store the intermediate results.
  23. 23. Aggregates ● A function call like Agg(distinct) does not allow the planner to use a HashAgg node, hence leading to slower query plans.
  24. 24. Solution ● Use subqueries instead.
  25. 25. EXISTS and NOT EXISTS ● Using EXISTS and NOT EXISTS in your queries is a good idea and will generally lead to better plans. Use them instead of complete queries if you only want to test presence or absence of specific values.
  26. 26. Reason ● EXISTS and NOT EXISTS allow the planner to use semi-joins and anti-joins, respectively. These are much efficient than full joins for just testing the presence or absence of values.
  27. 27. Prepared Statements ● ● Prepared statements are constantly a point of query performance lags. They perform much better now, though, thanks to some recent work done.
  28. 28. What is bad with prepared statements? ● ● Pre 9.2, user had a choice of using prepared statements and using generic plans, or not use them and re plan every time. Post 9.2, user doesnt have that restriction, but the downside is that the user doesnt have a control on whether a prepared statement is replanned.
  29. 29. Pre 9.2 ● Prepared statements are good for workloads where the selectivity of the queries is in a small range. Widely varying selectivity values can lead to bad performance for prepared statements.
  30. 30. Post 9.2 ● The point to look at post 9.2 is if the planner is overlooking a better alternate plan and choosing a sub optimal plan.
  31. 31. Example ● ● ● select * from sometable where id = any ($1); where $1 was a small array. The generic plan for this is of course trivial and would have worked fine. But the planner has to estimate the costs of the generic plan without knowing how many elements are in the array.
  32. 32. Example continued ● ● It estimated a value of size of the array for the generic plan which was much higher than the actual size of the array. What that meant was that the generic plan was about twice the cost of the custom plan, even though it was the exact same plan.
  33. 33. Take away lesson ● ● You need to be wary when using prepared statements, pre 9.2 or post 9.2. Prepared statements are much better than before, but some surprise may still turn up.
  34. 34. Join Reordering ● ● ● ● The planner may reorder the joins present in a query based on some heuristics that it thinks will lead to better plans. Generally, this is a good idea. We should not assume that we have a better idea about the optimal join order. Still, if we are inclined to do so, we have a setting known as join_collapse_limit
  35. 35. Gotcha! ● ● ● We have a catch here, though. Setting join_collapse_limit can lead to suboptimal plans, but faster planning time durations. If you dont care as much about suboptimal plans as much you care about faster planner speeds, set join_collapse_limit to a lower value.
  36. 36. Genetic Query Optimizer ● ● The genetic query optimizer, or GEQO, is used when the number of tables involved in a join exceed a specific limit. GEQO is used when searching the entire search space for executing the join is not feasible.
  37. 37. Genetic Query Optimizer Continued ● GEQO uses genetic algorithms to generate query plans and evaluate their fitness, generating better plans in each successive population.
  38. 38. GEQO Gotcha ● GEQO is useful for generating plans when the number of tables is high. ● It is also useful for generating plans quickly. ● However, it can lead to suboptimal plans. ● Lower geqo_threshold if you care much about the planner speed rather than the optimality of the query plan.
  39. 39. Note about join_collapse_limit and geqo_threshold ● Since the default value for geqo_threshold is higher than default value of join_collapse_limit, GEQO isnt used by default for explicit join syntax.
  40. 40. Lack of planner heuristics ● ● ● Refer to 'Planner Conceptual Error when Joining a Subquery -- Outer Query Condition not Pulled Into Subquery' on performance list It tells us that the planner does not understand that 'a = b' and (b = c OR b = d) is equivalent to (a = c OR a = d) Planner lacks such heuristics. Another example is SELECT * from foo WHERE (foo.a + 0 = foo.a);
  41. 41. So, queries like...
  42. 42. ● SELECT * FROM parent INNER JOIN child ON child.parent_id = parent.id LEFT JOIN (SELECT parent.id, parent.bin, COUNT(*) AS c FROM parent INNER JOIN child ON child.parent_id = parent.id WHERE child.data > 25 GROUP BY 1, 2) agg1 ON agg1.id = parent.id AND agg1.bin = parent.bin WHERE parent.bin IN (1, 2) AND agg1.c >= 3;
  43. 43. Should be changed to...
  44. 44. ● SELECT * FROM parent INNER JOIN child ON child.parent_id = parent.id LEFT JOIN (SELECT parent.id, parent.bin, COUNT(*) AS c FROM parent INNER JOIN child ON child.parent_id = parent.id WHERE child.data > 25 GROUP BY 1, 2) agg1 ON agg1.id = parent.id AND agg1.bin = parent.bin WHERE parent.bin = 1 AND agg1.c >= 3; Notice the 'bin = 1' here
  45. 45. NOT IN Vs. NOT EXISTS ● SELECT ProductID, ProductName FROM Northwind..Products p WHERE p.ProductID NOT IN ( SELECT ProductID FROM Northwind..[Order Details])
  46. 46. NOT IN Vs. NOT EXISTS Continued ● SELECT ProductID, ProductName FROM Northwind..Products p WHERE NOT EXISTS ( SELECT 1 FROM Northwind..[Order Details] od WHERE p.ProductId = od.ProductId)
  47. 47. What the planner does for NOT IN ● ● NOT IN has two possible plans, a hashed subplan (which requires a hashtable of the entire subselect, so it's only done when that would fit in work_mem) If the subselect does not fit into work_mem, a plain subplan is made, which is O(N^2).
  48. 48. What the planner does for NOT EXISTS ● ● ● The query planner generates an anti-join for NOT EXISTS if the planner can sufficiently disentangle a join clause from the conditions in the subquery. If not, then the not exists remains as a (correlated) subplan. A correlated subplan is one which has to be re-run for every row.
  49. 49. Comparison of performance ● ● NOT IN can be faster when the subquery result is small since that result set can be hashed, leading to better performance. Then as your data gets larger, it might exceed work_mem, and without warning, your query might get very slow.
  50. 50. NOT IN Gotcha ● ● NOT IN has some pretty well known issues with NULL values. If a column has NULL values, NOT IN might lead to more work, hence leading to even more drop in the performance of the query. This is a quite constraining, since we may have NULLs in our columns already or may have them in the future.
  51. 51. So, what should we use? ● ● The points discussed till now lead to the conclusion that we should use NOT IN for smaller data sets, and NOT EXISTS for larger data sets. However, since NOT IN has the constraint of NULLs, we should generally prefer NOT EXISTS in all cases unless our case specifically needs NOT IN.
  52. 52. General Advice 1 ● Use tables to store intermediate results and process over them wherever possible. This allows for avoiding pitfalls such as materialized view joins, UNION pitfalls, DISTINCT pitfalls among others.
  53. 53. General Advice 2 ● Whenever the planner's accuracy about plan costs seems to be reducing, run ANALYZE. This shall update the planner statistics and may lead to better results.
  54. 54. General Advice 3 ● When the accuracy of planner's costs doesnt seem right, you can consider increasing the statistics target (No. Of MCV values stored) by using the following command: ALTER COLUMN <column name> SET STATISTICS <value>
  55. 55. Future developments to the query planner ● ● ● ● The query planner still lacks some features. Understanding of foreign keys, inner join removal, more heuristics. Hopefully, we will have them all someday. CTE planning needs to be improved and done in line with postgres executor's data pull model.
  56. 56. Case Study – Adding foreign keys understanding to planner ● ● In context of removal of inner joins. The main issue identified is that if a row is deleted, in the time duration from the deletion of the row and running of the FK trigger, all plans generated and dependent on that row's foreign key will be invalid.
  57. 57. Thank You!

×