Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PostgreSql query planning and tuning

1,866 views

Published on

Slides from the PostgreSQL meetup in brighton. How postgresql executes a query and performance tips.

Published in: Technology

PostgreSql query planning and tuning

  1. 1. PostgreSQL query planning and tuning Federico Campoli 03 Mar 2016 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 1 / 47
  2. 2. Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 2 / 47
  3. 3. Jargon Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 3 / 47
  4. 4. Jargon Jargon source https://commons.wikimedia.org/wiki/File:Klingon Empire Flag.svg Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 4 / 47
  5. 5. Jargon Jargon Jargon OID Object ID, 4 bytes integer. Used to identify the system objects (tables, indices etc) class any relational object, table, index, view, sequence... attribute the table fields execution plan the steps required for executing a query plan nodes execution plan’s steps CBO cost based optimizer cost arbitrary value used to determine a score for the plan nodes Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 5 / 47
  6. 6. I find your lack of plan disturbing Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 6 / 47
  7. 7. I find your lack of plan disturbing I find your lack of plan disturbing source http://apbialek.deviantart.com/art/Darth-Vader-171921375 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 7 / 47
  8. 8. I find your lack of plan disturbing Query stages The query execution requires four stages Syntax validation Query tree generation Plan estimation Execution Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 8 / 47
  9. 9. I find your lack of plan disturbing Syntax validation Syntax validation The query parser validates the query syntax using fixed rules. Any error at this step will cause the execution to stop, returning a syntax error. This early stage doesn’t requires a system catalogue access. The parser returns a normalised parse tree for the next step. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 9 / 47
  10. 10. I find your lack of plan disturbing Query tree generation The query tree The query parser in the second stage look up the system catalogue and translates the parse tree into the query tree. The query tree is the query’s logical representation where any object involved is unique and described using the object id mapping. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 10 / 47
  11. 11. I find your lack of plan disturbing Query tree generation The query tree The query tree Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 11 / 47
  12. 12. I find your lack of plan disturbing Query tree generation The planner The planner stage The next stage is the query planner. The parser sends the generated query tree to the planner. The query planner reads the tree and generates all the possible execution plans. The planner, using the internal statistics,determines the estimated costs for each plan. The execution plan with the minimum estimated cost is sent to the executor. Old or missing statistics will result not-optimal execution plans and therefore slow queries. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 12 / 47
  13. 13. I find your lack of plan disturbing Query tree generation The executor The executor The executor performs the plan nodes in the execution plan generated by the planner. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 13 / 47
  14. 14. I find your lack of plan disturbing Query tree generation The workflow Figure : The query stages Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 14 / 47
  15. 15. I love it when a plan comes together Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 15 / 47
  16. 16. I love it when a plan comes together I love it when a plan comes together source http://clipperdata.com/blogpost/i-love-it-when-a-plan-comes-together/i- love-it-when-a-plan-comes-together/ Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 16 / 47
  17. 17. I love it when a plan comes together The plan nodes Scan nodes used by the executor to retrieve data from the relations Join nodes used by the executor to perform joins of the data streams Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 17 / 47
  18. 18. I love it when a plan comes together seq scan seq scan: reads sequentially the table and discards the unmatched rows. The output is a data stream. Returns unsorted rows. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 18 / 47
  19. 19. I love it when a plan comes together index scan index scan: read the index tree with random disk read and gets the heap blocks pointed by the index. Returns sorted rows. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 19 / 47
  20. 20. I love it when a plan comes together index only scan index only scan: read the index tree with random disk read and returns the data without accessing the heap page. Returns sorted rows. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 20 / 47
  21. 21. I love it when a plan comes together bitmap index/heap scan bitmap index/heap scan: read the index sequentially generating a bitmap used to recheck on heap pages. It’s a good compromise between seq scan and a full index scan. Returns unsorted rows. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 21 / 47
  22. 22. I love it when a plan comes together sort sort : reads the rows and returns them in an ordered way like in queries with an ORDER BY Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 22 / 47
  23. 23. I love it when a plan comes together nested loop nested loop join: The right relation is scanned once for every row found in the left relation. This strategy is easy to implement but can be very time consuming. However, if the right relation can be scanned with an index scan, this can be a good strategy. It is possible to use values from the current row of the left relation as keys for the index scan of the right. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 23 / 47
  24. 24. I love it when a plan comes together hash join hash join: the right relation is first scanned and loaded into a hash table, using its join attributes as hash keys. Next the left relation is scanned and the appropriate values of every row found are used as hash keys to locate the matching rows in the table. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 24 / 47
  25. 25. I love it when a plan comes together merge join merge join: Each relation is sorted on the join attributes before the join starts. Then the two relations are scanned in parallel, and matching rows are combined to form join rows. This kind of join is more attractive because each relation has to be scanned only once. The required sorting might be achieved either by an explicit sort step, or by scanning the relation in the proper order using an index on the join key. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 25 / 47
  26. 26. EXPLAIN! EXPLAIN! Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 26 / 47
  27. 27. EXPLAIN! EXPLAIN! EXPLAIN! EXPLAIN! source http://keepcalmandwatchdoctorwho.tumblr.com/post/5450959631/the- daleks-personally-i-like-it-when-they-shout Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 27 / 47
  28. 28. EXPLAIN! EXPLAIN! EXPLAIN Prepending EXPLAIN to any query will display the query’s estimated execution plan . Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 28 / 47
  29. 29. EXPLAIN! EXPLAIN! EXPLAIN Prepending EXPLAIN to any query will display the query’s estimated execution plan . The ANALYZE clause executes the query, discards the results and returns the real execution plan. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 28 / 47
  30. 30. EXPLAIN! EXPLAIN! EXPLAIN Prepending EXPLAIN to any query will display the query’s estimated execution plan . The ANALYZE clause executes the query, discards the results and returns the real execution plan. Using EXPLAIN ANALYZE with the DML queries will change the data. It is safe to wrap the EXPLAIN ANALYZE between BEGIN; and ROLLBACK; Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 28 / 47
  31. 31. EXPLAIN! EXPLAIN! Explain in action For our example we’ll create a test table with two fields, a serial and character varying. test =# CREATE TABLE t_test ( i_id serial , v_value character varying (50) ) ; CREATE TABLE Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 29 / 47
  32. 32. EXPLAIN! EXPLAIN! Explain in action Now let’s add some rows to our table Listing 1: Insert in table test =# INSERT INTO t_test (v_value) SELECT v_value FROM ( SELECT generate_series (1 ,1000) as i_cnt , md5(random ():: text) as v_value ) t_gen ; INSERT 0 1000 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 30 / 47
  33. 33. EXPLAIN! EXPLAIN! Explain in action Let’s generate the estimated plan for one row result test =# EXPLAIN SELECT * FROM t_test WHERE i_id =20; QUERY PLAN -- ------------------------------------------------------- Seq Scan on t_test (cost =0.00..16.62 rows =3 width =122) Filter: (i_id = 20) (2 rows) The cost is just an arbitrary value The values after the cost are the the startup and total cost The start up cost is the cost to deliver the first row to the next step The total cost is cost to deliver all the rows to the next step The value in rows is the planner’s estimation for the total rows returned by the plan node The value in width is the estimated average row width in bytes Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 31 / 47
  34. 34. EXPLAIN! EXPLAIN! EXPLAIN ANALYZE Now let’s generate the real execution plan for one row result test =# EXPLAIN ANALYZE SELECT * FROM t_test WHERE i_id =20; QUERY PLAN -- --------------------------------------------------------------------------- Seq Scan on t_test (cost =0.00..21.50 rows =1 width =37) (actual time =0.022..0.262 rows =1 loops =1) Filter: (i_id = 20) Rows Removed by Filter: 999 Planning time: 0.066 ms Execution time: 0.286 ms (5 rows) The values in actual time are the time, in milliseconds, required for the startup and total cost The value in rows is the number of rows returned by the step The value loops value is the number of times the step is executed On the bottom we have the planning time and the total execution time Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 32 / 47
  35. 35. EXPLAIN! EXPLAIN! Indices Let’s add an index on the i id field test =# CREATE INDEX idx_i_id ON t_test (i_id); CREATE INDEX Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 33 / 47
  36. 36. EXPLAIN! EXPLAIN! Indices test =# EXPLAIN ANALYZE SELECT * FROM t_test WHERE i_id =20; QUERY PLAN -- --------------------------------------------------------------------------- Index Scan using idx_i_id on t_test (cost =0.28..8.29 rows =1 width =37) ( actual time =0.035..0.036 rows =1 loops =1) Index Cond: (i_id = 20) Planning time: 0.252 ms Execution time: 0.058 ms (4 rows) The query is several times faster. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 34 / 47
  37. 37. EXPLAIN! EXPLAIN! Controlling the planner The cost based optimiser becomes cleverer for each major release. For example, if the query’s filter returns almost the entire table, the database will choose the sequential scan which is by default 4 times cheaper than a full index scan. test =# EXPLAIN ANALYZE SELECT * FROM t_test WHERE i_id >2; QUERY PLAN -- --------------------------------------------------------------------------- Seq Scan on t_test (cost =0.00..21.50 rows =999 width =37) (actual time =0.012..0.467 rows =998 loops =1) Filter: (i_id > 2) Rows Removed by Filter: 2 Planning time: 0.142 ms Execution time: 0.652 ms (5 rows) Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 35 / 47
  38. 38. EXPLAIN! EXPLAIN! Controlling the planner test =# SET enable_seqscan =’off’; SET test =# EXPLAIN ANALYZE SELECT * FROM t_test WHERE i_id >2; QUERY PLAN -- --------------------------------------------------------------------------- Index Scan using idx_i_id on t_test (cost =0.28..49.76 rows =999 width =37) ( actual time =0.029..0.544 rows =998 loops =1) Index Cond: (i_id > 2) Planning time: 0.145 ms Execution time: 0.741 ms (4 rows) Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 36 / 47
  39. 39. I fight for the users! Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 37 / 47
  40. 40. I fight for the users! I fight for the users! source http://tron.wikia.com/wiki/Tron Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 38 / 47
  41. 41. I fight for the users! ANALYZE ANALYZE ANALYZE gather statistics runs random reads on the tables The statistics are stored in the pg statistic system table The view pg stats translates the statistics in human readable format The parameter default statistics target sets the limit for the random read The statistic target can be fine tuned per column Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 39 / 47
  42. 42. I fight for the users! Table statistics Before starting the performance analysis check the statistics are up to date querying the view pg stat all tables test =# SELECT * FROM pg_stat_all_tables WHERE relname=’t_test ’; -[ RECORD 1 ]-- -----+------------------------------ relid | 16546 schemaname | public relname | t_test seq_scan | 6 seq_tup_read | 5000 idx_scan | 7 idx_tup_fetch | 2980 n_tup_ins | 1000 n_tup_upd | 0 n_tup_del | 0 n_tup_hot_upd | 0 n_live_tup | 1000 n_dead_tup | 0 n_mod_since_analyze | 0 last_vacuum | last_autovacuum | last_analyze | last_autoanalyze | 2015 -09 -24 04:51:51.622906+00 vacuum_count | 0 autovacuum_count | 0 analyze_count | 0 autoanalyze_count | 1 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 40 / 47
  43. 43. I fight for the users! index statistics Before starting the performance analysis check the indices are used querying the view pg stat all indexes test =# SELECT * FROM pg_stat_all_indexes WHERE relname=’t_test ’; -[ RECORD 1 ]-+ --------- relid | 16546 indexrelid | 16552 schemaname | public relname | t_test indexrelname | idx_i_id idx_scan | 7 idx_tup_read | 2980 idx_tup_fetch | 2980 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 41 / 47
  44. 44. I fight for the users! Controlling the planner enable bitmapscan enable hashagg enable hashjoin enable indexscan enable indexonlyscan enable material enable mergejoin enable nestloop enable seqscan enable sort enable tidscan Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 42 / 47
  45. 45. Wrap up Table of contents 1 Jargon 2 I find your lack of plan disturbing 3 I love it when a plan comes together 4 EXPLAIN! EXPLAIN! 5 I fight for the users! 6 Wrap up Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 43 / 47
  46. 46. Wrap up Questions Questions? Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 44 / 47
  47. 47. Wrap up Boring legal stuff The join nodes descriptions are excerpts from the PostgreSQL 9.4 on line manual. Copyright by PostgreSQL Global Development Group. http://www.postgresql.org/ The scan node images are derived from the pgadmin3 scan nodes. Copyright by pgadmin development group. http://www.pgadmin.org/ All the images copyright is owned by the respective authors. The sources the author’s attribution is provided with a link alongside with image. Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 45 / 47
  48. 48. Wrap up Contacts and license Twitter: 4thdoctor scarf Blog:http://www.pgdba.co.uk Brighton PostgreSQL Meetup: http://www.meetup.com/Brighton-PostgreSQL-Meetup/ This document is distributed under the terms of the Creative Commons Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 46 / 47
  49. 49. Wrap up PostgreSQL query planning and tuning Federico Campoli 03 Mar 2016 Federico Campoli PostgreSQL query planning and tuning 03 Mar 2016 47 / 47

×