I’m a Postgres person. Period. After talking to many Rails developers about their application performance, I realized many performance issues can be solved by understanding your database a bit better. So I thought I’d share the statistics Postgres captures for you and how you can use them to find slow queries, un-used indexes, or tables which are not getting vacuumed correctly. This talk will cover Postgres tools and tips for the above, including pgstatstatements, useful catalog tables, and recently added Postgres features such as CREATE STATISTICS.
Optimizing your app by understanding your Postgres | RailsConf 2019 | Samay Sharma
1. Samay Sharma | RailsConf 2019
Optimizing your app by understanding
your Postgres database
Samay Sharma
Solutions Engineering Manager
RailsConf 2019 | Minneapolis | April 2019
2. Samay Sharma | RailsConf 20192
• Citus - Open Source Extension to
Scale out Postgres
• Manage a team of Solutions
Engineers
• Work with clients
• Fun(?) fact : Recently got married J
4. Samay Sharma | RailsConf 2019
Challenge: Relating issues to the database
• Problems you’re facing from an application perspective
• Tied to database
• How do I find the root cause?
• How do I solve it?
4
5. Samay Sharma | RailsConf 2019
Solution: Using Postgres’ statistics
• Postgres captures, uses and exposes many statistics.
• Monitoring statistics – Statistics about system activity
• Query planner statistics – Statistics used by the planner to choose the right query plan.
• Server Administrator statistics – Statistics about replication, size of database, tables, etc.
• It’s important to know how to use them.
5
6. Samay Sharma | RailsConf 2019
Problem: My application is slow
• Symptoms: Application users are seeing slow query performance.
• Your monitoring tool shows you that most of the time is going in database
calls.
• Your page is making 100s of database calls, you don’t know which query
is slow.
6
7. Samay Sharma | RailsConf 2019
Cache hit ratio
• Gives an idea of how much of your data is coming from the PostgreSQL
buffercache.
• For transactional apps, want to keep if >95%, ideally around 99%.
7
8. Samay Sharma | RailsConf 2019
How do I measure?
SELECT
sum(heap_blks_read) as heap_read,
sum(heap_blks_hit) as heap_hit,
sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM
pg_statio_user_tables;
8
10. Samay Sharma | RailsConf 2019
Digging further
• If cache hit ratio is low, that could mean a lot of things.
• Maybe you have a lot of bloat.
• Maybe autovacuum is not tuned well enough.
• Maybe your queries are not optimized and you are doing a lot of sequential scans.
• Maybe you have a lot of unused indexes.
• Or, you actually have less resources and you need to scale out with Citus.
10
12. Samay Sharma | RailsConf 2019
Uses for pg_stat_database
• Find out rows fetched vs returned by queries to the database
• Find out the insert / update / delete ratio for your database
• Total number of transactions executed / throughput
12
14. Samay Sharma | RailsConf 2019
Uses for pg_stat_user_tables
• Approximate number of live / dead tuples (rows) in a table
• Find out if the table is an insert / update / delete heavy table.
• Find out if your table is being auto vacuumed sufficiently or not.
• Find out if most scans on the table are sequential scans or index scans.
14
16. Samay Sharma | RailsConf 2019
Uses for pg_stat_user_indexes
• Figuring out unused indexes
• Ratio of index rows read vs fetched
16
17. Samay Sharma | RailsConf 2019
Back to cache hit rate
• Understand why you had a bad hit rate
• You see database patterns which don’t align with the application – Dive deeper into them
• Is there a lot of bloat, are some tables not vacuumed often enough – Tune autovacuum
• Are you doing a lot of sequential scans - Create indexes
• Maybe you have a lot of unnecessary indexes – Delete them
17
18. Samay Sharma | RailsConf 2019
Generic vs specific optimizations
• Follow above practices to make generic optimizations when your
database is slow.
• How do you speed up a particular webpage?
• Or how do you find out what specific queries are slow?
18
19. Samay Sharma | RailsConf 2019
How do I find what queries are slow?
19
20. Samay Sharma | RailsConf 2019
What is `pg_stat_statements` ?
• PostgreSQL extension which tracks execution statistics of all SQL
statements executed by a server
• Normalized so similar queries grouped together
• Exposed to user w/ view named `pg_stat_statements`
20
21. Samay Sharma | RailsConf 2019
`pg_stat_statements` view
• Query: User_id, db_id, query_id, query text, # of times executed
• Timing: min_time, max_time, total_time, mean_time, stddev_time
• Shared blocks hit, read, written, dirtied
• Local blocks hit, read, written, dirtied
• I/O times: total time spent in reading & writing blocks.
• Enable track_io_timing to get these, otherwise will be empty
21
22. Samay Sharma | RailsConf 2019
Examples
• Top 10 time consuming queries
select query, total_time from pg_stat_statements ORDER BY total_time DESC limit 10
• Top 10 queries spending most time on disk reads
select query, blk_read_time from pg_stat_statements ORDER BY blk_read_time DESC limit 10
• Top 10 most common queries
select query, calls from pg_stat_statements ORDER BY calls DESC limit 10
22
23. Samay Sharma | RailsConf 2019
SELECT rolname,
calls,
total_time,
mean_time,
max_time,
stddev_time,
rows,
regexp_replace(query, '[ tn]+', ' ', 'g') AS query_text
FROM pg_stat_statements
JOIN pg_roles r ON r.oid = userid
WHERE calls > 100
AND rolname NOT LIKE '%backup'
ORDER BY mean_time DESC
LIMIT 15;
-[ RECORD 1] ----------------------------------------------------------------------------------------------------
- ---------------------------------------------------------------------------------------------------------------
---
rolname | citus
calls | 141
total_time | 121404.290523
mean_time | 861.023337042553
max_time | 3718.828292
stddev_time | 913.641891936023
rows | 846
query_text | SELECT ads.campaign_id, COUNT(*) FROM ads JOIN impressions i ON (ads.id = ad_id AND ads.company_id
= i.company_id) WHERE ads.campaign_id IN ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14) AND ads.company_id =
$15 AND seen_at > now()::date GROUP BY ads.campaign_id
Source: http://geekmonkey.org/2017/11/optimizing-postgresql-queries-using-pg_stat_statements-and-pg_buffercache/
24. Samay Sharma | RailsConf 2019
EXPLAIN ANALYZE and tune them!
EXPLAIN ANALYZE SELECT * FROM tbl where col1 = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Seq Scan on tbl (cost=0.00..169247.80 rows=9584 width=8) (actual time=0.641..622.851 rows=10000 loops=1)
Filter: (col1 = 1)
Rows Removed by Filter: 9990000
Planning time: 0.051 ms
Execution time: 623.185 ms
(5 rows)
24
25. Samay Sharma | RailsConf 2019
Interesting things about EXPLAIN
EXPLAIN ANALYZE SELECT * FROM tbl where col1 = 1 and col2 = 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------
Seq Scan on tbl (cost=0.00..194248.69 rows=100 width=8) (actual time=0.640..630.130 rows=10000 loops=1)
Filter: ((col1 = 1) AND (col2 = 0))
Rows Removed by Filter: 9990000
Planning time: 0.072 ms
Execution time: 630.467 ms
(5 rows)
25
26. Samay Sharma | RailsConf 2019
Estimates vs actual
• Postgres tells you what it thinks even in the EXPLAIN plans.
• Sometimes reason for slow queries might just be that Postgres has
incorrect statistics. Notice and fix those mismatches.
• Try to ANALYZE with larger default_statistics_target.
26
27. Samay Sharma | RailsConf 2019
Multi-column statistics
• Postgres only captures only single column statistics.
• Sometimes, those are not enough because they can’t cover correlations.
• But, you can tell PG two columns are related.
27
28. Samay Sharma | RailsConf 2019
CREATE STATISTICS
CREATE STATISTICS s1 (dependencies) on col1, col2 from tbl;
ANALYZE tbl;
EXPLAIN ANALYZE SELECT * FROM tbl where col1 = 1 and col2 = 0;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Seq Scan on tbl (cost=0.00..194247.76 rows=9584 width=8) (actual time=0.638..629.741 rows=10000 loops=1)
Filter: ((col1 = 1) AND (col2 = 0))
Rows Removed by Filter: 9990000
Planning time: 0.115 ms
Execution time: 630.076 ms
(5 rows)
28
30. Samay Sharma | RailsConf 2019
Summary of specific query optimization
• Find the right queries to optimize using pg_stat_statements
• Use EXPLAIN ANALYZE and tune the slowest parts
• Look at differences between estimated and actual and find ways to fix
them.
30
33. Samay Sharma | RailsConf 2019
Uses for pg_stats
• What is the null ratio for a particular column?
• Find the approximate number of distinct values for a column.
• What are the most common values for a column and what are the
approximate number of rows with those values?
33
34. Samay Sharma | RailsConf 2019
Approximate count
• Do you know how to get a very fast approximate count of rows in a table?
SELECT
relname, reltuples
FROM
pg_class
WHERE
relname = 'tbl’;
relname | reltuples
---------+-----------
tbl | 1e+07
(1 row)
34
35. Samay Sharma | RailsConf 2019
Summary
• Postgres exposes a lot of stats to you. In this talk, we covered mainly
performance.
• Pg_stat_* tables to find index, table, IO, vacuum statistics.
• Enable pg_stat_statements to find out your slow queries.
35
36. Samay Sharma | RailsConf 2019
Summary
• Use EXPLAIN ANALYZE or EXPLAIN (ANALYZE, BUFFERS) to see what
the planner is actually doing.
• Use pg_stats and pg_class to get good approximations and understand
postgres’ understanding of your data.
36