2022 COSCUP SPEED UP! 奔跑吧大象! Let's speed up your PostgreSQL services!

1. SPEED UP! 奔跑吧大象!
Let's speed up your PostgreSQL services!
Speaker: 林宗禧 @ COSCUP 2022
Taiwan PostgreSQL User Group
1

2. About
林宗禧
• PostgreSQL愛好者(2012-)
• PostgreSQL推廣者(2017-)
How to use ?
• 以前: 開發FDW套件 (C, Python都有)
• 後來: 到處整合PG的應用
• 推 Industry 4.0，讓業主不經意的導入 PG
• 推 Smart City Solutions ，拿PG做基礎
2022/7/30 2
Taiwan PostgreSQL User Group

3. 為何討論這個Topic?
(1) 避免緊急時，才發現有地方該調未調…
(2) 硬體優化不討論，畢竟各公司資源不同
(3) 本篇從架構設計、查詢優化、設定面進行討論
Taiwan PostgreSQL User Group 3
7/30/2022

4. Taiwan PostgreSQL User Group 4
2022/7/30
Agenda
01. 架構設計
02. 查詢優化
03. 參數配置

5. Taiwan PostgreSQL User Group 5
2022/7/30
Agenda
01. 架構設計
02. 查詢優化
03. 參數配置

6. 01. 架構設計
1) 用 Connection Pool 管理加速 - PgBouncer
• https://www.pgbouncer.org/
• 1.15 - Nov 19, 2020
• 輕量級的 connection pool
• 提供重複使用session機制
• PG 9.5 沒有自動於時斷線機制
• PG 9.6 Transaction Timeout 參數
Taiwan PostgreSQL User Group 6
2022/7/30
https://mlog.club/article/2987957

7. 01. 架構設計
1) 用 Connection Pool 管理加速 – PgBouncer (cont.)
Taiwan PostgreSQL User Group 7
2022/7/30
https://www.percona.com/blog/2018/06/27/scaling-postgresql-with-
pgbouncer-you-may-need-a-connection-pooler-sooner-than-you-expect/

8. 01. 架構設計
2) 用讀寫分離加速：Pgpool – II
• https://pgpool.net
• 4.2.3 – May 20, 2021
• Connection Pooling
• Load Balance
• Replication
• Parallel Query
• pgpoolAdmin
Taiwan PostgreSQL User Group 8
2022/7/30
https://www.pgpool.net/docs/pgpool-II-4.1.0/en/html/example-cluster.html

9. 01. 架構設計
2) 用讀寫分離加速：Pgpool – II (cont.)
Taiwan PostgreSQL User Group 9
2022/7/30
https://cloud.tencent.com/document/product/409/49547
https://blog.pythian.com/comparing-pgpool-ii-
and-pgbouncer/

10. 01. 架構設計
3) 依場景挑選儲存核心
• 如時序型資料處理: TimescaleDB (Time Series PostgreSQL)
• JOIN較多使用Columnar Storage: citusdata 的 cstore_fdw
Taiwan PostgreSQL User Group 10
2022/7/30
https://m.blog.naver.com/geartec82/221987422533
https://www.timescale.com/blog/timescaledb-vs-6a696248104e/

11. Taiwan PostgreSQL User Group 11
2022/7/30
Agenda
01. 架構設計
02. 查詢優化
03. 參數配置

12. 02. 查詢優化
a) 透過 pg_stat_activity 找到慢查詢
• 語法
• 選擇 datname,usename,
current_query 執行
• 歡迎參與 PG 正體中文手冊
https://docs.postgresql.tw/server-administration/monitoring-database-
activity/the-statistics-collector#27.2.3.-pg_stat_activity
Taiwan PostgreSQL User Group 12
7/30/2022
select * from pg_stat_activity;
postgres=# SELECT
datname,usename,current_query FROM
pg_stat_activity ;
datname | usename | current_query
----------+----------+---------------------
-----------------------------------------
postgres | postgres | SELECT
datname,usename,current_query FROM
pg_stat_activity ;
postgres | joe | <IDLE> (2 rows)

13. 02. 查詢優化
a) 透過 pg_stat_activity 找到慢查詢
• 檢視耗時較長的查詢
• 如右例： runtime 34s
Taiwan PostgreSQL User Group 13
7/30/2022
select
current_timestamp - query_start as
runtime, datname, usename,
current_query from
pg_stat_activity
where current_query != '<IDLE>'
order by 1 desc;

14. 02. 查詢優化
b) 異常SQL診斷及修復
指令執行很久沒結果，可檢查該指令還在執行中還是已經被block
(1)找出被Block的SQL (2)找出被Block進一步資訊
(3)透過cancel取消
Taiwan PostgreSQL User Group 14
7/30/2022
SELECT
datname,usename,current_query
FROM pg_stat_activity
WHERE waiting;
SELECT pg_cancel_backend(pid)

15. 02. 查詢優化
c) 使用 explain 解析查詢
透過explain 找出查詢成本處，避掉Full Table Scan
Taiwan PostgreSQL User Group 15
7/30/2022
explain select * from demotable;
explain analyze select * from demotable;
https://medium.com/@sandeep.patle/explain-the-query-in-postgresql-dab3b321da54

16. 02. 查詢優化
d) 使用 Parallel Queries 查詢
Taiwan PostgreSQL User Group 16
7/30/2022
https://www.2ndquadrant.com/en/blog/how-will-postgres-xl-
exploit-the-parallel-query-capabilities-of-postgresql-9-6/

17. Taiwan PostgreSQL User Group 17
7/30/2022
d) 使用 Parallel Queries 查詢 (cont.)
MPP架構如 GreenplumDB 、 Postgres-XL
https://www.linkedin.com/pulse/what-greenplum-
database-intro-big-data-kristi-anderson
https://zhuanlan.zhihu.com/p/26391728
補充 01. 架構設計

18. 02. 查詢優化
d) 設定 materialized view
Taiwan PostgreSQL User Group 18
7/30/2022
https://docs.microsoft.com/zh-tw/azure/architecture/patterns/_images/materialized-view-pattern-diagram.png
 在COSCUP 2018 的「What‘s
new in PostgreSQL 11 ?」找得
到

19. 02. 查詢優化
e) 針對 INSERT 優化，TimescaleDB提出下列做法
a. 適度使用索引
b. 重新考慮外鍵約束
c. 避免不必要的 UNIQUE KEY
d. 為 WAL 與資料使用單獨的硬碟
e. 使用高性能磁盤: SSD
f. 使用平行寫入
g. 批量插入(Batch INSERT)
h. 正確配置shared_buffers
i. 在 Linux 主機上使用 TimescaleDB Docker Image
j. 以鬆散的時間順序寫入數據
k. 避免“太大”的Chunk (TimescaleDB)
l. 避免太多或太小的Chunk (TimescaleDB)
Taiwan PostgreSQL User Group 19
7/30/2022
https://www.timescale.com/blog/13-tips-to-
improve-postgresql-insert-performance/
shared_buffers 的合理起始值是系統記憶體的 25％

20. Taiwan PostgreSQL User Group 20
2022/7/30
Agenda
01. 架構設計
02. 查詢優化
03. 參數配置

21. 03. 參數配置
Taiwan PostgreSQL User Group 21
7/30/2022
常用參數配置
Background Writer配置：過度設定會有不必要的IO
優化WAL的相關參數
maintenance_work_mem autovacuum_work_mem

22. 03. 參數配置
Taiwan PostgreSQL User Group 22
2022/7/30
https://github.com/le0pard/pgtune
https://pgtune.leopard.in.ua/
調校的參考工具 PGTune
Tuning PostgreSQL config by your hardware

23. 03. 參數配置
Taiwan PostgreSQL User Group 23
2022/7/30
# DB Version: 14
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 24 GB
# CPUs num: 2
# Connections num: 1000
# Data Storage: ssd
max_connections = 1000
shared_buffers = 6GB
effective_cache_size = 18GB
maintenance_work_mem = 1536MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 6291kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 2
max_parallel_workers_per_gather = 1
max_parallel_workers = 2
max_parallel_maintenance_workers = 1
調校的參考工具 PGTune (cont.)

24. 03. 參數配置
Taiwan PostgreSQL User Group 24
2022/7/30
# WARNING
# this tool not being optimal
# for very high memory systems
# DB Version: 14
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 128 GB
# CPUs num: 8
# Connections num: 1000
# Data Storage: ssd
max_connections = 1000
shared_buffers = 32GB
effective_cache_size = 96GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
# DB Version: 14
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 24 GB
# CPUs num: 2
# Connections num: 1000
# Data Storage: ssd
ALTER SYSTEM SET
max_connections = '1000';
ALTER SYSTEM SET
shared_buffers = '6GB';
ALTER SYSTEM SET
effective_cache_size = '18GB';
ALTER SYSTEM SET
maintenance_work_mem = '1536MB';
ALTER SYSTEM SET
checkpoint_completion_target = '0.9';
ALTER SYSTEM SET
wal_buffers = '16MB';
ALTER SYSTEM SET
default_statistics_target = '100';
建議用來評估
RAM 100GB內
調校的參考工具 PGTune (cont.)

25. 持續貢獻
DB-engine ranking
• https://db-engines.com/en/ranking
• 加入FB： PostgreSQL.TW
Taiwan PostgreSQL User Group 25
2022/7/30
DBMS of the Year 2020: PostgreSQL

26. 參考資料
• 提高 PostgreSQL 插入性能的 13 個技巧
• https://www.timescale.com/blog/13-tips-to-improve-postgresql-insert-performance/
• PostgreSQL 的性能調優方法 https://juejin.cn/post/7119489847529570334
• 利用pg_stat_activity做日常檢查 https://www.gushiciku.cn/pl/27i0/zh-tw
• PostgreSQL性能调优 http://bos.itdks.com/a5bd49bba1a342308515ad9b6687e430.pdf
• PGTune https://pgtune.leopard.in.ua/
• PostgreSQL ecosystem, https://www.slideshare.net/tsunghsilin/postgresql-ecosystem (COSCUP 2021)
Taiwan PostgreSQL User Group 26
7/30/2022

27. Thank you.
PostgreSQL ecosystem
@ COSCUP 2022
Taiwan PostgreSQL User Group
林宗禧 linjose@postgresql.tw
27
https://www.postgresql.fastware.com/