ABSTRACT OF THE TALK 6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve. cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only. BIO A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte! KEY TAKE-AWAYS FROM THE TALK - A methodology for benchmarking a PostgreSQL database in Kubernetes - Open source set of tools for benchmarking a PostgreSQL database in Kubernetes - Reasons why benchmarking both the storage and the database is important https://github.com/EnterpriseDB/cnp-sandbox https://github.com/EnterpriseDB/cnp-bench

1. Benchmarking for
PostgreSQL
workloads in
Kubernetes (part 2)
Gabriele Bartolini
#109 Data on Kubernetes (DOK) Webinar
16 December 2021

2. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Today’s speakers
Gabriele Bartolini
VP of Cloud Native at EDB
PostgreSQL user since ~2000
• Community member since 2006
• Co-founder of PostgreSQL Europe
Previously at 2ndQuadrant, from 2008 to 2020
• Co-founder
• Head of Global Support
• Cloud Native Initiative Lead
• Founding member of Barman
DevOps evangelist
2
Twitter: @_GBartolini_ / @EDBPostgres

3. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
EDB and Kubernetes
3
Major sponsor
of the PostgreSQL project
Kubernetes Certified Service
Provider (KCSP)
Silver Member of CNCF &
Linux Foundation
Platinum founding sponsor of
the Data on Kubernetes
Community
We have contributed to the PostgreSQL community every year since 2006, making major feature contributions.
We had 32 contributors in PostgreSQL 14, including 7 code committers and 3 core members.
Bringing PostgreSQL to Kubernetes

4. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Agenda
• Key takeaways from Dok #58
• A day in the life of a Postgres transaction
• Recommended architectures
• Our methodology
• Conclusions

5. DoK #58 webinar
Recap

6. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 6

7. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Why Kubernetes? Why PostgreSQL?
7
Cloud native culture with a highly versatile SQL driven database
● “Cloud Native” is much more than tools (Kubernetes)
○ Patterns/architectures (microservices, operators, ...)
○ Principles/culture (devops/lean/agile, velocity, automation, pervasive quality and security processes, …)
● Kubernetes is becoming popular for stateful workloads, including databases:
○ Please refer to dok.community/dokc-2021-report/ for details
○ Reasons: storage classes, local persistent volumes, the operator pattern
● PostgreSQL is based on 25+ years of evolutionary innovation
○ Linux : Operating System = Postgres : Database
○ Database of the year in 2017, 2018, and 2020 at db-engines.com
○ Some of its main features:
■ Native streaming replication, both physical and logical, sync and async, cascading
■ Online Continuous Backup and Point In Time Recovery
■ Declarative Partitioning
■ Parallel queries
■ Extensibility and extensions (e.g. PostGIS)
■ JSON support (SQL/noSQL hybrid databases)
■ ACID transactions

8. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Benchmarking PostgreSQL
8
Workloads, storage and the database
● Storage
○ Write Ahead Log (WAL), or historically xlog
■ Sequential writes and fsync
○ Shared buffers cleaning
■ By checkpoint, bgwriter, or the single backend
■ Random writes (and OS cache)
○ Page reads
■ Random reads
○ Optimization: Table scans
■ Sequential reads
○ Capping on cloud environments
● Database
○ Workloads: in-memory, OLTP, and OLAP
○ Initial focus: TPS on large OLTP workloads (RAM < DB size)
○ pgbench
● We introduced cnp-bench

9. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Know your storage
9
You need to trust it
● Make sure you benchmark your storage before you go in production
● Make sure you benchmark your storage before you test your database
○ Storage can become your bottleneck
■ If your storage is slow, your database will be slow
● Please refer to DoK #58 for more information on storage benchmarking
○ Use cnp-bench
○ Use fio directly

10. A day in the life of a transaction
(VERY simplified view)

11. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Disclaimer
Postgres internals are more complex than
this. The following is a simplified view for
clarity.

12. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Disk (pg_wal)
Disk (PGDATA)
12
Shared Buffers (Postgres cache)
8kb 8kb … WAL file segment
Checkpoint
8kb 8kb …
Postgres backend
Another “brick” in the WAL
8kb
8kb
Ready to be recycled
usually 16MB in size
Transaction log
Sequential writes
Fsync-ed
Regularly the
database cache is
flushed on disk
(“dirty pages”)
WAL file segment
DISCLAIMER: simplified view for didactic purposes
8kb
e.g Random writes
Random reads
Seq scans

13. Recommended
architectures for
PostgreSQL

14. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
PostgreSQL architectures in Business Continuity
14
Always plan for benchmarking the final production architecture
● Start with one instance to spot the major bottlenecks
○ e.g. storage
● Then move to a real life production architecture
● Consider your Business Continuity goals
○ Disaster recovery - primarily focused on Recovery Point Objective (RPO)
○ High Availability - primarily focused on Recovery Time Objective (RTO)
○ Plan your production database architectures with both RTO and RPO in mind
● PostgreSQL provides the fundamental blocks for Business Continuity
○ Continuous backup and Point In Time Recovery
■ Base backups
■ WAL archiving
○ Native streaming replication based on the Write Ahead Log (WAL)
● The WAL is central in PostgreSQL
● To keep your data safe, managing the above in Kubernetes requires an operator written by
Postgres experts

15. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Disk (pg_wal)
Disk (PGDATA)
15
Shared Buffers (Postgres cache)
WAL file segment
Postgres backend
The criticality of the WAL in day-to-day Postgres
Ready to be recycled
WAL file segment
DISCLAIMER: simplified view for didactic purposes
archive_command
wal_sender(s)
streaming replication
WAL
archive
Replicas
(standby)
Ready to be archived
Potential Bottlenecks!

16. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Kubernetes cluster / namespace
Recommended architecture
16
Node
Local storage
Node
Local storage
Node
Local storage
Primary Sync Standby Potential Sync
Standby
zone 1 zone 2 zone 3
base backups
and WAL archive
Continuous backup
(WAL archiving)
restore_command
Streaming
Replication

17. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Some potential bottlenecks and issues
17
A summary of the major issues that bottlenecks can cause
● WAL writing: local storage
○ Slow system
● WAL archiving: serialized process, network, remote storage, compression (if applicable)
○ Bottleneck cause WAL files to pile up on the volume where pg_wal is, causing Postgres to halt
● Streaming replication: network, remote storage
○ wal_keep_segments/wal_keep_size
■ Beyond this threshold, WAL files are recycled on the primary and the standby falls out of sync
○ replication slots
■ The primary keeps track of the location in the WAL needed by a standby and keeps the WAL file
● Same issue as WAL archiving - WAL files pile up and Postgres risks to halt
○ synchronous replication
■ A bottleneck here slows down writes on the primary
■ If all synchronous standby servers are down, the primary stops accepting writes (never use a single synchronous standby)
● Restore command: serialized process, network, remote storage, decompression (if applicable)
○ A standby cannot start streaming replication and relies on WAL files from the archive
○ Delayed standby - possible impact on RPO and RTO in case of failover of the primary
● Standby replay: single process
○ Delayed standby - possible impact on RPO and RTO in case of failover of the primary

18. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Dedicated resources
18
Prefer shared nothing architectures, even in Kubernetes
● If you can, dedicate a Kubernetes node to one Postgres instance only
○ Take advantage of Pod scheduling capabilities and availability zones (where available)
■ pod affinity/anti-affinity
■ node selectors
■ tolerations
○ Properly set resource requests and limits
■ Guaranteed QoS is recommended
● If you can, use local storage on the dedicated node
○ Benchmark throughput
○ In the public cloud, watch out for IOPS limitations
● Costs/benefits analysis
○ One more reason why benchmarking is fundamental in proper and effective capacity planning
○ It’s your choice, and yours only

19. Our methodology

20. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
How we’re benchmarking Postgres on K8s
20
Observe and let numbers and diagrams help you discover issues
● We rely on:
○ cnp-sandbox
■ Prometheus, Grafana, Cloud Native PostgreSQL operator (EDB)
○ cnp-bench
■ on existing clusters
○ pg_stat_statements
● You can use your own PostgreSQL setup
○ Your favourite operator
● You can use your favorite observability tools
○ Your own Prometheus/Grafana
○ Something else (you should know what to look for now!)

21. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
A sandbox for Cloud Native PostgreSQL
21
cnp-sandbox is an open source helm chart
● Deploys a sandbox environment in Kubernetes with:
○ Prometheus
○ Grafana
○ Cloud Native PostgreSQL with:
■ a selection of PostgreSQL metrics for the native Prometheus exporter in CNP
■ a custom Grafana dashboard developed by EDB for Cloud Native PostgreSQL
● Main goals:
○ Evaluate Cloud Native PostgreSQL’s observability with Prometheus and Grafana
○ Integrate benchmarks with real-time collected data
● Suitable for pre-production and staging environments
○ Production environments should have their own Prometheus and Grafana installations
○ Metrics and dashboards can be reused
● URL: github.com/EnterpriseDB/cnp-sandbox

22. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Deployment
22
helm repo add cnp-sandbox https://enterprisedb.github.io/cnp-sandbox/
helm repo update
helm upgrade --install cnp-sandbox
cnp-sandbox/cnp-sandbox

23. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 23

24. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
cnp-bench
24
Benchmarking the storage and a database PostgreSQL
● Storage benchmarking with fio
● Database benchmarking and stress testing with:
○ pgbench
○ HammerDB
● Can be run against an existing Postgres database
○ Integrated with Cloud Native PostgreSQL, including pgBouncer for connection pooling
● Suitable for pre-production and staging environments
● URL: https://github.com/EnterpriseDB/cnp-bench

25. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
An example of pgbench initialization
25
cnp:
existingCluster: true
existingCredentials: pg-14-app
existingHost: pg-14-rw
existingDatabase: pgbench
image: quay.io/enterprisedb/postgresql:14.1
pgbench:
nodeSelector:
workload: pgbench
scaleFactor: 8000
initialize: true

26. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
An example of pgbench run
26
cnp:
existingCluster: true
existingCredentials: pg-14-app
existingHost: pg-14-rw
existingDatabase: pgbench
image: quay.io/enterprisedb/postgresql:14.1
pgbench:
nodeSelector:
workload: pgbench
initialize: false
skipVacuum: true
reportLatencies: true
time: 600
clients: 64
jobs: 128

27. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 27
Notes:
● 5 x 10min pgbench tests
● scale factor 8000 (120GB)
● 3 dedicated nodes:
○ AKS Standard_E8s_v4
○ 7 cores/56Gi RAM
○ Guaranteed Qos
○ Premium P80 storage class
● 1 MinSync replication
● Azure Blob Container (backup)
● pgbench on AKS Standard_D64s_v4

28. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 28
Another example showing ~ 13k tps with 32 cores

29. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Bottleneck: serialized WAL archiving
29
Anticipate and avoid this scenario!
36k piled WALs!
What if the primary dies now?

30. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Parallel archiving (1)
30
Remediation: parallel WAL archiving and large segment size (64MB)

31. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Parallel archiving (2)
31
Might be OK (bulk loads or vacuums)

32. There’s more ...

33. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
What’s next with cnp-bench
33
Our plan for the H1/2022
● Manage increasing number of client connections
● Manage repetitions
● Support custom pgbench scripts
● Improve support for HammerDB
● Introduce application level benchmarking
○ Web application load generation with hey
○ Front end scalability

34. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
About Cloud Native PostgreSQL
34
The Kubernetes operator from EDB
● It is currently closed source
○ Available for trials
● Fully declarative
● Integrated with the Kubernetes API server (no external tool for failover)
● Directly manages persistent volumes
● Our intention is to open source Cloud Native PostgreSQL in 2022
● It is the component that manages PostgreSQL in the data layer of BigAnimal

35. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved
Conclusions
35
Why benchmarking PostgreSQL is important?
● Data is the most important asset of an organization
● Data can live in Kubernetes, in reliable databases like PostgreSQL
● Don’t leave anything to chance
○ Benchmark your storage and know its limits
○ Benchmark your database and know its limits
● Benchmark before you go to production
○ You might not be able to benchmark when in production
● Highly consider dedicating storage and nodes to a single PostgreSQL instance
○ First benchmark the single node, and focus on the storage primarily
○ Then benchmark the high availability cluster, with continuous backup and replicas
■ Pay attention to WAL archiving, streaming, WAL restore, replay, and so on …
● Evaluate introduction of failover and switchover events in benchmarks (chaos)
○ Observe the cluster and always consider your RPO and RTO goals
● Study Postgres, love Postgres!
○ There are so many features you might not know that Postgres already has!

36. “
Thank you!
DoK #109 webinar - Benchmarking for PostgreSQL workloads in Kubernetes (part 2)
Gabriele Bartolini - @_GBartolini_
Watch part 1!