In this presentation I will try to analyze the experience of using pgbouncer and other database proxies at prom.ua. What profit does it bring in organizational sense, what new flexibility it gives to the infrastructure and everyday maintenance and scaling, which additional enchancements we managed to implement and test, what worked and what did not.

2. Who we are?

3. 1. Prom.ua
2. Marketplace
3. Using PostgreSQL since the start(2008?)
4. Self hosted infrastructure
5. Multiple(~243) postgres instances across 5
datacenters

4. What is even this pgbouncer?

5. 1. Database Proxy for PostgreSQL
2. Single threaded
3. Can do load balance on DNS only
4. Cannot split READ/WRITE
5. Can multiplex inbound/outbound
connections
6. Very stable and robust

6. Some alternatives
1. PGPool II - can split read/write,
slow
2. PGCat - can do multi-core
3. Supavisor - can do multi-core,
web-scale

7. Why do we even *need* a db proxy?

8. 1. Highload
2. Control

9. 1. Highload
PostgreSQL implements a “process per user” client/server model. In this model, every client process connects to
exactly one backend process. As we do not know ahead of time how many connections will be made, we have to use a
“supervisor process” that spawns a new backend process every time a connection is requested. This supervisor
process is called postmaster and listens at a specified TCP/IP port for incoming connections. Whenever it detects a
request for a connection, it spawns a new backend process. Those backend processes communicate with each other
and with other processes of the instance using semaphores and shared memory to ensure data integrity throughout
concurrent data access.

10. 1. Highload
Session Pool Transaction Pool Statement Pool
Service
DB DB DB
Service Service

12. 2. Control
Service
A
Service
B
Service
C
Service
D
DB
DB
DB
Pool
Pool
Pool
Pool
Pool
Pool

13. Service
A
DB
PGB
2. Control
Service
B

14. 2. Control
Service
A
Old DB
New DB
PGB

15. 2. Control
Service
A
Old DB
New DB
PGB

16. Deployment evolution
2011(?) - 2025- ?

17. Dedicated Pgbouncer
APP
APP
APP
PGB
Server
PGB
Server
PGB
Server
DB Server
DB Server
DNS

18. Distributed Pgbouncer
DB Server
DB Server
DNS
App Server
App
PgB
App Server
App
PgB
App Server
App
PgB

19. DB Server
PG
PGB
DB Server
PG
PGB
Embedded Pgbouncer
APP
APP
APP
DNS

20. K8S Pgbouncer
APP Pod
APP Pod
APP Pod
PGB Pod
PGB Pod
PGB Pod
DB Server
DB Server
k8s
Service
DNS

21. Pod evolution
2018(?) - 2025- ?

22. PgBouncer Pod
PGB
1. Basic

23. PgBouncer Pod
PGB ConfigWatcher
2. Add config watcher

24. logit "Starting config watcher"
sleep 10s
config_file=/etc/pgbouncer/pgbouncer.ini
while true; do
if [ -f ${config_file} ]; then
logit "Start monitoring config file ${config_file}"
inotifywait -e DELETE_SELF ${config_file} &
wait $!
sleep 1s
if [ -f ${config_file} ]; then
logit "Settings changed! Applying new settings"
cat ${config_file}
logit "Reloading pgbouncer"
psql --host=127.0.0.1 --port 6543 -U pgbadmin pgbouncer -c "reload"
psql --host=127.0.0.1 --port 6543 -U pgbadmin pgbouncer -c "show databases"
# reload the dns
curl 127.0.0.1:8081/reload | true
else
logit "Config file ${config_file} is missing. Skipping"
fi
else
logit "Config file ${config_file} is missing. Sleeping 10s"
sleep 10s
fi
done &

26. PgBouncer Pod
PGB ConfigWatcher
MetricsExporter
3. Add metrics exporter

27. pgbouncer=# show help;
NOTICE: Console usage
DETAIL:
SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|USERS|
VERSION
SHOW PEERS|PEER_POOLS
SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM|STATE
SHOW DNS_HOSTS|DNS_ZONES
SHOW STATS|STATS_TOTALS|STATS_AVERAGES|TOTALS
SET key = arg
RELOAD
PAUSE [<db>]
RESUME [<db>]
DISABLE <db>
ENABLE <db>
RECONNECT [<db>]
KILL <db>
SUSPEND
SHUTDOWN
WAIT_CLOSE [<db>]
SHOW

28. 2. Add metrics exporter

29. 2. Add metrics exporter

30. 4. Add health check
PgBouncer Pod
PGB ConfigWatcher
ConfigWatcher HealthCheck

32. PgBouncer Pod
PGB Config Watcher
Metrics exporter Healthcheck
HaProxy
5. Add HaProxy

33. K8S Pgbouncer
APP Pod
APP Pod
APP Pod
DB Server
DB Server
k8s
Service
PGB
Haproxy
PGB
Haproxy
PGB
Haproxy

34. resolvers mydns
nameserver dns1 $RESOLVER:53
accepted_payload_size 8192
{{- range .Values.haproxy.dbs }}
frontend {{ .name }}_frontend
mode tcp
bind :{{ .localPort }}
default_backend {{ .name }}_backend
backend {{ .name }}_backend
option pgsql-check user {{ .user }}
balance leastconn
server-template {{ .name }} 16 {{ .host }}:{{ .port }} check resolvers mydns
init-addr none
{{- end }}
https://www.haproxy.com/documentation/haproxy-configuration-tutorials/dns-resolution/

35. THIS IS WHAT PROXY REALLY IS
Softirqs (software interrupt requests) are similar to
interrupts, but they are generated by the kernel itself rather
than hardware devices. Softirqs are used for a variety of
purposes, such as network processing, task scheduling,
and disk I/O. Softirqs are generally less time-critical than
hardware interrupts, but they can still impact system
performance.
https://www.netdata.cloud/blog/understanding-interrupts-s
oftirqs-and-softnet-in-linux/

39. 1. Haproxy uses DNS
2. Haproxy does ping
How do we achieve the same without proxy?

40. Ping
DNS

41. PgBouncer Pod
PGB Config Watcher
Metrics exporter Healthcheck
HaProxy
DNS Ping
5. Add DNS Ping, Remove HaProxy

42. K8S Pgbouncer
APP Pod
APP Pod
APP Pod
DB Server
DB Server
k8s
Service
PGB
DNS
Ping
PGB
DNS
Ping
PGB
DNS
Ping

44. PgBouncer Pod
PGB Config Watcher
Metrics exporter Healthcheck
DNS Ping
5. Plans: Add log Parser
LogParser

46. 5. Plans: Better Circuit Breaker
DNS Ping
Postmaster
Real
connection
If we run out of connections, Postmaster still can reject auth, we need real
credentials to initiate connection and check for “no more connections allowed”

47. 5. Plans: Investigate host_strategy
Add load_balance_hosts parameter, to support not load balancing between hosts. (#736)