This document provides an overview of replicating a PostgreSQL database. It discusses setting up a primary server for reads and writes and standby servers that are kept in sync with the primary to serve as backups. The primary server writes data to its write-ahead log (WAL) files, which are streamed in real-time to the standby servers via WAL shipping. This allows the standby servers to keep an identical copy of the database. The document also covers configuration of both the primary and standby servers for replication as well as tools for testing the replication setup.

1. Service Platform Architect
Brandon Kang
sangjinn@gmail.com
https://tech.brandonkang.net
July 2020
How to Replicate
PostgreSQL Database
/ / 0 2 /
.
2 : / 2 . /
A
@ /: /
/ /

2. Replication Overall
Primary Server
• A server for both READ and WRITE
Standby Server = Slave server, Backup server, Replica, etc.
• A server where the data is kept in sync with the master continuously
• A warm standby server can’t be connected until it is promoted to primary
• A hot standby server accepts connections and serves read-only queries
• Data is written to the master server and propagated to the slave servers
• When primary server has an issue, a standby server will take over and
continue to take ‘WRITE” process

3. WAL Shipping Replication
WAL(Write-Ahead Logging)
• A log file where all the modifications to the database
• WAL is used for recovery after a database crash, ensuring data integrity
• WAL is used in database systems to achieve atomicity and durability
WAL based Replication ­ Streaming WAL Records
• Write-ahead log records are used to keep the data in sync
• WAL record chunks are streamed by database servers to synchronize
• Standby server connects to the master to receive the WAL chunks
• WAL records are streamed as they are generated
• The streaming of WAL records need not wait for the WAL file to be filled

4. Streaming Replication
PRIMARY
Server
Standby
Server
WAL
Sender
WAL
Receiver
WAL Record
READ ONLYREAD/WRITE
pg_wal
directory
WAL
File
pg_wal
directory
WAL
File
Synchronize
- WAL Record1
- WAL Record2
- WAL Record3
……

5. Replication Configuration
Primary Node ­ pg_hba.conf
/PG_HOME/DATA/pg_hba.conf
# Allow replication connections from localhost,
# by a user with the replication privilege.
# TYPE DATABASE USER ADDRESS METHOD
host replication repl_user IPaddress(CIDR) md5
Restart PostgreSQL service on the primary node
Internal IP ranges
Firewall Open for this.

6. Replication Configuration
Primary Node ­/PG_HOME/DATA/postgres.conf
# Possible values are replica|minimal|logical
wal_level = hot_standby
archive_mode = on #Syncronize with Primany node using wal archive
archive_command = ‘cp %p /PG_TBS/backup/archive/%f’ #psql_superuser archive backup
# required for pg_rewind capability when standby goes out of sync with master
wal_log_hints = on
# sets the maximum number of concurrent connections from the standby servers.
max_wal_senders = 2
# The below parameter is used to tell the master to keep the minimum number of
# segments of WAL logs so that they are not deleted before standby consumes them.
# each segment is 16MB
wal_keep_segments = 16

7. Replication Configuration
Secondary Node ­/PG_HOME/DATA/postgres.conf
hot_standby = on
Secondary Node ­/PG_HOME/DATA/recovery.conf
standby_mode = ‘on’
restore_command = ‘cp %p /PG_TBS/backup/archive/%f “%p”’
Primary_conninfo = ‘host=… ports=… user=postgres password= application_name=slave1’
# Explaining a few options used for pg_basebackup utility
# -X option is used to include the required transaction log files (WAL files) in the
# backup. When you specify stream, this will open a second connection to the server
# and start streaming the transaction log at the same time as running the backup.
# -c is the checkpoint option. Setting it to fast will force the checkpoint to be
# created soon.
# -W forces pg_basebackup to prompt for a password before connecting
# to a database.
pg_basebackup -D <data_directory> -h <master_host> -X stream -c fast -U
repl_user -W

8. Replication Configuration
Secondary Node ­ pg_basebackup utility
# Explaining a few options used for pg_basebackup utility
# -X option is used to include the required transaction log files (WAL files) in the
# backup. When you specify stream, this will open a second connection to the server
# and start streaming the transaction log at the same time as running the backup.
# -c is the checkpoint option. Setting it to fast will force the checkpoint to be
# created soon.
# -W forces pg_basebackup to prompt for a password before connecting
# to a database.
pg_basebackup -D <data_directory> -h <master_host> -X stream -c fast -U repl_user -W
Example)
$pg_basebackup ­h localhost ­p 5170 -u postgres ­D /PG_TBS/cluster -checkpoint=fast --progress

9. Replication Configuration
Secondary Node ­ Replication configuration file
# Specifies whether to start the server as a standby. In streaming replication,
# this parameter must be set to on.
standby_mode = ‘on’
# Specifies a connection string which is used for the standby server to connect
# with the primary/master.
primary_conninfo = ‘host=<master_host> port=<postgres_port> user=<replication_user>
password=<password> application_name=”host_name”’
# Specifies recovering to a particular timeline. The default is to recover along the
# same timeline that was current when the base backup was taken.
# Setting this to latest recovers to the latest timeline found
# in the archive, which is useful in a standby server.
recovery_target_timeline = ‘latest’
Start PostgreSQL service on the Standby node

10. Replication Testing
postgres=# select * from pg_stat_replication ;
-[ RECORD 1 ]----+---------------------------------
pid | 1114
usesysid | 16384
usename | repuser
application_name | walreceiver
client_addr | 127.0.0.1
client_hostname |
client_port | 52444
backend_start | 08-MAR-20 19:54:05.535695 -04:00
state | streaming
..
sync_priority | 0
sync_state | async
ps ­ef | grep postgres
[Master] wal sender process check
[Standby] wal receiver & startup process check
SELECT * FROM pg_stat_replication; Query execution on the Primary/Standby nodes

11. Replication Testing
pg_is_in_recovery(): A function to check if standby server is still in recovery mode or not
postgres=# select pg_is_in_recovery();
pg_is_in_recovery
-------------------
t
(1 row)
pg_last_xact_replay_timestamp(): A function shows time stamp of last replica transaction
postgres=# select pg_last_xact_replay_timestamp();
pg_last_xact_replay_timestamp
----------------------------------
08-MAR-20 20:54:27.635591 -04:00 (1 row)
SELECT CASE WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location()
THEN 0
ELSE EXTRACT (EPOCH FROM now() - pg_last_xact_replay_timestamp())
END AS log_delay;
Calculating logs in seconds

12. Scaling PostgreSQL
PRIMARY
Server
Standby
Server
WAL
Sender
WAL
Receiver
WAL Record
WRITE
Synchronize
HAProxy /
NGINX / F5 READ
WRITE
READ
pgBouncer
primary_db
standby_db
Application

13. - Thank You. -
Service Platform Architect
Brandon Kang
sangjinn@gmail.com
https://tech.brandonkang.net