This document discusses PostgreSQL streaming replication over a VPN. It describes configuring a primary and standby server for streaming replication, including setting WAL archiving, recovery settings, and verification. Issues with low bandwidth, latency, WAL segment rotation, and downtime are addressed through combining WAL archiving with streaming replication to minimize data loss during failover. References for further information on PostgreSQL streaming replication, WAL logs, warm standby servers, and continuous archiving are also provided.

1. Derechos reservados © 2013-2016 Sandino Araico Sánchez
<sandino@1101.mx>
Se permite ilimitadamente el uso, copia, redistribución con o sin
modificaciones siempre y cuando se mantenga el aviso de
derecho de autor y se anoten al final de la presentación todas las
modificaciones que se llevan a cabo conservando la historia de
las modificaciones que hagan las demás personas e indicando la
fecha de cada modificación y el nombre de la persona que la
llevó a cabo.
2016-10-29
PostgreSQL streaming
replication sobre VPN

2. RDBMS =
Relational
Data
Base
Management
System
Manejador de bases de datos
relacionales

3. ACID =
Atomicity
Consistency
Isolation
Durability
http://en.wikipedia.org/wiki/ACID

4. WAL =
Write
Ahead
Log
https://www.postgresql.org/docs/9.6/static/wal-intro.html

5. #--------------------------------------------------
----------------------------
# RESOURCE USAGE (except WAL)
#--------------------------------------------------
----------------------------
# - Memory -
shared_buffers = 512MB!! ! # min 128kB
! ! ! ! ! # (change requires restart)
temp_buffers = 32MB! ! ! # min 800kB
#max_prepared_transactions = 0! ! # zero disables
the feature
! ! ! ! ! # (change requires restart)
postgresql.conf

6. Shared Buffers
SELECT * FROM dbmail_users;

7. Shared Buffers
SELECT * FROM dbmail_users ORDER
BY user_idnr DESC LIMIT 5;

8. Shared Buffers
INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31337, ‘sandino’);

9. Shared Buffers
COMMIT;

10. Shared Buffers
INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31338, ‘chalo’);
INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31339, ‘roa’);

11. Shared Buffers
COMMIT;

12. Shared Buffers
UPDATE dbmail_users SET userid =
‘sandino@1101.mx’ WHERE user_idnr =
31337;

13. Shared Buffers
UPDATE dbmail_users SET userid =
‘chalo@1101.mx’ WHERE user_idnr =
31338;
UPDATE dbmail_users SET userid =
‘roa@1101.mx’ WHERE user_idnr =
31339;

14. Shared Buffers
ROLLBACK;

15. Shared Buffers
UPDATE dbmail_users SET userid =
‘sandino@1101.mx’ WHERE user_idnr =
31337;

16. Shared Buffers
UPDATE dbmail_users SET userid =
‘chalo@1101.mx’ WHERE user_idnr =
31338;
UPDATE dbmail_users SET userid =
‘roa@1101.mx’ WHERE user_idnr =
31339;

17. Shared Buffers
COMMIT;

18. Shared Buffers
VACUUM;

19. #--------------------------------------------------
----------------------------
# WRITE AHEAD LOG
#--------------------------------------------------
----------------------------
# - Settings -
wal_level = hot_standby # minimal,
archive, or hot_standby
# (change requires
restart)
#fsync = on # turns forced
synchronization on or off
postgresql.conf

20. wal_buffers = 16MB # min 32kB, -1
sets based on shared_buffers
# (change requires
restart)
#wal_writer_delay = 200ms # 1-10000
milliseconds
#commit_delay = 0 # range
0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
postgresql.conf

21. Shared Buffers + WAL
INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31337, ‘sandino’);
DB WAL

22. COMMIT;
Shared Buffers + WAL
DB WAL

23. INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31338, ‘chalo’);
INSERT INTO dbmail_users (user_idnr,
userid) VALUES (31339, ‘roa’);
Shared Buffers + WAL
DB WAL

24. COMMIT;
Shared Buffers + WAL
DB WAL

25. UPDATE dbmail_users SET userid =
‘sandino@1101.mx’ WHERE user_idnr =
31337;
Shared Buffers + WAL
DB WAL

26. UPDATE dbmail_users SET userid =
‘chalo@1101.mx’ WHERE user_idnr =
31338;
UPDATE dbmail_users SET userid =
‘roa@1101.mx’ WHERE user_idnr =
31339;
Shared Buffers + WAL
DB WAL

27. CHECKPOINT
Shared Buffers + WAL
DB WAL

28. ROLLBACK;
Shared Buffers + WAL
DB WAL

29. UPDATE dbmail_users SET userid =
‘sandino@1101.mx’ WHERE user_idnr =
31337;
Shared Buffers + WAL
DB WAL

30. UPDATE dbmail_users SET userid =
‘chalo@1101.mx’ WHERE user_idnr =
31338;
UPDATE dbmail_users SET userid =
‘roa@1101.mx’ WHERE user_idnr =
31339;
Shared Buffers + WAL
DB WAL

31. COMMIT;
Shared Buffers + WAL
DB WAL

32. ... En algún punto del tiempo ....
Shared Buffers + WAL
DB WAL

33. Respaldo base
SELECT pg_start_backup('label');
rsync -vacHz --progress /var/lib/pgsql/
data/
postgres@bak.srvr:/var/lib/pgsql/
data/
SELECT pg_stop_backup();
rsync -vacHz --progress /var/lib/pgsql/
archive/
postgres@bak.srvr:/var/lib/pgsql/
archive/

34. recovery.conf
# Specifies a command to load archive segments
from the WAL archive. If
# wal_keep_segments is a high enough number to
retain the WAL segments
# required for the standby server, this may not be
necessary. But
# a large workload can cause segments to be
recycled before the standby
# is fully synchronized, requiring you to start
again from a new base backup.
restore_command = 'cp /path_to/archive/%f "%p"'

35. Respaldo incremental
DB WAL
DB WAL

36. Recuperación
DB WAL

37. log shipping
DB WAL
DB WAL

38. recovery.conf para streaming replication
# Specifies whether to start the server as a standby. In
streaming replication,
# this parameter must to be set to on.
standby_mode = 'on'
# Specifies a connection string which is used for the standby
server to connect
# with the primary.
primary_conninfo = 'host=192.168.0.10 port=5432 user=replicator'
# Specifies a trigger file whose presence should cause streaming
replication to
# end (i.e., failover).
trigger_file = '/var/lib/pgsql/trigger'
restore_command = 'cp /var/lib/pgsql/archive/%f "%p"'

39. Streaming replication
DB WAL
DB WAL

40. Configuración origen
$ vi postgresql.conf
listen_addresses = '192.168.0.10'
$ vi pg_hba.conf
# The standby server must connect with a user that has
replication privileges.
host replication replication 192.168.0.20/22 trust
http://wiki.postgresql.org/wiki/Streaming_Replication

41. $ vi postgresql.conf
wal_level = hot_standby
max_wal_senders = 5
wal_keep_segments = 32
archive_mode = on
archive_command = 'cp %p /path_to/archive/%f'
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración origen

42. $ psql -c "SELECT pg_start_backup('label', true)"
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --
exclude postmaster.pid
$ psql -c "SELECT pg_stop_backup()"
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración origen

43. $ vi postgresql.conf
hot_standby = on
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración destino

44. $ vi recovery.conf
standby_mode = 'on'
primary_conninfo = 'host=192.168.0.10 port=5432
user=postgres'
trigger_file = '/path_to/trigger'
restore_command = 'cp /path_to/archive/%f "%p"'
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración destino

45. $ psql -c "SELECT pg_current_xlog_location()" -
h192.168.0.10 (primary host)
pg_current_xlog_location
--------------------------
0/2000000
(1 row)
$ psql -c "select pg_last_xlog_receive_location()" -
h192.168.0.20 (standby host)
pg_last_xlog_receive_location
-------------------------------
0/2000000
(1 row)
$ psql -c "select pg_last_xlog_replay_location()" -
h192.168.0.20 (standby host)
pg_last_xlog_replay_location
------------------------------
0/2000000
(1 row)
http://wiki.postgresql.org/wiki/Streaming_Replication
Verificación

46. # The displayed LSNs indicate the byte position that the
standby server has
# written up to in the xlogs.
[primary] $ ps -ef | grep sender
postgres 6879 6831 0 10:31 ? 00:00:00 postgres:
wal sender process postgres 127.0.0.1(44663) streaming
0/2000000
[standby] $ ps -ef | grep receiver
postgres 6878 6872 1 10:31 ? 00:00:01 postgres:
wal receiver process streaming 0/2000000
http://wiki.postgresql.org/wiki/Streaming_Replication
Verificación

47. Problemática
Estrecho de banda (100 Mbps)
Latencia (200 ms)
Rotación de WAL segments
Downtime (48+ horas)

48. Problemática
Estrecho de banda (100 Mbps)
Latencia (200 ms)
Rotación de WAL segments
Downtime (48+ horas)

49. COMMIT;
Log archiving + streaming replication
DB WAL
DB
WAL
Archive

50. # - Archiving -
archive_mode = on # enables archiving; off, on, or
always
# (change requires restart)
archive_command = 'scp "%p" archive.example.com:/archive/"%f"'
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración origen

51. standby_mode = 'on'
primary_conninfo = 'host=192.168.0.10 port=5432
user=postgres'
trigger_file = '/data/pgsql/trigger'
restore_command = 'scp archive.example.com:/archive/"%f" "%p"'
http://wiki.postgresql.org/wiki/Streaming_Replication
Configuración destino

52. Referencias
http://wiki.postgresql.org/wiki/Streaming_Replication
https://www.postgresql.org/docs/9.6/static/wal-intro.html
https://www.postgresql.org/docs/9.6/static/warm-standby.html
https://www.postgresql.org/docs/9.6/static/continuous-archiving.html
http://en.wikipedia.org/wiki/Relational_database_management_system
http://en.wikipedia.org/wiki/ACID
Sandino Araico Sánchez
<sandino@1101.mx>
@KBrown
#mendozaaaa