Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
PostgreSQL Replication
in 10 Minutes

Josh Berkus
PostgreSQL Experts, Inc.
SCALE 2014
1. See a multistage replication
demo
2. Hear how to set up and
configure replication
3. Understand different
setups and re...
replication terms
master / slave
master / standby
master / replica
primary / secondary
primary / replica
replication mechanisms

1. statement
2. row
3. binary
replication mechanisms

1. queries
2. rows
3. data pages
also called ...
●

streaming replication
–

●

refers to the ability to stream new data pages
over a network connection

h...
more terms
●

recovery
–

●

snapshot, clone, basebackup
–

●

binary replication came from binary backup,
i.e. Point In T...
administering
replication
configuration files
●

postgresql.conf
–

●

same settings for master, replica

recovery.conf
–
–

presence turns on repli...
views & functions
process list
● pg_stat_replication
● pg_is_in_recovery()
● pg_xlog* functions
●
permissions & security
A.
B.
C.
D.

replication permission
pg_hba.conf
max_wal_senders
firewall/network
archiving replication
archiving replication
1.
2.
3.
4.
5.
6.

set up archiving
start archiving
pg_start_backup('label')
rsync all files
pg_stop...
reasons to archive
replica out-of-sync
● combine with PITR or DR
● very erratic connection to master
● need remastering be...
archiving tips
use a script which handles copy failure
● use a shared drive
● put archive on a partition
● monitor for arc...
failover, failback
& remastering
more terms
●

failover, promotion
–

●

failback
–

●

making a replica into a master/standalone
returning the original ma...
replica promotion
pg_ctl promote
● trigger file
● rm recovery.conf & restart
●
failover has 3 parts
1. failing over the database
2. failing over the connections
3. STONITH
manual failover
●

advantages:
–
–

●

easy to set up
fewer accidental failovers

disadvantages:
–
–

downtime
being woken...
automated failover
●

advantages:
–
–

●

low downtime
sleep through the night

disadvantages:
–
–
–

hard to set up corre...
automated
failover
logic
www.handyrep.org
STONITH
use corosync/VIP
● use connection failover
● use peer broker server
●
remastering
need the replica which is “furthest
ahead”
● measure both receive point and replay
point
● need 9.3 for “strea...
replication lag
&
query cancel
reasons for lag
●

network delay
–

speed of light

replica too busy
● file operations block
●

–
–

VACUUM
DROP TABLE
replication lag issues
inconsistency (if load-balancing)
● query cancel
●

–

applications need to retry queries

catch-up...
synchronous
replication
what synch rep does
guarantee against data loss
what it doesn't
●

enforce global consistency
–
–

●

master can be behind
replica snapshot can be behind

help availabili...
“I would rather be down
than potentially lose
data.”
how to synch rep
1. pick one (or a pool) of servers to be
your synch replicas
2. change application_name
3. change master'...
Postgres specialities
implements only 1-redundant model
● synch is per-transaction
●

–
–

not per replica
synch only impo...
synch rep design
1 replica is synch replica
● several asych replicas
● load-balance to asynch only
● always failover to sy...
cascading
replication
how to cascade
1.
2.
3.
4.

have master & replica
clone the master or the replica
point primary_conninfo to the replica
br...
why to cascade
limit connections to master
● don't clone master
● know which replica is ahead
●
why to cascade
Los Angeles

walsender

Shanghai

walsender

walreciever

walreciever

walreciever
why not cascade?
complexity
● cycles
● increases SPOFs
●
replication
in the
cloud
use a shared archive

WAL-E
ephemeral replicas
no sync to disk
● do not recover from crash
●

–
●

spin up a replacement instead

turn off all logging...
sharding and replication
server 1

server 2
sharding and replication
server 1

server 2
changes coming up
●

in dev: pg_rewind
–

failback without full clone

9.4: merge recovery.conf &
postgresql.conf
● 9.4+: ...
questions?
●

HandyRep: www.handyrep.org

●

Josh Berkus: josh@pgexperts.com
–
–

●

PGX: www.pgexperts.com
Blog: www.data...
PostgreSQL Replication in 10  Minutes - SCALE
PostgreSQL Replication in 10  Minutes - SCALE
PostgreSQL Replication in 10  Minutes - SCALE
PostgreSQL Replication in 10  Minutes - SCALE
Upcoming SlideShare
Loading in …5
×

PostgreSQL Replication in 10 Minutes - SCALE

22,384 views

Published on

A very brief review of how to do PostgreSQL 9.3 Streaming Replication. Meant to accompany a demo.

Published in: Technology

PostgreSQL Replication in 10 Minutes - SCALE

  1. 1. PostgreSQL Replication in 10 Minutes Josh Berkus PostgreSQL Experts, Inc. SCALE 2014
  2. 2. 1. See a multistage replication demo 2. Hear how to set up and configure replication 3. Understand different setups and replication issues
  3. 3. replication terms master / slave master / standby master / replica primary / secondary primary / replica
  4. 4. replication mechanisms 1. statement 2. row 3. binary
  5. 5. replication mechanisms 1. queries 2. rows 3. data pages
  6. 6. also called ... ● streaming replication – ● refers to the ability to stream new data pages over a network connection hot standby – refers to the ability of standbys to run readonly queries while in standby mode
  7. 7. more terms ● recovery – ● snapshot, clone, basebackup – ● binary replication came from binary backup, i.e. Point In Time Recovery taking a moment-in-time copy of a running database server standalone – a lone read-write server, neither master nor replica
  8. 8. administering replication
  9. 9. configuration files ● postgresql.conf – ● same settings for master, replica recovery.conf – – presence turns on replication must be in $PGDATA
  10. 10. views & functions process list ● pg_stat_replication ● pg_is_in_recovery() ● pg_xlog* functions ●
  11. 11. permissions & security A. B. C. D. replication permission pg_hba.conf max_wal_senders firewall/network
  12. 12. archiving replication
  13. 13. archiving replication 1. 2. 3. 4. 5. 6. set up archiving start archiving pg_start_backup('label') rsync all files pg_stop_backup() bring up replica
  14. 14. reasons to archive replica out-of-sync ● combine with PITR or DR ● very erratic connection to master ● need remastering before 9.3 ●
  15. 15. archiving tips use a script which handles copy failure ● use a shared drive ● put archive on a partition ● monitor for archive growth ● compression ●
  16. 16. failover, failback & remastering
  17. 17. more terms ● failover, promotion – ● failback – ● making a replica into a master/standalone returning the original master to master status remastering – designating a new master in a group of servers
  18. 18. replica promotion pg_ctl promote ● trigger file ● rm recovery.conf & restart ●
  19. 19. failover has 3 parts 1. failing over the database 2. failing over the connections 3. STONITH
  20. 20. manual failover ● advantages: – – ● easy to set up fewer accidental failovers disadvantages: – – downtime being woken up at 3am
  21. 21. automated failover ● advantages: – – ● low downtime sleep through the night disadvantages: – – – hard to set up correctly need broker server accidentally triggered failovers
  22. 22. automated failover logic
  23. 23. www.handyrep.org
  24. 24. STONITH use corosync/VIP ● use connection failover ● use peer broker server ●
  25. 25. remastering need the replica which is “furthest ahead” ● measure both receive point and replay point ● need 9.3 for “streaming-only” remastering ●
  26. 26. replication lag & query cancel
  27. 27. reasons for lag ● network delay – speed of light replica too busy ● file operations block ● – – VACUUM DROP TABLE
  28. 28. replication lag issues inconsistency (if load-balancing) ● query cancel ● – applications need to retry queries catch-up speed ● burden on master ●
  29. 29. synchronous replication
  30. 30. what synch rep does guarantee against data loss
  31. 31. what it doesn't ● enforce global consistency – – ● master can be behind replica snapshot can be behind help availability
  32. 32. “I would rather be down than potentially lose data.”
  33. 33. how to synch rep 1. pick one (or a pool) of servers to be your synch replicas 2. change application_name 3. change master's postgresql.conf 4. reload
  34. 34. Postgres specialities implements only 1-redundant model ● synch is per-transaction ● – – not per replica synch only important transactions
  35. 35. synch rep design 1 replica is synch replica ● several asych replicas ● load-balance to asynch only ● always failover to synch replica ●
  36. 36. cascading replication
  37. 37. how to cascade 1. 2. 3. 4. have master & replica clone the master or the replica point primary_conninfo to the replica bring up the new replica
  38. 38. why to cascade limit connections to master ● don't clone master ● know which replica is ahead ●
  39. 39. why to cascade Los Angeles walsender Shanghai walsender walreciever walreciever walreciever
  40. 40. why not cascade? complexity ● cycles ● increases SPOFs ●
  41. 41. replication in the cloud
  42. 42. use a shared archive WAL-E
  43. 43. ephemeral replicas no sync to disk ● do not recover from crash ● – ● spin up a replacement instead turn off all logging/disk – fsync off, bgwriter off, full_page_writes off
  44. 44. sharding and replication server 1 server 2
  45. 45. sharding and replication server 1 server 2
  46. 46. changes coming up ● in dev: pg_rewind – failback without full clone 9.4: merge recovery.conf & postgresql.conf ● 9.4+: “logical” streaming replication ●
  47. 47. questions? ● HandyRep: www.handyrep.org ● Josh Berkus: josh@pgexperts.com – – ● PGX: www.pgexperts.com Blog: www.databasesoup.com Upcoming Events – – NYC pgDay: April 2-3 pgCon, Ottawa, May 21-24 Copyright 2013 PostgreSQL Experts Inc. Released under the Creative Commons Share-Alike 3.0 License. All images are the property of their respective owners. The WAL-E image is the property of Disney Inc. and is use here as parody.

×