PostgreSQL replication


Published on

@Inaugural Indian PostgreSQL User Group meetup!

Published in: Technology, Business

PostgreSQL replication

  1. 1. Copyright © 2013 NTT DATA Corporation27 April 2013Fujii Masao (twitter: @fujii_masao)PostgreSQL replicationInaugural Indian PostgreSQL User Group meetup!
  2. 2. 2Copyright © 2013NTT DATA CorporationWho am I ? Database engineer in NTT DATA• NTT DATA is global IT service and consultingcompany PostgreSQL developer since 2008 One of main authors of PostgreSQL replication
  3. 3. 3Copyright © 2013NTT DATA CorporationIndex Whats replication? History PostgreSQL replication1. Features2. Synchronous vs. Asynchronous
  4. 4. Copyright © 2013 NTT DATA Corporation 4Whats replication?
  5. 5. 5Copyright © 2013 NTT DATA CorporationWhats replication?ClientClientChangeChangeDB serversChangeReplicateDB serversChangeProxy serverCreate a replica of the database on multiple servers
  6. 6. 6Copyright © 2013 NTT DATA CorporationWhy is replication required?Replication is an essential feature to get the system to work properly 24/7!HighAvailabilityLoadBalancingIf one server crashes, another can keep providing the serviceWe can easily reduce the downtime of the systemThe load of query execution can be distributed to multiple serversWe can improve the performance of whole systemClientClientSQL SQLSQLHigh Availability Load BalancingDB servers DB servers
  7. 7. Copyright © 2013 NTT DATA Corporation 7History
  8. 8. 8Copyright © 2013 NTT DATA CorporationHistory of PostgreSQL replication2007200820092010201220119.0(Sep 2010 release)• Async reploication9.1(Sep 2011 release)• Synchronous replication• Improve replication monitoring• pg_basebackup9.2(Sep 2012 release)• Cascade replication• Backup from standby• New sync rep modeSlony-IBucardo LondisteSequoiaPGClusterPostgresForestPostgres-RMammothPL/Proxypgpool-IIrubyrepPostgres-XCGridSQLsyncreplicatorReplication war !Historically the community policy has been to avoid putting replication into core. But,because of large user needs, replication has been introduced in 9.0.Then, replication keeps evolving steady.
  9. 9. Copyright © 2013 NTT DATA Corporation 9Features
  10. 10. 10Copyright © 2013 NTT DATA CorporationSingle master / Multiple standbysSingle masterMultiple standbys Multiple standbysReplicate ReplicateWrite SQLRead SQLRead SQLRead SQL Replication from single master to multiple standbys Cascade replication Only master accepts write query, both accepts read queryRead scalable, not write scalable
  11. 11. 11Copyright © 2013 NTT DATA CorporationRead-only query on standbyOnline backup– (Logical) pg_dump– (Physical) pg_basebackupMaintenance command– VACUUM, ANALYZE※No need to do VACUUM andANALYZE on standby becausethe result of mantenanceexecution on master isautomatically replicated tostandbyAllowQuery access– SELECT– PREPARE, EXECUTE– CURSOR operationDisallowDML– INSERT, UPDATE, DELETE– SELECT FOR UPDATEDDL– CREATE, DROP, ALTERTemporary table
  12. 12. 12Copyright © 2013 NTT DATA CorporationLog-shipping Transaction log (WAL) is shipped from master to standby Standby is in recovery mode Standby keeps the database current by replaying shipped WALRecoveryMaster StandbyClientWrite SQLWAL writingWAL shippingWAL writing
  13. 13. 13Copyright © 2013 NTT DATA CorporationLimitation by log-shippingThe followings must be the same between master and standby H/W and OS architecture PostgreSQL major versionSlony-I can be used for replication between differentarchitectures and major versionsMasterStandby64bit OSPostgreSQL9.2.1PostgreSQL9.1.032bit OS64bit OSPostgreSQL9.2.0NGNGOK21
  14. 14. 14Copyright © 2013 NTT DATA CorporationAll database objects are replicated Per-table granularity is not allowedSlony-I supports per-table granularity replicationPer database cluster granularityPer database cluster Per tableMaster Standby Master Standby
  15. 15. 15Copyright © 2013 NTT DATA CorporationSQL distributionPostgreSQL doesnt provide SQL distribution feature Implement SQL distribution logic into client application Use SQL distributor like pgpool-IIClientClientRead SQLWrite SQLマスタWrite SQLWrite/Read SQLDistributorスタンバイRead SQLMaster StandbyImplement logic Use SQL distributor
  16. 16. 16Copyright © 2013 NTT DATA CorporationFailoverPostgreSQL doesnt provide automatic failover feature Standby can be promoted to master anytime(pg_ctl promote) Automatic error detection and faliover requires clusterwareClientClientMasterpgpool-IIStandbyMaster スタンバイPacemaker pgpool-IIVIPPacemaker supports theresource agent for HA clusterusing PostgreSQL replication!
  17. 17. 17Copyright © 2013 NTT DATA CorporationMonitoring=# SELECT * FROM pg_stat_replication;-[ RECORD 1 ]----+------------------------------procpid | 26531usesysid | 10usename | postgresapplication_name | tokyoclient_addr | |client_port | 39654backend_start | 2012-02-01 18:54:49.429459+09state | streamingsent_location | 0/406E7CCwrite_location | 0/406E7CCflush_location | 0/406E7CCreplay_location | 0/406E1B0sync_priority | 1sync_state | syncReplication progressHow far has master sent WAL?How far has standby written, flushed orreplayed WAL?Replication connection informationStandby IP address, Port number,ROLE for replication、Replication start time, etcReplication statusCurrent synchronous mode,Standby has already caught up with master?
  18. 18. Copyright © 2013 NTT DATA Corporation 18Synchronous vs. Asynchronous
  19. 19. 19Copyright © 2013 NTT DATA CorporationReplication modeReplication mode is configurable Asynchronous Synchronous
  20. 20. 20Copyright © 2013 NTT DATA CorporationAsynchronous replicationCOMMIT doesnt wait for the completion of replication No guarantee that WAL has already arrived at standby at end ofCOMMITData loss window on failoverQuery on standby may see outdated dataLow performance impact on masterRecoveryMaster StandbyClientCOMMITWAL writingWAL shippingWAL writingOK123456Committed data getslost if failover happensbetween andoperations3 4
  21. 21. 21Copyright © 2013 NTT DATA CorporationSynchronous replicationCOMMIT waits for completion of replication WAL is guaranteed to be flushed in master and standby at end of COMMITNo data loss on failover!Relatively high performance impact on masterQuery on standby may see outdated data 「Synchronous ≠ Committed data is available on standby immediately」RecoveryMaster StandbyClientCOMMITWAL writingWAL shippingWAL writingOK126347Reply5WAL shipping is synchronous,but recovery is asynchronous
  22. 22. Copyright © 2011 NTT DATA CorporationCopyright © 2013 NTT DATA Corporation(Please omit notations when unnecessary)This document contains confidential Company information. Do not disclose it to third parties without permission from the Company.