Your SlideShare is downloading. ×
Built-in Replication in PostgreSQL
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Built-in Replication in PostgreSQL

1,404
views

Published on

@LINUXCON JAPAN 2010 …

@LINUXCON JAPAN 2010
https://events.linuxfoundation.org/2010/linuxcon-japan/masao

Published in: Technology, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,404
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
43
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Built-in Replication in PostgreSQL Fujii Masao NTT OSS Center 09/27/2010Copyright(c)2010 NTT, Inc. All Rights Reserved.
  • 2. Who am I? • Database engineer in NTT Open Source Software Center • PostgreSQL developer since 2008 • Author of new built-in replicationCopyright(c)2010 NTT, Inc. All Rights Reserved. 2
  • 3. Abstract • What’s replication? • Background • How does the built-in replication work? – Features – Architecture – Limitations – Future worksCopyright(c)2010 NTT, Inc. All Rights Reserved. 3
  • 4. What’s replication? • Create a replica of the database on multiple servers – Multiple servers have the same database Client Change Change Original ReplicasCopyright(c)2010 NTT, Inc. All Rights Reserved. 4
  • 5. Why replication? • High Availability – Reduces the system downtime • Load Balancing – Improve the system performance High Availability Load Balancing Client Client SQL SQL SQL DBMS DBMSCopyright(c)2010 NTT, Inc. All Rights Reserved. 5
  • 6. Background • Historical policy – Avoid putting replication into core PostgreSQL – No "one size fits all" replication solution • Replication war! Postgres-XC syncreplicator rubyrep PL/Proxy Londiste Postgres-R Sequoia Slony-I Bucardo pgpool-II Mammoth GridSQL PGCluster PyReplica PostgresForestCopyright(c)2010 NTT, Inc. All Rights Reserved. 6
  • 7. Road to core • No default choice – Too complex to install and use for simple cases – Low activity, easily-inactive – No Japanese document – Cannot work on other than linux – vs. other dbms • v9.0 – Simple, reliable basic replication in coreCopyright(c)2010 NTT, Inc. All Rights Reserved. 7
  • 8. Built-in replication in PostgreSQL 9.0 • Streaming Replication – Capability to stream changes on master to standby • Hot Standby – Capability to run read-only queries on standby • 1+1=3 Client Hot Standby R/W SQL R/O SQL Master Standby Change Streaming ReplicationCopyright(c)2010 NTT, Inc. All Rights Reserved. 8
  • 9. Master / Standbys • One master / Multiple standbys – Only master accepts write query – Both accepts read query • Read scalable – Not write scalable Client Standbys R/O SQL R/W SQL Change MasterCopyright(c)2010 NTT, Inc. All Rights Reserved. 9
  • 10. Cascading vs. Proxy Client Not allow Cascading Master Standby Standby Client Allow Proxy approach Master Proxy StandbysCopyright(c)2010 NTT, Inc. All Rights Reserved. 10
  • 11. Hot Standby Allow • Query access • Logical hot backup – SELECT – pg_dump Not allow • Data Manipulation • Maintenance Language (DML) – VACUUM, ANALYZE – INSERT, UPDATE, DELETE – (Replicated from master) – SELECT FOR UPDATE • Physical hot backup • Data Definition – pg_start/stop_backup Language (DDL) – CREATE, DROP, ALTERCopyright(c)2010 NTT, Inc. All Rights Reserved. 11
  • 12. Log-shipping • WAL is shipped from master to standby – WAL a.k.a transaction log • Standby in recovery mode – Keeps the database current by replaying receved WAL Client Write query Master Standby WAL Recovery WAL WAL DatabaseCopyright(c)2010 NTT, Inc. All Rights Reserved. 12
  • 13. Limitation by log-shipping • Must be the same between master and standby – H/W architecture – PostgreSQL major version Client Standby OS: 32bit NG Master NG PG: v9.1.0 OS: 64bit PG: v9.0.0 OK OS: 64bit PG: v9.0.2Copyright(c)2010 NTT, Inc. All Rights Reserved. 13
  • 14. Per database cluster granularity • All database objects are replicated – Per-table granularity is not allowed Per database cluster Per table Master Standby Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 14
  • 15. Easy migration • No need to change table definition – cf. Slony-I forces table to have a primary key • No need to rewrite SQL – cf. Slony-I doesn’t replicate DDL – All the SQL PostgreSQL supports are available in master • Easy to use existing database server as master Client Client Easy migration! Stand-alone Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 15
  • 16. No query distribution • Postgres doesn’t provide query distribution capability – Implement query distribution logic into application – Use query distributor Implement logic Use distributor Client Client Query Write query Read query Distributor Write query Read query Master Standby Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 16
  • 17. Shared nothing • WAL is shipped via network – No special H/W required – No distance limitation – No single point of failure Shared nothing Shared disk Master Standby Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 17
  • 18. Asynchronous • WAL is shipped asynchronously – Low performance impact on the master – Data loss window on failover – Query on the standby sees a bit outdated transactions Client thinks this Client transaction has been committed. But.. Transaction “success” Transaction has Master WAL Standby not been replicated yet.Copyright(c)2010 NTT, Inc. All Rights Reserved. 18
  • 19. Failover • Standby can be brought up anytime – Automatic failover requires clusterware • Failover time is relatively short Client Client Pacemaker pgpool-II VIP pgpool-II Master Standby Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 19
  • 20. Online standby addtion and deletion • Standby can be added or deleted without downtime of the master and the other standbys – This is useful for small start system Client Don’t need to Master stop master during adding new standby New StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 20
  • 21. Built-in • Easy to install and use – Need to install only Postgres – User-intuitive usage – Run on all the major operating systems • Highly active community – Volunteers translate the document into Japanese – Bug will be fixed soon – Continuous improvement and development – Many usersCopyright(c)2010 NTT, Inc. All Rights Reserved. 21
  • 22. Architecture Client Write query Read query access database backend backend database backend change apply send receive backend backend backend walsender walreceiver startup write read write read WAL WAL Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 22
  • 23. Multiple standbys • One-to-one relationship between walsender and standby – WAL is shipped to each standby in parallel backend backend walsender walreceiver startup backend WAL Standby WAL walsender walreceiver WAL startup Standby walsender walreceiver WAL startup Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 23
  • 24. Walsender and WAL • Walsender always reads WAL from disk – Prevent standby from going ahead of master – Avoid loss of consistency between master and standby • WAL is basically read from file cache – WAL is read just after written – I/O load by walsender is not high – But, WAL is read from disk if standby falls far behind masterCopyright(c)2010 NTT, Inc. All Rights Reserved. 24
  • 25. Recovery vs. Read-only query Client Read query access backend backend database backend apply startup read WAL RecoveryCopyright(c)2010 NTT, Inc. All Rights Reserved. 25
  • 26. Recovery vs. Read-only query • Until the conflict has been resolved, – Read query returns outdated result – Failover is blocked • Parameter specifying maximum delay in recovery – Increase the delay when running time-consuming job – Decrease the delay when we want to make the failover time shortCopyright(c)2010 NTT, Inc. All Rights Reserved. 26
  • 27. Recovery vs. Read-only query Client Write query Don’t interfere with log-shipping database change send receive backend backend backend walsender walreceiver write read write WAL WAL Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 27
  • 28. Future work - Synchronous • Synchronous replication is essential to avoid data loss on failover – Currently under development for 9.1 – Three synchronization levels: recv, fsync, apply Client Write query database “success” change send receive backend backend backend walsender walreceiver reply write read recv WAL Master waits until standby Master has received WAL StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 28
  • 29. Future work - Synchronous • Synchronous replication is essential to avoid data loss on failover – Currently under development for 9.1 – Three synchronization levels: recv, fsync, apply Client Write query fsync access database “success” Master waits until standby has written WAL change send receive backend backend backend walsender walreceiver reply write read write WAL WAL Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 29
  • 30. Future work - Synchronous • Synchronous replication is essential to avoid data loss on failover – Currently under development for 9.1 – Three synchronization levels: recv, fsync, apply Client Write query apply Read query Master waits until standby has applied WAL database “success” database change apply send receive backend backend backend walsender walreceiver startup reply write read write read WAL WAL Master StandbyCopyright(c)2010 NTT, Inc. All Rights Reserved. 30
  • 31. Future work - Synchronous • Per-transaction control – Some transactions are important, others are not • Quorum commit – Master waits for N standbysCopyright(c)2010 NTT, Inc. All Rights Reserved. 31
  • 32. If you find bug or problem • Bug report form – http://www.postgresql.org/support/submit bug • Mail – pgsql-jp@ml.postgresql.jp – masao.fujii@gmail.comCopyright(c)2010 NTT, Inc. All Rights Reserved. 32
  • 33. Thank you for listeningCopyright(c)2010 NTT, Inc. All Rights Reserved.

×