Building Tungsten Clusters with PostgreSQL Hot Standby and Streaming Replication Linas Virbalas and Alex Alexander Continu...
Agenda <ul><li>Introductions </li></ul><ul><li>PG 9 Hot Standby and Streaming Replication </li></ul><ul><li>What is Tungst...
Introductions
About Continuent <ul><li>Our Business:  Continuous Data Availability </li></ul><ul><li>Our Solution </li></ul><ul><ul><li>...
PostgresSQL 9:  Hot Standby and  Log Streaming
PostgreSQL 8.4 Warm Standby WAL Files PostgreSQL Master pg_xlogs Directory Archived WAL Files Archive Directory PostgreSQL...
Limitations of Warm Standby <ul><li>Utilization -- Cannot open the standby </li></ul><ul><ul><li>To bring up the standby f...
Introducing Hot Standby <ul><li>Allows users to connect to standby in read-only mode </li></ul><ul><ul><li>Allowed:  SELEC...
Introducing Log Streaming PostgreSQL Master PostgreSQL Standby Continuous replication to standby Recovery WAL Sender WAL R...
Configuration and Usage <ul><li>Log streaming layers on top of existing warm standby log shipping </li></ul><ul><li>Multip...
What is Tungsten?
What Is Tungsten?  <ul><li>Tungsten implements master/slave clusters to: </li></ul><ul><ul><li>Protect data </li></ul></ul...
What’s Inside Tungsten? <ul><li>Replication - Making copies </li></ul><ul><ul><li>Tungsten Replicator  -- Database-neutral...
Tungsten Clustering In Action Master DB Slave DB Master Host Slave Host Replicator Monitor Manager Manager Manager Applica...
Distributed Rule-Based Management Broadcast commands  and monitoring data Business Rules Manager (Coordinator) Manager Man...
Multiple Routes to Databases Java App Server Tungsten SQL Router PHP Application Tungsten Connector libpq.a Tungsten Clust...
Tungsten + PostgreSQL Hot Standby and Streaming Replication
Moving Tungsten to PostgreSQL <ul><li>Problem:  We can’t read PostgreSQL logs (yet) </li></ul><ul><li>Tungsten solution is...
What is Tungsten’s Added Value To PostgreSQL? <ul><li>15 minute cluster installation </li></ul><ul><li>Single commands to:...
Streaming Replication Setup (By Hand) <ul><li>Configure master postgresql.conf and reboot </li></ul><ul><ul><li>archive_mo...
Manual Failover (By Hand) <ul><li>Standby’s postgresql.conf should be prepared from the start to act as a master when need...
Gotchas (1/2) <ul><li>Wait for standby to finish initial recovery before routing applications to it </li></ul><ul><li>“ ps...
Captchas (2/2) <ul><li>Need to know current progress? </li></ul><ul><ul><li>Use  pg_current_xlog_location()  on a master (...
Ultimately <ul><li>Hot Standby & Streaming Replication is  awesome , but </li></ul><ul><li>Set-up/management is  harder  t...
How Tungsten covers all this? DBMS Replicator JMX Interface Replication State Model Backup Storage Plugin pg_dump/ pg_rest...
DEMO or A Summary in Action
Questions?
Contact Information EMEA and APAC Lars Sonckin kaari 16 02600 Espoo, Finland Tel +358 50 517 9059 Fax +358 9 863 0060  Con...
Upcoming SlideShare
Loading in …5
×

Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replication-pg-east-2010Building Tungsten Clusters with PostgreSQL Hot Standby and Streaming Replication

4,477 views

Published on

Alex Alexander & Linas Virbalas

Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.

This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:

* Configuration of warm standby and streaming replication

* Provisioning new standby instances

* Strategies for balancing reads across primary and standby database

* Managing failover

* Troubleshooting and gotchas

Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,477
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
85
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Notes here.
  • Todays session is very straightforward Bullets During todays session you may submit questions at any point during the presentation by using the Q&amp;A box on the lower right of your webex screen. At the end of todays session we will address each question in the order they were received. Contact information is provided at the end of this presentation. For additional questions please feel free to contact us directly.
  • How do I get an eval copy of the software Can you share any whitepapers on this technology? What is the price? Our dedicated server price is based upon CPU’s and the size of the cluster The monthly subscription starts at $125 list for a 2 node. Virtual Private Clusters are priced differently and I would be glad to discuss this off-line Is the entire database kept in memory? Can I make schema changes while the cluster is running How do you differ from Veritas Cluster Server?
  • Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replication-pg-east-2010Building Tungsten Clusters with PostgreSQL Hot Standby and Streaming Replication

    1. 1. Building Tungsten Clusters with PostgreSQL Hot Standby and Streaming Replication Linas Virbalas and Alex Alexander Continuent, Inc.
    2. 2. Agenda <ul><li>Introductions </li></ul><ul><li>PG 9 Hot Standby and Streaming Replication </li></ul><ul><li>What is Tungsten? </li></ul><ul><li>Tungsten + PostgreSQL Hot Standby and Streaming Replication </li></ul><ul><li>Demo! </li></ul><ul><li>Questions and Comments </li></ul>
    3. 3. Introductions
    4. 4. About Continuent <ul><li>Our Business: Continuous Data Availability </li></ul><ul><li>Our Solution </li></ul><ul><ul><li>Continuent Tungsten (Master/Slave Database Replication) </li></ul></ul><ul><li>Our Value: </li></ul><ul><ul><li>Ensure data are available when and where you need them </li></ul></ul><ul><ul><li>TCO less than 20% of comparable solutions </li></ul></ul><ul><li>Our Technical Expertise </li></ul><ul><ul><li>Database replication </li></ul></ul><ul><ul><li>Database cluster management </li></ul></ul><ul><ul><li>Application connectivity </li></ul></ul><ul><ul><li>Software-as-a-Service (SaaS) </li></ul></ul>
    5. 5. PostgresSQL 9: Hot Standby and Log Streaming
    6. 6. PostgreSQL 8.4 Warm Standby WAL Files PostgreSQL Master pg_xlogs Directory Archived WAL Files Archive Directory PostgreSQL Standby WAL Files pg_xlogs Directory pg_standby rsync to standby Continuous recovery
    7. 7. Limitations of Warm Standby <ul><li>Utilization -- Cannot open the standby </li></ul><ul><ul><li>To bring up the standby for queries you must end recovery </li></ul></ul><ul><ul><li>Standby hardware is idle </li></ul></ul><ul><ul><li>Difficult to track state of recovery since you cannot query log position </li></ul></ul><ul><li>Data Loss -- Warm standby transfers only full WAL files </li></ul><ul><ul><li>Can bound loss using archive_timeout </li></ul></ul><ul><ul><li>Low values create large numbers of WAL files; complicate point-in-time recovery </li></ul></ul><ul><ul><li>Workarounds using DRBD, etc. are complex </li></ul></ul>
    8. 8. Introducing Hot Standby <ul><li>Allows users to connect to standby in read-only mode </li></ul><ul><ul><li>Allowed: SELECT, SET, LOAD, COMMIT/ROLLBACK </li></ul></ul><ul><ul><li>Disallowed: INSERT, UPDATE, DELETE, CREATE, 2PC, SELECT … FOR SHARE/UPDATE, nextval(), LISTEN, LOCK, </li></ul></ul><ul><ul><li>No admin commands: ANALYZE, VACUUM, REINDEX, GRANT </li></ul></ul><ul><li>Can come out of recovery while queries are running </li></ul><ul><li>Thanks to Simon Riggs for this description </li></ul>
    9. 9. Introducing Log Streaming PostgreSQL Master PostgreSQL Standby Continuous replication to standby Recovery WAL Sender WAL Receiver Archived WAL Files Archive Directory Archiving
    10. 10. Configuration and Usage <ul><li>Log streaming layers on top of existing warm standby log shipping </li></ul><ul><li>Multiple standby servers allowed </li></ul><ul><li>Failure of one standby does not affect others </li></ul><ul><li>Management is not simple - must coordinate provisioning & WAL shipping to set up/restart </li></ul>
    11. 11. What is Tungsten?
    12. 12. What Is Tungsten? <ul><li>Tungsten implements master/slave clusters to: </li></ul><ul><ul><li>Protect data </li></ul></ul><ul><ul><li>Maintain high availability </li></ul></ul><ul><ul><li>Improve resource utilization </li></ul></ul><ul><ul><li>Raise performance </li></ul></ul><ul><li>Install and set up in a few minutes </li></ul><ul><li>Integrated backup/restore and data integrity checks </li></ul><ul><li>Efficient failover operations </li></ul><ul><li>Distributed, rule-driven management </li></ul><ul><li>No/minimal application changes </li></ul><ul><li>Highly pluggable </li></ul><ul><li>No specialized hardware requirements </li></ul>
    13. 13. What’s Inside Tungsten? <ul><li>Replication - Making copies </li></ul><ul><ul><li>Tungsten Replicator -- Database-neutral, platform independent master/slave replication </li></ul></ul><ul><li>Connectivity -- Finding databases </li></ul><ul><ul><li>Tungsten Connector -- Fast MySQL/PostgreSQL client to JDBC proxying </li></ul></ul><ul><ul><li>Tungsten SQL Router --JDBC wrapper for high-performance and transparent failover, load-balancing, and partitioning (no proxy required) </li></ul></ul><ul><li>Management -- Administering the database </li></ul><ul><ul><li>Tungsten Manager -- Distributed administration with autonomic, rule-based configuration and no single point of failure </li></ul></ul><ul><ul><li>Tungsten Monitor -- Track resource status and </li></ul></ul>
    14. 14. Tungsten Clustering In Action Master DB Slave DB Master Host Slave Host Replicator Monitor Manager Manager Manager Application Server SQL Router/Connector Application Server SQL Router/Connector Management Client Management Client Replicator Monitor Manager
    15. 15. Distributed Rule-Based Management Broadcast commands and monitoring data Business Rules Manager (Coordinator) Manager Manager Admin Client Admin Client Group Communications Admin Client Local Services Local Services Local Services
    16. 16. Multiple Routes to Databases Java App Server Tungsten SQL Router PHP Application Tungsten Connector libpq.a Tungsten Cluster PostgreSQL JDBC Driver Admin & Monitoring Admin & Monitoring Virtual IP Address
    17. 17. Tungsten + PostgreSQL Hot Standby and Streaming Replication
    18. 18. Moving Tungsten to PostgreSQL <ul><li>Problem: We can’t read PostgreSQL logs (yet) </li></ul><ul><li>Tungsten solution is to manage : </li></ul><ul><li>Warm Standby + WAL Shipping (for PostgreSQL <9) </li></ul><ul><ul><li>Good basic availability/fast failover </li></ul></ul><ul><ul><li>Slaves open up for reads only after failover </li></ul></ul><ul><ul><li>No load balancing/scaling </li></ul></ul><ul><li>Hot Standby + Streaming Replication (for PostgreSQL >=9) </li></ul><ul><ul><li>Slaves opened up for reads = Tungsten scaling facilities work </li></ul></ul><ul><ul><li>Add Streaming Replication = minimal delay in replicating data </li></ul></ul><ul><ul><li>A fully fledged clustering solution </li></ul></ul>
    19. 19. What is Tungsten’s Added Value To PostgreSQL? <ul><li>15 minute cluster installation </li></ul><ul><li>Single commands to: </li></ul><ul><ul><li>View cluster status </li></ul></ul><ul><ul><li>Provision a new standby </li></ul></ul><ul><ul><li>Confirm liveness of replication </li></ul></ul><ul><ul><li>Switch servers safely for maintenance </li></ul></ul><ul><ul><li>Failover a dead server to most current replica </li></ul></ul><ul><li>Automatic discovery of new database replicas </li></ul><ul><li>Automatic failover when databases fail </li></ul><ul><li>Simple procedures for provisioning </li></ul><ul><li>Transparent application routing </li></ul><ul><li>Easy scaling </li></ul>
    20. 20. Streaming Replication Setup (By Hand) <ul><li>Configure master postgresql.conf and reboot </li></ul><ul><ul><li>archive_mode = on </li></ul></ul><ul><ul><li>max_wal_senders = 10 </li></ul></ul><ul><ul><li>recovery_connections = on </li></ul></ul><ul><ul><li>archive_command =‘rsync -cz $1 ${STANDBY}:${PGHOME}/archive/$2 %p %f' </li></ul></ul><ul><ul><li>archive_timeout = 60 </li></ul></ul><ul><li>Set up standby recovery.conf </li></ul><ul><ul><li>standby_mode = 'on’ </li></ul></ul><ul><ul><li>primary_conninfo = 'host=${MASTER} port=5432 user=postgres’ </li></ul></ul><ul><ul><li>trigger_file = '/tmp/pgsql_stop_recovery' </li></ul></ul><ul><li>Provision standby </li></ul><ul><ul><li>psql# select pg_switch_xlog(); </li></ul></ul><ul><ul><li>psql# select pg_xlogfile_name(pg_start_backup('base_backup')); </li></ul></ul><ul><ul><li>rsync –azv --delete --exclude=*pg_xlog* --exclude=postgresql.conf ${PGHOME}/ $STANDBY:$PGHOME/archive </li></ul></ul><ul><ul><li>psql# select pg_xlogfile_name(pg_stop_backup()); </li></ul></ul><ul><li>Start standby, recovery starts </li></ul>
    21. 21. Manual Failover (By Hand) <ul><li>Standby’s postgresql.conf should be prepared from the start to act as a master when needed: </li></ul><ul><ul><li>archive_mode = on </li></ul></ul><ul><ul><li>max_wal_senders = 10 </li></ul></ul><ul><li>Touch /tmp/pgsql_stop_recovery on a standby to snap out of recovery mode </li></ul><ul><li>Wait for recovery to finish </li></ul><ul><ul><li>ERROR: recovery is in progress … LOG: database system is ready to accept connections </li></ul></ul><ul><li>Reroute applications to the new master </li></ul><ul><li>And to return to original cluster’s state: </li></ul><ul><li>Fix the failed master. Bring it online as a standby </li></ul><ul><li>Switch master with standby roles </li></ul>
    22. 22. Gotchas (1/2) <ul><li>Wait for standby to finish initial recovery before routing applications to it </li></ul><ul><li>“ psql: FATAL: the database system is starting up” – bad check for that. E.g.: try under French locale </li></ul><ul><li>After a switch/failover, need to wait for the new master to come up from recovery too </li></ul><ul><li>rsync of master data folder might return a non-zero exit code: 24 - “Partial transfer due to vanished source files” </li></ul><ul><li>Exclude pg_xlog, pg_log and postgresql.conf from rsync! </li></ul><ul><li>Careful: queries might get canceled on a standby! (E.g. DROP TABLE on a master doesn’t wait for SELECT queries to finish on a standby) </li></ul>
    23. 23. Captchas (2/2) <ul><li>Need to know current progress? </li></ul><ul><ul><li>Use pg_current_xlog_location() on a master (“ERROR:  recovery is in progress&quot; if used on a standby), pg_last_xlog_receive_location() and pg_last_xlog_replay_location() on a standby. </li></ul></ul><ul><li>Using pg_standby? Ensure it is available in the path for the user you’re running </li></ul><ul><ul><li>If it’s for the root, ‘sudo which pg_standby’ is not enough! </li></ul></ul><ul><li>Running Debian? Check for PostgreSQL status before starting it (it fails, if server is already running): </li></ul><ul><ul><li>sudo /etc/init.d/postgresql-8.4 start Starting PostgreSQL 8.4: pg_ctl: another server might be running; trying to start server anyway pg_ctl: could not start server </li></ul></ul><ul><li>Etc. </li></ul>
    24. 24. Ultimately <ul><li>Hot Standby & Streaming Replication is awesome , but </li></ul><ul><li>Set-up/management is harder than it looks. </li></ul><ul><li>Monitoring is critical. </li></ul><ul><li>Transparent application rerouting is essential. </li></ul>
    25. 25. How Tungsten covers all this? DBMS Replicator JMX Interface Replication State Model Backup Storage Plugin pg_dump/ pg_restore Plug-In Monitor DBMS Checker Plugin Tungsten Manager postgresql.conf recovery.conf pg_standby rsync Pg-wal Scripts Open Script Plugin
    26. 26. DEMO or A Summary in Action
    27. 27. Questions?
    28. 28. Contact Information EMEA and APAC Lars Sonckin kaari 16 02600 Espoo, Finland Tel +358 50 517 9059 Fax +358 9 863 0060 Continuent Web Site: http://www.continuent.com HQ and Americas 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e -mail: [email_address] , linas.virbalas@continuent.com

    ×