Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Logical Replication in PostgreSQL - FLOSSUK 2016

1,280 views

Published on

Slides from FLOSSUK Spring 2016 conference about the logical replication in PostgreSQL using pglogical extension.

Published in: Technology
  • Be the first to comment

Logical Replication in PostgreSQL - FLOSSUK 2016

  1. 1. © 2ndQuadrant 2016 Logical Replication in PostgreSQL Petr Jelínek PostgreSQL Contributor & Consultant
  2. 2. © 2ndQuadrant 2016 History
  3. 3. © 2ndQuadrant 2016 Postgres Replication History • PostgreSQL 7.x (~2000) – Replication should not be part of core Postgres • PostgreSQL 8.0 (2005) – Point-In-Time Recovery • PostgreSQL 9.0 (2010) – Streaming Replication (physical) • PostgreSQL 9.4 (2014) – Logical Decoding (changeset extraction)
  4. 4. © 2ndQuadrant 2016 Logical Replication History • Trigger based solutions – Slony (~2004) – Londiste (~2007) • Result of the no replication in core philosophy • Use table(s) as queue – Amplify writes on the upstream • Written in scripting languages • But work for the most part
  5. 5. © 2ndQuadrant 2016 BDR Project • Native logical replication • Multi-master • Firm design 2011 • Prototype 2012 • Resulted in many patches to PostgreSQL 9.3, 9.4, 9.5 and now 9.6 – For example Logical Decoding • Not yet fully integrated into PostgreSQL
  6. 6. © 2ndQuadrant 2016 Current state • BDR – Modified PostgreSQL 9.4 + Extension – Multi-master – Transparent DDL • pglogical – Extension for 9.4+ – One way replication – Replacement for trigger-based solutions – Integrating into PostgreSQL core
  7. 7. © 2ndQuadrant 2016 Streaming Replication
  8. 8. © 2ndQuadrant 2016 Physical Streaming Replication WAL Sender DatabaseWAL WAL Recvr Startup DatabaseWAL User Master Standby
  9. 9. © 2ndQuadrant 2016 Logical Streaming Replication WAL Sender Database Apply DatabaseWAL User Master Standby
  10. 10. © 2ndQuadrant 2016 Logical Decoding
  11. 11. © 2ndQuadrant 2016 Logical Decoding WAL Sender DatabaseWAL User Master
  12. 12. © 2ndQuadrant 2016 Logical Decoding • Extracts information from Write-Ahead Log into logical changes (INSERT/UPDATE/DELETE) • Per row and commit ordered • C API for output plugin • No DDL • SQL Interface • Streaming Interface
  13. 13. © 2ndQuadrant 2016 Existing output plugins • https://github.com/confluentinc/bottledwater-pg – Decodes change into AVRO – Sends data to Kafka • https://github.com/xstevens/decoderbufs – Decodes to protocol buffers • pglogical_output – Plugin that pglogical uses – “Native” protocol – JSON protocol
  14. 14. © 2ndQuadrant 2016 pglogical
  15. 15. © 2ndQuadrant 2016 pglogical WAL Sender Database Apply DatabaseWAL User Master Standby
  16. 16. © 2ndQuadrant 2016 Logical Replication • Target node is writeable – Allows temp tables – Allows different indexes – Allows different security – Allows (some) data transformation • Selective Replication – Can replicate subset of database • Cross-version
  17. 17. © 2ndQuadrant 2016 pglogical • Replicates transactions in commit order • Selective Replication • Online Upgrade • Data Transport – Data integration – Streaming changes to analytical database – Master configuration data management – … • Optionally semi-synchronous
  18. 18. © 2ndQuadrant 2016 Replication Sets • Replication is defined in terms of groups (sets) of tables, rather than individual tables – Need to be defined on each upstream node • Table is not replicated until added to a set • Tables may be defined in more than one set, but changes for the table will only be sent once to each subscriber • Predefined sets, “default”, “default_insert_only”, “ddl_sql”
  19. 19. © 2ndQuadrant 2016 Replication Actions • By default replication sets replicate all actions – INSERT, UPDATE, DELETE, TRUNCATE • It's possible to filter actions for given replication set • Useful for data aggregation, data warehousing etc.
  20. 20. © 2ndQuadrant 2016 Installation • Extension – All configuration in database • Provider + Subscriber – CREATE EXTENSION pglogical; – create_node(name, dsn) • Provider – replication_set_add_table(set_name, table_name) • Subscriber – create_subscription(name, provider_dsn, ...)
  21. 21. © 2ndQuadrant 2016 Example setup OLTP1 OLTP2 OLAP CFG
  22. 22. © 2ndQuadrant 2016 Performance 1 2 4 8 16 24 32 0 2000 4000 6000 8000 10000 12000 14000 pglocical slony londiste3 SR
  23. 23. © 2ndQuadrant 2016 DDL Replication • Initial schema either fully synchronized or not at all • The DDL commands are not automatically replicated yet • pglogical.replicate_ddl_command( command [, replication_sets])
  24. 24. © 2ndQuadrant 2016 Other caveats • Sequences not replicated yet – Fixed in upcoming 1.1 • Big transactions • Does not play well with physical replication yet – Failover – Fixed in PostgreSQL 9.6
  25. 25. © 2ndQuadrant 2016 Future plans • Make it available in PostgreSQL out of the box – 9.7? • Data transformation hooks • Filtering by expression • Push replication • Integrate BDR features – Multi-master – Transparent DDL replication
  26. 26. © 2ndQuadrant 2016 Questions? • petr@2ndquadrant.com • http://2ndquadrant.com/pglogical/

×