SlideShare a Scribd company logo
Need is the mother of all Inventions. It is very
true for PostgreSQL 10 Logical Replication
PostgreSQL 10 Logical Replication
By:
Rajni Baliyan
Agenda:
PostgreSQL replication
history.
Logical Replication:
Overview
Use cases-SR vs LR
Terminologies used in
LR.
 Publication, Publisher
 Subscription, subscriber
 Replication Slot
Architecture
Security
Configuration Settings
Quick Setup
Monitoring
Precautions- Conflicts
Restrictions/Limitations
Q&A
Overview of PostgreSQL replication 9.0 onwards:
9.0
hot standby,
streaming
replication
(async)
9.1
pg_basebackup,
streaming
replication
(sync)
9.2
9.3
standby switch
timeline
cascading
streaming
replication
Overview of PostgreSQL replication 9.0 onwards
cont..
9.4
Replication
Slots, Logical
Decoding (LD)
9.5 pg_rewind
9.6
10
Native support
for Logical
replication.
More support
to LD
Logical Replication: Overview
Replicating/capturing logical changes based upon their replica identity
using replication slots.
Propagate any changes (Insert, Update, Delete, all, combination) to
replica.
Uses a publish and subscribe model.
Publisher-Sender
Subscriber- Receiver
Subscribers pulls data from publications.
Generally ‘asynchronous’ in nature.
Why Logical Replication over Streaming Replication?
Features Streaming Replication Logical Replication
Replication Type Binary replication(byte-by-byte) Selective row level changes (logical
change)
Hardware/OS Same (Linux-Linux) Can be different(Linux-Windows)
Replication between major
versions
Not supported Supported.
Replication Level Instance level Object level(table)
Consolidation No consolidation Consolidating is possible
Replica/ Subscriber- Open
Mode
Standby is in read-only mode Subscriber can be used for write
operations
Use Case1: Cross platform replication
 Cross Platform : Linux  Windows
Db1 Db2
Db2
Logical Replication
PostgreSQL 10 server1 PostgreSQL 10 server2
tb1 tb1
Create Subs. on Windows using Linux pub.
Insert on server1 Select on server2
Use Case2: Cross version:
 Cross Version: Replication between different major versions of PostgreSQL
Db1 Db2 Db3
Logical Replication
PostgreSQL 11 server2PostgreSQL10 server1
tb1 tb1
Use Case3: Write operation @Subscriber:
 Modifying objects and adding new tables, insert records on subscriber.
Tbl1 Tbl1
Logical ReplicationID Name Name ID Phno
ID Title Name
Tbl2
PostgreSQL 10 server2PostgreSQL 10 server1
PUB SUB
Use Case4: Consolidating multiple databases
PostgreSQL Server1 PostgreSQL
Server2(reporting)
PostgreSQL Server3
 Can be helpful for reporting or analytical purposes
 Can replicate even few tables from one DB to another on different servers.
db1
db3
db2
db4
db5
Reporting db6
tb1 tb1 tb2 tb2
Terminology used in Logical Replication:
Publisher/Publication
Subscriber/Subscription
Replication Slot
Replica Identity
Publisher/Publication
CREATE
ALTER
DROP Replica Identity
First create tables to include in publication and then create publication.
Publisher/Publication
Publisher  node where a publication is defined.
Publication  set of changes defined for tables.
Publication can be defined on any physical replication master.
DB name can be different from Subscriber.
Create publication using the ‘CREATE’ command and can be later ALTERED and
DROPPED.
A published table should exists before creating publication and must have a “replica
identity“ (can be primary key or index).
Operations possible on tables- INSERT, UPDATE, and DELETE, combinations, or
ALL(default)
UPDATE or DELETE operations on publisher without replica identity will give error.
Same replica identity has to be set at subscription side as well.
Currently replicates only tables.
Commands related to Publication:
Create:
• CREATE PUBLICATION testpub FOR TABLE tb1, tb2;
• CREATE PUBLICATION testpub FOR TABLE tbl3 WITH (publish =
'insert, update');
• CREATE PUBLICATION testpub FOR ALL TABLES;
Alter :
• ALTER PUBLICATION testpub ADD TABLE tb4, tbl5;
• ALTER PUBLICATION testpub SET TABLE tbl4, tbl5;
• ALTER PUBLICATION testpub DROP TABLE tbl4, tbl5;
Drop:
• DROP PUBLICATION testpub;
• DROP PUBLICATION testpub, testpub1;
Subscriber/Subscription
First create tables with same name, same column name and type, same replica
identity as publication.
Later create subscription.
Subscription:
Subscription  Downstream side of logical replication and this node is called
subscriber.
It can be used as a publisher for other databases.
Multiple subscriptions on one subscriber is possible.
Subscription should be created using superuser.
Same table names at publisher and subscriber.
Tables should be created first before creating a subscription.
Same columns in source and target tables with same data type but can have
different order.
Commands related to Subscription:
Create:
• CREATE SUBSCRIPTION testsub1 CONNECTION 'host=<remote>
'dbname=<db> user=<user>' PUBLICATION testpub1,testpub2;
• CREATE SUBSCRIPTION mysub1 CONNECTION '...' PUBLICATION
...WITH (enabled = false,create_slot = false,slot_name =
'myslot', copy_data = false, ...);
Alter:
• ALTER SUBSCRIPTION testsub ENABLE/DISABLE;
• ALTER SUBSCRIPTION testsub CONNECTION 'host=newhost ...';
• ALTER SUBSCRIPTION testsub SET(slot_name = 'newslot‘/’NONE’);
• ALTER SUBSCRIPTION testsub REFRESH PUBLICATION;
Drop:
• DROP SUBSCRIPTION testsub;
Security
For creating publication : CREATE privilege in the database.
To add tables to a publication, the user must have Ownership rights on the table.
To create a subscription, the user must be a superuser.
Role used for the replication connection must have the REPLICATION attribute (or
be a superuser)
Entry for replication role in pg_hba.conf
Configuration Settings
Parameters: to be set in postgresql.conf
@Publisher
 wal_level = logical
 # max_replication_slots  default value= 10
 # max_wal_senders  default value= 10
@Subscribers
 # max_replication_slots  default value= 10
 # max_logical_replication_workers  default value= 4
 # max_worker_processes
 # max_sync_workers_per_subscription  default value= 8
* Note: For remote connections, set- listen_addresses and pg_hba.conf
Quick Setup and Demo
Settings required for demo:
1. Setting postgresql.conf @Publisher
wal_level = logical
 Note: Default values of other required parameters will suffice for this demo.
2. Use replication role(CREATE ROLE replication WITH REPLICATION PASSWORD ‘’ LOGIN;
3. Setting pg_hba.conf on both nodes.
 host all postgres 0.0.0.0/0 SCRAM-SHA-256
 host all replication 0.0.0.0/0 SCRAM-SHA-256
* Note :The values here depend on your actual network configuration
Overview of Demo:
Server1=pg10srv1 Server2=pg10srv2
SubscriberPublisher
PUB SUB
test emptest
Publication: testpub Subscription: testsub
User
Subscriber
Creating database ‘PUB’ on Publisher:
postgres=# CREATE DATABASE PUB;
CREATE DATABASE
Creating table ‘TEST’ on Publisher:
pub=# CREATE TABLE TEST(ID NUMERIC PRIMARY
KEY,NAME TEXT);
CREATE TABLE
Insert some records in table ‘TEST’:
pub=# INSERT INTO TEST VALUES(1,'TOM'),
(2,'DICK');
INSERT 0 2
Select records of table ‘TEST’ :
pub=# select * from test;
id | name
----+------
1 | TOM
2 | DICK
(2 rows)
Let’s see whether there is any Publication or not?
pub=# dRp
List of publications
Name | Owner | All tables | Inserts | Updates | Deletes
------+-------+------------+---------+---------+--------
(0 rows)
Let’s create a Publication on Publisher node:
pub=# CREATE PUBLICATION testpub FOR TABLE TEST;
CREATE PUBLICATION
Let’s check the publication status:
pub=# dRp
* Default operation performed on tables is ‘All’ i.e. Insert, update and delete.
List of publications
Name | Owner | All tables | Inserts | Updates | Deletes
---------+----------+------------+---------+---------+--------
-
testpub | postgres | f | t | t | t
(1 row)
This publication should now be added to the tables:
pub=# d test
Table "public.test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
id | numeric | | not null |
name | text | | |
Indexes:
"test_pkey" PRIMARY KEY, btree (id)
Publications:
"testpub"
Now let’s set up subscription on subscriber
Let’s create a new database ‘SUB’ on Subscriber:
postgres=# CREATE DATABASE SUB;
CREATE DATABASE
Creating table ‘TEST’ on Subscriber:
sub=# CREATE TABLE TEST(NAME TEXT ,ID NUMERIC
PRIMARY KEY);
 Order of columns can be different.
CREATE TABLE
Let’s see whether there is any Subscription or not?
sub=# dRs
List of subscriptions
Name | Owner | Enabled | Publication
------+-------+---------+-------------
(0 rows)
Let’s create a Subscription on Subscriber node:
sub=# CREATE SUBSCRIPTION testsub CONNECTION '
user=postgres host=pg10srv1 dbname=pub ' PUBLICATION
testpub;
NOTICE: created replication slot "testsub" on
publisher
CREATE SUBSCRIPTION
Let’s check the subscription status:
sub=# dRs
List of subscriptions
Name | Owner | Enabled | Publication
---------+----------+---------+-------------
testsub | postgres | t | {testpub}
(1 row)
Let’s see whether we got the data in table or not?
sub=# select * from test;
name | id
------+----
TOM | 1
DICK | 2
(2 rows)
Lets verify subscription log:
-bash-4.2$ tail -f postgresql-Tue.log
LOG: worker process: logical replication worker for
subscription 49304 (PID 12053) exited with exit code 1
LOG: logical replication apply worker for subscription
"testsub" has started
LOG: logical replication table synchronization worker for
subscription "testsub", table "test" has started
LOG: logical replication table synchronization worker for
subscription "testsub", table "test" has finished
Now, let’s try to insert some data on Publisher node:
pub=# INSERT INTO TEST VALUES(3,'AFTER REPLICATION');
INSERT 0 1
Select data of ‘Test’ table on Subscriber:
sub=# select * from test;
id | name
----+-------------------
1 | TOM
2 | DICK
3 | AFTER REPLICATION
(3 rows)
Write operations @Subscriber: Creating table ‘EMP’
sub=# CREATE TABLE EMP(EMPID NUMERIC PRIMARY
KEY, EMPNAME TEXT);
CREATE TABLE
sub=# INSERT INTO EMP VALUES(1,'LEE'), (2,'TOM');
empid | empname
-------+---------
1 | LEE
2 | TOM
(2 rows)
Monitoring:
On Publisher
pg_stat_replication
pg_replication_slots
pg_stat_activity
dRp
On Subscriber
pg_stat_subscription
pg_stat_activity
dRs
Precautions at Subscriber:
While Inserting rows.
While modifying tables DDL – adding/dropping columns etc.
If not, will leads to conflicts.
CONFLICT
If incoming data violates any constraints the replication will stop and is called CONFLICT
What if we insert a record with id=4 on Publisher?
User
Publisher
Db1PUB
Db3
SUB
testtest
Publication: testpub Subscription: testsub
Id=4
Inserted later
Id=4
inserted
first
Server1=pg10srv1 Server2=pg10srv2
conflict
Subscriber
Check conflict status in subscriber log:
Error in log file:
On Subscriber On Publisher
sub=# SELECT * FROM TEST;
name | id
-------------------+----
TOM | 1
DICK | 2
AFTER REPLICATION | 3
CONFLICT | 4
(4 rows)
pub=# INSERT INTO TEST
VALUES(4,'RESOLVE');
sub=# SELECT * FROM
TEST;
name | id
-------------------+----
TOM | 1
DICK | 2
AFTER REPLICATION | 3
CONFLICT | 4
(4 rows)
Tab1
Tab1
Tab1
Delete a column on subscriber:
ID Name
ID Name
ID Name
PUBLISHER SUBSCRIBER
Tab1(different order)
Tab1(add column)
Tab1(delete column)
Name ID
Name ID Phno
ID Phno
conflict
When
insert
Resolving Conflict:
Change data on subscriber i.e. delete conflicting key.
Skip the conflicting transaction from replication by calling the
pg_replication_origin_advance() function.
The current position of origins can be seen in the
pg_replication_origin_status system view.
Restrictions:
Schema/DDL replication
Sequences replication
TRUNCATE command replication
Large objects replication
Views, Materialized views ,Partition root
tables, or foreign tables replication
Conclusion:
Good feature but still room to include more features.
No need to replicate whole instance.
Subscriber can be used for Write operations but with care.
Good for cross platform replication, different major version replication, write
operations on subscriber, like use cases.
Not the replacement of SR.
Thank you
Your speaker
Copyright 2017 FUJITSU LAUSTRALIA SOFTWARE TECHNOLOGY
Rajni Baliyan
Database Administrator
Fujitsu Enterprise Postgres / PostgreSQL
+61 2 9452 9017
rajnib@fast.au.fujitsu.com
postgesql.fastware.com
twitter.com/fujitsupostgres
linkedin.com/showcase/fujitsu-enterprtise-postgres
Q&A

More Related Content

What's hot

Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
Sadayuki Furuhashi
 
アーキテクチャから理解するPostgreSQLのレプリケーション
アーキテクチャから理解するPostgreSQLのレプリケーションアーキテクチャから理解するPostgreSQLのレプリケーション
アーキテクチャから理解するPostgreSQLのレプリケーション
Masahiko Sawada
 
Koalas: How Well Does Koalas Work?
Koalas: How Well Does Koalas Work?Koalas: How Well Does Koalas Work?
Koalas: How Well Does Koalas Work?
Databricks
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
Alexey Lesovsky
 
Postgresql 12 streaming replication hol
Postgresql 12 streaming replication holPostgresql 12 streaming replication hol
Postgresql 12 streaming replication hol
Vijay Kumar N
 
Open Policy Agent Deep Dive Seattle 2018
Open Policy Agent Deep Dive Seattle 2018Open Policy Agent Deep Dive Seattle 2018
Open Policy Agent Deep Dive Seattle 2018
Torin Sandall
 
Webinar: PostgreSQL continuous backup and PITR with Barman
Webinar: PostgreSQL continuous backup and PITR with BarmanWebinar: PostgreSQL continuous backup and PITR with Barman
Webinar: PostgreSQL continuous backup and PITR with Barman
Gabriele Bartolini
 
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
흥배 최
 
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
NTT DATA Technology & Innovation
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
Jiangjie Qin
 
PostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical ReplicationPostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical Replication
Atsushi Torikoshi
 
Integrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect FrameworkIntegrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect Framework
confluent
 
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
NTT DATA Technology & Innovation
 
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
NTT DATA Technology & Innovation
 
PostgreSQL: XID周回問題に潜む別の問題
PostgreSQL: XID周回問題に潜む別の問題PostgreSQL: XID周回問題に潜む別の問題
PostgreSQL: XID周回問題に潜む別の問題
NTT DATA OSS Professional Services
 
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Julian Hyde
 
PostgreSQL HA
PostgreSQL   HAPostgreSQL   HA
PostgreSQL HA
haroonm
 
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
Flink Forward
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
Flink Forward
 
Lightweight Transactions at Lightning Speed
Lightweight Transactions at Lightning SpeedLightweight Transactions at Lightning Speed
Lightweight Transactions at Lightning Speed
ScyllaDB
 

What's hot (20)

Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
 
アーキテクチャから理解するPostgreSQLのレプリケーション
アーキテクチャから理解するPostgreSQLのレプリケーションアーキテクチャから理解するPostgreSQLのレプリケーション
アーキテクチャから理解するPostgreSQLのレプリケーション
 
Koalas: How Well Does Koalas Work?
Koalas: How Well Does Koalas Work?Koalas: How Well Does Koalas Work?
Koalas: How Well Does Koalas Work?
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 
Postgresql 12 streaming replication hol
Postgresql 12 streaming replication holPostgresql 12 streaming replication hol
Postgresql 12 streaming replication hol
 
Open Policy Agent Deep Dive Seattle 2018
Open Policy Agent Deep Dive Seattle 2018Open Policy Agent Deep Dive Seattle 2018
Open Policy Agent Deep Dive Seattle 2018
 
Webinar: PostgreSQL continuous backup and PITR with Barman
Webinar: PostgreSQL continuous backup and PITR with BarmanWebinar: PostgreSQL continuous backup and PITR with Barman
Webinar: PostgreSQL continuous backup and PITR with Barman
 
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
KGC 2016 오픈소스 네트워크 엔진 Super socket 사용하기
 
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
Memoizeの仕組み(第41回PostgreSQLアンカンファレンス@オンライン 発表資料)
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
 
PostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical ReplicationPostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical Replication
 
Integrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect FrameworkIntegrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect Framework
 
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
PostgreSQLモニタリング機能の現状とこれから(Open Developers Conference 2020 Online 発表資料)
 
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
PostgreSQLのfull_page_writesについて(第24回PostgreSQLアンカンファレンス@オンライン 発表資料)
 
PostgreSQL: XID周回問題に潜む別の問題
PostgreSQL: XID周回問題に潜む別の問題PostgreSQL: XID周回問題に潜む別の問題
PostgreSQL: XID周回問題に潜む別の問題
 
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
 
PostgreSQL HA
PostgreSQL   HAPostgreSQL   HA
PostgreSQL HA
 
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
 
Lightweight Transactions at Lightning Speed
Lightweight Transactions at Lightning SpeedLightweight Transactions at Lightning Speed
Lightweight Transactions at Lightning Speed
 

Similar to Basics of Logical Replication,Streaming replication vs Logical Replication ,Use Cases in PostgreSQL10

My sql with querys
My sql with querysMy sql with querys
My sql with querys
NIRMAL FELIX
 
Demystifying postgres logical replication percona live sc
Demystifying postgres logical replication percona live scDemystifying postgres logical replication percona live sc
Demystifying postgres logical replication percona live sc
Emanuel Calvo
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
Alex Zaballa
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
Alex Zaballa
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
Alex Zaballa
 
Postgre sql run book
Postgre sql run bookPostgre sql run book
Postgre sql run book
Vasudeva Rao
 
plProxy, pgBouncer, pgBalancer
plProxy, pgBouncer, pgBalancerplProxy, pgBouncer, pgBalancer
plProxy, pgBouncer, pgBalancer
elliando dias
 
Kafka zero to hero
Kafka zero to heroKafka zero to hero
Kafka zero to hero
Avi Levi
 
Apache Kafka - From zero to hero
Apache Kafka - From zero to heroApache Kafka - From zero to hero
Apache Kafka - From zero to hero
Apache Kafka TLV
 
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptxBuilt-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
nadirpervez2
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
Michael Renner
 
Plproxy
PlproxyPlproxy
Plproxy
Joshua Drake
 
MySQL 101 PHPTek 2017
MySQL 101 PHPTek 2017MySQL 101 PHPTek 2017
MySQL 101 PHPTek 2017
Dave Stokes
 
Webinar: MariaDB Provides the Solution to Ease Multi-Source Replication
Webinar: MariaDB Provides the Solution to Ease Multi-Source ReplicationWebinar: MariaDB Provides the Solution to Ease Multi-Source Replication
Webinar: MariaDB Provides the Solution to Ease Multi-Source Replication
Wagner Bianchi
 
My sql.ppt
My sql.pptMy sql.ppt
9.1 Grand Tour
9.1 Grand Tour9.1 Grand Tour
9.1 Grand Tour
PostgreSQL Experts, Inc.
 
Oracle Database 12.1.0.2 New Features
Oracle Database 12.1.0.2 New FeaturesOracle Database 12.1.0.2 New Features
Oracle Database 12.1.0.2 New Features
Alex Zaballa
 
MySQL database replication
MySQL database replicationMySQL database replication
MySQL database replication
PoguttuezhiniVP
 
DBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should KnowDBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should Know
Alex Zaballa
 
Firebird
FirebirdFirebird
Firebird
Chinsan Huang
 

Similar to Basics of Logical Replication,Streaming replication vs Logical Replication ,Use Cases in PostgreSQL10 (20)

My sql with querys
My sql with querysMy sql with querys
My sql with querys
 
Demystifying postgres logical replication percona live sc
Demystifying postgres logical replication percona live scDemystifying postgres logical replication percona live sc
Demystifying postgres logical replication percona live sc
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
 
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should KnowDBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
DBA Brasil 1.0 - DBA Commands and Concepts That Every Developer Should Know
 
Postgre sql run book
Postgre sql run bookPostgre sql run book
Postgre sql run book
 
plProxy, pgBouncer, pgBalancer
plProxy, pgBouncer, pgBalancerplProxy, pgBouncer, pgBalancer
plProxy, pgBouncer, pgBalancer
 
Kafka zero to hero
Kafka zero to heroKafka zero to hero
Kafka zero to hero
 
Apache Kafka - From zero to hero
Apache Kafka - From zero to heroApache Kafka - From zero to hero
Apache Kafka - From zero to hero
 
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptxBuilt-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
Built-in-Physical-and-Logical-Replication-in-Postgresql-Firat-Gulec.pptx
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
 
Plproxy
PlproxyPlproxy
Plproxy
 
MySQL 101 PHPTek 2017
MySQL 101 PHPTek 2017MySQL 101 PHPTek 2017
MySQL 101 PHPTek 2017
 
Webinar: MariaDB Provides the Solution to Ease Multi-Source Replication
Webinar: MariaDB Provides the Solution to Ease Multi-Source ReplicationWebinar: MariaDB Provides the Solution to Ease Multi-Source Replication
Webinar: MariaDB Provides the Solution to Ease Multi-Source Replication
 
My sql.ppt
My sql.pptMy sql.ppt
My sql.ppt
 
9.1 Grand Tour
9.1 Grand Tour9.1 Grand Tour
9.1 Grand Tour
 
Oracle Database 12.1.0.2 New Features
Oracle Database 12.1.0.2 New FeaturesOracle Database 12.1.0.2 New Features
Oracle Database 12.1.0.2 New Features
 
MySQL database replication
MySQL database replicationMySQL database replication
MySQL database replication
 
DBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should KnowDBA Commands and Concepts That Every Developer Should Know
DBA Commands and Concepts That Every Developer Should Know
 
Firebird
FirebirdFirebird
Firebird
 

Recently uploaded

Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
Aftab Hussain
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 

Recently uploaded (20)

Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 

Basics of Logical Replication,Streaming replication vs Logical Replication ,Use Cases in PostgreSQL10

  • 1. Need is the mother of all Inventions. It is very true for PostgreSQL 10 Logical Replication PostgreSQL 10 Logical Replication By: Rajni Baliyan
  • 2. Agenda: PostgreSQL replication history. Logical Replication: Overview Use cases-SR vs LR Terminologies used in LR.  Publication, Publisher  Subscription, subscriber  Replication Slot Architecture Security Configuration Settings Quick Setup Monitoring Precautions- Conflicts Restrictions/Limitations Q&A
  • 3.
  • 4. Overview of PostgreSQL replication 9.0 onwards: 9.0 hot standby, streaming replication (async) 9.1 pg_basebackup, streaming replication (sync) 9.2 9.3 standby switch timeline cascading streaming replication
  • 5. Overview of PostgreSQL replication 9.0 onwards cont.. 9.4 Replication Slots, Logical Decoding (LD) 9.5 pg_rewind 9.6 10 Native support for Logical replication. More support to LD
  • 6. Logical Replication: Overview Replicating/capturing logical changes based upon their replica identity using replication slots. Propagate any changes (Insert, Update, Delete, all, combination) to replica. Uses a publish and subscribe model. Publisher-Sender Subscriber- Receiver Subscribers pulls data from publications. Generally ‘asynchronous’ in nature.
  • 7. Why Logical Replication over Streaming Replication? Features Streaming Replication Logical Replication Replication Type Binary replication(byte-by-byte) Selective row level changes (logical change) Hardware/OS Same (Linux-Linux) Can be different(Linux-Windows) Replication between major versions Not supported Supported. Replication Level Instance level Object level(table) Consolidation No consolidation Consolidating is possible Replica/ Subscriber- Open Mode Standby is in read-only mode Subscriber can be used for write operations
  • 8. Use Case1: Cross platform replication  Cross Platform : Linux  Windows Db1 Db2 Db2 Logical Replication PostgreSQL 10 server1 PostgreSQL 10 server2 tb1 tb1
  • 9. Create Subs. on Windows using Linux pub.
  • 10. Insert on server1 Select on server2
  • 11. Use Case2: Cross version:  Cross Version: Replication between different major versions of PostgreSQL Db1 Db2 Db3 Logical Replication PostgreSQL 11 server2PostgreSQL10 server1 tb1 tb1
  • 12. Use Case3: Write operation @Subscriber:  Modifying objects and adding new tables, insert records on subscriber. Tbl1 Tbl1 Logical ReplicationID Name Name ID Phno ID Title Name Tbl2 PostgreSQL 10 server2PostgreSQL 10 server1 PUB SUB
  • 13. Use Case4: Consolidating multiple databases PostgreSQL Server1 PostgreSQL Server2(reporting) PostgreSQL Server3  Can be helpful for reporting or analytical purposes  Can replicate even few tables from one DB to another on different servers. db1 db3 db2 db4 db5 Reporting db6 tb1 tb1 tb2 tb2
  • 14. Terminology used in Logical Replication: Publisher/Publication Subscriber/Subscription Replication Slot Replica Identity
  • 15. Publisher/Publication CREATE ALTER DROP Replica Identity First create tables to include in publication and then create publication.
  • 16. Publisher/Publication Publisher  node where a publication is defined. Publication  set of changes defined for tables. Publication can be defined on any physical replication master. DB name can be different from Subscriber. Create publication using the ‘CREATE’ command and can be later ALTERED and DROPPED. A published table should exists before creating publication and must have a “replica identity“ (can be primary key or index). Operations possible on tables- INSERT, UPDATE, and DELETE, combinations, or ALL(default) UPDATE or DELETE operations on publisher without replica identity will give error. Same replica identity has to be set at subscription side as well. Currently replicates only tables.
  • 17. Commands related to Publication: Create: • CREATE PUBLICATION testpub FOR TABLE tb1, tb2; • CREATE PUBLICATION testpub FOR TABLE tbl3 WITH (publish = 'insert, update'); • CREATE PUBLICATION testpub FOR ALL TABLES; Alter : • ALTER PUBLICATION testpub ADD TABLE tb4, tbl5; • ALTER PUBLICATION testpub SET TABLE tbl4, tbl5; • ALTER PUBLICATION testpub DROP TABLE tbl4, tbl5; Drop: • DROP PUBLICATION testpub; • DROP PUBLICATION testpub, testpub1;
  • 18. Subscriber/Subscription First create tables with same name, same column name and type, same replica identity as publication. Later create subscription.
  • 19. Subscription: Subscription  Downstream side of logical replication and this node is called subscriber. It can be used as a publisher for other databases. Multiple subscriptions on one subscriber is possible. Subscription should be created using superuser. Same table names at publisher and subscriber. Tables should be created first before creating a subscription. Same columns in source and target tables with same data type but can have different order.
  • 20. Commands related to Subscription: Create: • CREATE SUBSCRIPTION testsub1 CONNECTION 'host=<remote> 'dbname=<db> user=<user>' PUBLICATION testpub1,testpub2; • CREATE SUBSCRIPTION mysub1 CONNECTION '...' PUBLICATION ...WITH (enabled = false,create_slot = false,slot_name = 'myslot', copy_data = false, ...); Alter: • ALTER SUBSCRIPTION testsub ENABLE/DISABLE; • ALTER SUBSCRIPTION testsub CONNECTION 'host=newhost ...'; • ALTER SUBSCRIPTION testsub SET(slot_name = 'newslot‘/’NONE’); • ALTER SUBSCRIPTION testsub REFRESH PUBLICATION; Drop: • DROP SUBSCRIPTION testsub;
  • 21. Security For creating publication : CREATE privilege in the database. To add tables to a publication, the user must have Ownership rights on the table. To create a subscription, the user must be a superuser. Role used for the replication connection must have the REPLICATION attribute (or be a superuser) Entry for replication role in pg_hba.conf
  • 22. Configuration Settings Parameters: to be set in postgresql.conf @Publisher  wal_level = logical  # max_replication_slots  default value= 10  # max_wal_senders  default value= 10 @Subscribers  # max_replication_slots  default value= 10  # max_logical_replication_workers  default value= 4  # max_worker_processes  # max_sync_workers_per_subscription  default value= 8 * Note: For remote connections, set- listen_addresses and pg_hba.conf
  • 24. Settings required for demo: 1. Setting postgresql.conf @Publisher wal_level = logical  Note: Default values of other required parameters will suffice for this demo. 2. Use replication role(CREATE ROLE replication WITH REPLICATION PASSWORD ‘’ LOGIN; 3. Setting pg_hba.conf on both nodes.  host all postgres 0.0.0.0/0 SCRAM-SHA-256  host all replication 0.0.0.0/0 SCRAM-SHA-256 * Note :The values here depend on your actual network configuration
  • 25. Overview of Demo: Server1=pg10srv1 Server2=pg10srv2 SubscriberPublisher PUB SUB test emptest Publication: testpub Subscription: testsub User Subscriber
  • 26. Creating database ‘PUB’ on Publisher: postgres=# CREATE DATABASE PUB;
  • 28. Creating table ‘TEST’ on Publisher: pub=# CREATE TABLE TEST(ID NUMERIC PRIMARY KEY,NAME TEXT);
  • 30. Insert some records in table ‘TEST’: pub=# INSERT INTO TEST VALUES(1,'TOM'), (2,'DICK');
  • 32. Select records of table ‘TEST’ : pub=# select * from test;
  • 33. id | name ----+------ 1 | TOM 2 | DICK (2 rows)
  • 34. Let’s see whether there is any Publication or not? pub=# dRp
  • 35. List of publications Name | Owner | All tables | Inserts | Updates | Deletes ------+-------+------------+---------+---------+-------- (0 rows)
  • 36. Let’s create a Publication on Publisher node: pub=# CREATE PUBLICATION testpub FOR TABLE TEST;
  • 38. Let’s check the publication status: pub=# dRp
  • 39. * Default operation performed on tables is ‘All’ i.e. Insert, update and delete. List of publications Name | Owner | All tables | Inserts | Updates | Deletes ---------+----------+------------+---------+---------+-------- - testpub | postgres | f | t | t | t (1 row)
  • 40. This publication should now be added to the tables: pub=# d test
  • 41. Table "public.test" Column | Type | Collation | Nullable | Default --------+---------+-----------+----------+--------- id | numeric | | not null | name | text | | | Indexes: "test_pkey" PRIMARY KEY, btree (id) Publications: "testpub"
  • 42. Now let’s set up subscription on subscriber
  • 43. Let’s create a new database ‘SUB’ on Subscriber: postgres=# CREATE DATABASE SUB;
  • 45. Creating table ‘TEST’ on Subscriber: sub=# CREATE TABLE TEST(NAME TEXT ,ID NUMERIC PRIMARY KEY);  Order of columns can be different.
  • 47. Let’s see whether there is any Subscription or not? sub=# dRs
  • 48. List of subscriptions Name | Owner | Enabled | Publication ------+-------+---------+------------- (0 rows)
  • 49. Let’s create a Subscription on Subscriber node: sub=# CREATE SUBSCRIPTION testsub CONNECTION ' user=postgres host=pg10srv1 dbname=pub ' PUBLICATION testpub;
  • 50. NOTICE: created replication slot "testsub" on publisher CREATE SUBSCRIPTION
  • 51. Let’s check the subscription status: sub=# dRs
  • 52. List of subscriptions Name | Owner | Enabled | Publication ---------+----------+---------+------------- testsub | postgres | t | {testpub} (1 row)
  • 53. Let’s see whether we got the data in table or not? sub=# select * from test;
  • 54. name | id ------+---- TOM | 1 DICK | 2 (2 rows)
  • 55. Lets verify subscription log: -bash-4.2$ tail -f postgresql-Tue.log
  • 56. LOG: worker process: logical replication worker for subscription 49304 (PID 12053) exited with exit code 1 LOG: logical replication apply worker for subscription "testsub" has started LOG: logical replication table synchronization worker for subscription "testsub", table "test" has started LOG: logical replication table synchronization worker for subscription "testsub", table "test" has finished
  • 57. Now, let’s try to insert some data on Publisher node: pub=# INSERT INTO TEST VALUES(3,'AFTER REPLICATION');
  • 59. Select data of ‘Test’ table on Subscriber: sub=# select * from test;
  • 60. id | name ----+------------------- 1 | TOM 2 | DICK 3 | AFTER REPLICATION (3 rows)
  • 61. Write operations @Subscriber: Creating table ‘EMP’ sub=# CREATE TABLE EMP(EMPID NUMERIC PRIMARY KEY, EMPNAME TEXT);
  • 63. sub=# INSERT INTO EMP VALUES(1,'LEE'), (2,'TOM');
  • 64. empid | empname -------+--------- 1 | LEE 2 | TOM (2 rows)
  • 66. Precautions at Subscriber: While Inserting rows. While modifying tables DDL – adding/dropping columns etc. If not, will leads to conflicts.
  • 67. CONFLICT If incoming data violates any constraints the replication will stop and is called CONFLICT
  • 68. What if we insert a record with id=4 on Publisher? User Publisher Db1PUB Db3 SUB testtest Publication: testpub Subscription: testsub Id=4 Inserted later Id=4 inserted first Server1=pg10srv1 Server2=pg10srv2 conflict Subscriber
  • 69. Check conflict status in subscriber log: Error in log file: On Subscriber On Publisher sub=# SELECT * FROM TEST; name | id -------------------+---- TOM | 1 DICK | 2 AFTER REPLICATION | 3 CONFLICT | 4 (4 rows) pub=# INSERT INTO TEST VALUES(4,'RESOLVE'); sub=# SELECT * FROM TEST; name | id -------------------+---- TOM | 1 DICK | 2 AFTER REPLICATION | 3 CONFLICT | 4 (4 rows)
  • 70. Tab1 Tab1 Tab1 Delete a column on subscriber: ID Name ID Name ID Name PUBLISHER SUBSCRIBER Tab1(different order) Tab1(add column) Tab1(delete column) Name ID Name ID Phno ID Phno conflict When insert
  • 71. Resolving Conflict: Change data on subscriber i.e. delete conflicting key. Skip the conflicting transaction from replication by calling the pg_replication_origin_advance() function. The current position of origins can be seen in the pg_replication_origin_status system view.
  • 72. Restrictions: Schema/DDL replication Sequences replication TRUNCATE command replication Large objects replication Views, Materialized views ,Partition root tables, or foreign tables replication
  • 73. Conclusion: Good feature but still room to include more features. No need to replicate whole instance. Subscriber can be used for Write operations but with care. Good for cross platform replication, different major version replication, write operations on subscriber, like use cases. Not the replacement of SR.
  • 75. Your speaker Copyright 2017 FUJITSU LAUSTRALIA SOFTWARE TECHNOLOGY Rajni Baliyan Database Administrator Fujitsu Enterprise Postgres / PostgreSQL +61 2 9452 9017 rajnib@fast.au.fujitsu.com postgesql.fastware.com twitter.com/fujitsupostgres linkedin.com/showcase/fujitsu-enterprtise-postgres
  • 76. Q&A