PostgreSQL major version upgrade using built in Logical Replication

Copyright©2018 NTT Corp. All Rights Reserved.
PostgreSQL Major Version
Upgrade Using built-in
Logical Replication
NTT OSS Center
Atsushi Torikoshi
PGConf.ASIA 2018

22Copyright©2018 NTT Corp. All Rights Reserved.
• Upgrading PostgreSQL cluster
• Introduction to Logical Replication
• Architecture and Behavior of Logical
Replication
• Major version upgrade using logical
replication
Agenda

Upgrading
PostgreSQL Cluster

Minor Version Upgrade
• e.g. 9.6.0 -> 9.6.1, 10.3 -> 10.6
• always compatible with earlier and later minor releases
• just need upgrade the PostgreSQL binary
Major Version Upgrade
• e.g. 9.5.1 -> 9.6.15, 10.2 -> 11.1
• not always compatible, the internal data storage format can be
changed
• upgrading the PostgreSQL binary is not enough, it needs also
upgrading data files
⇒ several ways for major version upgrade
2 types of version upgrade

pg_dump
o the traditional way for major version up
x service downtime during dump and restore
pg_upgrade
o faster than pg_dump
x shorter than pg_dump, but it still needs service downtime
link mode is much faster, but you cannot go back the original
cluster once you start up the new one
Logical Replication
o it doesn't need stop PostgreSQL service during replication
⇒ near zero downtime
x built-in logical replication is supported only after version 10
3 ways for major version upgrade

Introduction to
Logical Replication

Physical Replication replicates a whole DB cluster
by sending & replaying all the WAL.
Looking back Physical Replication
Upstream Downstream
sendTable
Table
Table
WALWAL WALWAL Table
Table
Table
replay

Physical Replication cannot do things like:
• partial replication(partial data, partial manipulation)
• replication between different OS arc/major version
PostgreSQL
Logical Replication has added flexibility to built-in
replication and made these things possible.
Motivations for Logical Replication
Upstream Downstream
decode, sendTable
Table
Table
WALWAL WALWAL Table
Table
apply
write

So, how can we upgrade our
cluster with Logical Replication?

1. Create a new cluster
How can we upgrade our cluster with
Logical Replication?
Current New
APP

2. Start Logical Replication
Current New
APP

3. Switch access point
Current New
APP

That’s all. It’s a piece of cake!
Current New
APP

That’s all. It’s a piece of cake!
Logical Replication has some limitations and does
characteristic behaviors.
Current New
APP

Architecture and Behaviors
of Logical Replication

• ‘walsender’ and ‘apply worker’ do most of the
works for Logical Replication.
Basics of the architecture
WAL
wal
sender
Publisher (upstream)
write
apply
worker
launcher
launch
Subscriber(downstream)
backend
process
read
decode
backend
process

• ‘walsender’ and ‘apply worker’ do most of the
works for Logical Replication.
• ‘sync worker’ and the corresponding ‘walsender’
run only at initial table sync.
Basics of the architecture
WAL
wal
sender
Publisher (upstream)
write
wal
sender
apply
worker
launcher
sync
worker
launch
launch
Subscriber(downstream)
backend
process
read
decode
backend
process

• walsender reads all WAL and decodes* them.
Then walsender sends some of them to the
subscriber.
• apply worker applies that change.
*decoding needs the most detailed WAL: wal_level = logical
Basics of the architecture ~replication
WAL
backend
process
wal
sender
Publisher
write
read
apply
worker
Subscriber
TableTableTable
write
decode
send
change

WAL
walsender
INSERT
UPDATE
UPDATE
DELETE
UPDATE apply
worker
Publisher Subscriber
:transaction
• walsender reassembles queries according to its
transaction.
• When INSERT, UPDATE or DELETE is placed,
walsender keeps that change in memory.

transaction.
WAL
walsender
INSERT
UPDATE
UPDATE
DELETE
UPDATE
1. read WAL
apply
worker
:transaction

WAL
walsender
INSERT
INSERT
UPDATE
UPDATE
DELETE
UPDATE
1. read WAL
2. decode
apply
worker
:transaction
transaction.

WAL
walsender
INSERT
INSERT
UPDATE
UPDATE
DELETE
UPDATE
1. read WAL
2. decode
3. reassemble
by transaction
apply
worker
:transaction
INSERT
transaction.

• When the decoded result is COMMIT,
‘walsender’ sends all the changes for that
transaction to subscriber.
:transaction
WAL
apply
worker
walsender
COMMIT
INSERT
UPDATE
UPDATE
DELETE
UPDATE
1. read WAL
2. decode
4. send
3. reassemble
by transaction
COMMIT

• When the decoded result is ROLLBACK,
walsender just throws away the changes for that
transaction.
:transaction
WAL
walsender
ROLLBACK
INSERT
UPDATE
UPDATE
DELETE
UPDATE
ROLLBACK
1. read WAL
2. decode
4. cleanup
apply
worker
3. reassemble
by transaction

• When the decoded result is ROLLBACK,
walsender just throws away the changes for that
transaction.
:transaction
WAL
walsender
ROLLBACK
INSERT
UPDATE
UPDATE
DELETE
1. read WAL
2. decode
4. cleanup
apply
worker
3. reassemble
by transaction

• walsender reassembles queries by its
transaction.
• When the decoded result is DDL, walsender does
NOT replicate it.
WAL
walsender
INSERT
UPDATE
UPDATE
DELETE
UPDATE apply
worker
:transaction
CREATE
NOT replicated

• At initial table synchronization, COPY runs.
• COPY is done by dedicated walsender and sync
worker. These processes exit after COPY is done.
Initial table synchronization
WAL
backend
process
wal
sender
Publisher
write
read
apply
worker
Subscriber
TableTableTable
sync
worker
wal
sender
write
(COPY)

1. Logical Replication does NOT replicate all
the data & manipulations
2. It needs additional information on WAL
3. It does initial table synchronization using
COPY
4. It transmits data per transaction
Summary of the architecture and behavior

Major version upgrade
using
Logical Replication

1. Logical Replication does NOT replicate
all the data & manipulation

Following objects are not replicated.
• view
• materialized view
• Foreign table
• Partition root table

Following objects are not replicated.
• view
• materialized view
• Foreign table
• Partition root table
⇒ These objects can be created data on new DB.
Create them before switching

• Large objects

• Large objects
⇒ After completion of the replication, run pg_dump
only on the large objects and restore them.
During this operation, APP needs stop updating

• Large objects
⇒ After completion of the replication, run pg_dump
only on the large objects and restore them.
During this operation, APP needs stop updating.
or
Migrate large objects to normal table

• Sequence
Data generated by sequence are replicated, but sequence objects
themselves are NOT replicated

• Sequence
Data generated by sequence are replicated, but sequence objects
themselves are NOT replicated
⇒ After completion of the replication, run
pg_dump only on the sequence and restore it.
During this operation, APP needs stop updating
sequences

• DDL

• DDL
⇒ If any DDL commands have to be run during
replication, run them also on subscriber

• DDL
⇒ If any DDL commands have to be run during
replication, run them also on subscriber
About partition child table
If partition child tables are created by trigger, it may be possible
to run the trigger on subscriber using ENABLE ALWAYS TRIGGER
NOTE: Even when using ENABLE ALWAYS TRIGGER, some kind of
triggers aren't replicated
e.g. UPDATE OF .. FOR EACH ROW

• TRUNCATE(~ver 10)

• TRUNCATE(~ver 10)
⇒ Substitute DELETE for TRUNCATE
PostgreSQL 11 replicates TRUNCATE☺

• Unlogged table
The basic idea of logical replication is decoding WAL
When there are no WAL, how can we do that?
• Global objects
The unit of Logical Replication is database, but global objects
belong to database cluster.

• Unlogged table
The basic idea of logical replication is decoding WAL
When there are no WAL, how can we do that?
⇒ pg_dump
• Global objects
The unit of Logical Replication is database, but global objects
belong to database cluster.
⇒ pg_dumpall

2. Logical Replication needs additional
information on WAL

Consideration for performance:
• wal_level = ‘logical’ is most detailed log level.
• UPDATE and DELETE on tables need ‘replica
identity’
• The default of the replica identity is primary key.
• If there are no primary key or suitable columns, set replica
identity 'full', which means the entire row becomes the key.
2. needs additional information on WAL

Performance and WAL size impacts by wal_level and replica
identity
• Amount of WAL
• Response time
2. needs additional information on WAL
wal_level replica identity ⊿LSN Ratio of ⊿LSN
logical full 21179088 1.49
logical default 16763896 1.18
replica - 14252616 1
wal_level replica identity avg(response) diff(avg(response))
Logical full 2.132ms -0.020ms
logical default 2.126ms -0.026ms
replica - 2.152ms -
1 minute pgbench on PostgreSQL 10

3. Initial table synchronization using COPY

initial table sync doesn't have parameters which
restrict the speed of COPY directly.
It may give performance impact on APP
3. initial table synchronization using COPY

Adding some delay on network using command tc
relieved the impact on client queries
No delay 100ms delay
ms

Under some environments, initial table
synchronization didn’t consume all the resources
because walsenders waited for client responses
In this case, we could improve the performance by
increasing max_sync_workers_per_subscription
sync worker = 1 sync worker = 8
kB/s kB/s

4. Transmission per transaction

walsender keeps each change of a transaction until
COMMIT or ROLLBACK
It may cause walsender to use a lot of memory.
walsender
INSERT
UPDATE
UPDATE
DELETE
UPDATE
Publisher
INSERT
INSERT
UPDATE
UPDATE
DELETE
UPDATE
UPDATE
DELETE
apply
worker
Subscriber

Type of manipulation Measuresto prevent memory use Risk
long transactions No feature high
many transactions
many savepoints
many changes in one
transaction
When the number of
changes in one transaction
exceeds 4096, It has a
feature to spill out the
change to disk.
Low

Major upgrade with Logical replication is possible, but it’s
not piece of cake.
We should watch out for its limitations and behavior such
as:
• Some data & manipulations are not replicated
⇒ there may be workarounds
• Additional information for replication is necessary
⇒ It seems that does not give critical impact
Summary

• Initial sync may give performance impact on APP
⇒ Tuning on PostgreSQL or network may moderate it
• Long or many transactions and many savepoints may lead
much memory consumption
⇒ Confirm the way of transactions run
Summary

Thanks for your cooperation!
Shinya Okano@Metro Systems
Hibiki Tanaka@Metro Systems

Thanks for listening!

Comparison between Logical and Physical
Replication
Physical Logical
way of the
replication
log shipping row-based replication
downstream DB copy of the
upstream DB
not necessarily the same as upstream DB
manipulations for
downstream DB
restricted
SELECT, PREPARE,
EXECUTE ..
No restrictions, but some manipulations
may lead to conflict
What is replicated ALL views, partition root tables, large objects
and some manipulations including DDL are
NOT replicated
purpose of use high availability
load balancing
fine-grained replication
replication between different
environments

PostgreSQL major version upgrade using built in Logical Replication

More Related Content

What's hot

Similar to PostgreSQL major version upgrade using built in Logical Replication

Recently uploaded

PostgreSQL major version upgrade using built in Logical Replication