PostgreSQL major version upgrade using built in Logical Replication

2. 22Copyright©2018 NTT Corp. All Rights Reserved. • Upgrading PostgreSQL cluster • Introduction to Logical Replication • Architecture and Behavior of Logical Replication • Major version upgrade using logical replication Agenda

4. 44Copyright©2018 NTT Corp. All Rights Reserved. Minor Version Upgrade • e.g. 9.6.0 -> 9.6.1, 10.3 -> 10.6 • always compatible with earlier and later minor releases • just need upgrade the PostgreSQL binary Major Version Upgrade • e.g. 9.5.1 -> 9.6.15, 10.2 -> 11.1 • not always compatible, the internal data storage format can be changed • upgrading the PostgreSQL binary is not enough, it needs also upgrading data files ⇒ several ways for major version upgrade 2 types of version upgrade

5. 55Copyright©2018 NTT Corp. All Rights Reserved. pg_dump o the traditional way for major version up x service downtime during dump and restore pg_upgrade o faster than pg_dump x shorter than pg_dump, but it still needs service downtime link mode is much faster, but you cannot go back the original cluster once you start up the new one Logical Replication o it doesn't need stop PostgreSQL service during replication ⇒ near zero downtime x built-in logical replication is supported only after version 10 3 ways for major version upgrade

7. 77Copyright©2018 NTT Corp. All Rights Reserved. Physical Replication replicates a whole DB cluster by sending & replaying all the WAL. Looking back Physical Replication Upstream Downstream sendTable Table Table WALWAL WALWAL Table Table Table replay

8. 88Copyright©2018 NTT Corp. All Rights Reserved. Physical Replication cannot do things like: • partial replication(partial data, partial manipulation) • replication between different OS arc/major version PostgreSQL Logical Replication has added flexibility to built-in replication and made these things possible. Motivations for Logical Replication Upstream Downstream decode, sendTable Table Table WALWAL WALWAL Table Table apply write

12. 1212Copyright©2018 NTT Corp. All Rights Reserved. 1. Create a new cluster 2. Start Logical Replication 3. Switch access point How can we upgrade our cluster with Logical Replication? Current New APP

13. 1313Copyright©2018 NTT Corp. All Rights Reserved. 1. Create a new cluster 2. Start Logical Replication 3. Switch access point That’s all. It’s a piece of cake! How can we upgrade our cluster with Logical Replication? Current New APP

14. 1414Copyright©2018 NTT Corp. All Rights Reserved. 1. Create a new cluster 2. Start Logical Replication 3. Switch access point That’s all. It’s a piece of cake! Logical Replication has some limitations and does characteristic behaviors. How can we upgrade our cluster with Logical Replication? Current New APP

16. 1616Copyright©2018 NTT Corp. All Rights Reserved. • ‘walsender’ and ‘apply worker’ do most of the works for Logical Replication. Basics of the architecture WAL wal sender Publisher (upstream) write apply worker launcher launch Subscriber(downstream) backend process read decode backend process

17. 1717Copyright©2018 NTT Corp. All Rights Reserved. • ‘walsender’ and ‘apply worker’ do most of the works for Logical Replication. • ‘sync worker’ and the corresponding ‘walsender’ run only at initial table sync. Basics of the architecture WAL wal sender Publisher (upstream) write wal sender apply worker launcher sync worker launch launch Subscriber(downstream) backend process read decode backend process

18. 1818Copyright©2018 NTT Corp. All Rights Reserved. • walsender reads all WAL and decodes* them. Then walsender sends some of them to the subscriber. • apply worker applies that change. *decoding needs the most detailed WAL: wal_level = logical Basics of the architecture ~replication WAL backend process wal sender Publisher write read apply worker Subscriber TableTableTable write decode send change

19. 1919Copyright©2018 NTT Corp. All Rights Reserved. Basics of the architecture ~replication WAL walsender INSERT UPDATE UPDATE DELETE UPDATE apply worker Publisher Subscriber :transaction • walsender reassembles queries according to its transaction. • When INSERT, UPDATE or DELETE is placed, walsender keeps that change in memory.

20. 2020Copyright©2018 NTT Corp. All Rights Reserved. • walsender reassembles queries according to its transaction. • When INSERT, UPDATE or DELETE is placed, walsender keeps that change in memory. Basics of the architecture ~replication WAL walsender INSERT UPDATE UPDATE DELETE UPDATE 1. read WAL apply worker Publisher Subscriber :transaction

21. 2121Copyright©2018 NTT Corp. All Rights Reserved. Basics of the architecture ~replication WAL walsender INSERT INSERT UPDATE UPDATE DELETE UPDATE 1. read WAL 2. decode apply worker Publisher Subscriber :transaction • walsender reassembles queries according to its transaction. • When INSERT, UPDATE or DELETE is placed, walsender keeps that change in memory.

22. 2222Copyright©2018 NTT Corp. All Rights Reserved. Basics of the architecture ~replication WAL walsender INSERT INSERT UPDATE UPDATE DELETE UPDATE 1. read WAL 2. decode 3. reassemble by transaction apply worker Publisher Subscriber :transaction INSERT • walsender reassembles queries according to its transaction. • When INSERT, UPDATE or DELETE is placed, walsender keeps that change in memory.

23. 2323Copyright©2018 NTT Corp. All Rights Reserved. • When the decoded result is COMMIT, ‘walsender’ sends all the changes for that transaction to subscriber. Basics of the architecture ~replication :transaction WAL apply worker walsender COMMIT INSERT UPDATE UPDATE DELETE UPDATE 1. read WAL 2. decode 4. send Publisher Subscriber 3. reassemble by transaction COMMIT

24. 2424Copyright©2018 NTT Corp. All Rights Reserved. • When the decoded result is ROLLBACK, walsender just throws away the changes for that transaction. Basics of the architecture ~replication :transaction WAL walsender ROLLBACK INSERT UPDATE UPDATE DELETE UPDATE ROLLBACK 1. read WAL 2. decode 4. cleanup apply worker Publisher Subscriber 3. reassemble by transaction

25. 2525Copyright©2018 NTT Corp. All Rights Reserved. • When the decoded result is ROLLBACK, walsender just throws away the changes for that transaction. Basics of the architecture ~replication :transaction WAL walsender ROLLBACK INSERT UPDATE UPDATE DELETE 1. read WAL 2. decode 4. cleanup apply worker Publisher Subscriber 3. reassemble by transaction

26. 2626Copyright©2018 NTT Corp. All Rights Reserved. • walsender reassembles queries by its transaction. • When the decoded result is DDL, walsender does NOT replicate it. Basics of the architecture ~replication WAL walsender INSERT UPDATE UPDATE DELETE UPDATE apply worker Publisher Subscriber :transaction CREATE NOT replicated

27. 2727Copyright©2018 NTT Corp. All Rights Reserved. • At initial table synchronization, COPY runs. • COPY is done by dedicated walsender and sync worker. These processes exit after COPY is done. Initial table synchronization WAL backend process wal sender Publisher write read apply worker Subscriber TableTableTable sync worker wal sender write (COPY)

28. 2828Copyright©2018 NTT Corp. All Rights Reserved. 1. Logical Replication does NOT replicate all the data & manipulations 2. It needs additional information on WAL 3. It does initial table synchronization using COPY 4. It transmits data per transaction Summary of the architecture and behavior

31. 3131Copyright©2018 NTT Corp. All Rights Reserved. Following objects are not replicated. • view • materialized view • Foreign table • Partition root table 1. Logical Replication does NOT replicate all

32. 3232Copyright©2018 NTT Corp. All Rights Reserved. Following objects are not replicated. • view • materialized view • Foreign table • Partition root table ⇒ These objects can be created data on new DB. Create them before switching 1. Logical Replication does NOT replicate all

34. 3434Copyright©2018 NTT Corp. All Rights Reserved. • Large objects ⇒ After completion of the replication, run pg_dump only on the large objects and restore them. During this operation, APP needs stop updating 1. Logical Replication does NOT replicate all

35. 3535Copyright©2018 NTT Corp. All Rights Reserved. • Large objects ⇒ After completion of the replication, run pg_dump only on the large objects and restore them. During this operation, APP needs stop updating. or Migrate large objects to normal table 1. Logical Replication does NOT replicate all

36. 3636Copyright©2018 NTT Corp. All Rights Reserved. • Sequence Data generated by sequence are replicated, but sequence objects themselves are NOT replicated 1. Logical Replication does NOT replicate all

37. 3737Copyright©2018 NTT Corp. All Rights Reserved. • Sequence Data generated by sequence are replicated, but sequence objects themselves are NOT replicated ⇒ After completion of the replication, run pg_dump only on the sequence and restore it. During this operation, APP needs stop updating sequences 1. Logical Replication does NOT replicate all

40. 4040Copyright©2018 NTT Corp. All Rights Reserved. • DDL ⇒ If any DDL commands have to be run during replication, run them also on subscriber About partition child table If partition child tables are created by trigger, it may be possible to run the trigger on subscriber using ENABLE ALWAYS TRIGGER NOTE: Even when using ENABLE ALWAYS TRIGGER, some kind of triggers aren't replicated e.g. UPDATE OF .. FOR EACH ROW 1. Logical Replication does NOT replicate all

43. 4343Copyright©2018 NTT Corp. All Rights Reserved. • Unlogged table The basic idea of logical replication is decoding WAL When there are no WAL, how can we do that? • Global objects The unit of Logical Replication is database, but global objects belong to database cluster. 1. Logical Replication does NOT replicate all

44. 4444Copyright©2018 NTT Corp. All Rights Reserved. • Unlogged table The basic idea of logical replication is decoding WAL When there are no WAL, how can we do that? ⇒ pg_dump • Global objects The unit of Logical Replication is database, but global objects belong to database cluster. ⇒ pg_dumpall 1. Logical Replication does NOT replicate all

46. 4646Copyright©2018 NTT Corp. All Rights Reserved. Consideration for performance: • wal_level = ‘logical’ is most detailed log level. • UPDATE and DELETE on tables need ‘replica identity’ • The default of the replica identity is primary key. • If there are no primary key or suitable columns, set replica identity 'full', which means the entire row becomes the key. 2. needs additional information on WAL

47. 4747Copyright©2018 NTT Corp. All Rights Reserved. Performance and WAL size impacts by wal_level and replica identity • Amount of WAL • Response time 2. needs additional information on WAL wal_level replica identity ⊿LSN Ratio of ⊿LSN logical full 21179088 1.49 logical default 16763896 1.18 replica - 14252616 1 wal_level replica identity avg(response) diff(avg(response)) Logical full 2.132ms -0.020ms logical default 2.126ms -0.026ms replica - 2.152ms - 1 minute pgbench on PostgreSQL 10

49. 4949Copyright©2018 NTT Corp. All Rights Reserved. initial table sync doesn't have parameters which restrict the speed of COPY directly. It may give performance impact on APP 3. initial table synchronization using COPY

50. 5050Copyright©2018 NTT Corp. All Rights Reserved. Adding some delay on network using command tc relieved the impact on client queries 3. initial table synchronization using COPY No delay 100ms delay ms

51. 5151Copyright©2018 NTT Corp. All Rights Reserved. Under some environments, initial table synchronization didn’t consume all the resources because walsenders waited for client responses In this case, we could improve the performance by increasing max_sync_workers_per_subscription 3. initial table synchronization using COPY sync worker = 1 sync worker = 8 kB/s kB/s

53. 5353Copyright©2018 NTT Corp. All Rights Reserved. walsender keeps each change of a transaction until COMMIT or ROLLBACK It may cause walsender to use a lot of memory. 4. Transmission per transaction walsender INSERT UPDATE UPDATE DELETE UPDATE Publisher INSERT INSERT UPDATE UPDATE DELETE UPDATE UPDATE DELETE apply worker Subscriber

54. 5454Copyright©2018 NTT Corp. All Rights Reserved. 4. Transmission per transaction Type of manipulation Measuresto prevent memory use Risk long transactions No feature high many transactions many savepoints many changes in one transaction When the number of changes in one transaction exceeds 4096, It has a feature to spill out the change to disk. Low

55. 5555Copyright©2018 NTT Corp. All Rights Reserved. Major upgrade with Logical replication is possible, but it’s not piece of cake. We should watch out for its limitations and behavior such as: • Some data & manipulations are not replicated ⇒ there may be workarounds • Additional information for replication is necessary ⇒ It seems that does not give critical impact Summary

56. 5656Copyright©2018 NTT Corp. All Rights Reserved. • Initial sync may give performance impact on APP ⇒ Tuning on PostgreSQL or network may moderate it • Long or many transactions and many savepoints may lead much memory consumption ⇒ Confirm the way of transactions run Summary

59. 5959Copyright©2018 NTT Corp. All Rights Reserved. Comparison between Logical and Physical Replication Physical Logical way of the replication log shipping row-based replication downstream DB copy of the upstream DB not necessarily the same as upstream DB manipulations for downstream DB restricted SELECT, PREPARE, EXECUTE .. No restrictions, but some manipulations may lead to conflict What is replicated ALL views, partition root tables, large objects and some manipulations including DDL are NOT replicated purpose of use high availability load balancing fine-grained replication replication between different environments

PostgreSQL major version upgrade using built in Logical Replication

Atsushi Torikoshi

PostgreSQL major version upgrade using built in Logical Replication