PGConf.ASIA 2019 Bali - 10 September 2019 Speaker: Muhammad Haroon Room: ACID Title: The Past, Present, and Future of PostgreSQL’s Fault Tolerance

1. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Fault Tolerance in
PostgreSQL
PGConf Asia
Sept, 2019
Muhammad Haroon
@ 2ndQuadrant

2. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
$psql~#: d haroon
● Working in PostgreSQL space @ 2ndQuadrant
● Part of PostgreSQL family for nearly a decade and a half
● Development, support, consulting, professional services and administration
● Past stints with PostgreSQL family include
○ EnterpriseDB
○ OpenSCG (now Amazon Web Services)
● Led Engineering Ops efforts @ IBEX Group
● Principal/Architect/Product Owner @ TRG
Email: haroon@2ndQuadrant.com
Skype: contact.haroon

3. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
What’s covered
- A little overview of PostgreSQL to set the tone
- PostgreSQL’s fault tolerance (WAL)
- Replication
- Available replication methods
- Physical Replication
- Replication and WAL management
- Timelines ? pg_rewind ?
- Trigger based replication
- Logical replication/decoding

4. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
PostgreSQL- world’s most advanced open source database
- Soft failures: Transactions combined with Transaction log means
automatic crash recovery
- Hard failures: Data block checksums
- Multiple backup mechanisms: Full Point-in-time-recovery
- Diagnostics tools
- Native database replication
- Synchronous replication with upto 99.999 percent uptime

5. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Olden days
- All database changes would eventually go to durable storage
- Data pages would be loaded to shared buffers
- Changes recorded there
- Changes synched with durable storage
- Crash would mean data loss

6. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Write Ahead Log (WAL)
PostgreSQL’s Write-Ahead-Log provides main fault tolerance system for
PostgreSQL to ensure durability of database changes.
- Series of binary log files containing transaction logs
- 16MB file segment (--with-wal-size)
- 8kB page size (--with-wal-blocksize)
- Stored under pg_wal (pg_xlog < PG10) directory
- All database changes recorded here
- Needed for crash and point in time recovery

7. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Transactions
- Begin?Commit?
- Changes recorded in WAL
- Written to disk later by checkpointer or background writer
- Standard SQL Transaction Isolation levels (PG docs for details)
- Read uncommitted
- Read committed
- Repeatable read
- Serializable

8. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Checkpoints
- Special checkpoint record in the log
- Heap and index data files updated
- All dirty data pages flushed to disk
- REDO operation to start from latest checkpoint record
- Scheduled/Immediate
Controls:
- Checkpoint_timeout (Def. 5 min)
- Max_wal_size (Def. 1 GB)

9. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Replication
(picture credit gulcin.yildirim)
- Term used to describe the technology to create copies of
database/set of data to some remote server
- Needed for reliable copies of data
- Replicas for redundancy

10. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Replication chronicle
● PostgreSQL 7.x (~2000)
○ Debate around not including Replication to core Postgres
○ Londiste – Slony (trigger based logical replication)
● PostgreSQL 8.0 (2005)
○ Point-In-Time Recovery (WAL)
● PostgreSQL 9.0 (2010)
○ Streaming Replication (physical)
● PostgreSQL 9.4 (2014)
○ Logical Decoding (changeset extraction)
● PostgreSQL 10 (2017)
○ Native Logical Replication

11. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Log-shipping standby servers
- base backup from primary
- primary in continuous archival
- standby in continuous recovery
- warm standby: cannot serve readonly queries
- hot standby: read-only queries while the server is in archive recovery or
standby mode

12. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Hot Standby
Warm standby Hot standby

13. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Physical replication
- WAL sent over to standby over the network
- Files sent via rsync, ftp, scp
- Streaming replication when WAL sent using PostgreSQL’s internal protocole that involves
wal-sender and wal-receiver processes

14. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
WAL levels
- wal_level (postgresql.conf)
- Determines how much information is recorded in WAL.
- minimal
- Crash recovery
- replica
- Physical replication
- File based archiving
- logical
- Information needed for logical replication

15. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Timelines
- You need to distinguish the series of WAL records generated after you’ve
done a point-in-time recovery from those that were generated in the original
database history
- Archive recovery results in a new timeline
- Timeline history file
- Small text file
- Needed to allow the system to pick the right WAL segment files when
recovering from an archive that contains multiple timelines
- Target timeline ID in recovery.conf

16. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Timelines

17. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Timelines (failover)
● There are outstanding changes in the old master (TL1)
● Timeline increase represents new history of changes (TL2)
● Changes from the old timeline can’t be replayed on the servers that
switched to new timeline
● The old master can’t follow the new master

18. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Timelines (switchover)
● There are no outstanding changes in the old master (TL1)
● Timeline increase represents new history of changes (TL2)
● The old master can become standby for the new master

19. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Timelines (pg_rewind)
● Outstanding changes are removed using data from the new master (TL1)
● The old master can follow the new master (TL2)
PostgreSQL docs for detailed information on pg_rewind

20. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous replication
- Synchronous replication guarantees that data is written to at least two nodes
before the user or application is told that a transaction has committed
- Needs more time to complete the transaction
- the replication delay directly affects the elapsed time of transactions
- Zero data loss
- Default is asynchronous
- data is streamed out whenever convenient for the server
- Doesn’t wait for acknowledgement from standby

21. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous Replication

22. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Processing of commit

23. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
synchronous_commit = local (default off)
COMMIT waits until the transaction record is flushed to the local disk

24. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous_commit = remote_write
the COMMIT will wait until the server(s) specified by synchronous_standby_names confirm write of the transaction
record to the operating system but has not necessarily reached the disk

25. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous_commit = on
the COMMIT will wait until the server(s) specified by synchronous_standby_names confirm that the transaction
record was safely written to disk

26. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous_commit = remote_apply
the COMMIT will wait until the server(s) specified by synchronous_standby_names confirm that the transaction
record was applied to the database

27. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Synchronous_commit
synchronous_commit parameter can be changed at any time; the behaviour for any
one transaction is determined by the setting in effect when it commits. It is
therefore possible, and useful, to have some transactions commit synchronously
and others asynchronously

28. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Logical Replication
- Trigger based replication
- Slony (~2004), Londiste (~2007) Predates the physical replication (as a result of the "no replication
in core philosophy")
- Uses triggers to capture the changes to individual table
- Increases the amount of work needed to be done for each write
- Use table(s) as queue
- Duplicates all writes
- Logical decoding (Changeset Extraction)
- Extracts information from Write-Ahead Log into logical changes (INSERT/UPDATE/DELETE)
- Per row based changes
- Commit order maintained
- No write amplification
- No DDL replication
- Uses same transport mechanism as streaming replication (sender & apply)

29. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Logical Replication
- Publish / Subscribe model
- Multiple upstream (publisher) servers into a single subscriber
- Selective Replication
- Replication across major versions
- Online major version upgrades

30. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Futures
- Multi-master/Bi-Directional Replication
- Geographic distributed clusters
- Concurrent writes
- Conflict resolution
- Reduced latency
- High availability (Standby nodes)

31. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Acknowledgement
Big _Thanks_ to Gulcin Yildirim for the amazing pictures used in the talk

32. https://www.2ndQuadrant.com
PGConf ASIA
Bali, Sept, 2019
Thank you!!