PostgreSQL Write-Ahead Log                                     Heikki Linnakangaspyright 2009 EnterpriseDB Corporation. Al...
What is Write-Ahead Log?        • Also known as the transaction log or redo log        • Practically every database manage...
Write-Ahead Log concepts        • A transaction log is a sequence of log records        • One log record for every change ...
PostgreSQL Write-Ahead-Log        • Introduced in version 7.1 by Vadim B. Mikheev               – Released in 2001        ...
PostgreSQL Example        INSERT INTO tbl (id, data) VALUES (123, foo)        @ 0/D023940: xid 734: Heap - insert: rel    ...
PostgreSQL WAL structure        • WAL is divided into WAL segments               – Each segment is a file in pg_xlog direc...
Tools        • Developer tools               – WAL_DEBUG compile option               – Xlogdump http://xlogviewer.project...
WAL-first rule        • The log record for an operation is always written          to disk before the affected data pages ...
Checkpoints        • WAL grows indefinitely        • Checkpoints allow us to truncate it        1. Flush all data pages to...
WAL filenames              00000001000000000000003E                                             Log id                 Seg...
Crash-safety        • Many database actions involve modifying more          than one page.           – Example: Splitting ...
WAL Summary        Write-Ahead Logging is critical for:        • Durability               – Once you commit, your data is ...
Advanced features        But theres more!        The Write-Ahead Log also allows:        • Online backup        • Point-in...
WAL Archiving        • Normally, old WAL is deleted at a checkpoint.        • But it can also be archived        • Archive...
WAL Archiving        In your postgresql.conf file:        archive_mode=on        wal_level=archive        archive_command ...
WAL archive        ~$ ls -l /mnt/walarchive/        yhteensä 114688        -rw------- 1 heikki heikki 16777216 30.3. 18:11...
Online Backup        • Normally, you need to shut down the database          while you take a backup               – or us...
Online Backup        1. SELECT pg_start_backup()        2. rsync / tar / whatever        3. SELECT pg_stop_backup();      ...
Online backup: Step 1        /mnt$ ls ­l        yhteensä 8        drwxr­xr­x  2 heikki heikki 4096 30.3. 18:11 archivedir ...
Online backup: Step 2        /mnt$ tar czf ~/backup.tar.gz data        /mnt$ ls -l ~/backup.tar.gz        -rw-r--r-- 1 hei...
Online backup: Step 3        /mnt$ psql -c "SELECT pg_stop_backup()"        NOTICE: pg_stop_backup complete, all required ...
Online backup: Step 4        /mnt$ tar czf ~/backup-xlog.tar.gz archivedir/*        /mnt$ ls -l /home/heikki/backup*.tar.g...
Point-in-Time Recovery        postgres=# DROP TABLE customers;        DROP TABLE        Oops!        • Once youve enabled ...
Point-in-Time Recovery        Recovery.conf:        # By default, recovery will rollforward to the end of the WAL log.    ...
Replication        • You can transmit WAL as it is generated to a          standby server               – Replication!    ...
Thank you        Write-Ahead Logging makes it possible for a        database to maintain consistency. It also allows      ...
Upcoming SlideShare
Loading in …5
×

PostgreSQL Write-Ahead Log (Heikki Linnakangas)

2,826 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,826
On SlideShare
0
From Embeds
0
Number of Embeds
319
Actions
Shares
0
Downloads
54
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

PostgreSQL Write-Ahead Log (Heikki Linnakangas)

  1. 1. PostgreSQL Write-Ahead Log Heikki Linnakangaspyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 1
  2. 2. What is Write-Ahead Log? • Also known as the transaction log or redo log • Practically every database management system has one • Also used by journaling file systems, transaction managers etc.pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 2
  3. 3. Write-Ahead Log concepts • A transaction log is a sequence of log records • One log record for every change • Each WAL record is assigned an LSN (Log Sequence Number)pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 3
  4. 4. PostgreSQL Write-Ahead-Log • Introduced in version 7.1 by Vadim B. Mikheev – Released in 2001 • REDO log only – No UNDO log – Instantaneous rollbacks – No limit on transaction sizepyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 4
  5. 5. PostgreSQL Example INSERT INTO tbl (id, data) VALUES (123, foo) @ 0/D023940: xid 734: Heap - insert: rel 1663/12040/16402; tid 0/2 @ 0/D023988: xid 734: Btree - insert: rel 1663/12040/16404; tid 1/2 @ 0/D0239D0: xid 734: Transaction - commit: 2012-03-30 19:08:10.262228+03 LOG: xlog flush request 0/D023A00pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 5
  6. 6. PostgreSQL WAL structure • WAL is divided into WAL segments – Each segment is a file in pg_xlog directory $ ls ­l pgsql/data/pg_xlog/ ­rw­­­­­­­ 1 heikki heikki      239 2009­11­03 10:39  000000010000000000000034.00000020.backup ­rw­­­­­­­ 1 heikki heikki 16777216 2009­11­03 15:26 00000001000000000000003A ­rw­­­­­­­ 1 heikki heikki 16777216 2009­11­03 15:26 00000001000000000000003B ­rw­­­­­­­ 1 heikki heikki 16777216 2009­11­03 10:39 00000001000000000000003C ­rw­­­­­­­ 1 heikki heikki 16777216 2009­11­03 13:49 00000001000000000000003D ­rw­­­­­­­ 1 heikki heikki 16777216 2009­11­03 15:14 00000001000000000000003E drwx­­­­­­ 2 heikki heikki      160 2009­11­03 15:26 archive_statuspyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 6
  7. 7. Tools • Developer tools – WAL_DEBUG compile option – Xlogdump http://xlogviewer.projects.postgresql.org/pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 7
  8. 8. WAL-first rule • The log record for an operation is always written to disk before the affected data pages – WAL first rule! • In case of a crash, the WAL is replayed to reconstruct the unsaved changespyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 8
  9. 9. Checkpoints • WAL grows indefinitely • Checkpoints allow us to truncate it 1. Flush all data pages to disk 2. Write a checkpoint record 3. Truncate away old WALpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 9
  10. 10. WAL filenames 00000001000000000000003E Log id Seg no TLI LSN • Timeline ID is used to distinguish WAL generated before and after a PITR recoverypyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 10
  11. 11. Crash-safety • Many database actions involve modifying more than one page. – Example: Splitting an index page. • Writing two data pages A and B is not atomic. • One write must happen before the other. – We need to give the operating system and disk controller freedom to reorder the writes; we dont even know which one will happen first. • We could crash in between.pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 11
  12. 12. WAL Summary Write-Ahead Logging is critical for: • Durability – Once you commit, your data is safe • Consistency – No corruption on crashpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 12
  13. 13. Advanced features But theres more! The Write-Ahead Log also allows: • Online backup • Point-in-time Recovery • Replicationpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 13
  14. 14. WAL Archiving • Normally, old WAL is deleted at a checkpoint. • But it can also be archived • Archive can be anything – Directory on another server – Tape drivepyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 14
  15. 15. WAL Archiving In your postgresql.conf file: archive_mode=on wal_level=archive archive_command = cp %p /mnt/walarchive/%f Restart server, and it will copy all WAL files to /mnt/walarchive as theyre generatedpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 15
  16. 16. WAL archive ~$ ls -l /mnt/walarchive/ yhteensä 114688 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000001 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000002 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000003 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000004 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000005 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000006 -rw------- 1 heikki heikki 16777216 30.3. 18:11 000000010000000000000007pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 16
  17. 17. Online Backup • Normally, you need to shut down the database while you take a backup – or use a filesystem snapshot (e.g ZFS or a SAN) • WAL archiving makes that unnecessarypyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 17
  18. 18. Online Backup 1. SELECT pg_start_backup() 2. rsync / tar / whatever 3. SELECT pg_stop_backup(); 4. Include all archived WAL files from archive directory in the backuppyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 18
  19. 19. Online backup: Step 1 /mnt$ ls ­l yhteensä 8 drwxr­xr­x  2 heikki heikki 4096 30.3. 18:11 archivedir drwx­­­­­­ 14 heikki heikki 4096 30.3. 18:10 data /mnt$ psql ­c "SELECT pg_start_backup(my first backup)"  pg_start_backup  ­­­­­­­­­­­­­­­­­  0/C000020 (1 row)pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 19
  20. 20. Online backup: Step 2 /mnt$ tar czf ~/backup.tar.gz data /mnt$ ls -l ~/backup.tar.gz -rw-r--r-- 1 heikki heikki 39670143 30.3. 18:33 /home/heikki/backup.tar.gzpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 20
  21. 21. Online backup: Step 3 /mnt$ psql -c "SELECT pg_stop_backup()" NOTICE: pg_stop_backup complete, all required WAL segments have been archived pg_stop_backup ---------------- 0/C0000A8 (1 row)pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 21
  22. 22. Online backup: Step 4 /mnt$ tar czf ~/backup-xlog.tar.gz archivedir/* /mnt$ ls -l /home/heikki/backup*.tar.gz -rw-r--r-- 1 heikki heikki 39670143 30.3. 18:33 /home/heikki/backup.tar.gz -rw-r--r-- 1 heikki heikki 41937910 30.3. 18:38 /home/heikki/backup-xlog.tar.gzpyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 22
  23. 23. Point-in-Time Recovery postgres=# DROP TABLE customers; DROP TABLE Oops! • Once youve enabled WAL archiving and taken a backup, you can use the archived WAL to restore to any point in timepyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 23
  24. 24. Point-in-Time Recovery Recovery.conf: # By default, recovery will rollforward to the end of the WAL log. # If you want to stop rollforward at a specific point, you # must set a recovery target. ... #recovery_target_name = # e.g. daily backup 2011-01-26 # #recovery_target_time = # e.g. 2004-07-14 22:39:00 EST # #recovery_target_xid = # #recovery_target_inclusive = truepyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 24
  25. 25. Replication • You can transmit WAL as it is generated to a standby server – Replication! • This can be done using the WAL archive, or by streaming directly from the server over a TCP connection • Standby server can be used for read-only queries (Hot Standby)pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 25
  26. 26. Thank you Write-Ahead Logging makes it possible for a database to maintain consistency. It also allows many advanced features: online backup, PITR and replication Questions?pyright 2009 EnterpriseDB Corporation. All rights Reserved. <presentation title goes here, edit in the slide master > Slide: 26

×