In this talk we'll talk in details on how modern databases are capable of performing backups without downtimes and how these backups later can be used to restore database to any point in time.
While the talk describes generally applicable approach, a Litestream (SQLite backup service) is used for reference implementation.
11. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
Locate page
with the
record
12. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
P2
Copy pages to
Rollback
Journal
P1
13. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
P2
P1
Modify
affected
pages
17. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
Locate page
with the
record
18. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2
Write new
page versions
19. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2
Pager
redirects
pages to new
versions
24. EVERY CHANGED PAGE MUST BE
FIRST APPENDED AT THE END OF
THE WRITE-AHEAD LOG FILE
25. EVERY CHANGE IN THE DATABASE
FILE COMES FROM READING
FRONT-TO-BACK WRITE-AHEAD
LOG FILE
EVERY CHANGED PAGE MUST BE
FIRST APPENDED AT THE END OF
THE WRITE-AHEAD LOG FILE
38. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
restore(2024/02/11/10:00:00)
39. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
40. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1
41. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1
P1
F4
P2
F5
F4-F5-2024/02/10/07:54:10
42. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
43. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
PRAGMA CHECKPOINT(TRUNCATE);
44. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
45. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
46. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
47. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
48. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
60. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1 P1
F2
size_after=0 size_after=3
T1
P3
F3 P1
F4
size_after=0 size_after=0
ROLLBACK;
EOF
what if we already backed up these
frames? :/
62. POINT-IN-TIME RECOVRERY PROBLEM
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
restore(2024/02/11/10:00:00)
63. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
POINT-IN-TIME RECOVRERY PROBLEM
F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
64. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
INSERT INTO t(name)
VALUES(‘John Doe’);
65. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
66. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00
67. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00
68. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00
Database history is no longer linear!
87. How does continuous backup and point-in-time recovery work in databases:
https://www.bartoszsypytkowski.com/db-backup-point-in-time-recovery
Litestream: https://litestream.io/
SQLite write-ahead log docs: https://www.sqlite.org/wal.html
REFERENCES