How Multiversion Concurrency Control is implemented in Postgreql. Slides from my wroc_love.rb talk in 2018.

1. MVCC
for Ruby developers
Michał Młoźniak
@roninek

2. Multiversion
Concurrency Control

3. Postgres internals

4. Motivation

5. To optimize your SQL queries

6. To quickly solve performance problems

8. MVCC, why?
● Support for many concurrent users
● Atomicity and Isolation (ACID)
● Performance
● Fewer locks

9. MVCC, how?
● Postgres stores multiple versions of the same row in the table
● INSERT is just plain insert
● DELETE is marking row as deleted
● UPDATE is DELETE old row and INSERT new row
● Postgres shows different versions of a row to different transactions
● After a while deleted rows are not visible to any running transactions
● They are called dead rows
● Postgres needs to cleanup dead rows from time to time
● It is like Garbage Collection in dynamic languages

10. Postgres stores multiple versions
of the same row in the table

11. MVCC, how?
● Postgres stores multiple versions of the same row in the table
● INSERT is just plain insert
● DELETE is marking row as deleted
● UPDATE is DELETE old row and INSERT new row
● Postgres shows different versions of a row to different transactions
● After a while deleted rows are not visible to any running transactions
● They are called dead rows
● Postgres needs to cleanup dead rows from time to time
● It is like Garbage Collection in dynamic languages

12. MVCC, details
● Postgres stores two additional columns for each row
● ID of transaction that created a row
● ID of transaction that deleted a row
● Each transaction gets its own id (TXID) at start of first modify statement
● TXIDs are 32-bits incremental integers
● Lower TXIDs mean earlier transactions

13. MVCC, details
● Two additional columns: xmin and xmax
● xmin is transaction ID that created the row
● xmax is transaction ID that deleted the row
● Those are hidden columns available in all tables
● You can see them by using explicit select statements
● You will get an error if you add columns with such names

15. MVCC, inspecting
● You can look into physical table files and find deleted rows
● Or you can use pageinspect extension
● It can fetch raw page data, page headers, page rows, etc

17. Transaction Snapshots

18. Transaction Snapshots
● Frozen view of current transactions status
● Snapshot has format xmin:xmax:xip, for example 12:16:12,14
● xmin = 12, this means that earliest running transaction id is 10
● All earliest transactions (less than 12) are either committed and visible, or
aborted and dead
● xmax = 16, first as-yet unassigned transaction id
● All transaction equal or greater than 16 are not yet started and thus invisible
● xip = [12, 14], active transactions only between xmin and xmax
● Transactions 13 and 15 are either committed and visible, or aborted and dead

21. MVCC, visibility checks
current snapshot 101:101:, all transactions were committed
xmin xmax visible?
25 YES
25 50 NO
50 110 YES
110 NO
110 120 NO

22. MVCC, visibility checks
Current snapshot 25:101:25,50,75, all transactions were committed
xmin xmax visible?
30 YES
50 NO
110 NO
30 80 NO
30 75 YES
30 110 YES

23. MVCC, visibility checks
Current snapshot 101:101:, transaction 75 was aborted.
xmin xmax visible?
30 YES
30 75 YES
75 NO

24. Snapshots and Isolation Levels
● Postgres supports 3 isolation levels (READ COMMITTED, REPEATABLE READ
and SERIALIZABLE)
● In READ COMMITTED snapshot is recorded at start of each SQL statement
● And at transaction start in higher isolation levels

28. Model.transaction(isolation: :repeatable_read) do
# transaction block ...
end
Model.transaction(isolation: :serializable) do
# transaction block ...
end

29. Commit Log

30. Commit Log
● 2 bits per transaction (in progress, committed, aborted, ...)
● Committing or aborting a transaction is just flipping a bit in Commit Log
● All transactions (committed and aborted) have side-effects
● Hint bits in table rows, optimization to avoid Commit Log lookups
● Innocent table scan can possibly update a lot of hint bits and perform heavy
table write

31. All transactions
(committed or aborted)
have side-effects

33. Commit Log
● 2 bits per transaction (in progress, committed, aborted, ...)
● Committing or aborting a transaction is just flipping a bit in Commit Log
● All transactions (committed and aborted) have side-effects
● Hint bits in table rows, optimization to avoid Commit Log lookups
● Innocent table scan can possibly update a lot of hint bits and perform heavy
table write

34. Vacuuming

35. MVCC, vacuuming
● Vacuum is like a Garbage Collector
● Looks for rows that are no longer visible to any running transactions and
removes them
● Avoid long-running transactions
● Makes room for new rows in existing pages
● Autovacuum can happen at any time

36. Avoid long-running transactions

37. MVCC, vacuuming
● Vacuum is like a Garbage Collector
● Looks for rows that are no longer visible to any running transactions and
removes them
● Avoid long-running transactions
● Makes room for new rows in existing pages
● Autovacuum can happen at any time

38. Autovacuum can happen at any time

39. Transaction Wraparound

42. Transaction Wraparound, huh?
● Transaction IDs (TIDs) are 32-bit integers
● That is ~ 4 billion transactions
● With enough traffic it can quickly wraparound
● Suddenly transactions that were in the past appear to be in the future
● And their output is invisible

43. Transaction Wraparound, solutions?
● Vacuum freezes old transactions, that are way in the past
● Freezing sets special flag on rows
● Set flag means that this row is visible to all transactions
● Can be done manually with VACUUM FREEZE

44. Main takeaways
● Postgres stores multiple versions of the same row in the table
● All transactions (committed or aborted) have side-effects
● All updates to the table create bloat
● Vacuum removes bloat and can happen at any time
● Avoid long-running transactions

45. More resources
● https://momjian.us/main/presentations/internals.html
● https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres.html
● https://www.joyent.com/blog/manta-postmortem-7-27-2015
● http://www.interdb.jp/pg/index.html
● https://queue.acm.org/detail.cfm?id=3099561

46. Thank You!
Michał Młoźniak
@roninek
www.michalmlozniak.com