Your SlideShare is downloading. ×
0
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Rapid Upgrades With Pg_Migrator
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Rapid Upgrades With Pg_Migrator

628

Published on

Bruce Momjian …

Bruce Momjian

Pg_Migrator allows data to be transfered between major Postgres versions without a dump/restore. This talk explains the internal workings of pg_migrator and includes a pg_migrator demonstration

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
628
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Rapid Upgrades With Pg_Migrator BRUCE MOMJIAN, ENTERPRISEDB March, 2010 Abstract Pg_Migrator allows migration between major releases of Postgres without a data dump/reload. This presentation explains how pg_migrator works. Creative Commons Attribution License http://momjian.us/presentations
  • 2. Traditional Postgres Upgrade Options pg_dump (logical dump)/restore   Slony   Rapid Upgrades With Pg-Migrator 1
  • 3. Why Upgrading Postgres Is Complex New features often require system table changes   However, the internal data format rarely changes   Rapid Upgrades With Pg-Migrator 2
  • 4. Why Pg_Migrator Very fast upgrades   Optionally no additional disk space   pg_migrator installs new system tables while using data files from the previous Postgres version. Rapid Upgrades With Pg-Migrator 3
  • 5. How It Works: Initial Setup Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog Rapid Upgrades With Pg-Migrator 4
  • 6. Decouple New Clog Via Freezing Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 3 5 6 8 9 2 3 Freeze 5 6 8 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 X X clog clog Rapid Upgrades With Pg-Migrator 5
  • 7. Transfer Clog and XID Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog controldata xid controldata Rapid Upgrades With Pg-Migrator 6
  • 8. Get Schema Dump pg_dumpall - -schema Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog Rapid Upgrades With Pg-Migrator 7
  • 9. Reserve TOAST OIDs Using Relfilenodes (pre-9.0) Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 18 13 19 25 14 20 26 26 15 21 27 clog clog This is necessary because heap references to TOAST tables contain the TOAST oids for easy lookup. Rapid Upgrades With Pg-Migrator 8
  • 10. Restore Schema In New Cluster pg_dumpall - -schema Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 9
  • 11. Connect TOAST Placeholders To the Proper Relations (pre-9.0) Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 X 41 47 clog clog Rapid Upgrades With Pg-Migrator 10
  • 12. Copy User Heap/Index Files Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 11
  • 13. Complete Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 12
  • 14. How It Works: In Detail Check for cluster compatability   – locale – encoding – integer datetimes (default changed from 8.3 -> 8.4) Use pg_dumpall to dump old cluster schema (no data)   Freeze all new cluster rows (remove reference to clog entries)   Rename tablespaces to *_old (pre-9.0)   New cluster uses old xid counter value (see freeze above)   – Set system table frozen xids to match the current xid Create new users/databases   Rapid Upgrades With Pg-Migrator 13
  • 15. Collect cluster information   Install support functions that call internal backend functions   Create placeholder files to reserve relfilenode file names (pre-9.0)   Create schema in new cluster   Adjust new cluster to use reserved relfilenode names (pre-9.0)   – Delete placeholder toast relfilenode files – Remove new cluster toast tables – Create new cluster toast table using reserved relfilenode – Assign new toast tables with proper relfilenodes to relations Copy or link files from old cluster to new cluster   – Toast tables have the same relfilenodes as in the old cluster Warn about any remaining issues, like REINDEX requirements   Rapid Upgrades With Pg-Migrator 14
  • 16. Sample Run: Performing Consistency Checks Performing Consistency Checks ----------------------------- Checking old data directory (/u/pgsql.old/data) ok Checking new data directory (/u/pgsql/data) ok Checking for columns with user-defined composite types ok Checking for columns with user-defined array types ok Checking for columns with user-defined enum types ok Checking for /contrib/isn with bigint-passing mismatch ok Checking for invalid ’name’ user columns ok Checking for tsquery user columns ok Creating script to adjust sequences ok Creating catalog dump ok Checking for presence of required libraries ok | If pg_migrator fails after this point, you must | re-initdb the new cluster before continuing. | You will also need to remove the ".old" suffix | from /u/pgsql.old/data/global/pg_control.old and old | tablespace directory names. Many of these checks are unnecessary when migrating to Postgres 9.0. Rapid Upgrades With Pg-Migrator 15
  • 17. Sample Run: Performing Migration Performing Migration -------------------- Adding ".old" suffix to tablespace directory names ok Adding ".old" suffix to old global/pg_control ok Analyzing all rows in the new cluster ok Freezing all rows on the new cluster ok Deleting new commit clogs ok Copying old commit clogs to new server ok Setting next transaction id for new cluster ok Resetting WAL archives ok Setting frozenxid counters in new cluster ok Creating databases in the new cluster ok Creating placeholder relfiles for toast relations /u/pgsql/data/base/16397/16586 Adding support functions to new cluster ok Restoring database schema to new cluster ok Restoring relations to use fixed toast file names /u/pgsql/data/base/16397/16586 Removing support functions from new cluster ok Restoring user relation files /u/pgsql.old/data/base/11511/2613 Setting next oid for new cluster ok Creating script to delete old cluster ok Adjusting sequences ok Rapid Upgrades With Pg-Migrator 16
  • 18. Sample Run: Completion Upgrade complete ---------------- | Optimizer statistics and free space information | are not transferred by pg_migrator so consider | running: | vacuumdb --all --analyze | on the newly-upgraded cluster. | Running this script will delete the old cluster’s data files: | /u/postgres/pg_migrator_output/delete_old_cluster.sh Rapid Upgrades With Pg-Migrator 17
  • 19. Possible Data Format Changes Change Conversion Method clog none heap page header, including bitmask convert to new page format on read tuple header, including bitmask convert to new page format on read data value format create old data type in new cluster index page format reindex, or recreate index methods TOAST page format convert to new page format on read Rapid Upgrades With Pg-Migrator 18
  • 20. Migration Timings Migration Method Minutes dump/restore 300.0 dump with parallel restore 180.0 pg_migrator in copy mode 44.0 pg_migrator in link mode 0.7 Database size: 150GB, 850 tables The last duration is 44 seconds. Timings courtesy of Stefan Kaltenbrunner (mastermind on IRC) Rapid Upgrades With Pg-Migrator 19
  • 21. Conclusion Rapid Upgrades With Pg-Migrator 20

×