Rapid Upgrades With Pg_Migrator

                                  BRUCE MOMJIAN,
                                   ENTER...
Traditional Postgres Upgrade Options



    pg_dump (logical dump)/restore
 




    Slony
 




Rapid Upgrades With Pg-Mi...
Why Upgrading Postgres Is Complex



    New features often require system table changes
 




    However, the internal d...
Why Pg_Migrator



    Very fast upgrades
 




    Optionally no additional disk space
 




pg_migrator installs new sys...
How It Works: Initial Setup
                           Old Cluster                     New Cluster


                     ...
Decouple New Clog Via Freezing
                          Old Cluster                     New Cluster


                   ...
Transfer Clog and XID
                             Old Cluster                           New Cluster


                   ...
Get Schema Dump
                                                   pg_dumpall - -schema

                             Old ...
Reserve TOAST OIDs Using Relfilenodes (pre-9.0)
                               Old Cluster                     New Cluster
...
Restore Schema In New Cluster
                                                  pg_dumpall - -schema

                    ...
Connect TOAST Placeholders To the Proper Relations
                    (pre-9.0)
                           Old Cluster   ...
Copy User Heap/Index Files
                           Old Cluster                     New Cluster


                      ...
Complete
                           Old Cluster                                New Cluster


                           Sy...
How It Works: In Detail

    Check for cluster compatability
 




    – locale
    – encoding
    – integer datetimes (de...
Collect cluster information
 




    Install support functions that call internal backend functions
 




    Create plac...
Sample Run: Performing Consistency Checks

    Performing Consistency Checks
    -----------------------------
    Checkin...
Sample Run: Performing Migration
    Performing Migration
    --------------------
    Adding ".old" suffix to tablespace ...
Sample Run: Completion



    Upgrade complete
    ----------------
    | Optimizer statistics and free space information
...
Possible Data
                                  Format Changes


                         Change                    Conver...
Migration Timings


                          Migration Method             Minutes
                          dump/restore ...
Conclusion




Rapid Upgrades With Pg-Migrator                20
Upcoming SlideShare
Loading in …5
×

Rapid Upgrades With Pg_Migrator

787 views
731 views

Published on

Bruce Momjian

Pg_Migrator allows data to be transfered between major Postgres versions without a dump/restore. This talk explains the internal workings of pg_migrator and includes a pg_migrator demonstration

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
787
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rapid Upgrades With Pg_Migrator

  1. 1. Rapid Upgrades With Pg_Migrator BRUCE MOMJIAN, ENTERPRISEDB March, 2010 Abstract Pg_Migrator allows migration between major releases of Postgres without a data dump/reload. This presentation explains how pg_migrator works. Creative Commons Attribution License http://momjian.us/presentations
  2. 2. Traditional Postgres Upgrade Options pg_dump (logical dump)/restore   Slony   Rapid Upgrades With Pg-Migrator 1
  3. 3. Why Upgrading Postgres Is Complex New features often require system table changes   However, the internal data format rarely changes   Rapid Upgrades With Pg-Migrator 2
  4. 4. Why Pg_Migrator Very fast upgrades   Optionally no additional disk space   pg_migrator installs new system tables while using data files from the previous Postgres version. Rapid Upgrades With Pg-Migrator 3
  5. 5. How It Works: Initial Setup Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog Rapid Upgrades With Pg-Migrator 4
  6. 6. Decouple New Clog Via Freezing Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 3 5 6 8 9 2 3 Freeze 5 6 8 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 X X clog clog Rapid Upgrades With Pg-Migrator 5
  7. 7. Transfer Clog and XID Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog controldata xid controldata Rapid Upgrades With Pg-Migrator 6
  8. 8. Get Schema Dump pg_dumpall - -schema Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 13 19 25 14 20 26 15 21 27 clog clog Rapid Upgrades With Pg-Migrator 7
  9. 9. Reserve TOAST OIDs Using Relfilenodes (pre-9.0) Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 11 17 23 12 18 24 18 13 19 25 14 20 26 26 15 21 27 clog clog This is necessary because heap references to TOAST tables contain the TOAST oids for easy lookup. Rapid Upgrades With Pg-Migrator 8
  10. 10. Restore Schema In New Cluster pg_dumpall - -schema Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 9
  11. 11. Connect TOAST Placeholders To the Proper Relations (pre-9.0) Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 X 41 47 clog clog Rapid Upgrades With Pg-Migrator 10
  12. 12. Copy User Heap/Index Files Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 11
  13. 13. Complete Old Cluster New Cluster System Tables System Tables 1 4 7 1 4 7 2 5 8 2 5 8 3 6 9 3 6 9 pg_class pg_class User Tables User Tables 10 16 22 30 36 42 11 17 23 31 37 43 12 18 24 32 18 44 13 19 25 33 39 45 14 20 26 34 40 26 15 21 27 35 41 47 clog clog Rapid Upgrades With Pg-Migrator 12
  14. 14. How It Works: In Detail Check for cluster compatability   – locale – encoding – integer datetimes (default changed from 8.3 -> 8.4) Use pg_dumpall to dump old cluster schema (no data)   Freeze all new cluster rows (remove reference to clog entries)   Rename tablespaces to *_old (pre-9.0)   New cluster uses old xid counter value (see freeze above)   – Set system table frozen xids to match the current xid Create new users/databases   Rapid Upgrades With Pg-Migrator 13
  15. 15. Collect cluster information   Install support functions that call internal backend functions   Create placeholder files to reserve relfilenode file names (pre-9.0)   Create schema in new cluster   Adjust new cluster to use reserved relfilenode names (pre-9.0)   – Delete placeholder toast relfilenode files – Remove new cluster toast tables – Create new cluster toast table using reserved relfilenode – Assign new toast tables with proper relfilenodes to relations Copy or link files from old cluster to new cluster   – Toast tables have the same relfilenodes as in the old cluster Warn about any remaining issues, like REINDEX requirements   Rapid Upgrades With Pg-Migrator 14
  16. 16. Sample Run: Performing Consistency Checks Performing Consistency Checks ----------------------------- Checking old data directory (/u/pgsql.old/data) ok Checking new data directory (/u/pgsql/data) ok Checking for columns with user-defined composite types ok Checking for columns with user-defined array types ok Checking for columns with user-defined enum types ok Checking for /contrib/isn with bigint-passing mismatch ok Checking for invalid ’name’ user columns ok Checking for tsquery user columns ok Creating script to adjust sequences ok Creating catalog dump ok Checking for presence of required libraries ok | If pg_migrator fails after this point, you must | re-initdb the new cluster before continuing. | You will also need to remove the ".old" suffix | from /u/pgsql.old/data/global/pg_control.old and old | tablespace directory names. Many of these checks are unnecessary when migrating to Postgres 9.0. Rapid Upgrades With Pg-Migrator 15
  17. 17. Sample Run: Performing Migration Performing Migration -------------------- Adding ".old" suffix to tablespace directory names ok Adding ".old" suffix to old global/pg_control ok Analyzing all rows in the new cluster ok Freezing all rows on the new cluster ok Deleting new commit clogs ok Copying old commit clogs to new server ok Setting next transaction id for new cluster ok Resetting WAL archives ok Setting frozenxid counters in new cluster ok Creating databases in the new cluster ok Creating placeholder relfiles for toast relations /u/pgsql/data/base/16397/16586 Adding support functions to new cluster ok Restoring database schema to new cluster ok Restoring relations to use fixed toast file names /u/pgsql/data/base/16397/16586 Removing support functions from new cluster ok Restoring user relation files /u/pgsql.old/data/base/11511/2613 Setting next oid for new cluster ok Creating script to delete old cluster ok Adjusting sequences ok Rapid Upgrades With Pg-Migrator 16
  18. 18. Sample Run: Completion Upgrade complete ---------------- | Optimizer statistics and free space information | are not transferred by pg_migrator so consider | running: | vacuumdb --all --analyze | on the newly-upgraded cluster. | Running this script will delete the old cluster’s data files: | /u/postgres/pg_migrator_output/delete_old_cluster.sh Rapid Upgrades With Pg-Migrator 17
  19. 19. Possible Data Format Changes Change Conversion Method clog none heap page header, including bitmask convert to new page format on read tuple header, including bitmask convert to new page format on read data value format create old data type in new cluster index page format reindex, or recreate index methods TOAST page format convert to new page format on read Rapid Upgrades With Pg-Migrator 18
  20. 20. Migration Timings Migration Method Minutes dump/restore 300.0 dump with parallel restore 180.0 pg_migrator in copy mode 44.0 pg_migrator in link mode 0.7 Database size: 150GB, 850 tables The last duration is 44 seconds. Timings courtesy of Stefan Kaltenbrunner (mastermind on IRC) Rapid Upgrades With Pg-Migrator 19
  21. 21. Conclusion Rapid Upgrades With Pg-Migrator 20

×