Recommended
More Related Content
What's hot
What's hot (20)
Similar to PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki
Similar to PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki (20)
More from Equnix Business Solutions
More from Equnix Business Solutions (20)
Recently uploaded
Recently uploaded (20)
PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki
- 1. https://www.2ndQuadrant.com PostgreSQL Database Migration and Maintenance Avoiding Long Downtime Koichi Suzuki PGConf.Asia 2019
- 2. https://www.2ndQuadrant.com PostgreSQL Now ● Features ○ Matured for very wide range of applications ● More production use case ○ Important business scenes ● More chance of database maintenance ○ Version up ○ Database design change ■ Physical ■ Logical ● Business Requirement (24/7 for example) ○ Less acceptance of long maintenance window
- 3. https://www.2ndQuadrant.com Database Migration and Design Change ● Column data type change ○ INT -> BIGINT ○ INT -> NUMERIC ○ VARCHAR -> TEXT Accomodate more records to a table Business Growth ALER TABLE? COPY? CREATE TABLE … SELECT?
- 5. https://www.2ndQuadrant.com Migration of too many tables Tables > 500,000 pg_dump too slow and hard to run with on-going business process
- 6. https://www.2ndQuadrant.com Most of these operations are blocking Typically take days Cannot simply perform in production environment Many use case require minimum down time (maintenance window), for example, < 15min. General Issue
- 7. https://www.2ndQuadrant.com ● Application characteristics ● Data characteristics Deeply depends upon use case specifics Hope following slides is applicable to many other cases. How to avoid long maintenance window
- 8. https://www.2ndQuadrant.com Partitioning an existing table Usual process: 1. CREATE TABLE PARTITIONED … 2. COPY from old table to new table Typical application/data characteristics: ● Log type: can use creation timestamp as partitioning key ● Data are not updated (may be deleted though) ● Has another primary key which is used as foreign key from other tables, but not declared explicitly.
- 9. https://www.2ndQuadrant.com Apps can continue to run. Partitioning Steps 1. Create new table with same column definition a. Creation timestamp as the partitioning key b. Exclude for PKEY/Unique constraint 2. Create partitions as needed 3. Copy row from original table 4. Lock original/new table 5. Do last copy 6. Rename original/new table 7. Unlock original/new table Maintenance Window, requres apps to stop.
- 13. https://www.2ndQuadrant.com First Data Copy: Don’t need to stop Apps.
- 14. https://www.2ndQuadrant.com Last Step: Need to stop Apps.
- 15. https://www.2ndQuadrant.com Maintain Unique Constraint Partitioning key must be a part of unique constraints Maintaining uniqueness is application’s responsibility 1. INSERT ○ Use of sequence/other default value 2. UPDATE ○ Use RULE/TRIGGER to prohibit unique value update All these depends upon application/date characteristics and/or requirements. Unique constraint in each partition may reduce a risk.
- 16. https://www.2ndQuadrant.com Discussed in the unconference, lead by Ibrar Ahmed on Sunday. Adding Unique Constraint
- 18. https://www.2ndQuadrant.com Change Sequence Datatype INTEGER to BIGINT: Beyond 4G limit
- 21. https://www.2ndQuadrant.com Migrate too many tables ● Some database have huge number of tables > 500,000 ● Migration by pg_dump is not practical ○ Takes too long ● Upgrade from very old version < even 9.0
- 22. https://www.2ndQuadrant.com Why FDW is good ● No specific feature/extension on the source ● Only libpq is needed ● Need postgres_fdw extension at the target
- 23. https://www.2ndQuadrant.com How FDW works Source PostgreSQL libpq libpq Target PostgreSQL postgres_fdw
- 24. https://www.2ndQuadrant.com Installing and setup postgres_fdw May need more, depending upon target database, for example, port and passwd.
- 25. https://www.2ndQuadrant.com Table migration (1) ● We can import all the remote schema ● No need to create each local table Create Foreign Table only: Actual data is still at remote server. Need to tweak max_locks_per_transaction GUC setting to import more tables in single statement.
- 26. https://www.2ndQuadrant.com Table Migration (2) ● Need to issue CREATE TABLE to import the data to local server To do this in single transaction for speedup, need to tweak max_locks_per_transaction GUC setting.
- 27. https://www.2ndQuadrant.com Evaluation ● Comparison with pg_dump ● 300,000 tables ● 1,000 rows for each table ● 3 columns: ○ one: integer, ○ two: varchar(32), ■ random length, random value
- 30. https://www.2ndQuadrant.com Environment SSD (SATA) PostgreSQL 9.3PostgreSQL 11.4 postgres_fdw pg_dump postgres 9.3 databasepostgres 11.4 database
- 31. https://www.2ndQuadrant.com Result pg_dump dump tables at PostgreSQL 9.3 1:51 pg_dump load tables to PostgreSQL 11.4 7:04 FDW Import table schema 1’18” FDW Import all the data 2:29 We need to consider the duration needed to copy pg_dump data from remote to local. In this case, the size of pg_dump output was about 11Gig.
- 32. https://www.2ndQuadrant.com Remarks/Summary ● No general steps to avoid long down time in database migration ● Need to observe characteristics of application and data itself ● Find what is dynamic and what is static ○ Beware DELETE and UPDATE ● Leverage static characteristics of the data ● Beware requirements/constraints ○ Find if you can ease requirements ■ May need to work with application ■ Plan carefully: schedule and resource ○ May need compromise
- 33. https://www.2ndQuadrant.com Possible PostgreSQL/Relevant Features ● Pkey/unique constraint over partitions ○ Ibrar Ahmed is working. ● View as partition ● Concurrent ALTER TABLE ● Logical replication
- 35. https://www.2ndQuadrant.com Change enum data type Enum type: customer_class class1 class2 class2 priority0 priority1 normal0 normal1 internal0 internal1 Business Rule Change -> Sometimes need both column for a while