Download to read offline
Uploaded byDatavail
Upgrading and Testing Rollback of a Multi-TB Amazon Aurora PostgreSQL Database Using Blue-Green Deployment
Upgrading and Testing Rollback of a Multi-TB Amazon Aurora PostgreSQL Database Using Blue-Green Deployment Managing upgrades for large, mission-critical databases is one of the most challenging tasks for DBAs and cloud architects. When dealing with multi-terabyte Amazon Aurora PostgreSQL databases, complexity multiplies—requiring meticulous planning, robust strategies, and proven best practices to ensure minimal downtime and zero data loss. In this PGConf NYC 2023 session, Shailesh Rangani, Director and Global Practice Lead for PostgreSQL Services at Datavail, shares a comprehensive roadmap for upgrading large Aurora PostgreSQL clusters from older versions (such as PG11) to newer versions (like PG15) using Blue-Green Deployment methodology. This approach enables near-zero downtime upgrades and provides a seamless rollback mechanism in case of unexpected issues. The presentation covers: Major Upgrade Options for Aurora PostgreSQL, including Import/Export, In-Place upgrades, and Logical Replication. Blue-Green Deployment Architecture: How to set up parallel environments (Blue for current version, Green for future version) and switch roles safely. Aurora Fast Database Cloning: Leveraging Amazon Aurora’s rapid cloning feature to create target environments quickly. Step-by-Step Upgrade Process: Preparing the source database (logical replication setup, WAL cache configuration, validating primary keys and sequences). Creating publications and replication slots. Cloning and preparing the target database. Performing optional maintenance tasks like VACUUM FULL, REINDEX, and ANALYZE. Upgrading clusters from PG11 to PG15 and validating the upgrade. Setting up replication between old and new clusters. Cutover and Rollback Strategies: Switching roles between Blue and Green environments. Validating replication lag and resetting sequences. Breaking replication and updating DNS endpoints. Implementing rollback procedures to revert if needed. Observability and Alerting: Monitoring schema changes, replication slot status, replication lag, and CloudWatch metrics. Real-World Use Cases: Minor and major upgrades, maintenance tasks, schema changes, and application deployments. Packed with actionable insights, best practices, and lessons learned from real-world implementations, this guide helps DBAs, DevOps engineers, and cloud architects confidently plan and execute large-scale PostgreSQL upgrades on AWS Aurora with minimal risk. About the Speaker Shailesh Rangani brings over 18 years of experience in database technologies and holds certifications across AWS, Azure, OCI, Oracle, PostgreSQL, MongoDB, and Db2 LUW. 👉 Download the full presentation here: https://www.datavail.com/resources/upgrading-and-testing-rollback/
More Related Content
Similar to Upgrading and Testing Rollback of a Multi-TB Amazon Aurora PostgreSQL Database Using Blue-Green Deployment
Upgrading and Testing Rollback of a Multi-TB Amazon Aurora PostgreSQL Database Using Blue-Green Deployment
- 1. Upgrading and Testing Rollback PRESENTEDBY Shailesh Rangani of a Multi-TB Amazon Aurora PostgreSQL Database Using Blue-Green Deployment Director and Global Practice Lead – PostgreSQL Datavail
- 2. www.datavail.com 2 Shailesh Ranganiis Practice Lead of PostgreSQL Services with 18+ years’ experience in the database domain. He holds certifications on cloud platforms like AWS, Azure, and OCI, along with database platforms like Oracle, PostgreSQL, MongoDB, and Db2 LUW. He is an expert in the design, deployment, administration, and management of data- intensive applications that enable organizations to effectively analyze and process large volumes of structured and unstructured data. S Shailesh Rangani Director & Global Practice Lead - PostgreSQL, Datavail Connect or follow me on LinkedIn: https://www.linkedin.com/in/ shaileshrangani/
- 3. www.datavail.com 3 Datavail at aGlance Delivering a superior approach to leverage data through application of a tech-enabled global delivery model & deep specialization in databases, data management, and application services. $25 M Invested in IP that improves the service experience and drives efficiency 16 + Years building and operating mission critical data and application systems 1,000 + Employees staffed 24x7, resolving over 2,000,000 incidents per year www.datavail.com 3
- 4. www.datavail.com 4 Fill outthe session evaluation and enter to
- 5. What Are theMajor Upgrade Options?
- 6. www.datavail.com 6 Aurora PostgreSQL– Major Upgrade Import/Export In-Place Aurora PostgreSQL – v11 Aurora PostgreSQL – v15 Aurora PostgreSQL – v11 EXPORT Aurora PostgreSQL – v15 IMPORT Logical Replication/DMS/Pub-Sub Aurora PostgreSQL – v11 Aurora PostgreSQL – v15 AWS DMS
- 7.
- 8. www.datavail.com 8 AWS Cloud Region AmazonRoute 53 DNS Endpoint App users Aurora PostgreSQL – v11 VPC Blue environment Live Version Aurora PostgreSQL – v15 Green environment Future Version
- 9. What is AuroraFast Cloning?
- 10. www.datavail.com 10 Aurora PostgreSQL– Source S1 S2 S3 S4
- 11. www.datavail.com 11 S1 S2S3 S4 Aurora PostgreSQL – Source Aurora PostgreSQL – Clone
- 12. www.datavail.com 12 S1 S2S3 S4 S1b Aurora PostgreSQL – Source Aurora PostgreSQL – Clone
- 13. www.datavail.com 13 S1 S2S3 S4 S1b S3g Aurora PostgreSQL – Source Aurora PostgreSQL – Clone
- 14. Steps for Aurora PostgreSQLMajor Upgrade - Blue/Green
- 15.
- 16. www.datavail.com 16 Enable logicalreplication rds.logical_replication = 1 Setup WAL cache (PG Version - 14.5, 13.8, 12.12, and 11.17 or higher) rds.logical_wal_cache=2GB Validate that all the tables have primary keys (including Partition tables) Analyze/Group tables based on number of transactions/loads Validate sequences Validate replication slot configuration max_logical_replication_workers; max_replication_slots; Preparing Source DB (PG11)
- 17. www.datavail.com 17 Create multiplepublications CREATE PUBLICATION pub_large_tab1; CREATE PUBLICATION pub_large_tab2; CREATE PUBLICATION pub_large_tab3; CREATE PUBLICATION pub_medium_tabs; CREATE PUBLICATION pub_small_tabs; Add Tables to Publications ALTER PUBLICATION pub_large_tab1 add table large_table1; : : Create replication slots SELECT pg_create_logical_replication_slot('pub_large_tab1', 'pgoutput’); : : Preparing Source DB (PG11)
- 18. www.datavail.com 18 pub_large_tab1 pub_large_tab2 pub_large_tab3 pub_medium_tabs pub_small_tabs Publications ReplicationSlots Slot1 Change data capture Large_Tab1 Slot2 Change data capture Large_Tab2 Slot3 Change data capture Large_Tab3 Slot4 Change data capture Medium_Tabs Slot5 Change data capture Small_Tabs
- 19. Create Clone and PrepareTarget
- 20. www.datavail.com 20 Preparing TargetDB (PG11) Create clone datavail=> SELECT * FROM aurora_volume_logical_start_lsn(); aurora_volume_logical_start_lsn --------------------------------- 1CC/324A7EA8 Note down start LSN from the clone Drop replication slots select pg_drop_replication_slot(‘pub_large_tab1’); : :
- 21.
- 22. www.datavail.com 22 Maintenance Taskson (PG11) Vacuum full on bloated tables datavail=> REINDEX DATABASE datavail; REINDEX REINDEX on bloated indexes Analyze database datavail=> analyze; ANALYZE datavail=> vacuum full large_table1; VACUUM datavail=> vacuum full large_table2; VACUUM : :
- 23. Upgrade Cluster from PG11to PG15
- 24. www.datavail.com 24 Upgrade Clusterto PG15 (PG11) Upgrade to PG15 datavail=> vacuum analyze; VACUUM OR vacuumdb -h datavail-cluster.abcdgfsdgsgs.us-west- 2.rds.amazonaws.com -p 5432 -d prod -U postgres -e -j 10 -v -Z Validate upgrade and execute vacuum analyze
- 25. Setup Replication Between PG11and PG15
- 26. www.datavail.com 26 Setup Replication(PG15) Create subscriptions datavail=> CREATE SUBSCRIPTION sub_large_tab1 CONNECTION 'host=datavail-cluster.cluster-czfcfxk38byn.us-east-1.rds.amazonaws.com dbname=datavail port=5432 user=sysdba password=************** PUBLICATION pub_large_tab1 WITH ( copy_data = false, create_slot = false, enabled = false, connect = true, slot_name = 'pub_large_tab1' ); CREATE SUBSCRIPTION : : Get replication origin names datavail=> SELECT * FROM pg_replication_origin; roident | roname ---------+------------- 1 | pg_53563397 2 | pg_53563398 3 | pg_53563399 4 | pg_53563400 5 | pg_53563401
- 27. www.datavail.com 27 Setup Replication(PG15) Start replication from the clone time (LSN) datavail=> SELECT pg_replication_origin_advance('pg_53563397','1CC/324A7EA8'); datavail=> SELECT pg_replication_origin_advance('pg_53563398','1CC/324A7EA8'); : : Enable Replication datavail=> ALTER SUBSCRIPTION sub_large_tab1 ENABLE; ALTER SUBSCRIPTION datavail=> ALTER SUBSCRIPTION sub_large_tab2 ENABLE; ALTER SUBSCRIPTION : :
- 28. www.datavail.com 28 Validate Replication(PG11) Validate replication status datavail=> select slot_name, active, restart_lsn, confirmed_flush_lsn from pg_catalog.pg_replication_slots; slot_name | active | restart_lsn | confirmed_flush_lsn ------------------+----------+--------------+-------------------- pub_large_tab1 | t | 1CC/3554DD50 | 1CC/355BBA58 pub_large_tab2 | t | 1CC/3554DD50 | 1CC/355BBA58 pub_large_tab3 | t | 1CC/3554DD50 | 1CC/355BBA58 pub_medium_tabs | t | 1CC/3554DD50 | 1CC/355BBA58 pub_small_tabs | t | 1CC/3554DD50 | 1CC/355BBA58 (5 rows) Validate replication status - By size datavail=> SELECT slot_name, active, active_pid, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS diff_size, pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS diff_bytes FROM pg_replication_slots WHERE slot_type = 'logical'; slot_name | active | active_pid | diff_size | diff_bytes -----------------+--------+------------+------------+-----------+------------ pub_large_tab1 | t | 1791 | 136 bytes | 136 pub_large_tab2 | t | 1966 | 136 bytes | 136 pub_large_tab3 | t | 2117 | 136 bytes | 136 pub_medium_tabs | t | 2117 | 136 bytes | 136 pub_small_tabs | t | 2117 | 136 bytes | 136 (5 rows)
- 29. www.datavail.com 29 AWS Cloud Region AmazonRoute 53 DNS Endpoint App users Aurora PostgreSQL – v11 VPC Blue environment Live Version Aurora PostgreSQL – v15 Green environment Future Version Replication
- 30.
- 31. www.datavail.com 31 AWS Cloud Region AmazonRoute 53 DNS Endpoint App users Aurora PostgreSQL – v11 VPC Blue environment Live Version Aurora PostgreSQL – v15 Green environment Future Version Replication Setup maintenance page OR Stop applications/update security groups Validate replication lag (Should be 0) Reset sequences on Green (PG15) Break replication between PG11 and PG15 • Drop Subscription on Green (PG15) • Drop Publication on Blue (PG11) Switch the roles of the Blue/Green PG15 • Create publications • Create replication slots PG11 • Create subscriptions Validate replication lag (Should be 0) Update CNAME Remove maintenance page/Start applications Blue environment Rollback Version Green environment Live Version
- 32.
- 33. www.datavail.com 33 AWS Cloud Region AmazonRoute 53 DNS Endpoint App users Aurora PostgreSQL – v11 VPC Blue environment Rollback Version Aurora PostgreSQL – v15 Green environment Live Version Replication Setup maintenance page OR Stop applications/update security groups Validate replication lag (Should be 0) Reset sequences on Blue (PG11) Break replication between PG15 and PG11 • Drop Subscription on Blue (PG11) • Drop Publication on Green (PG15) Switch the roles of the Blue/Green PG11 • Create publications • Create replication slots PG15 • Create subscriptions Validate replication lag (Should be 0) Update CNAME Remove maintenance page/Start applications Blue environment Live Version Green environment Future Version
- 34.
- 35. www.datavail.com 35 Observability/Alerting Schema changes onthe Source DB Replication slot status Replication lag increase Cloudwatch Metrics Sequence reset TransactionLogsDiskUsage OldestReplicationSlotLag ReplicationSlotDiskUsage
- 36.
- 37. www.datavail.com 37 Blue/Green DeploymentUse Cases Minor/Major database upgrade Maintenance tasks like REINDEX, FULL VACUMM, ANALYZE Partitioning table Database Parameter changes Application deployment, Schema changes, etc..
- 38. www.datavail.com 38 Fill outthe session evaluation and enter to
- 39.
- 40. www.datavail.com 40 Thank You ShaileshRangani Director & Global Practice Lead : PostgreSQL Services shailesh.rangani@datavail.com +1 866-623-4956
Editor's Notes
- #15 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #19 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #21 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #23 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #25 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #30 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #32 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #34 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.
- #36 And first one in this category is Design best Practices… In this we will go through some of the sample Design best Practice tasks where you as a DBA or DevOps person can spend time on some of these.