Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,677
On Slideshare
2,677
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
70
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Upgrade & Migrate on Multi-Terabyte Mission Critical RAC with Near Zero Downtime Jane Brown Senior Database Administrator, Oracle Certified Professional (OCP)
  • 2. Purpose • Minimize planned down time • Meet the 7x24 availability requirement for Dell Mission Critical Applications • Streamline Dell Mission Critical databases upgrades to 11g 2 Database Management Practice
  • 3. Current Versions • DB Version: 10.2.0.4 • OS Version: Suze Linux 64-bit Target Upgrade Versions • Database Version: 11.1.0.7 • OS Version: OEL 64-bit 3 Database Management Practice
  • 4. High Availability Data Guard Architecture Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby Other Key Data: Data Files: ASM Storage: EMC Symmetric lsdb01-06 LTST_LS Database Flashback: on Logical Standby Data Center 1 Data Center 2 4 Database Management Practice
  • 5. I. Reimage Logical Standby Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby Other Key Data: Data Files: ASM Storage: EMC Symmetric lsdb01-06 LTST_LS Database Flashback: on Logical Standby Data Center 1 Data Center 2 5 Database Management Practice
  • 6. Backup Essential Files 1) Backup spfile to pfile › lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/initltst.ora' from spfile; 2) Backup Oracle Home db_1 and save it on a different host › cd $ORACLE_HOME/.. › /u01/app/oracle/product/10.2.0> tar cvf db_1.tar db_1 › /u01/app/oracle/product/10.2.0> scp db_1.tar prdb01:/dbscratch/preReImange/lsdb01/. 3) Capture admin directories › /u01/app/oracle/admin/ltst> ls adump bdump cdump udump 4) Capture cluster name › grep -i name $ORA_CRS_HOME/install/cluster.ini cluster_name= sdr2ppramercrs 5) Capture LUN Mapping – by Linux System Engineer 6 Database Management Practice
  • 7. Reimage • Shutdown database • Re-image the host cluster servers to OEL 11g image by Linux System Engineer › Deploy Oracle Enterprise Linux 64-bit OS › Restore EMC LUN mapping › Deploy Oracle 11g CRS RPM › Deploy Oracle 11g Software RPM 7 Database Management Practice
  • 8. Start 11g ASM 1) Verify that CRS and Nodeapps are all up › crs_stat.sh 2) Create 11g ASM Initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora /u01/app/oracle/product/11.1.0/asm_1/dbs/. › remove *_dump_dest from the file 3) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapw+ASM1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 4) Source ASM environment › export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1 › export PATH=$ORACLE_HOME/bin:/usr/bin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/sbin:/usr/sbin:./ › export LD_LIBRARY_PATH=$ORACLE_HOME/lib › export ORACLE_SID=+ASM1 5) Startup 11g ASM › sqlplus / as sysasm › SQL> startup 8 Database Management Practice
  • 9. Register ASM to CRS 1. Add ASM to CRS › srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11.1.0/asm_1 › # Repeat to add all 6 nodes 2. Copy Initialization and Password files to the rest of RAC nodes › init+ASM1.ora › orapw+ASM1 3. Start ASM on nodes via srvctl › +ASM1 SQL>shutdown › srvctl start asm –n lsprddb01 #repeat for node 2-6 9 Database Management Practice
  • 10. Start 10g DB 1) Restore 10g Oracle Home to a temporary directory on node1 only › /dbscratch/> tar xvf db_1.tar 2) Create a symbolic link for the original 10g Oracle Home › mkdir –p /u01/app/oracle/product/10.2.0/ › cd /u01/app/oracle/product/10.2.0/ › ln –s /dbscratch/db_1 db_1 3) Create admin directories › mkdir -p /u01/app/oracle/admin/ltst/ › cd /u01/app/oracle/admin/ltst/ › mkdir adump bdump cdump udump 4) Source 10g environment › . 10gR2.env 5) Startup 10g Database › sqlplus / as sysdba › SQL> startup 10 Database Management Practice
  • 11. Verify Logical Standby logs 1. From Logical Standby: Startup 10g Listener › lsnrctl start LISTENER_LSPRDDB01 2. From Primary: Query the latest online logs › SQL> select THREAD#, SEQUENCE# from v$LOG where status='CURRENT'; THREAD# SEQUENCE# ---------- ---------- 1 5592 2 4076 3 4089 4 4149 5 4091 6 4092 3. From Primary: Force log archive › alter system archive log current; 4. From Logical Standby: Verify that the latest logs are applied › SQL> select THREAD#, MAX(SEQUENCE#), APPLIED from dba_logstdby_log where applied!='YES'; THREAD# MAX(SEQUENCE#) APPLIED ---------- -------------- -------- 1 5592 YES 2 4076 YES 3 4089 YES 4 4149 YES 5 4091 YES 6 4092 YES 11 Database Management Practice
  • 12. II. Upgrade Logical Standby Database
  • 13. Pre-Upgrade 1) Run pre-upgrade information tools › SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql 2) Resolve any outstanding issue 3) Create a guaranteed restore point › SQL> create restore point pre_upgrade guarantee flashback database; › SQL> select * from v$restore_point; 4) Stop 10g listener › .~ 10gR2.env › lsnrctl stop 5) Stop SQL Apply and shutdown 10g database › SQL> alter database stop logical standby apply; › SQL> shutdown 13 Database Management Practice
  • 14. Start 11g Database 1) Create 11g DB Initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/. › Edit initialization file to – remove *_dump_dest – set cluster_database=false – set dg_broker_start=false 2) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 3) Restore tnsnames.ora › cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora /u01/app/oracle/product/11.1.0/asm_1/network/admin/. 4) Startup 11g database › . .bash_profile › cd $ORACLE_HOME/rdbms/admin › SQL> startup upgrade 14 Database Management Practice
  • 15. Upgrade Database 1. Upgrade database › SQL> spool /tmp/upgrade.log › SQL> @catupgrd.sql #40 min, db shutdown at then end of upgrade 2. Run post update scripts › ltst1 SQL> startup › ltst1 SQL> @utlu111s.sql › ltst1 SQL> @catuppst.sql › ltst1 SQL> @utlrp.sql 3. Enable cluster database › ltst1 SQL> alter syste set cluster_database=true 4. Restart Instance 1 15 Database Management Practice
  • 16. Register Database to CRS 1. Add Database and Instances to CRS › srvctl add database -d ltst_ls -o /u01/app/oracle/product/11.1.0/db_1 › srvctl add instance -d ltst_ls -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance 2. Copy following files to the rest of RAC nodes › tnsnames.ora › initltst1.ora › orapwltst1 3. Restart DB › ltst1 SQL> shutdown › srvctl start database –d ltst_ls 16 Database Management Practice
  • 17. III. Synchronize Archive log (10g->11g) Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby 11g lsdb01-06 LTST_LS Logical Standby Data Center 1 Data Center 2 17 Database Management Practice
  • 18. Start Apply 1. Query outstanding log in Logical Standby › col file_name for a70 › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES' 2. Start log Apply › alter database start logical standby apply immediate; 3. Repeat step 1 until all logs are applied 18 Database Management Practice
  • 19. IV. Switchover to Logical Standby Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 19 Database Management Practice
  • 20. Transition Primary -> Logical Standby 1. Verify Switchover status › SQL> select switchover_status from v$database; SWITCHOVER_STATUS -------------------- TO STANDBY If the query returns “SESSIONS ACTIVE’, then ensure it’s ok to shut them down › SQL > SELECT SID, PROCESS, PROGRAM FROM V$SESSION WHERE TYPE = 'USER' AND SID <> (SELECT DISTINCT SID FROM V$MYSTAT); 2. Disable all RAC instances except instance 1 › srvctl stop instance –d ltst –i ltst2; #repeat for lsts2 - 6 › alter database disable thread 2; ; #repeat for thread 2 - 6 3. Switchover › SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO LOGICAL STANDBY;; 20 Database Management Practice
  • 21. Transition Logical Standby -> Primary 1. Defer Redo log transfer in Logical Standby › SQL> alter system set log_archive_dest_3='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=s drseprddb01- vip.us.dell.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ltst_XPT.prd.amer.dell.com)(INSTAN CE_NAME=ltst1)(SERVER=dedicated)))"',' LGWR ASYNC NOAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=30 db_unique_name="ltst" register net_timeout=15 valid_for=(online_logfile,primary_role)' scope=both › SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER; 2. Disable all RAC instances except instance 1 › srvctl stop instance –d ltst_ls –i ltst2; #repeat for instnace 2-6 › alter database disable thread 2; #repeat for thread 2-6 3. Commit Switchover › SQL> select switchover_status from v$database; SWITCHOVER_STATUS -------------------- TO PRIMARY › SQL> alter database commit to switchover to primary; 21 Database Management Practice
  • 22. Start Rest of RAC Instances 1. New Primary LTST_LS: Enable threads and start RAC instances 2-6 › alter database enable thread 2 › srvctl start instance –d ltst_ls –i ltst2; 2. New Logical Standby LTST: Enable threads and start RAC instances 2-6 › alter database enable thread 2; › srvctl start instance –d ltst –i ltst2; 22 Database Management Practice
  • 23. V. Reimage Original Primary Hosts Redo Apply 10g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 23 Database Management Practice
  • 24. Reimage Original Primary Host • Follow the process used in reimaging logical standby hosts (slice 7 – 10) • Do not sync archived log from 11g –> 10g (skip slice 11) 24 Database Management Practice
  • 25. VI. Upgrade Original Primary to 11g • Follow the process used in upgrading Logical Standby Database (slice 13– 16) 25 Database Management Practice
  • 26. VII.Synchronize Archive log (11g->11g) 1. Query outstanding log in Logical Standby › col file_name for a70 › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES' 2. Start log Apply › alter database start logical standby apply immediate; 3. Repeat step 1 until all logs are applied 26 Database Management Practice
  • 27. VIII.Reimage Physical Standby Hosts Redo Apply 10g 11g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 27 Database Management Practice
  • 28. Reimage Physical Standby Hosts • Follow the process used in reimaging Logical Standby Database (slice 7 – 10, skip step 5 in slice 10) • Mount 10g Database as standby › sqlplus / as sysdba › SQL> startup monut › SQL> alter database mount standby database 28 Database Management Practice
  • 29. Start 11g Physical Standby Database 1) Create 11g DB initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/. › Edit initialization file to remove *_dump_dest 2) Create spfile from pfile 3) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 4) Restore tnsnames.ora › cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora c/u01/app/oracle/product/11.1.0/asm_1/network/admin/. 5) Mount 11g standby database › . .bash_profile › SQL> startup nomount › SQL> alter database mount standby database Note: wait for dg broker configuration to apply log 29 Database Management Practice
  • 30. Register Physical Standby Database to CRS 1. Add Database and Instances to CRS › srvctl add database -d ltst_dr -o /u01/app/oracle/product/11.1.0/db_1 –s mount –s physical_standby -m dr.amer.dell.com › srvctl add instance -d ltst_dr -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance 2. Copy following files to the rest of RAC nodes › tnsnames.ora › initltst1.ora › orapwltst1 3. Restart DB › ltst1 SQL> shutdown › srvctl start database –d ltst_dr 30 Database Management Practice
  • 31. IX. Switchover to Original Primary Redo Apply 11g 11g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Bystand Physical Standby 11g lsdb01-06 LTST_LS Logical Standby Data Center 1 Data Center 2 31 Database Management Practice
  • 32. Switchover via Data Guard Broker I. Remove old Data Guard Broker configuration › dgmgrl > remove configuration II. Create new Data Guard Broker configuration › dgmgrl > create configuration dg_ltst as primary database is ltst_ls connect identifier is ltst_ls; › dgmgrl > add database ltst as connect identifier is ltst_prd maintained as logical ; III. Switchover to Logical Standby › dgmgrl > switchover to ltst IV. Add Physical Standby to Data Guard Broker 32 Database Management Practice
  • 33. X. Verify Data Guard Status 1) Verify Database Status › dgmgrl > show configuration 2) Verify Log Applying Status in Physical Standby › Run from primary › @dg_standby_log.sql 3) Verify Log Applying Status in Logical Standby › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= 'YES' 33 Database Management Practice
  • 34. Summary I. Reimage Logical Standby Hosts to 11g – Linux System Engineer II. Upgrade Logical Standby DB to 11g –Database Administrator III. Sync Archive log from 10g Primary to 11g Logical Standby - Database Administrator IV. Switchover to Logical Standby Manually –Database Administrator V. Reimage original Primary Hosts to 11g – Linux System Engineer VI. Upgrade the original Primary DB to 11g – Database Administrator VII. Sync Archive log from 11g Primary to 11g Logical Standby - Database Administrator VIII. Reimage Physical Standby Hosts to 11g – Linux System Engineer IX. Switchover to original Primary via DG Broker- Database Administrator X. Verify Data Guard Status 34 Database Management Practice
  • 35. Reference • Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02 • Oracle® Data Guard Concepts and Administration 11g R1 (11.1) B28294-03 • Oracle Data Guard Broker 11g Release 1 (11.1) B28295) • Database Rolling Upgrade Using Data Guard SQL Apply - Oracle Database 11g and 10gR2 35 Database Management Practice
  • 36. For more information: • Visit Dell at Oracle OpenWorld in Moscone South booth 521 • Visit www.dell.com/oracle for more information • Dell Services Oracle Practice information at http://content.dell.com/us/en/enterprise/oracle-solutions- services.aspx • Dell presentations at OOW 2010 at http://www.slideshare.net/dellenterprise • Dell | Oracle white papers at http://content.dell.com/us/en/enterprise/oracle- solutions.aspx#facets=whitepaper&p=1 • Dell | Oracle Tested and Validated documentation at http://content.dell.com/us/en/enterprise/d/solutions/product- configurations.aspx 36 Global Marketing
  • 37. Questions ? 37 Database Management Practice
  • 38. Versions - Production 38 Database Management Practice