Upgrade & Migrate on
Multi-Terabyte Mission Critical
RAC with Near Zero Downtime
Jane Brown Senior Database Administrator,...
Purpose
• Minimize planned down time
• Meet the 7x24 availability requirement for Dell
  Mission Critical Applications
• S...
Current Versions
• DB Version: 10.2.0.4
• OS Version: Suze Linux 64-bit




Target Upgrade Versions
• Database Version: 11...
High Availability Data Guard Architecture

                                    Redo Apply



     prddb01-06              ...
I. Reimage Logical Standby

                                  Redo Apply



     prddb01-06                               ...
Backup Essential Files
1)   Backup spfile to pfile
       › lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/i...
Reimage
• Shutdown database
• Re-image the host cluster servers to OEL 11g image by
  Linux System Engineer
    ›   Deploy...
Start 11g ASM
1)   Verify that CRS and Nodeapps are all up
       › crs_stat.sh

2)   Create 11g ASM Initialization file
 ...
Register ASM to CRS
1.   Add ASM to CRS
       ›      srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11...
Start 10g DB
1)   Restore 10g Oracle Home to a temporary directory on node1 only
      › /dbscratch/> tar xvf db_1.tar

2)...
Verify Logical Standby logs
1. From Logical Standby: Startup 10g Listener
      ›   lsnrctl start LISTENER_LSPRDDB01

2. F...
II. Upgrade Logical Standby
    Database
Pre-Upgrade
1)   Run pre-upgrade information tools
       ›    SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql

2) ...
Start 11g Database
1)   Create 11g DB Initialization file
       ›       cp /u01/app/oracle/product/10.2.0/db_1/dbs/initlt...
Upgrade Database
1.   Upgrade database
       ›   SQL> spool /tmp/upgrade.log
       ›   SQL> @catupgrd.sql #40 min, db sh...
Register Database to CRS
1.   Add Database and Instances to CRS
       ›   srvctl add database -d ltst_ls -o /u01/app/orac...
III. Synchronize Archive log (10g->11g)

                                  Redo Apply



     prddb01-06                  ...
Start Apply
1.   Query outstanding log in Logical Standby
       ›   col file_name for a70
       ›   select thread#, sequ...
IV. Switchover to Logical Standby

                                    Redo Apply



     prddb01-06                      ...
Transition Primary -> Logical Standby
1.   Verify Switchover status
       ›   SQL> select switchover_status from v$databa...
Transition Logical Standby -> Primary
1.   Defer Redo log transfer in Logical Standby
       ›   SQL> alter system set
   ...
Start Rest of RAC Instances
1.   New Primary LTST_LS: Enable threads and start RAC instances 2-6
       ›   alter database...
V. Reimage Original Primary Hosts

                                    Redo Apply

                       10g

     prddb0...
Reimage Original Primary Host
•    Follow the process used in reimaging logical standby hosts (slice 7 – 10)
•    Do not s...
VI. Upgrade Original Primary to 11g
•    Follow the process used in upgrading Logical Standby Database (slice 13–
     16)...
VII.Synchronize Archive log (11g->11g)
1.   Query outstanding log in Logical Standby
       ›   col file_name for a70
    ...
VIII.Reimage Physical Standby Hosts

                                    Redo Apply
                                      ...
Reimage Physical Standby Hosts
•    Follow the process used in reimaging Logical Standby Database (slice 7 –
     10, skip...
Start 11g Physical Standby Database
1)   Create 11g DB initialization file
       ›       cp /u01/app/oracle/product/10.2....
Register Physical Standby Database to CRS
1.   Add Database and Instances to CRS
       ›   srvctl add database -d ltst_dr...
IX. Switchover to Original Primary

                                    Redo Apply
                                       ...
Switchover via Data Guard Broker
I.    Remove old Data Guard Broker configuration
       ›       dgmgrl > remove configura...
X. Verify Data Guard Status
1)   Verify Database Status
      ›      dgmgrl > show configuration


2) Verify Log Applying ...
Summary
I.    Reimage Logical Standby Hosts to 11g – Linux System Engineer
II.   Upgrade Logical Standby DB to 11g –Databa...
Reference
• Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02
• Oracle® Data Guard Concepts and Administration 11g R1...
For more information:
•    Visit Dell at Oracle OpenWorld in Moscone South booth 521

•    Visit www.dell.com/oracle for m...
Questions ?

37           Database Management Practice
Versions - Production




38                      Database Management Practice
Upcoming SlideShare
Loading in...5
×

Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime

2,280

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,280
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
71
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime

  1. 1. Upgrade & Migrate on Multi-Terabyte Mission Critical RAC with Near Zero Downtime Jane Brown Senior Database Administrator, Oracle Certified Professional (OCP)
  2. 2. Purpose • Minimize planned down time • Meet the 7x24 availability requirement for Dell Mission Critical Applications • Streamline Dell Mission Critical databases upgrades to 11g 2 Database Management Practice
  3. 3. Current Versions • DB Version: 10.2.0.4 • OS Version: Suze Linux 64-bit Target Upgrade Versions • Database Version: 11.1.0.7 • OS Version: OEL 64-bit 3 Database Management Practice
  4. 4. High Availability Data Guard Architecture Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby Other Key Data: Data Files: ASM Storage: EMC Symmetric lsdb01-06 LTST_LS Database Flashback: on Logical Standby Data Center 1 Data Center 2 4 Database Management Practice
  5. 5. I. Reimage Logical Standby Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby Other Key Data: Data Files: ASM Storage: EMC Symmetric lsdb01-06 LTST_LS Database Flashback: on Logical Standby Data Center 1 Data Center 2 5 Database Management Practice
  6. 6. Backup Essential Files 1) Backup spfile to pfile › lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/initltst.ora' from spfile; 2) Backup Oracle Home db_1 and save it on a different host › cd $ORACLE_HOME/.. › /u01/app/oracle/product/10.2.0> tar cvf db_1.tar db_1 › /u01/app/oracle/product/10.2.0> scp db_1.tar prdb01:/dbscratch/preReImange/lsdb01/. 3) Capture admin directories › /u01/app/oracle/admin/ltst> ls adump bdump cdump udump 4) Capture cluster name › grep -i name $ORA_CRS_HOME/install/cluster.ini cluster_name= sdr2ppramercrs 5) Capture LUN Mapping – by Linux System Engineer 6 Database Management Practice
  7. 7. Reimage • Shutdown database • Re-image the host cluster servers to OEL 11g image by Linux System Engineer › Deploy Oracle Enterprise Linux 64-bit OS › Restore EMC LUN mapping › Deploy Oracle 11g CRS RPM › Deploy Oracle 11g Software RPM 7 Database Management Practice
  8. 8. Start 11g ASM 1) Verify that CRS and Nodeapps are all up › crs_stat.sh 2) Create 11g ASM Initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora /u01/app/oracle/product/11.1.0/asm_1/dbs/. › remove *_dump_dest from the file 3) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapw+ASM1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 4) Source ASM environment › export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1 › export PATH=$ORACLE_HOME/bin:/usr/bin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/sbin:/usr/sbin:./ › export LD_LIBRARY_PATH=$ORACLE_HOME/lib › export ORACLE_SID=+ASM1 5) Startup 11g ASM › sqlplus / as sysasm › SQL> startup 8 Database Management Practice
  9. 9. Register ASM to CRS 1. Add ASM to CRS › srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11.1.0/asm_1 › # Repeat to add all 6 nodes 2. Copy Initialization and Password files to the rest of RAC nodes › init+ASM1.ora › orapw+ASM1 3. Start ASM on nodes via srvctl › +ASM1 SQL>shutdown › srvctl start asm –n lsprddb01 #repeat for node 2-6 9 Database Management Practice
  10. 10. Start 10g DB 1) Restore 10g Oracle Home to a temporary directory on node1 only › /dbscratch/> tar xvf db_1.tar 2) Create a symbolic link for the original 10g Oracle Home › mkdir –p /u01/app/oracle/product/10.2.0/ › cd /u01/app/oracle/product/10.2.0/ › ln –s /dbscratch/db_1 db_1 3) Create admin directories › mkdir -p /u01/app/oracle/admin/ltst/ › cd /u01/app/oracle/admin/ltst/ › mkdir adump bdump cdump udump 4) Source 10g environment › . 10gR2.env 5) Startup 10g Database › sqlplus / as sysdba › SQL> startup 10 Database Management Practice
  11. 11. Verify Logical Standby logs 1. From Logical Standby: Startup 10g Listener › lsnrctl start LISTENER_LSPRDDB01 2. From Primary: Query the latest online logs › SQL> select THREAD#, SEQUENCE# from v$LOG where status='CURRENT'; THREAD# SEQUENCE# ---------- ---------- 1 5592 2 4076 3 4089 4 4149 5 4091 6 4092 3. From Primary: Force log archive › alter system archive log current; 4. From Logical Standby: Verify that the latest logs are applied › SQL> select THREAD#, MAX(SEQUENCE#), APPLIED from dba_logstdby_log where applied!='YES'; THREAD# MAX(SEQUENCE#) APPLIED ---------- -------------- -------- 1 5592 YES 2 4076 YES 3 4089 YES 4 4149 YES 5 4091 YES 6 4092 YES 11 Database Management Practice
  12. 12. II. Upgrade Logical Standby Database
  13. 13. Pre-Upgrade 1) Run pre-upgrade information tools › SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql 2) Resolve any outstanding issue 3) Create a guaranteed restore point › SQL> create restore point pre_upgrade guarantee flashback database; › SQL> select * from v$restore_point; 4) Stop 10g listener › .~ 10gR2.env › lsnrctl stop 5) Stop SQL Apply and shutdown 10g database › SQL> alter database stop logical standby apply; › SQL> shutdown 13 Database Management Practice
  14. 14. Start 11g Database 1) Create 11g DB Initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/. › Edit initialization file to – remove *_dump_dest – set cluster_database=false – set dg_broker_start=false 2) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 3) Restore tnsnames.ora › cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora /u01/app/oracle/product/11.1.0/asm_1/network/admin/. 4) Startup 11g database › . .bash_profile › cd $ORACLE_HOME/rdbms/admin › SQL> startup upgrade 14 Database Management Practice
  15. 15. Upgrade Database 1. Upgrade database › SQL> spool /tmp/upgrade.log › SQL> @catupgrd.sql #40 min, db shutdown at then end of upgrade 2. Run post update scripts › ltst1 SQL> startup › ltst1 SQL> @utlu111s.sql › ltst1 SQL> @catuppst.sql › ltst1 SQL> @utlrp.sql 3. Enable cluster database › ltst1 SQL> alter syste set cluster_database=true 4. Restart Instance 1 15 Database Management Practice
  16. 16. Register Database to CRS 1. Add Database and Instances to CRS › srvctl add database -d ltst_ls -o /u01/app/oracle/product/11.1.0/db_1 › srvctl add instance -d ltst_ls -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance 2. Copy following files to the rest of RAC nodes › tnsnames.ora › initltst1.ora › orapwltst1 3. Restart DB › ltst1 SQL> shutdown › srvctl start database –d ltst_ls 16 Database Management Practice
  17. 17. III. Synchronize Archive log (10g->11g) Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Physical Standby 11g lsdb01-06 LTST_LS Logical Standby Data Center 1 Data Center 2 17 Database Management Practice
  18. 18. Start Apply 1. Query outstanding log in Logical Standby › col file_name for a70 › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES' 2. Start log Apply › alter database start logical standby apply immediate; 3. Repeat step 1 until all logs are applied 18 Database Management Practice
  19. 19. IV. Switchover to Logical Standby Redo Apply prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 19 Database Management Practice
  20. 20. Transition Primary -> Logical Standby 1. Verify Switchover status › SQL> select switchover_status from v$database; SWITCHOVER_STATUS -------------------- TO STANDBY If the query returns “SESSIONS ACTIVE’, then ensure it’s ok to shut them down › SQL > SELECT SID, PROCESS, PROGRAM FROM V$SESSION WHERE TYPE = 'USER' AND SID <> (SELECT DISTINCT SID FROM V$MYSTAT); 2. Disable all RAC instances except instance 1 › srvctl stop instance –d ltst –i ltst2; #repeat for lsts2 - 6 › alter database disable thread 2; ; #repeat for thread 2 - 6 3. Switchover › SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO LOGICAL STANDBY;; 20 Database Management Practice
  21. 21. Transition Logical Standby -> Primary 1. Defer Redo log transfer in Logical Standby › SQL> alter system set log_archive_dest_3='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=s drseprddb01- vip.us.dell.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ltst_XPT.prd.amer.dell.com)(INSTAN CE_NAME=ltst1)(SERVER=dedicated)))"',' LGWR ASYNC NOAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=30 db_unique_name="ltst" register net_timeout=15 valid_for=(online_logfile,primary_role)' scope=both › SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER; 2. Disable all RAC instances except instance 1 › srvctl stop instance –d ltst_ls –i ltst2; #repeat for instnace 2-6 › alter database disable thread 2; #repeat for thread 2-6 3. Commit Switchover › SQL> select switchover_status from v$database; SWITCHOVER_STATUS -------------------- TO PRIMARY › SQL> alter database commit to switchover to primary; 21 Database Management Practice
  22. 22. Start Rest of RAC Instances 1. New Primary LTST_LS: Enable threads and start RAC instances 2-6 › alter database enable thread 2 › srvctl start instance –d ltst_ls –i ltst2; 2. New Logical Standby LTST: Enable threads and start RAC instances 2-6 › alter database enable thread 2; › srvctl start instance –d ltst –i ltst2; 22 Database Management Practice
  23. 23. V. Reimage Original Primary Hosts Redo Apply 10g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 23 Database Management Practice
  24. 24. Reimage Original Primary Host • Follow the process used in reimaging logical standby hosts (slice 7 – 10) • Do not sync archived log from 11g –> 10g (skip slice 11) 24 Database Management Practice
  25. 25. VI. Upgrade Original Primary to 11g • Follow the process used in upgrading Logical Standby Database (slice 13– 16) 25 Database Management Practice
  26. 26. VII.Synchronize Archive log (11g->11g) 1. Query outstanding log in Logical Standby › col file_name for a70 › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES' 2. Start log Apply › alter database start logical standby apply immediate; 3. Repeat step 1 until all logs are applied 26 Database Management Practice
  27. 27. VIII.Reimage Physical Standby Hosts Redo Apply 10g 11g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Logical Standby Bystand Physical Standby 11g lsdb01-06 LTST_LS Primary Data Center 1 Data Center 2 27 Database Management Practice
  28. 28. Reimage Physical Standby Hosts • Follow the process used in reimaging Logical Standby Database (slice 7 – 10, skip step 5 in slice 10) • Mount 10g Database as standby › sqlplus / as sysdba › SQL> startup monut › SQL> alter database mount standby database 28 Database Management Practice
  29. 29. Start 11g Physical Standby Database 1) Create 11g DB initialization file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/. › Edit initialization file to remove *_dump_dest 2) Create spfile from pfile 3) Restore password file › cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/. 4) Restore tnsnames.ora › cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora c/u01/app/oracle/product/11.1.0/asm_1/network/admin/. 5) Mount 11g standby database › . .bash_profile › SQL> startup nomount › SQL> alter database mount standby database Note: wait for dg broker configuration to apply log 29 Database Management Practice
  30. 30. Register Physical Standby Database to CRS 1. Add Database and Instances to CRS › srvctl add database -d ltst_dr -o /u01/app/oracle/product/11.1.0/db_1 –s mount –s physical_standby -m dr.amer.dell.com › srvctl add instance -d ltst_dr -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance 2. Copy following files to the rest of RAC nodes › tnsnames.ora › initltst1.ora › orapwltst1 3. Restart DB › ltst1 SQL> shutdown › srvctl start database –d ltst_dr 30 Database Management Practice
  31. 31. IX. Switchover to Original Primary Redo Apply 11g 11g prddb01-06 drdb01-06 LTST SQL Apply LTST_DR Primary Bystand Physical Standby 11g lsdb01-06 LTST_LS Logical Standby Data Center 1 Data Center 2 31 Database Management Practice
  32. 32. Switchover via Data Guard Broker I. Remove old Data Guard Broker configuration › dgmgrl > remove configuration II. Create new Data Guard Broker configuration › dgmgrl > create configuration dg_ltst as primary database is ltst_ls connect identifier is ltst_ls; › dgmgrl > add database ltst as connect identifier is ltst_prd maintained as logical ; III. Switchover to Logical Standby › dgmgrl > switchover to ltst IV. Add Physical Standby to Data Guard Broker 32 Database Management Practice
  33. 33. X. Verify Data Guard Status 1) Verify Database Status › dgmgrl > show configuration 2) Verify Log Applying Status in Physical Standby › Run from primary › @dg_standby_log.sql 3) Verify Log Applying Status in Logical Standby › select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= 'YES' 33 Database Management Practice
  34. 34. Summary I. Reimage Logical Standby Hosts to 11g – Linux System Engineer II. Upgrade Logical Standby DB to 11g –Database Administrator III. Sync Archive log from 10g Primary to 11g Logical Standby - Database Administrator IV. Switchover to Logical Standby Manually –Database Administrator V. Reimage original Primary Hosts to 11g – Linux System Engineer VI. Upgrade the original Primary DB to 11g – Database Administrator VII. Sync Archive log from 11g Primary to 11g Logical Standby - Database Administrator VIII. Reimage Physical Standby Hosts to 11g – Linux System Engineer IX. Switchover to original Primary via DG Broker- Database Administrator X. Verify Data Guard Status 34 Database Management Practice
  35. 35. Reference • Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02 • Oracle® Data Guard Concepts and Administration 11g R1 (11.1) B28294-03 • Oracle Data Guard Broker 11g Release 1 (11.1) B28295) • Database Rolling Upgrade Using Data Guard SQL Apply - Oracle Database 11g and 10gR2 35 Database Management Practice
  36. 36. For more information: • Visit Dell at Oracle OpenWorld in Moscone South booth 521 • Visit www.dell.com/oracle for more information • Dell Services Oracle Practice information at http://content.dell.com/us/en/enterprise/oracle-solutions- services.aspx • Dell presentations at OOW 2010 at http://www.slideshare.net/dellenterprise • Dell | Oracle white papers at http://content.dell.com/us/en/enterprise/oracle- solutions.aspx#facets=whitepaper&p=1 • Dell | Oracle Tested and Validated documentation at http://content.dell.com/us/en/enterprise/d/solutions/product- configurations.aspx 36 Global Marketing
  37. 37. Questions ? 37 Database Management Practice
  38. 38. Versions - Production 38 Database Management Practice
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×