Upgrading to Oracle 11gR2

Do I actually need to care about upgrading my
existing environment to race with
Oracle latest release?

Is it mandatory?

Presented by : Syed Jaffer Hussain red-gate/AllThingsOracle Slide # 1

Does it really matter, which version of Oracle
do we use?


Upgrading to Oracle 11gR2
A Successful case study


We really had a roller coaster journey…..
A successful upgrade
A successful downgrade
De-installed failed GI home before start over
Manual upgrade
Tried out of the box options….


Disclaimer

This views/content in this slides are those of the author and
do not necessarily reflect that of Oracle Corporation and/or its
affiliates/subsidiaries. The material in this document is for
informational purposes only and is published with no
guarantee or warranty, express or implied..

This material should not be reproduced or used without the
authors' written permission.


Foreword

The objective of this presentation is to explain you the best
practices that we had adopted/followed/practiced to upgrade
successfully our FOUR Oracle 10.2.0.4 clusterware
environments with over 100 databases to Oracle 11g R2 on
HPUX Servers.


What will be covered

o Why upgrade?
o Upgrade compatibility matrix.
o Upgrade pattern [best practices].
o Overview about our environment.
o Preparing to upgrade.
o Clusterware and ASM upgrade.
o 11gR2 RDBMS software installation.
o Database upgrade.
o Post upgrade steps.
o Upgrade aftermath stories.
o References.


Know your presenter
Syed Jaffer Hussain
Database Support Manager
Over 20 years IT experience
12 years as an Oracle DBA
Oracle ACE Director
Oracle 10g Certified Master(OCM)
Oracle 10g RAC Certified Expert
OCP v8i,9i,10g & 11g
ITIL v3 Foundation Certified
Authored Oracle 11g R1/R2 Real Application Clusters
Essentials
Twitter: @sjaffarhussain
http://jaffardba.blogspot.com


Know your presenter

Technologist of the Year, DBA 2011
http://www.oracle.com/technetwork/issue-archive/2012/12-
jan/o12awards-tech-1403083.html



o WHY upgrade?
o Overview of our setup.
o Database upgrade.
o References.


Why upgrade? – A burning question

Do we really need to care which
version are we using?

Is it mandatory to upgrade an
existing environment with every
Oracle new release?


Why upgrade?


Why upgrade?

To best of my knowledge the
following situations may
contributes to oppose an
upgrade:

 Legacy application.
 Doesn’t want to take the risk.
 Lack of skills and support.


Why upgrade?

Situations that are likely to
invite you to embrace for an
upgrade:

 Organization standards.
 Policy standards by Oracle for the
old releases.
 Tantalizing new features.
 Bug fixes.



o Why upgrade?
o Upgrade compatibility MATRIX.
o Database upgrade.
o References.


Upgrade compatibility matrix

Courtesy from Oracle Corporation
Thanks to Roy Swonger & Mike Dietrich


Minimum Version of the database that can be directly upgraded to Oracle 11g
Release 2

Source Database Target Database
9.2.0.8 or higher 11.2.x
10.1.0.5 or higher 11.2.x
10.2.0.2 or higher 11.2.x
11.1.0.6 or higher 11.2.x

The following database version will require an indirect upgrade path.

Source Database ---> Upgrade Path for Target Database--->Target Database
7.3.3 (or lower)-----> 7.3.4 ---> 9.2.0.8 ---->11.2.x
8.0.5 (or lower)----> 8.0.6 ---> 9.2.0.8 ---->11.2.x
8.1.7 (or lower)----> 8.1.7.4---> 10.2.0.4---->11.2.x
9.0.1.3 (or lower)----> 9.0.1.4-- ->10.2.0.4---->11.2.x
9.2.0.7(or lower)---->9.2.0.8---->11.2.x



o Why upgrade?
o Upgrade pattern [BEST PRACTICE].
o Database upgrade.
o References.


Upgrade pattern [BEST PRACTICE]

UPGRADE
Clusterware
+ ASM


Upgrade pattern [best practice]

UPGRADE INSTALL
Clusterware RDBMS
+ ASM Software



UPGRADE INSTALL
UPGRADE
Clusterware RDBMS
databases
+ ASM Software



UPGRADE INSTALL
UPGRADE Post
Clusterware RDBMS
databases upgrade
+ ASM Software



o Why upgrade?
o Overview about our environment.
o Database upgrade.
o References.


Overview about our environment
CISCO Switch

HP Superdome NetBackup 6.5.6
Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4


Production 1 Production 2

8 nodes
6 nodes

Non-Production 1 Non-Production 2

7 nodes 3 nodes


Production 1 Production 2

8 nodes
6 nodes
•31 databases •4 logical processors •11 databases •3 logical processors
•51 instances •22GB Memory •20 instances •12GB Memory
Non-Production 1 Non-Production 2

7 nodes 3 nodes

•60 databases •3 logical processors •12 databases •3 logical processors
•80 instances •16GB Memory •20 instances •12GB Memory


The framework that we have followed for the upgrade

V11.31
V11.23 /var/opt
/var/opt

HP UX upgrade from
V11.23 TO v11.31



V11.31
V11.23 /var/opt
/var/opt

HP UX upgrade from EMC Storage
V11.23 TO v11.31 Configuration
update


Overview of our environment

10.2.0.4 11.2.0.2

V11.31
V11.23 /var/opt
/var/opt

HP UX upgrade from EMC Storage Clusterware
V11.23 TO v11.31 Configuration Databases
upgrade upgrade



o Why upgrade?
o Database upgrade.
o References.



Preparation is the headstone to
every successful installation or
upgrade projects.


Prepare for upgrade – prerequisites checks
 Verify:
o OS certification for Oracle 11gR2.
o OS kernel, packages.
o Additional OS groups.
o Oracle base location.
o GIH location.
 Must set GI home different from OB.
 Configure Single Client Access Name(SCAN)
o Maximum of 3 IP’s (same across nodes).
o DNS round robin resolution.
o Must be on the subnet as Public and VIP IP’s.
o Not recommended to configure in HOSTS file.
 root user access
 UNSET the following
o ORACLE_BASE
o ORACLE_HOME
o ORACLE_CRS_HOME
o TNS_ADMIN
o ORA_NLS10


 Space:
o Additional space for Oracle software
home, typically /u00
o Enough /tmp space

 Back up:
o Back up the Cluster and Oracle homes.
o Back up Voting disk.
o Back up OCR.

 CRS version:
o Remember, the crs active and software version
must be the same.
crsctl query crs activeversion
crsctl query crs softwareversion

 Nodes readiness:
runcluvfy.sh stage –pre crsinst –n nodes_list
–fixup –fixupdir <dirname> -verbose



Verify nodes readiness for upgrade:
While upgrading from pre 11gR2, use the following syntax

runcluvfy.sh stage –pre crsinst –n nodes_list –fixup –fixupdir <dirname> -verbose

While upgrading 11.2.0.1 to 11.2.0.2 or higher, use the following syntax

runcluvfy.sh stage –pre crsinst -upgrade –n nodes_list –rolling
-scr_crshome existing_GI_HOME –dest_crshome new_GI_HOME
-dest_version 11.2.0.3.0
–fixup –fixupdirpath <dirname> -verbose



o Why upgrade?
o Database upgrade.
o References.


Clusterware, ASM upgrade – What’s new?
Grid Infrastructure

Clusterware
Automatic Storage
Management (ASM)

Out-of-Place
and
In-place upgrades

New GI home


Clusterware, ASM upgrade

Is it necessary to upgrade
Clusterware AND ASM
together?


Clusterware, ASM upgrade
Clusteware +
ASM =
no rolling
upgrade


Clusterware and ASM upgrade



ORACLE BASE must be
different from the SW
location……


Cause :
False warning generated by unpublished CVU bug 10024549
Solution:
The warning can be ignored



Can I run the
rootupgrade.sh script in
PARALLEL across nodes?



Can I run the
rootupgrade.sh script in
PARALLEL across nodes?

The answer is YES, however….



Always run the script on the LOCAL node first.




Run the script in parallel on all nodes, except the LAST node.




Run the script in parallel on all nodes, except the LAST node.

Run the script on the LAST node.



rootupgrade.sh

Shuts down the existing clusterware stack.

Brings up the new version clusterware stack.

Up on successfully running the script on the LAST node, the CRS active/release version
will set to the new version.



Average rootupgrade.sh execution time on each node was just about 15 min.

Took more time on couple nodes, mostly while stopping the existing cluster stack.

Look closely for the logs under /u00/app/11.2.0/grid/cfgtoollogs.



rootupgrade.sh
execution failed?



How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix
MOS [ID 969254.1]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix
OUI Failure
Grid Infrastructure Cluster rootupgrade.sh Failure
Grid Infrastructure Standalone rootupgrade.sh Failure
OUI Plug-in Failure
Old Clusterware Home Removal After Upgrade
Patching Pre-upgrade Clusterware Home
Scalability RAC Community
References



Use the –force flag to force the upgrade to complete in case some nodes becomes
unreachable whilst performing an upgrade.

/u01/app/11.2.0/grid/rootupgrade –force (>=11.2.0.2 only)



Cried for HELP!


Internet

Cried for HELP!


Internet MOS

Cried for HELP!


Internet MOS

Cried for HELP!

ASM alert.log, trace files & log files

Doesn’t help much!



Life Savior

MOS [ID 398374.1]

Presented by : Syed Jaffer Hussain Slide # 79

cd $ORA_ASM_HOME/log/hostname/racg/ora.<hostname>.ASM.asm.log

SQL> ORACLE instance shut down.
SQL> Disconnected

[PRSR][1]prsr_get_enabled_key: OCR api procr_open_key failed for key DATABASE.ASM.node01.+asm1.ENABLED with 4
[ PRSR][1]OCR error code : 4
[ PRSR][1]OCR error msg : PROC-4: The cluster registry key to be operated on does not exist.
2009-06-23 16:12:26.332: [ RACG][1] [28830][1][ora.node01.ASM1.asm]: prsr_fetch_asm_config() error, rc = 8

2009-06-23 16:12:27.443: [ RACG][1] [28830][1][ora.node01.ASM1.asm]:
SQL*Plus: Release 10.2.0.4.0 - Production on Tue Jun 23 16:12:26 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Enter user-name: Connected to an idle instance.

SQL> ORA-01506: missing or illegal database name
SQL> Disconnected


What might have gone wrong???

 ASM instance name registered as ASMn
instead of +ASMn in the OCR.

 Could be due to a manual entry.

 GI is very particular about + in contrast to
previous cluster version.

 Bug 5644904: WRONG ASM KEY PROC-4:
THE CLUSTER REGISTRY KEY TO BE OPERATED ON
DOES NOT EXIST


Remedy
 Stop ASM and dependent instances.

 Remove dependency on ASM instance.

 Remove ASM from OCR.

 Add ASM back in with ‘+’ correct name

 Add dependency to ASM.

 Startup ASM.

 Resume the failed step.


In action
 srvctl stop asm –n <hostname>

 srvctl modify instance –d DBNAME –i <instance_name> -f
[do it for all active instances on the node]

 srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

 srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>
-o <ORA_ASM_HOME>

 srvctl modify instance –d <DBNAME> -i <instance_name>
-s +ASM_INSTANCE1
 srvctl start asm –n <hostname>


In action
 srvctl stop asm –n <hostname>

 srvctl modify instance –d DBNAME –i <instance_name> -f

 srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

 srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>
-o <ORA_ASM_HOME>

 srvctl modify instance –d <DBNAME> -i <instance_name>
-s +ASM_INSTANCE1
 srvctl start asm –n <hostname>

Repeat on all nodes!!!



Post upgrade checks – active/software version

#crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.2.0]

# crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.2.0]



Post upgrade checks – active/software version

#crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.2.0]

# crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.2.0]

Ensure active/release version reflect to 11.2.0.2
for a successful upgrade.


Post upgrade checks – cluster stats across nodes

# crsctl check cluster -all
**************************************************************
node01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node02:
**************************************************************
node03:
**************************************************************



Post upgrade checks – query voting disk

# ocrcheck

Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1051708
Used space (kbytes) : 27048
Available space (kbytes) : 1024660
ID : 1837247566
Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_001
Device/File integrity check succeeded
Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_002
Device/File integrity check succeeded
Up to FIVE
Device/File not configured OCR files
Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user



Post upgrade checks – query voting disk

# crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 3ee7c55614f24fdcfff130e1abf94384 (/dev/rdsk/oracle/vote/ln1/ora_vote_001) []
2. ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (/dev/rdsk/oracle/vote/ln1/ora_vote_002) []
3. ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (/dev/rdsk/oracle/vote/ln1/ora_vote_003) []


Post upgrade checks – b/g and daemon processes
#ps -ef |grep d.bin
oracle 14458 1 0 15:14:12 ? 0:41 /u00/app/11.2.0/grid/bin/oraagent.bin
oracle 14462 1 0 15:14:13 ? 0:00 /u00/app/11.2.0/grid/bin/mdnsd.bin
root 14539 1 0 15:14:28 ? 0:24 /u00/app/11.2.0/grid/bin/cssdagent
oracle 14549 1 0 15:14:29 ? 0:01 /u00/app/11.2.0/grid/bin/diskmon.bin -d -f
root 14541 1 0 15:14:28 ? 4:17 /u00/app/11.2.0/grid/bin/orarootagent.bin
root 14732 1 0 15:15:48 ? 7:35 /u00/app/11.2.0/grid/bin/crsd.bin reboot
oracle 14469 1 0 15:14:15 ? 0:11 /u00/app/11.2.0/grid/bin/gpnpd.bin
oracle 14739 1 0 15:15:49 ? 3:21 /u00/app/11.2.0/grid/bin/evmd.bin
oracle 14535 1 0 15:14:25 ? 2:51 /u00/app/11.2.0/grid/bin/gipcd.bin
oracle 14545 1 0 15:14:29 ? 4:42 /u00/app/11.2.0/grid/bin/ocssd.bin
root 20675 1 0 15:36:03 ? 4:28 /u00/app/11.2.0/grid/bin/orarootagent.bin
root 14595 1 0 15:14:50 ? 3:06 /u00/app/11.2.0/grid/bin/octssd.bin
root 13999 1 0 15:13:20 ? 2:47 /u00/app/11.2.0/grid/bin/ohasd.bin reboot
root 14533 1 0 15:14:24 ? 0:26 /u00/app/11.2.0/grid/bin/cssdmonitor
oracle 20671 1 0 15:36:03 ? 0:08 /u00/app/11.2.0/grid/bin/appagent.bin
oracle 20685 1 0 15:36:03 ? 0:26 /u00/app/11.2.0/grid/bin/oraagent.bin

#ps -ef |grep init.d

root 66 0 0 Feb 9 ? 0:00 pagetable_init_daemon
oracle 8645 15234 0 19:31:02 pts/0 0:00 grep init.d
root 13977 1 0 15:13:19 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run


Remember

 You can easily resume/downgrade from a failed or successful upgrade.

 Can fall back to the pre 11gR2 version.

 You hardly need a 3-step procedure to do that.


Remove OLD homes
Removing OLD cluster home

It’s a best practice to keep the
OLD_HOME for a while after a successful
Upgrade.
The following procedure can be used to
remove the OLD_HOMES.

export ORACLE_HOME=$OLD_HOME

## detach OLD_HOME
$OLD_HOME/oui/bin/runInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME

#confirm $OLD_HOME is removed from central inventory:
$OLD_HOME/OPatch/opatch lsinventory -all

## remove files in OLD_HOME manually:
/bin/rm -rf $OLD_HOME

unset ORACLE_HOME



o Why upgrade?
o Prepare for upgrade.
o Database upgrade.
o References.


11gR2 RDBMS software installation

Verify the READINESS of the nodes for
DB software installation

cluvfy stage -pre dbinst -n
node01,node02,node03 -d
$ORACLE_HOME –verbose



How to manage the pre
11gR2 databases under
11gR2 GIH?


PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDER
The Grid Infrastructure umbrella.

root@node01: /u00/app/11.2.0/grid/bin #
./crsctl pin css -n node01 node02 node03

CRS-4664: Node node01 successfully pinned.



o Why upgrade?
o Database upgrade.
o Post DB upgrade steps.
o References.


Database upgrade – Caution

Ensure a valid database backup
BEFORE you kick off the db
upgrade procedure.


Database upgrade – upgrade paths

UPGRADE

SQL> @catupgrade.sql

• RAC aware and most recommended for RAC dbs.
• Also can be run in –silent mode.


Database upgrade
For a smooth rise, follow the best practice, run
@OH_11G/rdbms/admin/utlu112i.sql
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 788 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
--> SYSAUX tablespace is adequate for the upgrade.
--> TEMP tablespace is adequate for the upgrade.

PURGE DBA_RECYCLEBIN
.... must be executed immediately prior to executing your upgrade.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:

EXECUTE dbms_stats.gather_dictionary_stats;


Database upgrade

As a best practice, run @OH_11G/rdbms/admin/utlu112i.sql
**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views [upgrade] VALID
--> Oracle Packages and Types [upgrade] VALID
--> JServer JAVA Virtual Machine [upgrade] VALID
--> Oracle XDK for Java [upgrade] VALID
--> Real Application Clusters [upgrade] VALID
--> Oracle Workspace Manager [upgrade] VALID
--> OLAP Analytic Workspace [upgrade] VALID
--> OLAP Catalog [upgrade] VALID
--> EM Repository [upgrade] VALID
--> Oracle Text [upgrade] VALID
--> Oracle XML Database [upgrade] VALID
--> Oracle Java Packages [upgrade] VALID
--> Oracle interMedia [upgrade] VALID
--> Spatial [upgrade] VALID
--> Data Mining [upgrade] VALID
--> Expression Filter [upgrade] VALID
--> Rule Manager [upgrade] VALID
--> Oracle OLAP API [upgrade] VALID


Database upgrade


Database upgrade

Listener will
be created
on all
nodes


Database upgrade


Database upgrade
Note 453903.1 - Enabling and Disabling Oracle Database Vault in UNIX
Note 453902.1 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk dv_off
$ cd $ORACLE_HOME/bin
$ relink all

(For RAC installations, run these commands on all nodes.)

#cd /u00/app/oracle/product/11.2.0/db_1/rdbms/lib
# make -f ins_rdbms.mk dv_off /usr/ccs/bin/ar d
/u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a kzvidv.o
/usr/ccs/bin/ar cr
/u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a
/u00/app/oracle/product/11.2.0/db_1/rdbms/lib/kzvndv.o


Database upgrade



o Why upgrade?
o Database upgrade.
o References.


Database post upgrade – list

 Run @utlrp.sql to compile ANY invalid objects.

 Run @utlu112s.sql to check post upgrade
components status.

 Adjust time zone data in the database.

 Gather statistics for system and fixed objects.

 Amend scripts [if any] to affect the new OH settings.

 Verify the database upgrade
srvctl config database –d <DBNAME>

 Back up the database.

 Modify the COMPATIBLE init. Parameter to 11gR2.


Database upgrade – measuring upgrade


Database upgrade – trivia

 Upgraded over 100 databases successfully using
MANUAL and DBUA methods.

 Avg. database upgrade time was just about 2 hrs.

 Upgrade using DBUA for RAC DB proved a wise
decision.

 Nearly identical duration between CLI and DBUA
upgrade methods.

 Duration is typically proportional to the number of
installed database components.


Database upgrade – CLI upgrade steps




o Why upgrade?
o Database upgrade.
o References.


Upgrade aftermath stories
 Snapshot Controlfile must be placed on a shared storage for a RAC database.

 ORA-24247: network access denied by access control list (ACL)
Solution: UTL_TCP, UTL_HTTP, UTL_SMTP, UTL_MAIL is used
grant the privileges explicitly
DBMS_NETWORK_ACL_ADMIN.CREATE_ACL
ADD_PRIVILEGE
ASSIGN_ACL

 ASM instance resource status MIGHT turn into INTERMEDIATE with
CHECK TIMED OUT state_details [BUG 11807012]
Solution: crsctl modify resource “ora.asm” –attr “CHECK_TIMEOUT=132”
MOS [ID 1323679.1]

 IMPDP fails with ORA-29913, 31011, 19202, LPX-00217
[BUG 11877267]
Solution: no solution… workaround exists

 ORA-07445: exception core dump [evaopn3()]
Solution: _simple_view_merge, _complex_view_merging=false


Very important, apply the
latest PSU patch, right after
your upgrade.



o Why upgrade?
o Database upgrade.
o How to proceed from failed upgrade.
o References.


References

 Oracle Database Upgrade Path Reference List [ID 730365.1]
 Complete Checklist for Manual Upgrades to 11gR2 [ID 837570.1]
 Complete checklist to upgrade the database to 11g R2 using DBUA [ID 870814.1]
 Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 1096952.1]
 How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix [ID 969254.1]
 Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]
 Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]

 Oracle 11g R1/R2 Real Application Clusters Essentials

 Move to Oracle Database 11g – The whole Story blogs.oracle.com/UPGRADE
 http://download.oracle.com/docs/cd/E11882_01/server.112/e17222/toc.htm
 http://www.oracle.com/technetwork/database/upgrade/index.html
 http://oukc.oracle.com/static09/opn/core09/oracle9i_database/95960/060311_95960/index.htm


A big thank to red-gate
and you all
for listening ...
You can write me at sjaffarhussain@gmail.com


Upgrading to Oracle 11gR2

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Upgrading to Oracle 11gR2

Similar to Upgrading to Oracle 11gR2 (20)

Recently uploaded

Recently uploaded (20)

Upgrading to Oracle 11gR2