11g r1 rac administration and maintainance tasks utilities crsctl srvctl ocrconfig changing public i ps and vip

1. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 1/11
11g R1 RAC Administration and Maintenance Tasks and Utilities
By Bhavin Hingu
bhavin@oracledba.org
<<HOME>>
 Checking CRS Status
 Viewing Name Of the Cluster
 Viewing Nodes Configuration
 Checking Votedisk Information
 Checking OCR Disk information
 Timeout Settings in Cluster
 ADD/Remove OCR files
 ADD/Remove Votedisk
 Backing Up OCR
 Backing Up Votedisk
 Restoring OCR Devices
 Restoring Voting Disk Devices
 Changing Public IPs as well as Virtual IPs
Checking CRS Status:
The below two commands are generally used to check the status of CRS. The first command lists the
status of CRS on the local node where as the other command shows the CRS status across all the nodes in
Cluster.
crsctl check crs <<­­ for the local node
crsctl check cluster <<­­ for remote nodes in the cluster
[root@node1­pub ~]# crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
[root@node1­pub ~]#
For the below command to run, CSS needs to be running on the local node. The "ONLINE" status for
remote node says that CSS is running on that node. When CSS is down on the remote node, the status of
"OFFLINE" is displayed for that node.
[root@node1­pub ~]# crsctl check cluster
node1­pub ONLINE
node2­pub ONLINE
Viewing Cluster name:
I use below command to get the name of Cluster. The similar information can be retrieved from the
dump file.

2. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 2/11
ocrdump ­stdout ­keyname SYSTEM | grep ­A 1 clustername | grep ORATEXT | awk
'{print $3}'
OR
ocrconfig ­export /tmp/ocr_exp.dat ­s online
for i in `strings /tmp/ocr_exp.dat | grep ­A 1 clustername` ; do if [ $i !=
'SYSTEM.css.clustername' ]; then echo $i; fi; done
OR
Oracle creates a directory with the same name as Cluster under the $ORA_CRS_HOME/cdata.
No. Of Nodes configured in Cluster:
The below command can be used to find out the number of nodes registered into the cluster. It
also displays the node's Public name, Private name and Virtual name along with their numbers.
olsnodes ­n ­p ­i
[root@node1­pub ~]# olsnodes ­n ­p ­i
node1­pub 1 node1­prv node1­vip
node2­pub 2 node2­prv node2­vip
Viewing Votedisk Information:
The below command is used to view the no. of Voting disks configured in the Cluster.
crsctl query css votedisk
Viewing OCR Information:
The ocrcheck command displays the no. of OCR files configured in the Cluster. It is primarily
used to chck the integrity of the OCR files. It also displays the version of OCR as well as storage
space information. You can only have 2 OCR files at max.
[root@node1­pub ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 3848
Available space (kbytes) : 258272
ID : 744414276
Device/File Name : /u02/ocfs2/ocr/OCRfile_0
Device/File integrity check succeeded
Device/File Name : /u02/ocfs2/ocr/OCRfile_1
Device/File integrity check succeeded
Cluster registry integrity check succeeded
Various Timeout Settings in Cluster:
Disktimeout: Disk Latencies in seconds from node‐to‐Votedisk. Default Value is 200. (Disk IO)

3. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 3/11
Misscount: Network Latencies in second from node‐to‐node (Interconnect). Default Value is 60 Sec
(Linux) and 30 Sec in Unix platform. (Network IO) Misscount < Disktimeout
IF
(Disk IO Time > Disktimeout) OR (Network IO time > Misscount)
THEN
REBOOT NODE
ELSE
DO NOT REBOOT
END IF;
crsctl get css disktimeout
crsctl get css misscount
crsctl get css reboottime
[root@node1­pub ~]# crsctl get css disktimeout
200
[root@node1­pub ~]# crsctl get css misscount
Configuration parameter misscount is not defined.
The above message indicates that the Misscount is not set manually and it is set to its default Value
which is 60 seconds on Linux. It can be changed as below.
[root@node1­pub ~]# crsctl set css misscount 100
Configuration parameter misscount is now set to 100.
[root@node1­pub ~]# crsctl get css misscount
100
The below command sets the value of misscount back to its default value.
crsctl unset css misscount
[root@node1­pub ~]# crsctl unset css misscount
[root@node1­pub ~]# crsctl get css reboottime
Add/Remove OCR file in Cluster:
Removing OCR File
(1) Get the Existing OCR file information by running ocrcheck utility.
[root@node1­pub ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 3852
Available space (kbytes) : 258268
ID : 744414276
Device/File Name : /u02/ocfs2/ocr/OCRfile_0 <­­ OCR
Device/File integrity check
succeeded
Device/File Name : /u02/ocfs2/ocr/OCRfile_1 <­­ OCR
Mirror
Device/File integrity check
succeeded

4. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 4/11
Cluster registry integrity check succeeded
(2) The First command removes the OCR mirror (/u02/ocfs2/ocr/OCRfile_1). If you want to remove the
OCR file (/u02/ocfs2/ocr/OCRfile_1) run the next command.
ocrconfig ­replace ocrmirror
ocrconfig ­replace ocr
[root@node1­pub ~]# ocrconfig ­replace ocrmirror
[root@node1­pub ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 3852
Available space (kbytes) : 258268
ID : 744414276
Device/File Name : /u02/ocfs2/ocr/OCRfile_0 <<­­ OCR
File
Device/File integrity check
succeeded
Device/File not configured <­­ OCR
Mirror not existed any more
Cluster registry integrity check succeeded
Adding OCR
You need to add OCR or OCR mirror file in a case where you want to move the existing OCR file location
to the different devices. The below command add the OCR mirror file if OCR file already exists.
(1) Get the Current status of OCR:
[root@node1­pub ~]# ocrconfig ­replace ocrmirror
[root@node1­pub ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 3852
Available space (kbytes) : 258268
ID : 744414276
Device/File Name : /u02/ocfs2/ocr/OCRfile_0 <<­­ OCR
File
Device/File integrity check
succeeded
Device/File not configured <­­ OCR
Mirror does not exist
Cluster registry integrity check succeeded
As it can be seen, there is only one OCR file but not the second file (OCR Mirror). Below command adds
the second OCR file.
ocrconfig ­replace ocrmirror <File name>

5. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 5/11
[root@node1­pub ~]# ocrconfig ­replace ocrmirror
/u02/ocfs2/ocr/OCRfile_1
[root@node1­pub ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 3852
Available space (kbytes) : 258268
ID : 744414276
Device/File Name : /u02/ocfs2/ocr/OCRfile_0
Device/File integrity check
succeeded
Device/File Name : /u02/ocfs2/ocr/OCRfile_1
Device/File integrity check
succeeded
Cluster registry integrity check succeeded
You can have at most 2 OCR devices (OCR itself and its single Mirror) in a cluster. Adding extra Mirror
gives you below error message
[root@node1­pub ~]# ocrconfig ­replace ocrmirror /u02/ocfs2/ocr/OCRfile_2
PROT­21: Invalid parameter
[root@node1­pub ~]#
Add/Remove Votedisk file in Cluster:
Add/Remove Voting Disk in Cluster:
Adding Votedisk:
Get the existing Vote Disks associated into the cluster. To be safe, Bring crs cluster stack down on all
the nodes but one on which you are going to add votedisk from.
(1) Stop CRS on all the nodes in cluster but one.
[root@node2­pub ~]# crsctl stop crs
(2) Get the list of Existing Vote Disks
crsctl query css votedisk
[root@node1­pub ~]# crsctl query css votedisk
0. 0 /u02/ocfs2/vote/VDFile_0
1. 0 /u02/ocfs2/vote/VDFile_1
2. 0 /u02/ocfs2/vote/VDFile_2
Located 3 voting disk(s).
(3) Backup the Votedisk file
Backup the existing votedisks as below as oracle:
dd if=/u02/ocfs2/vote/VDFile_0 of=$ORACLE_BASE/bkp/vd/VDFile_0

6. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 6/11
[root@node1­pub ~]# su ­ oracle
[oracle@node1­pub ~]$ dd if=/u02/ocfs2/vote/VDFile_0
of=$ORACLE_BASE/bkp/vd/VDFile_0
41024+0 records in
41024+0 records out
[oracle@node1­pub ~]$
(4) Add an Extra Votedisk into the Cluster:
If it is a OCFS, then touch the file as oracle. On raw devices, initialize the raw devices using "dd"
command
touch /u02/ocfs2/vote/VDFile_3 <<­­ as oracle
crsctl add css votedisk /u02/ocfs2/vote/VDFile_3 <<­­ as oracle
crsctl query css votedisks
[root@node1­pub ~]# su ­ oracle
[oracle@node1­pub ~]$ touch /u02/ocfs2/vote/VDFile_3
[oracle@node1­pub ~]$ crsctl add css votedisk /u02/ocfs2/vote/VDFile_3
Now formatting voting disk: /u02/ocfs2/vote/VDFile_3.
Successful addition of voting disk /u02/ocfs2/vote/VDFile_3.
(5) Confirm that the file has been added successfully:
[root@node1­pub ~]# ls ­l /u02/ocfs2/vote/VDFile_3
­rw­r­­­­­ 1 oracle oinstall 21004288 Oct 6 16:31
/u02/ocfs2/vote/VDFile_3
[root@node1­pub ~]# crsctl query css votedisks
Unknown parameter: votedisks
[root@node1­pub ~]# crsctl query css votedisk
0. 0 /u02/ocfs2/vote/VDFile_0
1. 0 /u02/ocfs2/vote/VDFile_1
2. 0 /u02/ocfs2/vote/VDFile_2
3. 0 /u02/ocfs2/vote/VDFile_3
Located 4 voting disk(s).
Removing Votedisk:
Removing Votedisk from the cluster is very simple. The below command removes the given
votedisk from cluster configuration.
crsctl delete css votedisk /u02/ocfs2/vote/VDFile_3
[root@node1­pub ~]# crsctl delete css votedisk /u02/ocfs2/vote/VDFile_3
Successful deletion of voting disk /u02/ocfs2/vote/VDFile_3.
[root@node1­pub ~]#
[root@node1­pub ~]# crsctl query css votedisk
0. 0 /u02/ocfs2/vote/VDFile_0
1. 0 /u02/ocfs2/vote/VDFile_1
2. 0 /u02/ocfs2/vote/VDFile_2
Located 3 voting disk(s).
[root@node1­pub ~]#
Backing up OCR:

7. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 7/11
Oracle performs physical backup of OCR devices every 4 hours under the default backup directory
$ORA_CRS_HOME/cdata/<CLUSTER_NAME> and then it rolls that forward to Daily, weekly and monthly
backup. You can get the backup information by executing below command.
ocrconfig ­showbackup
[root@node1­pub ~]# ocrconfig ­showbackup
node2­pub 2007/09/03 17:46:47 /u01/app/crs/cdata/test­crs/backup00.ocr
node2­pub 2007/09/03 13:46:45 /u01/app/crs/cdata/test­crs/backup01.ocr
node2­pub 2007/09/03 09:46:44 /u01/app/crs/cdata/test­crs/backup02.ocr
node2­pub 2007/09/03 01:46:39 /u01/app/crs/cdata/test­crs/day.ocr
node2­pub 2007/09/03 01:46:39 /u01/app/crs/cdata/test­crs/week.ocr
[root@node1­pub ~]#
Manually backing up the OCR
ocrconfig ­manualbackup <<‐‐Physical Backup of OCR
The above command backs up OCR under the default Backup directory. You can export the contents of
the OCR using below command (Logical backup).
ocrconfig ­export /tmp/ocr_exp.dat ­s online <<‐‐ Logical Backup of OCR
Restoring OCR:
The below command is used to restore the OCR from the physical backup. Shutdown CRS on all nodes.
ocrconfig ­restore <file name>
Locate the available Backups
[root@node1­pub ~]# ocrconfig ­showbackup
node2­pub 2007/09/03 17:46:47 /u01/app/crs/cdata/test­crs/backup00.ocr
node2­pub 2007/09/03 13:46:45 /u01/app/crs/cdata/test­crs/backup01.ocr
node2­pub 2007/09/03 09:46:44 /u01/app/crs/cdata/test­crs/backup02.ocr
node2­pub 2007/09/03 01:46:39 /u01/app/crs/cdata/test­crs/day.ocr
node2­pub 2007/09/03 01:46:39 /u01/app/crs/cdata/test­crs/week.ocr
node1­pub 2007/10/07 13:50:41 /u01/app/crs/cdata/test­
crs/backup_20071007_135041.ocr
Perform Restore from previous Backup
[root@node2­pub ~]# ocrconfig ­restore /u01/app/crs/cdata/test­crs/week.ocr
The logical backup of OCR (taken using export option) can be imported using the below command.
ocrconfig ­import /tmp/ocr_exp.dat
Restoring Votedisks:
 Shutdown CRS on all the nodes in Cluster.

8. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 8/11
 Locate the current location of the Votedisks
 Restore each of the votedisks using "dd" command from the previous good backup of Votedisk
taken using the same "dd" command.
 Start CRS on all the nodes.
crsctl stop crs
crsctl query css votedisk
dd if=<backup of Votedisk> of=<Votedisk file> <<­­ do this for all
the votedisks
crsctl start crs
Changing Public and Virtual IP Address:
Current Config Changed
to
Node 1:
Public IP: 216.160.37.154
192.168.10.11
VIP: 216.160.37.153
192.168.10.111
subnet: 216.160.37.159
192.168.10.0
Netmask: 255.255.255.248
255.255.255.0
Interface used: eth0 eth0
Hostname: node1­pub.hingu.net node1­
pub.hingu.net
Node 2:
Public IP: 216.160.37.156
192.168.10.22
VIP: 216.160.37.157
192.168.10.222
subnet: 216.160.37.159
192.168.10.0
Netmask: 255.255.255.248
255.255.255.0
Interface used: eth0 eth0
Hostname: node1­pub.hingu.net node2­
pub.hingu.net
(A) Take the Services, Database, ASM Instances and nodeapps down on both the Nodes in Cluster. Also
disable the nodeapps, asm and database instances to prevent them from restarting in case if this
node gets rebooted during this process.

9. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 9/11
srvctl stop service ­d test
srvctl stop database ­d test
srvctl stop asm ­n node1­pub
srvctl stop asm ­n node2­pub
srvctl stop nodeapps ­n node1­pub,node1­pub2
srvctl disable instance ­d test ­i test1,test2
srvctl disable asm ­n node1­pub
srvctl disable asm ­n node2­pub
srvctl disable nodeapps ­n node1­pub
srvctl disable nodeapps ­n node2­pub
(B) Modify the /etc/hosts and/or DNS, ifcfg‐eth0 (local node) with the new IP values on All the Nodes
(C) Restart the specific network interface in order to use the new IP.
ifconfig eth0 down
ifconfig eth0 up
Or, you can restart the network. CAUTION: on NAS, restarting entire network may cause the node to be
rebooted.
(D) Update the OCR with the New Public IP information.
In case of public IP, you have to delete the interface first and then add it back with the new IP
address. As oracle user, Issue the below command:
oifcfg delif ­global eth0
oifcfg setif ­global eth0/192.168.10.0:public
(E) Update the OCR with the New Virtual IP.
Virtual IP is part of the nodeapps and so you can modify the nodeapps to update the Virtual IP
information. As privileged user (root), Issue the below commands:
srvctl modify nodeapps ­n node1­pub ­A
192.168.10.111/255.255.255.0/eth0 <‐‐ for Node 1
srvctl modify nodeapps ­n node1­pub ­A
192.168.10.222/255.255.255.0/eth0 <‐‐ for Node 2
(F) Enable the nodeapps, ASM, database Instances for all the Nodes.
srvctl enable instance ­d test ­i test1,test2
srvctl enable asm ­n node1­pub
srvctl enable asm ­n node2­pub
srvctl enable nodeapps ­n node1­pub
srvctl enable nodeapps ­n node2­pub
(G) Update the listener.ora file on each nodes with the correct IP addresses in case if it uses the IP
address instead of the hostname.
(H) Restart the Nodeapps, ASM and Database instance

10. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 10/11
Name
srvctl start nodeapps ­n node1­pub
srvctl start nodeapps ­n node2­pub
srvctl start asm ­n node1­pub
srvctl start asm ­n node2­pub
srvctl start database ­d test
<<HOME>>
Comments
Enter your comment here
Comment
(Sept 11, 2014) RICHARD SANCHEZ said:
good info ....
2
(Feb 21, 2014) Anonymous said:
very very good info
1
(Sept 4, 2013) KG said:
Nice, very nice.
1
(July 15, 2013) Anonymous said:
Very Useful !!!
1
(Apr 30, 2013) JPJ said:
Good Info ,,, olsnodes ­c ­­> give cluster name
2
(Feb 3, 2013) Anonymous said:
very good
2
(May 16, 2012) Kish said: 2

11. 10/2/2015 11g R1 RAC Administration and Maintainance Tasks Utilities CRSCTL SRVCTL OCRCONFIG Changing Public IPs and VIP
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html 11/11
(Showing 1 to 10)
Excellent Bro
(May 7, 2012) Gurvinder Singh said:
Super
3
(Apr 16, 2012) Anonymous said:
great great...!
2
(Mar 29, 2012) pnair said:
wonderfull!!! thx...
4