2. Objectives
• After completing this lesson, you should be able to:
– Manually control the Oracle Clusterware stack
– Change voting disk configuration
– Back up or recover your voting disks
– Manually back up OCR
– Recover OCR
– Replace an OCR mirror
– Repair the OCR configuration
– Change VIP addresses
– Use the CRS framework
– Prevent automatic instance restarts
3. Oracle Clusterware: Overview
– Portable cluster infrastructure that provides HA to
RAC databases and/or other applications:
• Monitors applications’ health
• Restarts applications on failure
• Can fail over applications on node failure
Oracle Clusterware
system files
ORACLE_HOME
CRS HOME
ORACLE_HOME
CRS HOME CRS HOME
Listener
RAC DB Inst
Protected App A
Listener
RAC DB Inst
Protected App B
Node 1 Node 2 Node 3
4. Oracle Clusterware Run-Time View
init
crsd
ocssd
evmd
action
action
action
racgevt
f
callout
evmlogger
oprocd
callout
callout
OCR
Voting
disk racgimon
racgimon
racgimo
n
racgwrap
+
racgmain
6. CRS Resources
– A resource is a CRS-managed application.
– Application profile attributes are stored in OCR:
• Check interval
• Failure policies
– An action script must be able to:
• Start the application
• Stop the application
• Check the application
– Life cycle of a resource:
crs_profile crs_register crs_start crs_stat
crs_relocate crs_stop crs_unregister
– Action script
– Privileges
– Dependencies
– …
8. Resource Attributes: Example
$ <CRS HOME>/bin/crs_stat -p ora.JFDB.JFDB1.inst
NAME=ora.JFDB.JFDB1.inst
TYPE=application
ACTION_SCRIPT=/u01/app/oracle/product/10g/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for Instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=atlhp8
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.atlhp8.ASM1.asm
RESTART_ATTEMPTS=5
…
$ <CRS HOME>/bin/crs_stat –t ora.xwkE.xwkE1.inst
Name Type Target State Host
-----------------------------------------------------
ora....E1.inst application ONLINE ONLINE atlhp8
9. Main Voting Disk Function
Node1 Node2
CSS CSS
Voting disk
Node3
CSS
Node1 Node2
CSS CSS
Voting disk
Node3
CSS
Nodes can
see each other.
Node3 can no longer
communicate through
private interconnect.
Others no longer see
its heartbeats and
evict that node by
using the voting disk.
I do not see 3
I see 1&2
2 see 1&2
=>
We should
evict 3!
1&I see 1&2
I see 3
I’ve been
evicted!
I’d better stop
We all see
1&2&3
We all see
1&2&3
We all see 1&2&3
Split-brain
10. Important CSS Parameters
– MISSCOUNT:
• Represents network heartbeat timeouts
• Determines disk I/O timeouts during reconfiguration
• Defaults to 30 seconds
• Should not be changed
– DISKTIMEOUT:
• Represents disk I/O timeouts outside reconfiguration
• Defaults to 200 seconds
• Can be temporarily changed when experiencing very long I/O latencies
to voting disks:
1. Shut down Oracle Clusterware on all nodes but one.
2. As root on available node, use: crsctl set css disktimeout M+1
3. Reboot available node.
4. Restart all other nodes.
– Can be changed ONLY under explicit guidance from Oracle Support
11. Multiplexing Voting Disks
– Voting disk is a vital resource for your cluster
availability.
– Use one voting disk if it is stored on a reliable disk.
– Otherwise, use multiplexed voting disks:
• There is no need to rely on multipathing solutions.
• Multiplexed copies should be stored on independent
devices.
• Make sure that there is no I/O starvation for your voting
disks devices.
• Use at least three multiplexed copies.
– CSS uses a simple majority rule to decide whether
voting disk reads are consistent:
v = f*2+1
12. Change Voting Disk Configuration
– Voting disk configuration can be changed dynamically.
– To add a new voting disk:
– To remove a voting disk:
– If Oracle Clusterware is down on all nodes, use the
–force option:
# crsctl add css votedisk <new voting disk path>
# crsctl delete css votedisk <old voting disk path>
# crsctl delete css votedisk <old voting disk path> -force
# crsctl add css votedisk <new voting disk path> -force
13. Back Up and Recover Your Voting Disks
– Should not be needed. Instead, you should add/remove.
– Recommendation is to use symbolic links.
– Back up one voting disk by using the dd command.
• After Oracle Clusterware installation
• After node addition or deletion
• Cannot be done online
– Recover voting disks by restoring the first one using the
dd command, and then multiplex it if necessary.
– If no voting disk backup is available, reinstall Oracle
Clusterware.
$ crsctl query css votedisk
$ dd if=<voting disk path> of=<backup path> bs=4k
17. Automatic OCR Backups
– The OCR content is critical to Oracle Clusterware.
– OCR is automatically backed up physically:
• Every four hours: CRS keeps the last three copies.
• At the end of every day: CRS keeps the last two copies.
• At the end of every week: CRS keeps the last two copies.
– Change the default automatic backup location:
$ cd $ORACLE_BASE/Crs/cdata/jfv_clus
$ ls -lt
-rw-r--r-- 1 root root 4784128 Jan 9 02:54 backup00.ocr
-rw-r--r-- 1 root root 4784128 Jan 9 02:54 day_.ocr
-rw-r--r-- 1 root root 4784128 Jan 8 22:54 backup01.ocr
-rw-r--r-- 1 root root 4784128 Jan 8 18:54 backup02.ocr
-rw-r--r-- 1 root root 4784128 Jan 8 02:54 day.ocr
-rw-r--r-- 1 root root 4784128 Jan 6 02:54 week_.ocr
-rw-r--r-- 1 root root 4005888 Dec 30 14:54 week.ocr
# ocrconfig –backuploc /shared/bak
18. Back Up OCR Manually
– Daily backups of your automatic OCR backups to a
different storage device:
• Use your favorite backup tool.
– On demand physical backups:
– Logical backups of your OCR before and after
making significant changes:
– Make sure that you restore OCR backups that match
your current system configuration.
# ocrconfig –export file name
# ocrconfig –manualbackup
19. Recover OCR Using Physical Backups
1. Locate a physical backup:
2. Review its contents:
3. Stop Oracle Clusterware
on all nodes:
4. Restore the physical OCR backup:
5. Restart Oracle Clusterware
on all nodes:
6. Check OCR integrity:
# ocrconfig –restore <CRS HOME>/cdata/jfv_clus/day.ocr
$ ocrconfig –showbackup
# ocrdump –backupfile file_name
# crsctl stop crs
# crsctl start crs
$ cluvfy comp ocr -n all
20. Recover OCR Using Logical Backups
1. Locate a logical backup created using an OCR export.
2. Stop Oracle Clusterware on all nodes:
3. Restore the logical OCR backup:
4. Restart Oracle Clusterware on all nodes:
5. Check OCR integrity:
# ocrconfig –import /shared/export/ocrback.dmp
# crsctl stop crs
# crsctl start crs
$ cluvfy comp ocr -n all
21. Replace an OCR Mirror: Example
# ocrcheck
Status of Oracle Cluster Registry is as follows:
Version : 2
Total space (kbytes) : 200692
Used space (kbytes) : 3752
Available space (kbytes) : 196940
ID : 495185602
Device/File Name : /oradata/OCR1
Device/File integrity check succeeded
Device/File Name : /oradata/OCR2
Device/File needs to be synchronized with the other device
# ocrconfig –replace ocrmirror /oradata/OCR2
22. Repair OCR Configuration: Example
1. Stop Oracle Clusterware on Node2:
2. Add OCR mirror from Node1:
3. Repair OCR mirror location on Node2:
4. Start Oracle Clusterware on Node2:
# crsctl stop crs
# ocrconfig –replace ocrmirror /OCRMirror
# ocrconfig –repair ocrmirror /OCRMirror
# crsctl start crs
23. OCR Considerations
– If using raw devices to store OCR files, make sure they
exist before add or replace operations.
– You must be the root user to be able to add, replace,
or remove an OCR file while using ocrconfig.
– While adding or replacing an OCR file, its mirror needs
to be online.
– If you remove a primary OCR file, the mirror OCR file
becomes primary.
– Never remove the last remaining OCR file.
24. Change VIP Addresses
1. Determine the interface used to support your VIP:
2. Stop all resources depending on the VIP:
3. Verify that the VIP is no longer running:
4. Change IP in /etc/hosts and DNS.
$ ifconfig -a
$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1
$ ifconfig -a $ crs_stat
+
[ ]
25. Change VIP Addresses
5. Modify your VIP address using srvctl:
6. Start nodeapps and all resources depending on it:
7. Repeat from step 1 for the next node.
# srvctl modify nodeapps -n node1 -A
192.168.2.125/255.255.255.0/eth0
# srvctl start nodeapps -n node1
26. Change Public/Interconnect IP Subnet
Configuration: Example
• Use oifcfg to add or delete network
interface information in OCR:
$ <CRS HOME>/bin/oifcfg getif
eth0 139.2.156.0 global public
eth1 192.168.0.0 global cluster_interconnect
$ oifcfg delif -global eth0
$ oifcfg setif –global eth0/139.2.166.0:public
$ oifcfg delif –global eth1
$ oifcfg setif –global eth1/192.168.1.0:cluster_interconnect
$ oifcfg getif
eth0 139.2.166.0 global public
eth1 192.168.1.0 global cluster_interconnect
1
2
3
4
27. Third-Party Application Protection:
Overview
– High Availability framework:
• Command-line tools to register applications with CRS
• Calls control application agents to manage applications
• OCR used to describe CRS attributes for the applications
– High Availability C API:
• Modify directly CRS attributes in OCR
• Modify CRS attributes on the fly
– Application VIPs:
• Used for applications accessed by network means
• NIC redundancy
• NIC failover
– OCFS:
• Store application configuration files
• Share files between cluster nodes
28. Application VIP and RAC VIP
Differences
– RAC VIP is mainly used in case of node down
events:
• VIP is failed over to a surviving node.
• From there it returns NAK to clients forcing them to
reconnect.
• There is no need to fail over resources associated to the
VIP.
– Application VIP is mainly used in case of application
down events:
• VIP is failed over to another node together with the
application(s).
• From there, clients can still connect through the VIP.
• Although not recommended, one VIP can serve many
applications.
29. Use CRS Framework: Overview
1. Create an application VIP, if necessary:
a. Create a profile: Network data + usrvip predefined script
b. Register the application VIP.
c. Set user permissions on the application VIP.
d. Start the application VIP by using crs_start.
2. Write an application action script that accepts three
parameters:
• start: Script should start the application.
• check: Script should confirm that the application is up.
• stop: Script should stop the application.
30. Use CRS Framework: Overview
3. Create an application profile:
• Action script location
• Check interval
• Failover policies
• Application VIP, if necessary
4. Set permissions on your application.
5. Register the profile with Oracle Clusterware.
6. Start your application by using crs_start.
34. Summary
• In this lesson, you should have learned how to:
– Manually control the Oracle Clusterware stack
– Change voting disk configuration
– Backup and recover your voting disks
– Manually back up OCR
– Recover OCR
– Replace an OCR mirror
– Repair the OCR configuration
– Change VIP addresses
– Use the CRS framework
– Prevent automatic instance restarts
35. Practice 9: Overview
• This practice covers the following topics:
– Mirroring the OCR
– Backing up and restoring OCR
– Multiplexing the voting disk
– Using Oracle Clusterware to protect the xclock
application
