Session : Troubleshooting Oracle 11g Real Application Clusters 101: Insider Tips and Tricks Ben Prusinski Ben Prusinski and Associates http://www.ben-oracle.com [email_address] CLOUG/ Santiago, Chile Tuesday 14 April 2009
oprocd: Runs on Unix when vendor Clusterware is not running. On Linux, only starting with 188.8.131.52.
oclsvmon.bin: Usually runs when a third party clusterware is used oclsomon.bin: Checks program of the ocssd.bin (starting in 184.108.40.206) diskmon.bin : new 220.127.116.11 process for Oracle Exadata Machine oclskd.bin: new 18.104.22.168 process to reboot nodes in case RDBMS instances for 11g RAC are in a hang condition
There are three fatal processes, i.e. processes whose abnormal halt or kill will provokes a node reboot (Metalink Note:265769.1) 1. the ocssd.bin 2. the oprocd.bin 3. the oclsomon.bin
The other processes are automatically restarted when they go away.
Quick Review- 11g RAC Concepts OCR and Vote Disk
What is the OCR?
Oracle Cluster Registry purpose is to hold cluster and database configuration information for RAC and Cluster Ready Services (CRS) such as the cluster node list, and cluster database instance to node mapping, and CRS application resource profiles.
The OCR must be stored on either shared raw devices or OCFS/OCFS2 (Oracle Cluster Filesystem)
What is the Voting Disk?
The Voting disk manages cluster node membership and must be stored on either shared raw disk or OCFS/OCFS2 cluster filesystem.
Common problem unable to start Clusterware resources
The command for crs_stat -t output shows VIP is offline and trying to start it gives error : CRS-0215: Could not start resource 'ora.dbtest2.vip'. Example: crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....st2.gsd application ONLINE ONLINE rac01 ora....st2.ons application ONLINE ONLINE rac01 ora....st2.vip application ONLINE OFFLINE
Metalink Note 265769.1 good reference point for node reboot issues and provides useful Decision Tree for these issues with RAC.
If there is a ocssd.bin problem/failure, the oprocd daemon detected a scheduling problem, or some other fatal problem, a node will reboot in a RAC cluster. This functionality is used for I/O fencing to ensure that writes from I/O capable clients can be cleared avoiding potential corruption scenarios in the event of a network split, node hang, or some other fatal event.
When ocssd.bin process dies it notifies the oprocd process to shoot the node in the head and cause the to node reboot (STONITH).
OCSSD (aka CSS daemon) - This process is spawned in init.cssd. It runs in both vendor clusterware and non-vendor clusterware environments and is armed with a node kill via the init script. OCSSD's primary job is internode health monitoring and RDBMS instance endpoint discovery. It runs as the Oracle user.
INIT.CSSD - In a normal environment, init spawns init.cssd, which in turn spawns OCSSD as a child. If ocssd dies or is killed, the node kill functionality of the init script will kill the node.
OPROCD - This process is spawned in any non-vendor clusterware environment, except on Windows where Oracle uses a kernel driver to perform the same actions and Linux prior to version 10.2.0.4. If oprocd detects problems, it will kill a node via C code. It is spawned in init.cssd and runs as root. This daemon is used to detect hardware and driver freezes on the machine. If a machine were frozen for long enough that the other nodes evicted it from the cluster, it needs to kill itself to prevent any IO from getting reissued to the disk after the rest of the cluster has remastered locks."
OCLSOMON (10.2.0.2 and above) - This process monitors the CSS daemon for hangs or scheduling issues and can reboot a node if there is a perceived hang.
Data collection is vital
OSWatcher tool Metalink Note 301137.1 and 433472.1 have the details on how to setup this diagnosis tool for Linux/UNIX and Windows
Network failure or latency between nodes. It would take at least 30 consecutive missed checkins to cause a reboot, where heartbeats are issued once per second. Example of missed checkins in the CSS log: WARNING: clssnmPollingThread: node <node> (1) at 50% heartbeat fatal, eviction in 29.100 seconds
Review messages file to determine root cause for OCSSD failures.
If the messages file reboot time < missed checkin time then the node eviction was likely not due to these missed checkins. If the messages file reboot time > missed checkin time then the node eviction was likely a result of the missed checkins.
Problems writing to or reading from the CSS voting disk. Check CSS logs: ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds)
High load averages due to lack of CPU resources.
Misconfiguration of CRS. Possible misconfigurations: