Cloug Troubleshooting Oracle 11g Rac 101 Tips And Tricks


Published on

Oracle 11g RAC (Real Application Clusters) presentation that I gave at CLOUG in Chile this year.

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Cloug Troubleshooting Oracle 11g Rac 101 Tips And Tricks

  1. 1. Session : Troubleshooting Oracle 11g Real Application Clusters 101: Insider Tips and Tricks Ben Prusinski Ben Prusinski and Associates [email_address] CLOUG/ Santiago, Chile Tuesday 14 April 2009
  2. 2. Speaker Qualifications Ben Prusinski <ul><li>Oracle ACE and Oracle Certified Professional with 14 plus years of real world experience with Oracle since version 7.3.4 </li></ul><ul><li>Oracle Author of two books on Oracle database technology </li></ul>
  3. 3. Agenda
  4. 4. Agenda: Troubleshooting Oracle 11g RAC <ul><li>Proactive checks to keep Oracle 11g RAC happy and healthy </li></ul><ul><li>Common RAC problems and solutions </li></ul><ul><li>Root cause analysis for RAC </li></ul><ul><li>Understanding Clusterware problems </li></ul><ul><li>Solving critical tuning issues for RAC </li></ul><ul><li>DBA 101 Toolkit for RAC problem solving </li></ul>
  5. 5. Checks and Balances for 11g RAC
  6. 6. Proactive checks to keep Oracle 11g RAC happy and healthy <ul><li>Setup monitoring system to automate checks before major problems occur! </li></ul><ul><li>Verify status for RAC processes and Clusterware </li></ul><ul><li>Check for issues with ASM </li></ul><ul><li>Check status for hardware, network, OS </li></ul>
  7. 7. Monitoring Systems for 11g RAC <ul><li>Oracle Grid Control provides monitoring alerts for Oracle 11g RAC </li></ul><ul><li>System level OS scripts to monitor Clusterware and Oracle 11g RAC processes </li></ul><ul><li>Check for 11g ASM processes and 11g RAC database processes </li></ul>
  8. 8. Verification 11g RAC Processes <ul><li>First, check operating system level that all 11g RAC processes up and running for Clusterware: </li></ul><ul><li>Oracle Metalink Note # 761259.1 How to Check the Clusterware Processes </li></ul><ul><li>[oracle@sdrac01 11.1.0]$ ps -ef|grep crsd </li></ul><ul><li>root 2853 1 0 Apr04 ? 00:00:00 </li></ul><ul><li>/u01/app/oracle/product/11.1.0/crs/bin/crsd.bin reboot </li></ul><ul><li>[oracle@sdrac01 11.1.0]$ ps -ef|grep cssd </li></ul><ul><li>root 2846 1 0 Apr04 ? 00:03:15 /bin/sh /etc/init.d/init.cssd fatal </li></ul><ul><li>root 3630 2846 0 Apr04 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon </li></ul><ul><li>/u01/app/oracle/product/11.1.0/crs/bin/ocssd.bin </li></ul><ul><li>[oracle@sdrac01 11.1.0]$ ps -ef|grep evmd </li></ul><ul><li>oracle 3644 2845 0 Apr04 ? 00:00:00 </li></ul><ul><li>/u01/app/oracle/product/11.1.0/crs/bin/evmd.bin </li></ul><ul><li>oracle 9595 29413 0 23:59 pts/3 00:00:00 grep evmd </li></ul>
  9. 9. Verify 11g RAC Processes <ul><li>oprocd: Runs on Unix when vendor Clusterware is not running. On Linux, only starting with </li></ul><ul><li>oclsvmon.bin: Usually runs when a third party clusterware is used oclsomon.bin: Checks program of the ocssd.bin (starting in diskmon.bin : new process for Oracle Exadata Machine oclskd.bin: new process to reboot nodes in case RDBMS instances for 11g RAC are in a hang condition </li></ul><ul><li>There are three fatal processes, i.e. processes whose abnormal halt or kill will provokes a node reboot (Metalink Note:265769.1) 1. the ocssd.bin 2. the oprocd.bin 3. the oclsomon.bin </li></ul><ul><li>The other processes are automatically restarted when they go away. </li></ul>
  10. 10. Scripts for RAC monitoring <ul><li>Metalink 135714.1 provides racdiag.sql script to collect health status for 11g RAC environments. </li></ul><ul><li>TIME </li></ul><ul><li>-------------------- </li></ul><ul><li>FEB-11-2009 10:06:36 </li></ul><ul><li>1 row selected. </li></ul><ul><li>INST_ID INSTANCE_NAME HOST_NAME VERSION STATUS STARTUP_TIME </li></ul><ul><li>------- --------- --------- -------- ------- ---------- </li></ul><ul><li>1 rac01 sdrac01 OPEN FEB-01-2009 </li></ul><ul><li>2 rac02 sdrac02 OPEN FEB-01-2009 </li></ul><ul><li>2 rows selected </li></ul>
  11. 11. Check Status 11g RAC Clusterware <ul><li>CRSCTL is your friend </li></ul><ul><li>[oracle@sdrac01 11.1.0]$ crsctl </li></ul><ul><li>Usage: crsctl check crs - checks the viability of the CRS stack </li></ul><ul><li>crsctl check cssd - checks the viability of CSS </li></ul><ul><li>crsctl check crsd - checks the viability of CRS </li></ul><ul><li>crsctl check evmd - checks the viability of EVM </li></ul><ul><li>Worked Example of using CRSCTL for 11g RAC </li></ul><ul><li>[oracle@sdrac01 11.1.0]$ crsctl check crs </li></ul><ul><li>CSS appears healthy </li></ul><ul><li>CRS appears healthy </li></ul><ul><li>EVM appears healthy </li></ul>
  12. 12. More Checks for 11g RAC <ul><li>Use srvctl to get quick status check for 11g RAC: </li></ul><ul><li>[oracle@sdrac01]$ srvctl </li></ul><ul><li>Usage: srvctl <command> <object> [<options>] </li></ul><ul><li>command: enable|disable|start|stop|relocate|status|add|remove|modify|getenv|setenv|unsetenv|config </li></ul><ul><li>objects: database|instance|service|nodeapps|asm|listener </li></ul>
  13. 13. Using SRVCTL with 11g RAC <ul><li>Using SRVCTL to Check Database and Instances for 11g RAC </li></ul><ul><li>11g RAC Database Status: </li></ul><ul><li>srvctl status database -d <database-name> [-f] [-v] [-S <level>] </li></ul><ul><li>srvctl status instance -d <database-name> -i <instance-name> >[,<instance-name-list>] </li></ul><ul><li>[-f] [-v] [-S <level>] </li></ul><ul><li>srvctl status service -d <database-name> -s <service-name>[,<service-name-list>] </li></ul><ul><li>[-f] [-v] [-S <level>] </li></ul><ul><li>srvctl status nodeapps [-n <node-name>] </li></ul><ul><li>srvctl status asm -n <node_name> </li></ul>
  14. 14. SRVCTL for 11g RAC- Syntax <ul><li>Status of the database, all instances and all services. </li></ul><ul><li>$ srvctl status database -d ORACLE -v </li></ul><ul><li>Status of named instances with their current services. </li></ul><ul><li>$srvctl status instance -d ORACLE -i RAC01, RAC02 -v </li></ul><ul><li>Status of a named services. </li></ul><ul><li>$srvctl status service -d ORACLE -s ERP -v </li></ul><ul><li>Status of all nodes supporting database applications. </li></ul><ul><li>$srvctl status nodeapps –n {nodename} </li></ul>
  15. 15. SRVCTL Worked Examples 11g RAC <ul><li>Database and Instance Status Checks </li></ul><ul><li>$ srvctl status database -d RACDB -v </li></ul><ul><li>Instance RAC01 is not running on node sdrac01 </li></ul><ul><li>Instance RAC02 is not running on node sdrac02 </li></ul><ul><li>Node Application Checks </li></ul><ul><li>$ srvctl status nodeapps -n sdrac01 </li></ul><ul><li>VIP is not running on node: sdrac02 </li></ul><ul><li>GSD is running on node: sdrac01 </li></ul><ul><li>Listener is not running on node: sdrac01 </li></ul><ul><li>ONS daemon is running on node: sdraco1 </li></ul><ul><li>ASM Status Check for 11g RAC </li></ul><ul><li>$ srvctl status asm -n sdrac01 </li></ul><ul><li>ASM instance +ASM1 is not running on node sdrac01. </li></ul>
  16. 16. Don’t forget about CRS_STAT <ul><li>CRS_STAT useful for quick check for 11g RAC! </li></ul><ul><li>$ crs_stat -t </li></ul><ul><li>Name Type Target State Host </li></ul><ul><li>---------------------------------------------------------- </li></ul><ul><li>ora....B1.inst application ONLINE OFFLINE </li></ul><ul><li>ora....B2.inst application ONLINE OFFLINE </li></ul><ul><li>ora....ux1.gsd application ONLINE ONLINE sdrac01 </li></ul><ul><li>ora....ux1.ons application ONLINE ONLINE sdrac01 </li></ul><ul><li> application ONLINE OFFLINE </li></ul><ul><li>ora....t1.inst application ONLINE OFFLINE </li></ul><ul><li>ora.test.db application OFFLINE OFFLINE </li></ul><ul><li>ora....t1.inst application ONLINE OFFLINE </li></ul>
  17. 17. 11g Checks for ASM with RAC <ul><li>11g ASM has new features but still mostly the same as far as monitoring is concerned. </li></ul><ul><li>Check at the operating system level to ensure all critical 11g ASM processes are up and running: </li></ul><ul><li>$ ps -ef|grep asm </li></ul><ul><li>oracle 23471 1 0 01:46 ? 00:00:00 asm_pmon_+ASM1 </li></ul><ul><li>oracle 23483 1 1 01:46 ? 00:00:00 asm_diag_+ASM1 </li></ul><ul><li>oracle 23485 1 0 01:46 ? 00:00:00 asm_psp0_+ASM1 </li></ul><ul><li>oracle 23494 1 1 01:46 ? 00:00:00 asm_lmon_+ASM1 </li></ul><ul><li>oracle 23496 1 1 01:46 ? 00:00:00 asm_lmd0_+ASM1 </li></ul><ul><li>oracle 23498 1 1 01:46 ? 00:00:00 asm_lms0_+ASM1 </li></ul><ul><li>oracle 23534 1 0 01:46 ? 00:00:00 asm_mman_+ASM1 </li></ul><ul><li>oracle 23536 1 1 01:46 ? 00:00:00 asm_dbw0_+ASM1 </li></ul><ul><li>oracle 23546 1 0 01:46 ? 00:00:00 asm_lgwr_+ASM1 </li></ul><ul><li>oracle 23553 1 0 01:46 ? 00:00:00 asm_ckpt_+ASM1 </li></ul><ul><li>oracle 23561 1 0 01:46 ? 00:00:00 asm_smon_+ASM1 </li></ul><ul><li>oracle 23570 1 0 01:46 ? 00:00:00 asm_rbal_+ASM1 </li></ul><ul><li>oracle 23572 1 0 01:46 ? 00:00:00 asm_gmon_+ASM1 </li></ul><ul><li>oracle 23600 1 0 01:47 ? 00:00:00 asm_lck0_+ASM1 </li></ul>
  18. 18. More checks for 11g ASM <ul><li>Use the ASMCMD command to check status for 11g ASM with RAC </li></ul><ul><li>The ls and lsdg commands provide summary for ASM configuration </li></ul><ul><li>$ asmcmd </li></ul><ul><li>ASMCMD> ls </li></ul><ul><li>MY_DG1/ </li></ul><ul><li>MY_DG2/ </li></ul><ul><li>ASMCMD> lsdg </li></ul><ul><li>State Type Rebal Unbal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name </li></ul><ul><li>MOUNTED EXTERN N N 512 4096 1048576 3920 1626 0 1626 0 MY_DG1/ </li></ul><ul><li>MOUNTED EXTERN N N 512 4096 1048576 3920 1408 0 1408 0 MY_DG2/ </li></ul>
  19. 19. SQL*Plus with 11g ASM <ul><li>Useful query to check status for 11g ASM with RAC from SQL*PLUS: </li></ul><ul><li>SQL> select name, path, state from v$asm_disk; </li></ul><ul><li>NAME PATH STATE </li></ul><ul><li>------------------------- -------------------- ---------- </li></ul><ul><li>MY_DG1_0001 /dev/raw/raw12 NORMAL </li></ul><ul><li>MY_DG1_0000 /dev/raw/raw11 NORMAL </li></ul><ul><li>MY_DG1_0002 /dev/raw/raw13 NORMAL </li></ul><ul><li>MY_DG2_0000 /dev/raw/raw15 NORMAL </li></ul><ul><li>MY_DG2_0001 /dev/raw/raw16 NORMAL </li></ul><ul><li>MY_DG1_0003 /dev/raw/raw14 NORMAL </li></ul>
  20. 20. Healthchecks- OCR and Votedisk for 11g RAC
  21. 21. Quick Review- 11g RAC Concepts OCR and Vote Disk <ul><li>What is the OCR? </li></ul><ul><li>Oracle Cluster Registry purpose is to hold cluster and database configuration information for RAC and Cluster Ready Services (CRS) such as the cluster node list, and cluster database instance to node mapping, and CRS application resource profiles. </li></ul><ul><li>The OCR must be stored on either shared raw devices or OCFS/OCFS2 (Oracle Cluster Filesystem) </li></ul><ul><li>What is the Voting Disk? </li></ul><ul><li>The Voting disk manages cluster node membership and must be stored on either shared raw disk or OCFS/OCFS2 cluster filesystem. </li></ul>
  22. 22. OCR and Vote Disk Health Check <ul><li>Without the OCR and Vote Disk 11g RAC will fail! </li></ul><ul><li>Useful health checks for OCR with OCRCHECK command: </li></ul><ul><li>$ ocrcheck </li></ul><ul><li>Status of Oracle Cluster Registry is as follows : </li></ul><ul><li>Version : 2 </li></ul><ul><li>Total space (kbytes) : 297084 </li></ul><ul><li>Used space (kbytes) : 3848 </li></ul><ul><li>Available space (kbytes) : 293236 </li></ul><ul><li>ID : 2007457116 </li></ul><ul><li>Device/File Name : /dev/raw/raw5 </li></ul><ul><li>Device/File integrity check succeeded </li></ul><ul><li>Device/File Name : /dev/raw/raw6 </li></ul><ul><li>Device/File integrity check succeeded </li></ul><ul><li>Cluster registry integrity check succeeded </li></ul>
  23. 23. Healthcheck for Vote Disk <ul><li>Use the CRSCTL command: </li></ul><ul><li>$ crsctl query css votedisk </li></ul><ul><li>0. 0 /dev/raw/raw7 </li></ul><ul><li>1. 0 /dev/raw/raw8 </li></ul><ul><li>2. 0 /dev/raw/raw9 </li></ul><ul><li>located 3 votedisk(s). </li></ul>
  24. 24. Problems and Solutions: 11g RAC
  25. 25. 11g RAC Problems and Solutions <ul><li>Missing Clusterware resources offline </li></ul><ul><li>Failed or corrupted vote disk </li></ul><ul><li>Failed or corrupted OCR disks </li></ul><ul><li>RAC node reboot issues </li></ul><ul><li>Hardware, Storage, Network problems with RAC </li></ul>
  26. 26. Root Cause Analysis 11g RAC <ul><li>First step- locate and examine 11g RAC log files. </li></ul><ul><li>Metalink Note 781632.1 and 311321.1 are useful </li></ul><ul><li>CRS_HOME Log Files </li></ul><ul><li>$ CRS_HOMElog odename acg contains log files for VIP and ONS resources </li></ul><ul><li>RBDMS_HOME log files under ORACLE_HOME/log/nodename/racg </li></ul><ul><li>Example: </li></ul><ul><li>/u01/app/oracle/product/11.1.0/db_1/log/sdrac01/racg </li></ul><ul><li>Errors are reported to imon<DB_NAME>.log files </li></ul><ul><li>$ view imon.log </li></ul><ul><li>2009-03-15 21:39:38.497: [ RACG][3002129328] [13876][3002129328][ora.RACDB.RACDB2.inst]: clsrfdbe_enqueue: POST_ALERT() failed: evttypname='down' type='1' resource='ora.RACDB.RACDB2.inst' node='sdrac01' time='2009-03-15 21:39:36.0 -05:00' card=0 </li></ul><ul><li>2009-03-15 21:40:08.521: [ RACG][3002129328] [13876][3002129328][ora.RACDB.RACDB2.inst]: CLSR-0002: Oracle error encountered while executing DISCONNECT </li></ul><ul><li>2009-03-15 21:40:08.521: [ RACG][3002129328] [13876][3002129328][ora.RACDB.RACDB2.inst]: ORA-03114: not connected to ORACLE </li></ul>
  27. 27. 11g RAC Log Files <ul><li>ASM Log Files for 11g RAC root cause analysis </li></ul><ul><li>ASM_HOME/log/nodename/racg if ASM is separate from the RDBMS otherwise these logs are located under RDBMS_HOME </li></ul><ul><li>ASM log files for 11g RAC analysis are named in format convention of ora.nodename.asm.log </li></ul><ul><li>$ view ora.sdrac01.ASM1.asm.log </li></ul>
  28. 28. 11g RAC ASM Log File <ul><li>$ view ora.sdrac01.ASM1.asm.log </li></ul><ul><li>2009-03-15 21:40:03.725: [ RACG][3086936832] [11200][3086936832][ora.sdrac01.ASM1.asm]: Real Application Clusters, Oracle Label Security, OLAP </li></ul><ul><li>and Data Mining Scoring Engine options </li></ul><ul><li>SQL> ASM instance shutdown </li></ul><ul><li>SQL> Disconnected from Oracle Database 10g Enterprise Edition Release – Production </li></ul><ul><li>With the Partitioning, Real Application </li></ul><ul><li>2009-03-15 21:40:03.725: [ RACG][3086936832] [11200][3086936832][ora.sdrac01.ASM1.asm]: Clusters, Oracle Label Security, OLAP </li></ul><ul><li>and Data Mining Scoring Engine options </li></ul>
  29. 29. Missing Clusterware resources offline <ul><li>Common problem unable to start Clusterware resources </li></ul><ul><li>The command for crs_stat -t output shows VIP is offline and trying to start it gives error : CRS-0215: Could not start resource ''. Example: crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....st2.gsd application ONLINE ONLINE rac01 ora....st2.ons application ONLINE ONLINE rac01 application ONLINE OFFLINE </li></ul>
  30. 30. Offline Clusterware Resources <ul><li>[root@sdrac01]# ./srvctl start nodeapps -n sdrac01 </li></ul><ul><li> eth0 checked failed ( </li></ul><ul><li> parameters, or failed to bring up VIP ( </li></ul><ul><li> eth0 checked failed ( </li></ul><ul><li> parameters, or failed to bring up VIP ( </li></ul><ul><li>CRS-1006: No more members to consider </li></ul><ul><li>CRS-0215: Could not start resource ''. </li></ul><ul><li> eth0 checked failed ( </li></ul><ul><li> parameters, or failed to bring up VIP ( </li></ul><ul><li> eth0 checked failed ( </li></ul><ul><li> parameters, or failed to bring up VIP ( </li></ul><ul><li>CRS-1006: No more members to consider </li></ul><ul><li>CRS-0215: Could not start resource 'ora.sdrac01.LISTENER_SDRAC01.lsnr'. </li></ul>
  31. 31. Solution for Offline Clusterware Resources <ul><li>Metalink Notes 781632.1 and 356535.1 have some good troubleshooting advice with failed CRS resources. </li></ul><ul><li>First, we need to diagnose current settings for VIP: </li></ul><ul><li>[root@sdrac011 bin]# ./srvctl config nodeapps -n sdrac01 -a -g -s -l </li></ul><ul><li>VIP exists.: / </li></ul><ul><li>GSD exists. </li></ul><ul><li>ONS daemon exists. </li></ul><ul><li>Listener exists. </li></ul><ul><li>Start Debug for failed resources by either setting the environment variable _USR_ORA_DEBUG=1 in the script $ORA_CRS_HOME/bin/racgvip or using crsctl debug command shown in below example: </li></ul><ul><li># ./crsctl debug log res &quot;; </li></ul><ul><li>Set Resource Debug Module: Level: 5 </li></ul>
  32. 32. Useful debug output 11g RAC VIP Issue <ul><li># ./srvctl start nodeapps -n sdrac01 </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Broadcast = </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Checking interface existance </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Calling getifbyip </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] getifbyip: started for </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Completed getifbyip </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Calling getifbyip -a </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] getifbyip: started for </li></ul><ul><li> Apr 8 01:12:43 EDT 2009 [ 27550 ] Completed getifbyip </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] Completed with initial interface test </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] Interface tests </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] checkIf: start for if=eth0 </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] /sbin/mii-tool eth0 error </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] defaultgw: started </li></ul><ul><li> Apr 8 01:12:44 EDT 2009 [ 27550 ] defaultgw: completed with </li></ul><ul><li> Apr 8 01:12:47 EDT 2009 [ 27550 ] checkIf: ping and RX packets checked if=eth0 failed </li></ul><ul><li> eth0 checked failed ( </li></ul>
  33. 33. Failed VIP Resource 11g RAC <ul><li>Start the VIP using srvctl start nodeapps again. This will create a log for VIP starting problem in the directory $ORA_CRS_HOME/log/racg/*vip.log </li></ul><ul><li>Review the log files </li></ul><ul><li># cd/u01/app/oracle/product/11.1.0/crs/log/sdrac01/racg </li></ul><ul><li>[root@sdrac01 racg]# ls </li></ul><ul><li>evtf.log ora.sdrac01.ons.log ora.test.db.log racgmain </li></ul><ul><li>ora.RACDB.db.log racgeut </li></ul><ul><li>ora.sdrac01.gsd.log racgevtf </li></ul><ul><li>Turn off debugging with command : </li></ul><ul><li># ./crsctl debug log res &quot;; </li></ul>
  34. 34. Example: 11g RAC Resource Offline <ul><li># view </li></ul><ul><li>2009-04-08 00:45:36.447: [ RACG][3086936832] [22614][3086936832][]: clsrcexecut: rc = 1, time = 6.210s </li></ul><ul><li>2009-04-08 00:45:42.765: [ RACG][3086936832] [22614][3086936832][]: Interface eth0 checked failed ( </li></ul><ul><li>Invalid parameters, or failed to bring up VIP ( </li></ul><ul><li>2009-04-08 00:45:42.765: [ RACG][3086936832] [22614][3086936832][]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/oracle/product/11.1.0/crs </li></ul><ul><li>2009-04-08 00:45:42.765: [ RACG][3086936832] [22614][3086936832][]: clsrcexecut: cmd = /u01/app/oracle/product/11.1.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/app/oracle/product/11.1.0/crs/bin/racgvip check sdrac01 </li></ul><ul><li>2009-04-08 00:45:42.765: [ RACG][3086936832] [22614][3086936832][]: clsrcexecut: rc = 1, time = 6.320s </li></ul><ul><li>2009-04-08 00:45:42.765: [ RACG][3086936832] [22614][3086936832][]: end for resource =, action = start, status = 1, time = 12.560s </li></ul>
  35. 35. Solution for Offline VIP Resource <ul><li>Stop nodeapps with srvctl stop nodeapps –n sdrac01 </li></ul><ul><li>Login as root and edit $ORA_CRS_HOME/bin/racgvip </li></ul><ul><li>Change the value of variable FAIL_WHEN_DEFAULTGW_NOT_FOUND=0 </li></ul><ul><li>Start nodeapps with srvctl start nodeapps –n sdrac01 and you should see the resources ONLINE </li></ul>
  36. 36. Failed or Corrupted Vote Disk <ul><li>Best practice- multiple copies of vote disk on different disk volumes to eliminate single point of failure (SPOF). </li></ul><ul><li>Metalink Note 279793.1 has tips on vote disk for RAC </li></ul><ul><li>Make sure you take backups with dd utility (UNIX/Linux) or ocopy utility (Windows) </li></ul><ul><li>Take frequent backups if using dd should be 4k blocksize on Linux/UNIX platform to ensure complete blocks are backed up for voting disk. </li></ul><ul><li>Without backup you must re-install CRS! </li></ul>
  37. 37. Failed or corrupted OCR disks <ul><li>Best practice- maintain frequent backups of OCR on separate disk volumes to avoid single point of failure (SPOF) </li></ul><ul><li>OCRCONFIG utility to perform recovery </li></ul><ul><li>Metalink Notes 220970.1 , 428681.1 and 390880.1 are useful </li></ul><ul><li>Find backup for OCR </li></ul>
  38. 38. Recover OCR from backup <ul><li># ./ocrconfig </li></ul><ul><li>Name: </li></ul><ul><li>ocrconfig - Configuration tool for Oracle Cluster Registry. </li></ul><ul><li>Synopsis: </li></ul><ul><li>ocrconfig [option] </li></ul><ul><li>option: </li></ul><ul><li>-export <filename> [-s online] </li></ul><ul><li>- Export cluster register contents to a file </li></ul><ul><li>-import <filename> - Import cluster registry contents from a file </li></ul><ul><li>-upgrade [<user> [<group>]] </li></ul><ul><li>- Upgrade cluster registry from previous version </li></ul><ul><li>-downgrade [-version <version string>] </li></ul><ul><li>- Downgrade cluster registry to the specified version </li></ul><ul><li>-backuploc <dirname> - Configure periodic backup location </li></ul><ul><li>-showbackup - Show backup information </li></ul><ul><li>-restore <filename> - Restore from physical backup </li></ul><ul><li>-replace ocr|ocrmirror [<filename>] - Add/replace/remove a OCR device/file </li></ul><ul><li>-overwrite - Overwrite OCR configuration on disk </li></ul><ul><li>-repair ocr|ocrmirror <filename> - Repair local OCR configuration </li></ul><ul><li>-help - Print out this help information </li></ul><ul><li>Note: </li></ul><ul><li>A log file will be created in </li></ul><ul><li>$ORACLE_HOME/log/<hostname>/client/ocrconfig_<pid>.log. Please ensure </li></ul><ul><li>you have file creation privileges in the above directory before </li></ul><ul><li>running this tool. </li></ul>
  39. 39. Using OCRCONFIG for recover lost OCR <ul><li>First we need to find our backups of the OCR with ocrconfig utility </li></ul><ul><li># ./ocrconfig -showbackup </li></ul><ul><li>rac01 2009/04/07 23:01:40 /u01/app/oracle/product/11.1.0/crs/cdata/crs </li></ul><ul><li>rac01 2009/04/07 19:01:39 /u01/app/oracle/product/11.1.0/crs/cdata/crs </li></ul><ul><li>rac01 2009/04/07 01:40:31 /u01/app/oracle/product/11.1.0/crs/cdata/crs </li></ul><ul><li>rac01 2009/04/06 21:40:30 /u01/app/oracle/product/11.1.0/crs/cdata/crs </li></ul><ul><li>rac01 2009/04/03 14:12:46 /u01/app/oracle/product/11.1.0/crs/cdata/crs </li></ul>
  40. 40. Recovery lost/corrupt OCR <ul><li>We check the status of OCR backups: </li></ul><ul><li>$ ls -l </li></ul><ul><li>total 24212 </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 29 2008 backup00.ocr </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 21 2008 backup01.ocr </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 20 2008 backup02.ocr </li></ul><ul><li>-rw-r--r-- 1 root root 2949120 Apr 4 19:26 day_.ocr </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 29 2008 day.ocr </li></ul><ul><li>-rw-r--r-- 1 root root 4116480 Apr 7 23:01 temp.ocr </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 29 2008 week_.ocr </li></ul><ul><li>-rw-r--r-- 1 oracle oinstall 2949120 Aug 19 2008 week.ocr </li></ul><ul><li>Next we use OCRCONFIG –restore to recover the lost or corrupted OCR from a valid backup </li></ul><ul><li>$ ocrconfig –restore backup00.ocr </li></ul>
  41. 41. 11g RAC node reboot issues <ul><li>What causes node reboots in 11g RAC? </li></ul><ul><li>Root cause can be difficult to diagnose </li></ul><ul><li>Can be due to network and storage issues </li></ul><ul><li>Metalink Note 265769.1 good reference point for node reboot issues and provides useful Decision Tree for these issues with RAC. </li></ul><ul><li>If there is a ocssd.bin problem/failure, the oprocd daemon detected a scheduling problem, or some other fatal problem, a node will reboot in a RAC cluster. This functionality is used for I/O fencing to ensure that writes from I/O capable clients can be cleared avoiding potential corruption scenarios in the event of a network split, node hang, or some other fatal event. </li></ul>
  42. 42. 11g RAC Clusterware Processes – Node Reboot Issues <ul><li>When ocssd.bin process dies it notifies the oprocd process to shoot the node in the head and cause the to node reboot (STONITH). </li></ul><ul><li>OCSSD (aka CSS daemon) - This process is spawned in init.cssd. It runs in both vendor clusterware and non-vendor clusterware environments and is armed with a node kill via the init script. OCSSD's primary job is internode health monitoring and RDBMS instance endpoint discovery. It runs as the Oracle user. </li></ul><ul><li>INIT.CSSD - In a normal environment, init spawns init.cssd, which in turn spawns OCSSD as a child. If ocssd dies or is killed, the node kill functionality of the init script will kill the node. </li></ul><ul><li>OPROCD - This process is spawned in any non-vendor clusterware environment, except on Windows where Oracle uses a kernel driver to perform the same actions and Linux prior to version If oprocd detects problems, it will kill a node via C code. It is spawned in init.cssd and runs as root. This daemon is used to detect hardware and driver freezes on the machine. If a machine were frozen for long enough that the other nodes evicted it from the cluster, it needs to kill itself to prevent any IO from getting reissued to the disk after the rest of the cluster has remastered locks.&quot; </li></ul><ul><li>OCLSOMON ( and above) - This process monitors the CSS daemon for hangs or scheduling issues and can reboot a node if there is a perceived hang. </li></ul><ul><li>Data collection is vital </li></ul><ul><li>OSWatcher tool Metalink Note 301137.1 and 433472.1 have the details on how to setup this diagnosis tool for Linux/UNIX and Windows </li></ul>
  43. 43. Root Cause: Node Reboots 11g RAC <ul><li>Find the process that caused the node to reboot </li></ul><ul><li>Review all log and trace files to determine failed process for 11g RAC </li></ul><ul><li>* Messages file locations: Sun: /var/adm/messages HP-UX: /var/adm/syslog/syslog.log Tru64: /var/adm/messages Linux: /var/log/messages IBM: /bin/errpt -a > messages.out ** CSS log locations: 11.1 and 10.2: <CRS_HOME>/log/<node name>/cssd 10.1: <CRS_HOME>/css/log LINUX: </li></ul><ul><li>*** Oprocd log locations: In /etc/oracle/oprocd or /var/opt/oracle/oprocd depending on version/platform. </li></ul>
  44. 44. 11g RAC Log Files for Troubleshooting <ul><li>For 10.2 and above, all files under: <CRS_HOME>/log For 10.1: <CRS_HOME>/crs/log <CRS_HOME>/crs/init <CRS_HOME>/css/log <CRS_HOME>/css/init <CRS_HOME>/evm/log <CRS_HOME>/evm/init <CRS_HOME>/srvm/log </li></ul><ul><li>Useful tool called RAC DDT to collect all 11g RAC log and trace files </li></ul><ul><li>Metalink Note 301138.1 covers use of RAC DDT </li></ul><ul><li>Also important to collect OS and network information: </li></ul><ul><li>netstat, iostat, vmstat and ping outputs from 11g RAC cluster nodes </li></ul>
  45. 45. OCSSD Reboots and 11g RAC <ul><li>Network failure or latency between nodes. It would take at least 30 consecutive missed checkins to cause a reboot, where heartbeats are issued once per second. Example of missed checkins in the CSS log: WARNING: clssnmPollingThread: node <node> (1) at 50% heartbeat fatal, eviction in 29.100 seconds </li></ul><ul><li>Review messages file to determine root cause for OCSSD failures. </li></ul><ul><li>If the messages file reboot time < missed checkin time then the node eviction was likely not due to these missed checkins. If the messages file reboot time > missed checkin time then the node eviction was likely a result of the missed checkins. </li></ul><ul><li>Problems writing to or reading from the CSS voting disk. Check CSS logs: ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds) </li></ul><ul><li>High load averages due to lack of CPU resources. </li></ul><ul><li>Misconfiguration of CRS. Possible misconfigurations: </li></ul>
  46. 46. OPROCD Failure and Node Reboots <ul><li>Four things cause OPROC to fail and node reboot with 11g RAC: </li></ul><ul><li>1) An OS scheduler problem. </li></ul><ul><li>2) The OS is getting locked up in a driver or hardware issue. </li></ul><ul><li>3) Excessive amounts of load on the machine, thus preventing the scheduler from behaving reasonably. </li></ul><ul><li>4) An Oracle bug such as Bug 5015469 </li></ul>
  47. 47. OCLSOMON- RAC Node Reboot <ul><li>Four root causes to the OCLSOMON process failure that causes 11g RAC node reboot condition: </li></ul><ul><li>1) Hung threads within the CSS daemon. </li></ul><ul><li>2) OS scheduler problems </li></ul><ul><li>3) Excessive amounts of load on the machine </li></ul><ul><li>4) Oracle bugs </li></ul>
  48. 48. Hardware, Storage, Network problems <ul><li>Check certification matrix on Metalink for supported versions for network drivers, storage and firmware releases with 11g RAC. </li></ul><ul><li>Develop close working relationship with system and network team. Educate them on RAC. </li></ul><ul><li>System utilities such as ifconfig, netstat, ping, and traceroute are essential for diagnosis and root cause analysis. </li></ul>
  49. 49. Summary <ul><li>What happened to my 11g RAC Clusterware? </li></ul><ul><li>Failed resources in 11g RAC Clusterware </li></ul><ul><li>Upgrade and Migration issues for 11g RAC and Clusterware </li></ul><ul><li>Patch Upgrade issues with Clusterware </li></ul><ul><li>Node eviction issues </li></ul>
  50. 50. Tuning 11g RAC
  51. 51. Solving critical tuning issues for RAC <ul><li>Tune for single instance first and then RAC </li></ul><ul><li>Interconnect Performance Tuning </li></ul><ul><li>Cluster related wait issues </li></ul><ul><li>Lock/Latch Contention </li></ul><ul><li>Parallel tuning tips for RAC </li></ul><ul><li>ASM Tuning for RAC </li></ul>
  52. 52. Interconnect Tuning for 11g RAC <ul><li>Invest in best network for 11g RAC Interconnect </li></ul><ul><li>Infiniband offers robust performance </li></ul><ul><li>Majority of performance problems in 11g RAC are due to poorly sized network for interconnect </li></ul>
  53. 53. DBA Toolkit for 11g RAC
  54. 54. DBA 101 Toolkit for 11g RAC <ul><li>Oracle 11g DBA Tools: </li></ul><ul><li>Oracle 11g ADDM </li></ul><ul><li>Oracle 11g AWR </li></ul><ul><li>Oracle 11g Enterprise Manager/Grid Control </li></ul><ul><li>Operating System Tools </li></ul>
  55. 55. Using Oracle 11g Tools <ul><li>ADDM and AWR now provide RAC specific monitoring checks and reports </li></ul><ul><li>AWR Report Sample for 11g RAC via OEM Grid Control or awrrpt.sql </li></ul><ul><li>SQL> @?/rdbms/admin/awrrpt.sql </li></ul><ul><li>WORKLOAD REPOSITORY report for </li></ul><ul><li>DB Name DB Id Instance Inst Num Startup Time Release RAC </li></ul><ul><li>---- ------ --------- ----- --------------- -------- ----- </li></ul><ul><li>RACDB 2057610071 RAC01 1 20-Jan-09 20:50 YES </li></ul><ul><li>Host Name Platform CPUs Cores Sockets Memory(GB) </li></ul><ul><li>------- ---------- ---- ----- ------- ---------- </li></ul><ul><li>sdrac01 Linux x86 64-bit 8 8 4 31.49 </li></ul><ul><li>Snap Id Snap Time Sessions Curs/Sess </li></ul><ul><li>--------- ------------------- -------- --------- </li></ul><ul><li>Begin Snap: 12767 21-Jan-09 00:00:06 361 25.9 </li></ul><ul><li>End Snap: 12814 21-Jan-09 08:40:09 423 22.0 </li></ul><ul><li>Elapsed: 520.05 (mins) </li></ul><ul><li>DB Time: 102,940.70 (mins) </li></ul>
  56. 56. Using AWR with 11g RAC <ul><li>We want to examine the following areas from AWR for 11g RAC Performance: </li></ul><ul><li>RAC Statistics DB/Inst: RACDB/RAC01 Snaps: 12767-12814 </li></ul><ul><li>Begin End </li></ul><ul><li>----- ----- </li></ul><ul><li>Number of Instances: 3 3 </li></ul><ul><li>Global Cache Load Profile </li></ul><ul><li>~~~~~~~~~~~~~~~~~~~~~~~~~ Per Second Per Transaction </li></ul><ul><li>--------------- --------------- </li></ul><ul><li>Global Cache blocks received: 88.89 2.41 </li></ul><ul><li>Global Cache blocks served: 92.32 2.51 </li></ul><ul><li>GCS/GES messages received: 906.54 24.63 </li></ul><ul><li>GCS/GES messages sent: 755.21 20.52 </li></ul><ul><li>DBWR Fusion writes: 5.56 0.15 </li></ul><ul><li>Estd Interconnect traffic (KB) 1,774.22 </li></ul>
  57. 57. AWR for 11g RAC (Continued) <ul><li>Global Cache Efficiency Percentages (Target local+remote 100%) </li></ul><ul><li>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ </li></ul><ul><li>Buffer access - local cache %: 99.59 </li></ul><ul><li>Buffer access - remote cache %: 0.12 </li></ul><ul><li>Buffer access - disk %: 0.29 </li></ul><ul><li>Global Cache and Enqueue Services - Workload Characteristics </li></ul><ul><li>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ </li></ul><ul><li>Avg global enqueue get time (ms): 2.7 </li></ul><ul><li>Avg global cache cr block receive time (ms): 3.2 </li></ul><ul><li>Avg global cache current block receive time (ms): 1.1 </li></ul><ul><li>Avg global cache cr block build time (ms): 0.0 </li></ul><ul><li>Avg global cache cr block send time (ms): 0.0 </li></ul><ul><li>Global cache log flushes for cr blocks served %: 11.3 </li></ul><ul><li>Avg global cache cr block flush time (ms): 29.4 </li></ul><ul><li>Avg global cache current block pin time (ms): 11.6 </li></ul><ul><li>Avg global cache current block send time (ms): 0.1 </li></ul><ul><li>Global cache log flushes for current blocks served %: 0.3 </li></ul><ul><li>Avg global cache current block flush time (ms): 61.8 </li></ul><ul><li>Global Cache and Enqueue Services - Messaging Statistics </li></ul><ul><li>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ </li></ul><ul><li>Avg message sent queue time (ms): 4902.6 </li></ul><ul><li>Avg message sent queue time on ksxp (ms): 1.2 </li></ul><ul><li>Avg message received queue time (ms): 0.1 </li></ul><ul><li>Avg GCS message process time (ms): 0.0 </li></ul><ul><li>Avg GES message process time (ms): 0.0 </li></ul><ul><li>% of direct sent messages: 70.13 </li></ul><ul><li>% of indirect sent messages: 28.36 </li></ul><ul><li>% of flow controlled messages: 1.51 </li></ul><ul><li>------------------------------------------------------------- </li></ul><ul><li>Cluster Interconnect </li></ul><ul><li>~~~~~~~~~~~~~~~~~~~~ </li></ul><ul><li>Begin End </li></ul><ul><li>-------------------------------------------------- ----------- </li></ul><ul><li>Interface IP Address Pub Source IP Pub Src </li></ul><ul><li>---------- --------------- --- ------------------------------ --- --- --- </li></ul><ul><li>bond0 N Oracle Cluster Repository </li></ul>
  58. 58. Interconnect Performance 11g RAC <ul><li>Interconnect Performance key for identify performance issues with 11g RAC! </li></ul><ul><li>Interconnect Throughput by Client DB/Inst: RACDB/RAC01 Snaps: 12767-12814 </li></ul><ul><li>-> Throughput of interconnect usage by major consumers. </li></ul><ul><li>-> All throughput numbers are megabytes per second </li></ul><ul><li>Send Receive </li></ul><ul><li>Used By Mbytes/sec Mbytes/sec </li></ul><ul><li>---------------- ----------- ----------- </li></ul><ul><li>Global Cache .72 .69 </li></ul><ul><li>Parallel Query .01 .01 </li></ul><ul><li>DB Locks .16 .17 </li></ul><ul><li>DB Streams .00 .00 </li></ul><ul><li>Other .02 .02 </li></ul><ul><li>------------------------------------------------------------- </li></ul><ul><li>Interconnect Device Statistics DB/Inst: RACDB/RAC01 Snaps: 12767-12814 </li></ul><ul><li>-> Throughput and errors of interconnect devices (at OS level). </li></ul><ul><li>-> All throughput numbers are megabytes per second </li></ul><ul><li>Device Name IP Address Public Source </li></ul><ul><li>--------------- ---------------- ------ ------------------------------- </li></ul><ul><li>Send Send </li></ul><ul><li>Send Send Send Buffer Carrier </li></ul><ul><li>Mbytes/sec Errors Dropped Overrun Lost </li></ul><ul><li>----------- -------- -------- -------- -------- </li></ul><ul><li>Receive Receive </li></ul><ul><li>Receive Receive Receive Buffer Frame </li></ul><ul><li>Mbytes/sec Errors Dropped Overrun Errors </li></ul><ul><li>----------- -------- -------- -------- -------- </li></ul><ul><li>bond0 NO Oracle Cluster Repository </li></ul><ul><li>1.43 0 0 0 0 </li></ul><ul><li>1.44 0 0 0 0 </li></ul><ul><li>------------------------------------------------------------- </li></ul><ul><li>End of Report </li></ul>
  59. 59. ADDM for 11g RAC <ul><li>ADDM nicer interface than AWR and available via OEM Grid Control or addmrpt.sql script. </li></ul><ul><li>SQL> @?/admin/rdbms/addmrpt.sql </li></ul><ul><li>---------------------------------- </li></ul><ul><li>Analysis Period </li></ul><ul><li>--------------- </li></ul><ul><li>AWR snapshot range from 12759 to 12814. </li></ul><ul><li>Time period starts at 20-JAN-09 10.40.17 PM </li></ul><ul><li>Time period ends at 21-JAN-09 08.40.10 AM </li></ul><ul><li>Analysis Target </li></ul><ul><li>--------------- </li></ul><ul><li>Database 'RACDB' with DB ID 2057610071. </li></ul><ul><li>Database version </li></ul><ul><li>ADDM performed an analysis of instance RAC01, numbered 1 and hosted at </li></ul><ul><li>sdrac01 </li></ul><ul><li>Activity During the Analysis Period </li></ul><ul><li>----------------------------------- </li></ul><ul><li>Total database time was 7149586 seconds. </li></ul><ul><li>The average number of active sessions was 198.64. </li></ul><ul><li>Summary of Findings </li></ul><ul><li>------------------- </li></ul><ul><li>Description Active Sessions Recommendations </li></ul><ul><li>Percent of Activity </li></ul><ul><li>---------------------------- ------------------- --------------- </li></ul><ul><li>1 Unusual &quot;Network&quot; Wait Event 192.91 | 97.12 3 </li></ul>
  60. 60. Operating System Tools for 11g RAC <ul><li>Strace for Linux </li></ul><ul><li># ps -ef|grep crsd </li></ul><ul><li>root 2853 1 0 Apr05 ? 00:00:00 /u01/app/oracle/product/11.1.0/crs/bin/crsd.bin reboot </li></ul><ul><li>root 20036 2802 0 01:53 pts/3 00:00:00 grep crsd </li></ul><ul><li>[root@sdrac01 bin]# strace -p 2853 </li></ul><ul><li>Process 2853 attached - interrupt to quit </li></ul><ul><li>futex(0xa458bbf8, FUTEX_WAIT, 7954, NULL </li></ul><ul><li>Truss for Solaris </li></ul><ul><li>Both are excellent OS trace level tools to find out exactly what a specific Oracle 11g RAC process is doing. </li></ul>
  61. 61. Preguntas? <ul><li>Hay algunas preguntas? Questions? </li></ul><ul><li>I’ll also be available in the Oracle ACE lodge </li></ul><ul><li>Tambien se puede enviarme sus preguntas : </li></ul><ul><li>Email: </li></ul>
  62. 62. Conclusion <ul><li>Muchas gracias! </li></ul><ul><li>Please complete your evaluation form </li></ul><ul><li>Ben Prusinski [email_address] </li></ul><ul><li>Oracle 11g Real Application Clusters 101: Insider Tips and Tricks </li></ul><ul><li>My company- Ben Prusinski and Associates </li></ul><ul><li> </li></ul><ul><li>Oracle Blog </li></ul><ul><li> </li></ul>