1,539 views

Oracle utilities in 11g R2

This document discusses several new features and tools in Oracle 11g R2 for monitoring, performance tuning, and troubleshooting. It introduces the Automatic Diagnostic Repository (ADR) as the new paradigm for monitoring Oracle databases in 11g. ADR provides centralized access to alert logs, trace files, and incidents. The document demonstrates how to use the command line ADR interface (ADRCI) to view alerts, traces, and incidents. It also covers performance tools like Autotrace, SQL Trace, TKPROF, and DBMS_STATS and how to use them with Oracle 11g R2.

Road less tRAveled Oracle 11g R2 utilitiesGuenadi Nedkov JilevskiE-mail: gjilevski@gmail.comBGOUG : Bulgaria April 2010

Agenda: Road less traveled Oracle utilitiesTop new features in Oracle 11g R1 & 11g R2Monitoring & Performance ToolsAutotrace, SQL trace, TKPROF, DBMS_STATS, DBMS_SQLTUNEOradebugHidden Oracle 11g R2 parametersUseful V$ and X$ tables for Oracle 11g R2Powerful system tools: dbx, strace, orakillORA-0600 Tips and tricksKFOD for ASM Monitoring

Top features for Oracle 11g R2ADR for monitoringData Recovery Advisor with RMAN for Oracle Backup and Recovery with Oracle 11gReal Application Testing (RAT) features with 11gNew partitioning features for Oracle 11g Data partitioning such as (automatic interval, partitioning by reference, composite range-range, composite list-range, composite list-list, composite list-hash, virtual column) and partition advisor Enhancements for Oracle 11g performance tuning (SQLPA,HM, SQL plan management, Server/Client result cache, Enhanced statistics, SQL tuning automation, PL/SQL native compilation etc...)Invisible indexes new feature for Oracle 11gIn-Memory Parallel Execution in 11g R2Auto Degree of Parallelism (DOP) and Queuing in 11g R2The DBMS_PARALLEL_EXECUTE package in 11g R2Performance improvement of On-Commit Fast Refresh in 11g R2

What is ADR for Oracle 11gR2?ADR is the new Automatic Diagnostic monitor with 11g release. Many new monitoring and reporting features.Completely new paradigm for Oracle DBA monitoring.Available in both command line interface (adrci) or from Oracle 11g Grid/Database control.Replaces alert.log, trace files text based monitoring in releases prior to 11g.Alert.log still available but recommended you learn ADR.

Introduction to Oracle 11g R2 ADRThe new Oracle 11g Automatic Diagnostic Repository (ADR) has the following new features:One stop shop for monitoring and reporting on Oracle 11gNew feature for incident reporting and packaging (IPS)Available with Oracle 11g Database/Grid Control GUI as well as from command line utility (adrci)Many useful reporting and monitoring tools.Provide access to all of the Oracle 11g trace files for networking, alert.log etc.

Inside Automatic Disgnostic Monitor (ADR) for Oracle 11g R2ADR stores the main configuration details in log.xml fileDefault ADR directory set Oracle 11g initialization diagnostic_dest parameterSQL> show parameter diagnostic_destNAME TYPE VALUE------------------------------------ ----------- ------------------------------diagnostic_dest string /u01/app/oracleSQL> SQL> SELECT * FROM V$DIAG_INFO; INST_ID NAME VALUE---------- ---------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 Diag Enabled TRUE 1 ADR Base /u01/app/oracle 1 ADR Home /u01/app/oracle/diag/rdbms/racdb/RACDB_1 1 Diag Trace /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace 1 Diag Alert /u01/app/oracle/diag/rdbms/racdb/RACDB_1/alert 1 Diag Incident /u01/app/oracle/diag/rdbms/racdb/RACDB_1/incident 1 Diag Cdump /u01/app/oracle/diag/rdbms/racdb/RACDB_1/cdump 1 Health Monitor /u01/app/oracle/diag/rdbms/racdb/RACDB_1/hm 1 Default Trace File /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_6723.trc 1 Active Problem Count 0 1 Active Incident Count 011 rows selected.

Inside the (ADR) for Oracle 11g R2ADR home directory located under default:<ADR_BASE>/diag/rdbms/<db_name>/<instance_id>[oracle@raclinux1 RACDB_1]$ pwd/u01/app/oracle/diag/rdbms/racdb/RACDB_1[oracle@raclinux1 RACDB_1]$ lsalert cdump hm incident incpkg ir lck metadata stage sweep trace[oracle@raclinux1 RACDB_1]$ [oracle@raclinux1 RACDB_1]$ adrci ADRCI: Release 11.2.0.1.0 - Production on Mon Jan 25 15:51:01 2010 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. ADR base = "/u01/app/oracle" adrci>

Using the Automatic Diagnostic Repository (ADR)Command line utility (adrci)Help command with ADR: help and extended help adrci> help HELP [topic] Available Topics: CREATE REPORT ECHO EXIT HELP HOST IPS PURGE RUN SET BASE SET BROWSER SET CONTROL SET ECHO SET EDITOR SET HOMES | HOME | HOMEPATH SET TERMOUT SHOW ALERT SHOW BASE SHOW CONTROL SHOW HM_RUN SHOW HOMES | HOME | HOMEPATH SHOW INCDIR SHOW INCIDENT SHOW PROBLEM SHOW REPORT SHOW TRACEFILE SPOOL There are other commands intended to be used directly by Oracle, type "HELP EXTENDED" to see the listadrci>

Using extended help with Oracle 11g R2 ADR[oracle@raclinux1 ~]$ adrciadrci> help extended HELP [topic] Available Topics: BEGIN BACKUP CD CREATE STAGING XMLSCHEMA CREATE VIEW DDE DEFINE DELETE DESCRIBE DROP VIEW END BACKUP INSERT LIST DEFINE MERGE ALERT MERGE FILE MIGRATE QUERY REPAIR SELECT SET COLUMN SHOW CATALOG SHOW DUMP SHOW SECTION SHOW TRACE SHOW TRACEMAP SWEEP UNDEFINE UPDATE VIEWadrci>

Viewing alert.log and trace files with ADR for Oracle 11g R2Using ADR to view alert.logUse the following command with ADR: show alert[oracle@raclinux1 ~]$ adrciADRCI: Release 11.2.0.1.0 - Production on Fri Jan 29 18:10:28 2010Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.ADR base = "/u01/app/oracle"adrci> show alertChoose the alert log from the following homes to view:1: diag/asm/+asm/+ASM12: diag/tnslsnr/raclinux1/listener_scan13: diag/tnslsnr/raclinux1/listener4: diag/rdbms/rac11g/RAC11G5: diag/rdbms/rac11gr1/RAC11gR16: diag/rdbms/rac11gr1/RAC11GR17: diag/rdbms/racdb/RACDB_1Q: to quitPlease select option: 7

Viewing trace files with ADR for Oracle 11g R2 Use the show tracefile command for ADR with Oracle 11gadrci> show tracefile diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4592.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_10352.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_11557.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_5887.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_2678.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_22084.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_16417.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ping_30558.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25283.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4914.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25584.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_qmnc_7024.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_15058.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25409.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_18403.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23833.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23604.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_30828.trcadrci>

Incident Reporting with ADR for Oracle 11g R2ADR uses incidents for critical errors with Oracle 11g R2These incidents can be packaged by ADR and sent to Oracle support for analysisUseful new feature with Oracle 11g for problem resolution and reportingPerform many of the tasks used by the RDA (Remote Diagnostic Assistant) for releases prior to Oracle 11g.

Show incident with ADR and Oracle 11g R2Critical error appear as incidents. You can use show incident command for ADR with Oracle 11gadrci> set homepath diag/rdbms/racdb/RACDB_1adrci> show incidentADR Home = /u01/app/oracle/diag/rdbms/racdb/RACDB_1:*************************************************************************INCIDENT_ID PROBLEM_KEY CREATE_TIME-----------------------------------------------------------------------------------------------------------------------11009 ORA 600 [kssadpm: null parent] 2010-01-15 19:08:30.464549 -04:001 rows fetched

Show incident detail for ADR and Oracle 11g R2Detail view for incidents with ADR and Oracle 11gadrci> show incident –mode detail –p “incident_id=11009” ADR Home = /u01/app/oracle/diag/rdbms/ora11g/ORA11G:***********************************************************************************************************************************INCIDENT INFO RECORD 1**********************************************************INCIDENT_ID 11009STATUS readyCREATE_TIME 2010-01-15 19:08:30.464549 -04:00PROBLEM_ID 1

Packaging Incidents for Oracle SupportADR provide the following commands for managing incidents:IPS CREATE PACKAGE for new incident with Oracle 11gIPS ADD INCIDENT <insident_id> package <package_id>IPS GENERATE PACKAGE <package_id> in <directory>This procedure creates a ZIP file with the trace diagnostics for Oracle support to analyze for critical errors.Use the IPS SHOW CONFIGURATION command to check configuration for ADR incidents.

Example: Incident packages for ADR and Oracle 11g R2First we need to create a new package for the incident:adrci> ips create packageCreated package 1 without any contents, correlation level typicalThen we need to add the incident to the new package for ADR:adrci> ips add incident 11009 package 1;Added incident 11009 to package 1Finally, we need to generate the package with data:adrci> ips generate package 1 in /u01/app/oracle/adrci/supportGenerated package 1 in file /u01/app/oracle/adrci/support/IPSPKG_20080817214731_COM_1.zip, mode completeThe new file is generated as a zip file in the directory for support

Performance toolkit for Oracle 11g R2Every DBA should use these tools for tuningOracle!AutotraceSQL traceTKPROFDBMS_STATSDBMS_SQLTUNE packageORADEBUG

Using Autotrace with Oracle 11g R2Autotrace useful quick and dirty tool for tracing with Oracle 11gProvides many tracing and tuning options:Explain planCosts associated with SQL queriesTuning statisticsUsed with SQL*Plus interfaceExcellent starting point for tuning SQL queries.

Using Autotrace with Oracle 11g R2Use the set autotrace on command in SQL*Plus to enable Autotrace and set autotrace off to disable autotrace tool.Syntax for autotrace:SQL> set autotraceUsage: SET AUTOT[RACE] {OFF | ON | TRACE[ONLY]} [EXP[LAIN]] [STAT[ISTICS]]SQL>

Autotrace Example for Oracle 11g R2SQL> set autotrace onSQL> select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99;

Autotrace with Oracle 11g R252 rows selected.Execution Plan----------------------------------------------------------Plan hash value: 1954719464------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 3 | 93 | 3 (0)| 00:00:01 ||* 1 | TABLE ACCESS FULL| PRODUCTS | 3 | 93 | 3 (0)| 00:00:01 |------------------------------------------------------------------------------Predicate Information (identified by operation id):--------------------------------------------------- 1 - filter("PROD_LIST_PRICE"<69.99 AND "PROD_LIST_PRICE">7.99)Statistics---------------------------------------------------------- 7880 recursive calls 0 db block gets 1769 consistent gets 63 physical reads 0 redo size 2555 bytes sent via SQL*Net to client 452 bytes received via SQL*Net from client 5 SQL*Net roundtrips to/from client 88 sorts (memory) 0 sorts (disk) 52 rows processedSQL>

How to use SQL trace with Oracle 11g R2SQL Trace is powerful tool for the serious Oracle Performance expertEvent based: 10046 (SQL Trace), 10053 ( Cost based Optimizer – CBO)Can be set at instance or session levelUse DBMS_SUPPORT, DBMS_SYSTEM, DBMS_SESSION, ORADEBUG, or SQL*PLUS commands to enable tracing activities with Oracle 11g R2.

PL/SQL packages for tracing in 11gR2SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => TRUE);SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => FALSE);SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>TRUE);SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>FALSE);SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>8, nm=>' ');SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>0, nm=>' ');-- All versions, requires DBMS_SUPPORT package to be loaded.SQL> EXEC DBMS_SUPPORT.start_trace(waits=>TRUE, binds=>FALSE);SQL> EXEC DBMS_SUPPORT.stop_trace;SQL> EXEC DBMS_SUPPORT.start_trace_in_session(sid=>123, serial=>1234, waits=>TRUE, binds=>FALSE);SQL> EXEC DBMS_SUPPORT.stop_trace_in_session(sid=>123, serial=>1234);The DBMS_SUPPORT package is not present by default, but can be loaded as the SYS user by executing the @$ORACLE_HOME/rdbms/admin/dbmssupp.sql script.For methods that require tracing levels the following are valid values:0 - No trace. Like switching sql_trace off.2 - The equivalent of regular sql_trace.4 - The same as 2, but with the addition of bind variable values.8 - The same as 2, but with the addition of wait events.12 - The same as 2, but with both bind variable values and wait events.

How to perform SQL trace for Oracle 11g R2Oracle Metalink Note 199081.1 provides good overview for SQL traceUsing SQL*PLUS commands for SQL trace at session/instance levels:alter system/session set sql_trace=true;alter system/session set timed_statistcs=true;Optional for SQL timing tracing detailsALTER SYSTEM/SESSION SET EVENTS’10046 trace name context forever, level <level>’;Output SQL trace files to user_dump_dest as .trc fileSHOW PARAMETER USER_DUMP_DEST

SQL trace example for Oracle 11g R2Should use SYS enabled account to run the SQL trace.Need to have explain plan enabled (i.e.: utlxplan.sql)Enable parameters for SQL tracing: timed_statistics, sql_traceRecommended use ALTER SESSION as ALTER SYSTEM can place load on Oracle 11g system and impact performanceSQL> alter session set timed_statistics=true;Session altered.SQL> alter session set sql_trace=true;Session altered.SQL> show parameter user_dump_destNAME TYPE VALUE-----------------------------------------------------------------------------user_dump_dest string /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace

Contents of SQL trace fileUse the ALTER SESSION SET TRACEFILE_IDENTIFIER command to name trace file for easy identificationSQL> alter session set tracefile_identifire=‘products_sqltune’;Session alteredSQL> show parameter tracefile_identifiercontent of sample SQL Trace file=====================PARSING IN CURSOR #2 len=109 dep=0 uid=88 oct=3 lid=88 tim=1264443780269154 hv=1524029202 ad='8e65be74' sqlid='gkc6r0xdddpsk'select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99END OF STMTPARSE #2:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269152EXEC #2:c=0,e=24,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269240FETCH #2:c=0,e=45,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=1954719464,tim=1264443780269346FETCH #2:c=0,e=19,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269523FETCH #2:c=0,e=26,p=0,cr=2,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269889FETCH #2:c=0,e=16,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780270211FETCH #2:c=0,e=14,p=0,cr=1,cu=0,mis=0,r=6,dep=0,og=1,plh=1954719464,tim=1264443780270463*** 2010-01-25 20:23:01.766CLOSE #2:c=0,e=13,dep=0,type=3,tim=1264443781766794=====================

Identifying the trace file nameTRACEFILE_IDENTIFIER - Another useful parameter Many a times you need to enable tracing at a session level. And when you do that, you have to make a little bit of effort in digging out your session's trace file. Why? because the trace file is created in the user_dump_dest directory and there would be so many other trace files, and all of them would have similar naming convention "SID_ora_nnnn.trc". However with the help of the parameter "TRACEFILE_IDENTIFIER", you can easily distinguish your trace file. Very useful specifically when you want to use trace analyzer etc. Here is how?1. Set the tracefile identifier as you want, this will be appended to trace file name.alter session set tracefile_identifier=‘filename_sql_trace';2. Enable your session level tracingalter session set sql_trace=true;alter session set events '10046 trace name context forever, level 12' ;3. Execute your queries/statements4. Turn-off tracing alter session set events '10046 trace name context off';alter session set sql_trace=false;5. Check the user_dump_dest folder and you will find a file name "SID_ora_nnnn_filname_sql_trace.trcSee now it's so easy to identify. Having said that you can still find out the trace file name without using TRACEFILE_IDENTIFIER parameter using the following SQL but when Oracle has provided an easier method, why not use it?-- sql to find out trace file name for your session.select c.value || '/' || d.instance_name ||'_ora_' || a.spid || '.trc' tracefrom v$process a, v$session b, v$parameter c, v$instance dwhere a.addr = b.paddrand b.audsid = userenv('sessionid')and c.name = 'user_dump_dest'/

What is TKPROF utilityTKPROF is a trace kernel profiler – useful for formatting trace files for tuning purposesFlexible options for performance report statisticsCleans up the raw SQL trace files into human readable tuning reports.Oracle Metalink Note: 41634.1 TKPROF Simplistic overview

$How to use TKPROFTKPROF takes the trace file and output report file plus performance specific parametersMany parameter options for using tkprofSyntax for tkprof:tkprof trace_file.trc output_file.rpt [waits={yes|no}] [sort={yes|no}] [print=option] [aggregate={yes|no}][insert={filename}] [sys={yes|no}][table={schema.table}] [explain={schema/password}] [record=filename] [width=n]$

Getting help with TKPROFTKPROF Help :[oracle@raclinux1 ~]$ tkprofUsage: tkprof tracefile outputfile [explain= ] [table= ] [print= ] [insert= ] [sys= ] [sort= ] table=schema.tablename Use 'schema.tablename' with 'explain=' option. explain=user/password Connect to ORACLE and issue EXPLAIN PLAN. print=integer List only the first 'integer' SQL statements. aggregate=yes|no insert=filename List SQL statements and data inside INSERT statements. sys=no TKPROF does not list SQL statements run as user SYS. record=filename Record non-recursive statements found in the trace file. waits=yes|no Record summary for any wait events found in the trace file. sort=option Set of zero or more of the following sort options: prscnt number of times parse was called prscpu cpu time parsing prsela elapsed time parsing prsdsk number of disk reads during parse prsqry number of buffers for consistent read during parse prscu number of buffers for current read during parse prsmis number of misses in library cache during parse execnt number of execute was called execpu cpu time spent executing exeela elapsed time executing exedsk number of disk reads during execute exeqry number of buffers for consistent read during execute execu number of buffers for current read during execute exerow number of rows processed during execute exemis number of library cache misses during execute fchcnt number of times fetch was called fchcpu cpu time spent fetching fchela elapsed time fetching fchdsk number of disk reads during fetch fchqry number of buffers for consistent read during fetch fchcu number of buffers for current read during fetch fchrow number of rows fetched userid userid of user that parsed the cursor[oracle@raclinux1 ~]$

Formatting SQL trace file with TKPROFExample[oracle@raclinux1 trace]$ tkprof RACDB1_ora_9163_products_sqltune.trc tune_sql.rpt waits=yes aggregate=yes sys=yes explain=sys/oracle

Output report for SQL Trace with TKPROF********************************************************************************SQL ID: gkc6r0xdddpskPlan Hash: 1954719464select prod_name, prod_list_pricefrom sh.products where prod_list_price > 7.99 and prod_list_price < 69.99call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 3 0.00 0.02 0 0 0 0Execute 4 0.00 0.00 0 0 0 0Fetch 15 0.00 0.00 0 24 0 156------- ------ -------- ---------- ---------- ---------- ---------- ----------total 22 0.00 0.03 0 24 0 156Misses in library cache during parse: 0Optimizer mode: ALL_ROWSParsing user id: 88********************************************************************************

DBMS_SQLTUNE packageWhat is the DBMS_SQLTUNE package?Automates SQL tuning by using profiles to tune SQL.DBMS_SQLTUNE package is used as part of the SQL Tune Advisors (STA) with Oracle 11g

Using DBMS_SQLTUNE for Oracle 11g R2ExampleSQL> DECLAREsql_task_name VARCHAR2(30);tune_sqltext CLOB;BEGINtune_sqltext :='select * '||'FROM sh.sales '||' where prod_id=20 and '||' cust_id=200 ';sql_task_name := DBMS_SQLTUNE.CREATE_TUNING_TASK( sql_text=>tune_sqltext,User_name=>'SH',Scope=>'COMPREHENSIVE',Time_limit=>60,Task_name=>'TUNE_SH_TASK',Description=>'New task for tuning SH query');END;/ 2 3 4 5 6 7 8 9 10 11 12 13 14 PL/SQL procedure successfully completed.

Using DBMS_SQLTUNE for Oracle 11g R2 (Continued)DBMS_SQLTUNE uses the CREATE_TUNING_TASK function as part of SQL Tuning Advisor (STA) with 11gOnce we have created the tuning task, we need to execute it with the DBMS_SQLTUNE.EXECUTE_TUNING_TASK:SQL> beginDBMS_SQLTUNE.EXECUTE_TUNING_TASK(task_name=>'TUNE_SH_TASK');End;/ 2 3 4 PL/SQL procedure successfully completed.SQL>

Viewing SQL tuning Advisor Report for 11g R2To view SQL Tuning report query the DBMS_SQLTUNE.REPORT_TUNING_TASK;SQL> select dbms_sqltune.report_tuning_task(‘TUNE_SH_TASK’) drom DUAL;SQL> /DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')--------------------------------------------------------------------------------GENERAL INFORMATION SECTION-------------------------------------------------------------------------------Tuning Task Name : TUNE_SH_TASKTuning Task Owner : SYSWorkload Type : Single SQL StatementExecution Count : 2Current Execution : EXEC_86Execution Type : TUNE SQLScope : COMPREHENSIVETime Limit(seconds): 60Completion Status : COMPLETEDDBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')--------------------------------------------------------------------------------Started at : 01/26/2010 16:34:26Completed at : 01/26/2010 16:34:27-------------------------------------------------------------------------------Schema Name: SHSQL ID : 8z3nbpj9f0nsaSQL Text : select * FROM sh.sales where prod_id=20 and cust_id=200-------------------------------------------------------------------------------FINDINGS SECTION (1 finding)-------------------------------------------------------------------------------DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK')--------------------------------------------------------------------------------1- Index Finding (see explain plans section below)-------------------------------------------------- The execution plan of this statement can be improved by creating one or more indices. Recommendation (estimated benefit: 93.13%) ------------------------------------------ - Consider running the Access Advisor to improve the physical schema design or creating the recommended index. If you choose to create the recommended index, consider dropping the index "SH"."SALES_CUST_BIX”

Introducing OradebugWhat is Oradebug?Why should you use it?Performance TuningHang AnalysisDump Analysis for Oracle 11g MemorySuspend and Resume Oracle processes

Inside OradebugOradebug introduced in version 7.X release of OraclePoorly documented utilityRecommend caution when using oradebugSyntax for Oradebug

Help option for OradebugOradebug has MANY options!!!Use the oradebug help command to list all the main options available:SQL> oradebug helpHELP [command] Describe one or all commandsSETMYPID Debug current processSETOSPID <ospid> Set OS pid of process to debugSETORAPID <orapid> ['force'] Set Oracle pid of process to debugSETORAPNAME <orapname> Set Oracle process name to debugSHORT_STACK Get abridged OS stackCURRENT_SQL Get current SQLDUMP <dump_name> <lvl> [addr] Invoke named dumpDUMPSGA [bytes] Dump fixed SGADUMPLIST Print a list of available dumpsEVENT <text> Set trace event in processSESSION_EVENT <text> Set trace event in sessionDUMPVAR <p|s|uga> <name> [level] Print/dump a fixed PGA/SGA/UGA variableDUMPTYPE <address> <type> <count> Print/dump an address with type infoSETVAR <p|s|uga> <name> <value> Modify a fixed PGA/SGA/UGA variablePEEK <addr> <len> [level] Print/Dump memoryPOKE <addr> <len> <value> Modify memoryWAKEUP <orapid> Wake up Oracle processSUSPEND Suspend executionRESUME Resume executionFLUSH Flush pending writes to trace fileCLOSE_TRACE Close trace fileTRACEFILE_NAME Get name of trace fileLKDEBUG Invoke global enqueue service debuggerNSDBX Invoke CGS name-service debugger-G <Inst-List | def | all> Parallel oradebug command prefix-R <Inst-List | def | all> Parallel oradebug prefix (return outputSETINST <instance# .. | all> Set instance list in double quotesSGATOFILE <SGA dump dir> Dump SGA to file; dirname in double quotesDMPCOWSGA <SGA dump dir> Dump & map SGA as COW; dirname in double quotesMAPCOWSGA <SGA dump dir> Map SGA as COW; dirname in double quotesHANGANALYZE [level] [syslevel] Analyze system hangFFBEGIN Flash Freeze the InstanceFFDEREGISTER FF deregister instance from clusterFFTERMINST Call exit and terminate instanceFFRESUMEINST Resume the flash frozen instanceFFSTATUS Flash freeze status of instanceSKDSTTPCS <ifname> <ofname> Helps translate PCs to namesWATCH <address> <len> <self|exist|all|target> Watch a region of memoryDELETE <local|global|target> watchpoint <id> Delete a watchpointSHOW <local|global|target> watchpoints Show watchpointsDIRECT_ACCESS <set/enable/disable command | select query> Fixed table accessCORE Dump core without crashing processIPC Dump ipc informationUNLIMIT Unlimit the size of the trace filePROCSTAT Dump process statisticsCALL [-t count] <func> [arg1]...[argn] Invoke function with argumentsSQL>

How to use oradebug – Tracing a sessionFirst we need to obtain SPID from v$SESSION and V$PROCESSSQL> select username, sid, serial#, paddr from v$session where username='SH';USERNAME SID SERIAL# PADDR------------------------------ ---------- ---------- --------SH 153 3370 994694F4SQL> select addr, pid , spid from v$process where addr='994694F4';ADDR PID SPID-------- ---------- ------------------------994694F4 35 20933SQL>

Tracing a session with OradebugThen we use the SPID in oradebug to trace session:SQL> oradebug setospid 20933Oracle pid: 35, Unix process pid: 20933, image: oracle@raclinux1.gj.com (TNS V1-V3)SQL> We then set the unlimit parameter for the trace file with Oradebug:SQL> oradebug unlimitStatement processed.SQL>Now we are ready to start the trace for our session with OradebugSQL> oradebug event 10046 trace name context forever, level 4Statement processed.SQL>

Tracing SQL for session with oradebugOradebug uses Events for tracing optionsSome common trace events:10053 – Cost Based Optimizer (CBO) trace10046 – SQL TraceTo start tracing with OradebugSQL> oradebug event 10046 trace name context forever, level 4Statement processed.SQL> To stop and turn off tracing for Oradebug:SQL> oradebug event 10046 trace name context offStatement processed.

Locate the session trace file for OradebugTrace file output based on user_dump_dest parameterNaming convention for the trace filesid_ora_pid_traceid.trcEasy way find trace file is to set trace_identifier parameter:alter session set tracefile_identifier=‘mytrace’;Useful query to find trace fileSelect u_dump.value||’/’||db_name.value||’_ora_’||v$PROCESS.spid||nvl2(v$process.traceid,’_’||v$process.traceid,Null )||’.trc’ “Trace File” from v$parameter u_dump cross join v$parameter db_name cross join v$process join v$session on v$process.addr = ‘usr_dump_dest’ and db_name.name = ‘db_name’ and v$session.audsid=sys_context(‘userenv’,’sessionid’);

Viewing the oradebug trace file for user sessionLocate the trace file for oradebugFormat it with tkprofcp RACDB_1_ora_20933_test_sh_query.trc trc.trc[oracle@raclinux1 trace]$ tkprof trc.trc rpt.rpt aggregate=yes waits=yes sys=no explain=sh/shTKPROF: Release 11.2.0.1.0 - Development on Tue Jan 26 18:29:25 2010Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.[oracle@raclinux1 trace]$ Open the trace file with text editor$vi rpt.rptSQL ID: 3j4064f628jmkPlan Hash: 1954719464select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 6 0.00 0.00 0 0 0 0Execute 6 0.00 0.00 0 0 0 0Fetch 30 0.00 0.00 0 48 0 312------- ------ -------- ---------- ---------- ---------- ---------- ----------total 42 0.00 0.00 0 48 0 312Misses in library cache during parse: 1Optimizer mode: ALL_ROWSParsing user id: 88 (SH)Rows Row Source Operation------- --------------------------------------------------- 52 TABLE ACCESS FULL PRODUCTS (cr=8 pr=0 pw=0 time=0 us cost=3 size=1820 card=52)Rows Execution Plan------- --------------------------------------------------- 0 SELECT STATEMENT MODE: ALL_ROWS 52 TABLE ACCESS MODE: ANALYZED (FULL) OF 'PRODUCTS' (TABLE)

Using oradebug for Hang AnalysisOradebug is excellent tool for solving database hang issue.Use the HANGANALYZE option with oradebugSyntax:oradebug hanganalyze <level>For Oracle RAC environment the syntax is:oradebug –g all hanganalyze <level>Level indicates amount of details to collect for a hang.Refer to Metalink Note: 175006.1 and Note:215858.1

Oracle Database Hang analysis with OradebugThe debug levels defined for HANGANALYZE with oradebug are as follows Note 215858.1: Level 10: Dump all process ( IGN state)Level5: Level 4 + Dump all processes involved with chain (NLEAF)Level 4: Level 3 + Dump leaf nodes (blocker) in wait chains (LEAF, LEAF_NW, ING_DMP state)Level 3: Level 2 + Dump only processes considered in hang condition (IN_HANG state)Levels 1-2: Generate ONLY HANGANALYZE output with no process dump

Hang Analysis Example for Oracle 11g R2Set environment for hang analysis:SQL> oradebug setmypidStatement process.SQL> oradebug unlimitStatement processed.Perform Hang Analysis:SQL> oradebug -g all hanganalyze 10Hang Analysis Dumps in /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_diag_6112.trc SQL>exit

Review Hang Analysis Oracle 11g R2Review output hang analysis from trace file:Trace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_diag_6112.trcOracle Database 11g Enterprise Edition Release 11.2.0.1.0 - ProductionWith the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,Data Mining, Oracle Database Vault and Real Application Testing optionsORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1System name: LinuxNode name: raclinux1.gj.comRelease: 2.6.18-164.el5PAEVersion: #1 SMP Thu Sep 3 02:28:20 EDT 2009Machine: i686Instance name: RACDB_1Redo thread mounted by this instance: 0 <none>Oracle process number: 5Unix process pid: 6112, image: oracle@raclinux1.gj.com (DIAG)*** 2010-01-26 13:02:55.529*** SESSION ID:(127.1) 2010-01-26 13:02:55.529*** CLIENT ID:() 2010-01-26 13:02:55.529*** SERVICE NAME:() 2010-01-26 13:02:55.529*** MODULE NAME:() 2010-01-26 13:02:55.529*** ACTION NAME:() 2010-01-26 13:02:55.529Node id: 0List of nodes: 0, *** 2010-01-26 13:02:55.565Reconfiguration starts [incarn=0]*** 2010-01-26 13:02:55.566I'm the master node*** 2010-01-26 13:02:55.566Reconfiguration completes [incarn=1]Group reconfiguration cleanup

Dump Analysis for Oracle 11g R2 Memory and eventsUse oradebug to dump contents of UGA and PGA memoryUseful for debugging memory leak issuesDump Analysis for Oracle 11g MemoryDescribe eventsTo start diagnosing a problem with the process size, such as a suspected leak,a heapdump of the offending process is required (Also helps to check with pmap –x <pid> for memory leaks): $sqlplus SQL> connect '/ as sysdba' SQL> oradebug setospid <pid> SQL> oradebug unlimit SQL> oradebug dump heapdump 5 --this dumps PGA and UGA heaps The above information should be provided to Oracle Support Services.Oracle 8i / 9i / 10gSQL> oradebug dump events <level> Level 1 - Dump session group's event settings Level 2 - Dump process group's event settings Level 4 - Dump system group's event settings (Ie the instance wide events) Oracle 10g / 11g and aboveSQL> oradebug eventdump <level> SESSION - Dump session group's event settings PROCESS - Dump process group's event settings SYSTEM - Dump system group's event settings (Ie the instance wide events) SQL> oradebug setmypidStatement processed.SQL> oradebug eventdump sessionStatement processed.SQL> oradebug eventdump systemStatement processed.SQL> alter system set events '942 trace name errorstack level 3'; System altered.SQL> oradebug eventdump system942 trace name errorstack level 3SQL> SQL> oradebug eventdump session942 trace name errorstack level 3

Dump the SGA with Oradebug for Oracle 11g R2Use the oradebug dumpsga command:SQL> oradebug setmypidStatement processed.SQL> oradebug unlimitStatement processed.SQL> oradebug dumpsgaStatement processed.SQL> oradebug tracefile_name/u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25351.trc SQL> exit

Viewing SGA memory dump for Oracle 11g R2Contents of SGA dump in oradebug trace fileTrace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25351.trcOracle Database 11g Enterprise Edition Release 11.2.0.1.0 - ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,OLAP, Data Mining, Oracle Database Vault and Real Application Testing optionsORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1System name: LinuxNode name: raclinux1.gj.comRelease: 2.6.18-164.el5PAE*** SESSION ID:(154.6347) 2010-01-26 19:14:21.438*** CLIENT ID:() 2010-01-26 19:14:21.438*** SERVICE NAME:(SYS$USERS) 2010-01-26 19:14:21.438*** MODULE NAME:(sqlplus@raclinux1.gj.com (TNS V1-V3)) 2010-01-26 19:14:21.438*** ACTION NAME:() 2010-01-26 19:14:21.438Processing Oradebug command 'setmypid'*** 2010-01-26 19:14:21.438Oradebug command 'setmypid' console output: <none>*** 2010-01-26 19:14:30.092Processing Oradebug command 'unlimit'*** 2010-01-26 19:14:30.092Oradebug command 'unlimit' console output: <none>*** 2010-01-26 19:15:11.671----- Fixed Areas Dump (level=2) ---------- Dump of the Fixed SGA -----ksmsgft ksmsgf_ [20000000, 20001000) = 00000000 00000000 00000000 00000000 ...Dump of memory from 0x20000010 to 0x2000100020000010 00000000 00000000 00000000 00000000 [................] Repeat 254 timeskywmr * kywmrsga_ [20001000, 20001004) = 990CD020

Suspend and resume runaway Oracle processeOradebug useful for stopping and restarting Oracle sessionsBe careful when using oradebug in production environments!Syntax for suspending process with oradebugoradebug suspendTo resume a process that has been suspendedoradebug resumeUseful when need to delay batch process for more critical job.

Find the session to suspendFind the session information to suspend:SQL> select s.username, s.pid, p.spidFromV$session s, v$process pWhere s.paddr=p.addrAndS.username <> ‘SYS’;USERNAME SID SPID---------------------- -------- ----------SH 140 18335

Suspend the session with oradebugOradebug suspendSQL> oradebug setospid 18335SQL> oradebug suspendStatement processed.

Resume process for Oracle 11g R2 with OradebugOradebug resumeSQL> oradebug resumeStatement processed.

Using Oradebug for Oracle RAC eventsoradebug is used to trace issues related to IPC in interconnects and RACSyntax for IPC memory dump with Oracle RAC:SQL> oradebug ipc

Oracle RAC tracing with oradebug for IPCExampleSQL> oradebug setmypidStatement processed.SQL>oradebug unlimitStatement processed.SQL>oradebugipcInformation written to trace file.SQL> oradebug trace_file /u01/app/oracle/admin/RACDB/udump/racdb1_ora_6391.trcSQL>exit

Oracle11g R2 RAC trace with oradebugReview trace file for interconnect trace with Oradebug for RAC:SSKGXPT 0xcc75e9c flags SSKGXPT_READPENDING info for network 0socket no 7 IP 10.10.10.11 UDP 2247sflagsSSKGXPT_UPinfo for network 1socket no 0 IP 0.0.0.0 UDP 0sflagsSSKGXPT_DOWNactive 0 actcnt1context timestamp 0no portssconnoacconoerttstateseq# sent asyncsyncrtransacks

Hidden Oracle 11g R2 parameterOracle 11g has many new additional hidden parameters!There are more than 2050 hidden parametrs for Oracle 11g R2!SQL> select count(*) from x$ksppi where substr(ksppinm,1,1)='_'; COUNT(*)---------- 2057SQL>Query to find out these hidden Oracle 11g R2 parameters:Select ksppinm from x$ksppi where substr(ksppinm,1,1)=‘_’;

Some new hidden parameters for Oracle 11g R2Some hidden parameters for 11g new featuresHang manager hidden parameters for Oracle 11g R2:_hm_analysis_ouput_disk_hm_analysis_oradebug_node_dump_level_hm_analysis_oradebug_sys_dump_level_hm_verification_interval_hm_log_incidentsBe careful using theseFirst change in test and sandboxBest to work with Oracle support since those are undocumented parameters!

Oracle 11g V$ and X$ tables for troubleshooting and tuning 11g R2Each release of Oracle adds many new V$ and X$ performance view tables.More than 945 different X$ tables in Oracle 11g R2.There are over 525 different v$ performance views in Oracle 11g R2 .Query to find the V$ performance views for Oracle 11g R2:Select name, type from v$fixed_table where name like ‘V$%’;Query to find the list of X$ tables for Oracle 11g R2:Select name, type from v$fixed_tablewhere name like ‘X$%’;

Useful v$ and X$ table for Oracle 11g R2Some interesting new V$ Dynamic Performance ViewsV$MEMORY_DYNAMIC_COMPONENTS - displays information about the dynamic SGA components. This view summarizes information based on all completed SGA resize operations since instance startup. All sizes are expressed in bytes. V$MEMORY_TARGET_ADVICE - information about how the MEMORY_TARGET parameter should be sized based on current sizing and satisfaction metrics. V$ALERT_TYPES – information for server alert typesV$IR_FAILURE – provides information on the failure. Note that records in this view can have parent records within this view. (DRA)V$IR_REPAIR – show the results of repair failure commands. (DRA)V$IR_MANUAL_CHECKLIST – detailed information messages related to the failure. Messages provide information on how to manually correct the problem. (DRA)V$IR_FAILURE_SET – list of advice records associated with the failure. (DRA)V$HM_CHECK - information about all the checks registered with Health Monitor. Each check is uniquely identified by a name or an ID. V$HM_CHECK_PARAM - information about the input parameters of all Health Monitor checks. V$HM_RUN - information about all Health Monitor checks and their status. All checks ran.V$HM_FINDING - information about all the findings of various Health Monitor runs. V$HM_RECOMMENDATION - information about any recommendation from Health check runs. V$HM_INFO - Information about runs, finding and recommendations. Perform a describe on each of these V$ dynamic performance views for detail information.

Some useful X$ tables for Oracle 11g R2These are some useful new X$ tables for Oracle 11g R2:X$DIAG_INFOX$IR_WORKING_FAILURE_SET – incident reportingX$IR_WORKING_REPAIR_SET – incident reportingX$IR_REPAIR_OPTION – incident reportingX$IR_MANUAL_OPTION – incident reportingX$IR_REPAIR_STEP – incident reporting

Power tools for UNIX and WindowsWhen Oracle tools are not enough, these OS tools are essential:dbx (Solaris and Linux)gdb (Linux) similar to dbx truss (Solaris)strace (Linux)orakill (Windows)STACKX (UNIX) Metalink Note: 362791.1Useful References on Metalink for core Debug Analysis:Oracle Metalink Note:16534.1 “debug Core Dumps / core files on Unix”Oracle Metalink Note: 1007808.6 “How to handle core dumps on UNIX”

DBX – Debugging Solaris and LinuxDBX is trace debugger for Solaris and also available for free download from Sun.com as part of Solaris Developer Studio.Provides comprehensive debugger level commands for Solaris and Linux OS systems calls and Oracle 11g instance and network level internal processes.DBX is an advanced complex tool with complex syntax.Complete reference guide from SUN:Sun Studio 12: Debugging a Program with dbx:http:/docs.sun.com/app/docs

Using dbx for Oracle 11g R2 TroubleshootingDBX can be used to trace Oracle 11g processesGetting help with dbx[root@raclinux1 bin]# ./dbxFor information about new features see `help changes'To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc(dbx) helpdbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8'dbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8'command summaryUse `commands' to see a command summary consisting of one-linedescriptions of each dbx command.Execution and Tracing cancel catch clear cont delete fix fixed handler ignore intercept next pop replay rerun restore run runargs save status step stop trace unintercept when whocatches Displaying and Naming Data assign call dis display down dump examine exists fortran_modules frame hide print undisplay unhide up watch whatis where whereami whereis which Accessing Source Files bsearch cd edit file files func funcs line list loadobject loadobjects module modules pathmap pwd searchDebugging Multiple Threads lwp lwps sync syncs thread threads

Using dbx for Oracle 11g R2Find the PID for Oracle process[root@raclinux1 bin]# ps -ef | grep oracleRACDBoracle 9546 9545 0 18:26 ? 00:00:00 oracleRACDB_1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))root 15103 12453 0 20:02 pts/3 00:00:00 grep oracleRACDB[root@raclinux1 bin]# Now we need to get a trace for Oracle[root@raclinux1 bin]# ./dbx /u01/app/oracle/product/11.2.0/db_1/bin/oracle 9546For information about new features see `help changes'To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrcReading oracleReading ld-linux.so.2……………………………Reading libnque11.soAttached to process 9546t@3671744 (l@9546) stopped in (unknown) at 0xdcd4100x00dcd410: popl %ebp(dbx) Enable Collection Process and continue trace - The collector command collects performance data for analysis by the Performance Analyzer. It is valid only in native mode. (dbx) collector enableBack to our DBX trace session:(dbx) cont

Continued – Tracing Oracle 11g R2 with dbxUse the dbx thread command to look at Oracle 11g R2 memory process stack. Use the dbx thread command. Displays the current thread. (dbx) thread t@3671744 t@3671744 (l@9546) stopped in (unknown) at 0xdcd4100x00dcd410: popl %ebp(dbx) wherecurrent thread: t@3671744 [1] 0xdcd410(0x2010, 0x108ba596, 0x7cb763, 0x105d4874, 0xf4351a0, 0x9), at 0xdcd410 [2] __read_nocancel(0x108b1108, 0x108b1770, 0x108ae8f8, 0x108ba596, 0xbfaa3d10, 0x0), at 0x7cb763 [3] ntpfprd(0x108ae790, 0x108ba596, 0x2010, 0xbfaa3e24, 0x8), at 0xf4350d9 [4] nsbasic_brc(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41b0b7 [5] nsbrecv(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41dba6 [6] nioqrc(0x106dad04, 0x0, 0xbfaa5f0c, 0x1, 0x0, 0x1), at 0xf422429 [7] __PGOSF25_opikndf2(0x106dad04, 0x2, 0xbfaa5f0c, 0x1, 0x0), at 0xf1c88e6 [8] opitsk(0x0, 0x0), at 0x8f3b9fb [9] opiino(0x3c, 0x4, 0xbfaa6e14), at 0x8f3f89b [10] opiodr(0x3c, 0x4, 0xbfaa6e14, 0x0), at 0xf1cb036 [11] opidrv(0x3c, 0x4, 0xbfaa6e14), at 0x8f382f3 [12] sou2o(0xbfaa6df8, 0x3c, 0x4, 0xbfaa6e14), at 0x92ed66c [13] opimai_real(0x2, 0xbfaa6f48, 0x108721a8, 0x0, 0x0, 0x651f14), at 0x8542cdb [14] ssthrdmain(0x2, 0xbfaa6f48), at 0x92f1f2d [15] main(0x2, 0xbfaa7044, 0xbfaa7050, 0x640810, 0x0, 0x1), at 0x8542c57 (dbx) (dbx) thread -info Thread t@3671744 (0x0) at priority 0 state: active on l@9546 base function: 0x0: 0x00000000() stack: 0x0[0] flags: (none) masked signals: (none) Currently active at 0x00dcd410(dbx)

Debug Oracle 11g R2 with dbxThe debug function with dbx is useful for Oracle 11g R2(debug lists the program being debugged) :(dbx) debug –fDebugging: oracle ( process id 24976)

Truss: Solaris powerful tool for tracing Oracle 11g R2Truss provides ability to walk through trace at step by step level for Oracle 11g internal processSyntax:truss [-fcaeil ] [-[tvx] [!]syscall..] [-s [!] signal..] [-m [!]fault..] [-[rw] [!]fd..] [-o outfile] command | -p pidTypical options for truss:-o output to file-f trace child processes-c count system calls-p trace calls based on Unix pid

How to use truss with Solaris for Oracle 11g R2Examplesolaris02$truss –cp 3316^CSyscall seconds calls errorsRead .000 2Write .000 2Times .000 33Yield .000 226

Strace tool for Linux and Oracle 11g R2 tracingStrace is similar to truss for Linux O/S tracing with Oracle 11gPowerful tool with command line syntax:usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file] [-p pid] ... [-s strsize] [-u username] [-E var=val] ... [command [arg ...]] or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ... [command [arg ...]]

How to use strace with Linux and Oracle 11g R2Options available for strace[root@raclinux1 ~]# straceusage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file] [-p pid] ... [-s strsize] [-u username] [-E var=val] ... [command [arg ...]] or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ... [command [arg ...]]-c -- count time, calls, and errors for each syscall and report summary-f -- follow forks, -ff -- with output into separate files-F -- attempt to follow vforks, -h -- print help message-i -- print instruction pointer at time of syscall-q -- suppress messages about attaching, detaching, etc.-r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs-T -- print time spent in each syscall, -V -- print version-v -- verbose mode: print unabbreviated argv, stat, termio[s], etc. args-x -- print non-ascii strings in hex, -xx -- print all strings in hex-a column -- alignment COLUMN for printing syscall results (default 40)-e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]... options: trace, abbrev, verbose, raw, signal, read, or write-o file -- send trace output to FILE instead of stderr-O overhead -- set overhead for tracing syscalls to OVERHEAD usecs-p pid -- trace process with process id PID, may be repeated-s strsize -- limit length of print strings to STRSIZE chars (default 32)-S sortby -- sort syscall counts by: time, calls, name, nothing (default time)-u username -- run command as username handling setuid and/or setgid-E var=val -- put var=val in the environment for command-E var -- remove var from the environment for command[root@raclinux1 ~]#

$Example for strace for Oracle 11g R2Perform system call trace for Oracle 11g:[oracle@raclinux1 ~]$ strace oracle execve("/u01/app/oracle/product/11.2.0/db_1/bin/oracle", ["oracle"], [/* 36 vars */]) = 0brk(0) = 0x10792000mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x8b9000access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory)stat64("/u01/app/oracle/product/11.2.0/db_1/lib/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory)open("/u01/app/oracle/product/11.2.0/db_1/lib/libodm11.so", O_RDONLY) = 3read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\254\6\0\0004\0\0\0"..., 512) = 512fstat64(3, {st_mode=S_IFREG|0644, st_size=8857, ...}) = 0mmap2(NULL, 6768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40d000mmap2(0x40e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x40e000close(3) = 0open("/u01/app/oracle/product/11.2.0/db_1/lib/libcell11.so", O_RDONLY) = 3read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340|\0\0004\0\0\0"..., 512) = 512fstat64(3, {st_mode=S_IFREG|0644, st_size=383547, ...}) = 0$

Tracing System calls with strace with Oracle 11g R2Similar to truss:Check for system call errors for Oracle 11g with strace[oracle@raclinux1 ~]$ strace -c oracle% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 40.53 0.000308 308 1 execve 15.79 0.000120 0 338 49 open 15.13 0.000115 7 16 16 mkdir 12.37 0.000094 1 75 read 8.29 0.000063 0 132 munmap 7.89 0.000060 0 293 close 0.00 0.000000 0 20 write 0.00 0.000000 0 1 chmod 0.00 0.000000 0 3 lseek 0.00 0.000000 0 3 3 access 0.00 0.000000 0 1 dup 0.00 0.000000 0 4 times 0.00 0.000000 0 4 brk 0.00 0.000000 0 2 setrlimit 0.00 0.000000 0 5 gettimeofday 0.00 0.000000 0 5 uname 0.00 0.000000 0 9 mprotect 0.00 0.000000 0 3 _llseek 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 5 rt_sigprocmask 0.00 0.000000 0 5 getrlimit 0.00 0.000000 0 307 mmap2 0.00 0.000000 0 48 19 stat64 0.00 0.000000 0 7 2 lstat64 0.00 0.000000 0 26 fstat64 0.00 0.000000 0 2 getuid32 0.00 0.000000 0 1 geteuid32 0.00 0.000000 0 6 fcntl64 0.00 0.000000 0 1 gettid 0.00 0.000000 0 2 futex 0.00 0.000000 0 1 set_thread_area 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 3 socket 0.00 0.000000 0 1 bind 0.00 0.000000 0 2 2 connect 0.00 0.000000 0 5 shmat 0.00 0.000000 0 5 shmdt 0.00 0.000000 0 12 9 shmget------ ----------- ----------- --------- --------- ----------------100.00 0.000760 1362 100 total[oracle@raclinux1 ~]$

Tracing system calls for Oracle 11g R2 DBWRExample : using strace to find system calls for Oracle DBWR processoracle 6123 1 0 12:16 ? 00:00:01 ora_dbw0_RACDB_1oracle 31050 8191 0 20:24 pts/2 00:00:00 grep dbw[oracle@raclinux1 admin]$ strace -c -p 6123Process 6123 attached - interrupt to quitProcess 6123 detached% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 57.30 0.049795 732 68 io_submit 35.52 0.030866 123 250 pread64 3.14 0.002729 0 20064 getrusage 1.55 0.001350 11 123 io_getevents 1.15 0.000999 4 235 125 semtimedop 0.34 0.000297 2 151 semctl 0.32 0.000279 0 17913 gettimeofday 0.25 0.000219 4 58 mmap2 0.25 0.000214 4 50 sendmsg 0.10 0.000087 2 57 open 0.08 0.000068 1 57 read 0.00 0.000000 0 57 close 0.00 0.000000 0 766 times------ ----------- ----------- --------- --------- ----------------100.00 0.086903 39849 125 total[oracle@raclinux1 admin]$

Using orakill in Windows and Oracle 11g R2Useful tool available for Oracle 11g on Windows called orakillOracle Metalink Note: 69882.1Useful for killing runaway processes on Oracle 11g on Windows as well as for zombie and hung sessionsSyntax:orakill <sid> ,<thread>Can use QSLICE.exe to find threads ( available on Windows NT/200X Resource kit from Microsoft)

$How to use orakill for windows and Oracle 11g R2Query if not using QSLICE.EXE for details:C:\Users\User>orakillUsage: orakill sid thread where sid = the Oracle instance to target thread = the thread id of the thread to kill The thread id should be retrieved from the spid column of a query such as: select spid, osuser, s.program from v$process p, v$session s where p.addr=s.paddr;C:\Users\User>$

Example: Using orakill for Windows and Oracle 11g R2Qslise.exe outputUse Qslice to find Oracle processesThan use orakill to end process that is hungC:> orakill ora11g 985Kill of thread id 985 in instance ora11g successfully signaled.

ORA-0600: Tips and TricksORA-0600: Internal errorsCall Oracle support firstLeverage Metalink ORA-0600 support matrixCollect dumps with oradebug if possible

What is ORA-0600 error ?Internal Oracle errors generate kernel memory dump.Dump core file and trace files.Useful Metalink Notes: 146580.1, 146581.1Use the Oracle Metalink ORA-0600 lookup tool for addressing these internal errors. Note 153788.1 – Troubleshoot an ORA-0600 Error using the ORA-600 Argument lookup tool

Using the ORA-0600 lookup toolFrom Oracle support see Note 153788.1Troubleshoot an ORA-600 Error using the ORA-600 argument lookup toolCan also be used to review and analyze ORA-7445 errors

Resolving ORA-0600 errorsWork closely with Oracle SupportUsually caused by bug such as memory leakTest recent changes between development, test, QA and production to ensure continuity of issue.Ideal to have same release of Oracle to be able to reproduce the error for all environments and platforms.

ORA-0600 troubleshootingUse ADR to review incident and associated trace fileDump continued from file: /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora_12562.trcORA-00600: internal error code, arguments: [kssadpm: null parent], [], [], [], [], [], [], []Plug in the arguments from trace file to ORA-0600 Metalink troubleshooting lookup tool.The related solution will then be displayed:Known Issues: Bug# 5605370 * See Note 5605370.8* This bug is alerted in Note 454464.1Various dumps / instance crash possible Fixed: 10.2.0.4, 11.1.0.6

KFOD Utility in 11g R2KFOD for ASM monitoringKFOD help=y[oracle@raclinux1 ~]$ kfod help=y_asm_a/llow_only_raw_disks KFOD allow only raw devices [_asm_allow_only_raw_disks=TRUE/(FALSE)]_asm_l/ibraries ASM Libraries[_asm_libraries=lib1,lib2,...]_asms/id ASM Instance[_asmsid=sid]a/sm_diskstring ASM Diskstring [asm_diskstring=discoverystring, discoverystring ...]c/luster KFOD cluster [cluster=TRUE/(FALSE)]db/_unique_name db_unique_name for ASM instance[db_unique_name=dbname]di/sks Disks to discover [disks=raw,asm,all]ds/cvgroup Include group name [dscvgroup=TRUE/(FALSE)]g/roup Disks in diskgroup [group=diskgroup]h/ostlist hostlist[hostlist=host1,host2,...]metadata_a/usize AU Size for Metadata Size Calculationmetadata_c/lients Client Count for Metadata Size Calculationmetadata_d/isks Disk Count for Metadata Size Calculationmetadata_n/odes Node Count for Metadata Size Calculationmetadata_r/edundancy Redundancy for Metadata Size Calculationn/ohdr KFOD header suppression [nohdr=TRUE/(FALSE)]o/p KFOD options type [OP=DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/CLIENTS/RM/RMVERS/DFLTDSTR/GPNPDSTR/METADATA/ALL]p/file ASM parameter file [pfile=parameterfile]s/tatus Include disk header status [status=TRUE/(FALSE)]v/erbose KFOD verbose errors [verbose=TRUE/(FALSE)]KFOD-01000: USAGE: kfod op=<op> asm_diskstring=... | pfile=...

KFOD utility 11g R2[oracle@raclinux1 ~]$ kfod disk=all-------------------------------------------------------------------------------- Disk Size Path User Group ================================================================================ 1: 15358 Mb /dev/sdb1 oracle oinstall 2: 15358 Mb /dev/sdc1 oracle oinstall 3: 15358 Mb /dev/sdd1 oracle oinstall 4: 40954 Mb /dev/sde1 oracle oinstall--------------------------------------------------------------------------------ORACLE_SID ORACLE_HOME ================================================================================ +ASM1 /u01/app/11.2.0/grid [oracle@raclinux1 ~]$ kfod op=groups--------------------------------------------------------------------------------Group Size Free Redundancy Name ================================================================================ 1: 40954 Mb 30440 Mb EXTERN FLASH 2: 46074 Mb 42080 Mb NORMAL DATA [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=insts--------------------------------------------------------------------------------ORACLE_SID ORACLE_HOME ================================================================================ +ASM1 /u01/app/11.2.0/grid [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=version--------------------------------------------------------------------------------ORACLE_SID RAC VERSION ================================================================================ +ASM1 YES 11.2.0.1.0 [oracle@raclinux1 ~]$ kfod op=clients--------------------------------------------------------------------------------ORACLE_SID VERSION ================================================================================ RACDB_1 11.2.0.1.0 +ASM1 11.2.0.1.0 +ASM1 11.2.0.1.0 RACDB_1 11.2.0.1.0 [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=rm--------------------------------------------------------------------------------Rolling Migration State================================================================================Inactive[oracle@raclinux1 ~]$ kfod op=rmvers--------------------------------------------------------------------------------Rolling Migration Compatible Versions================================================================================11.1.0.6.011.1.0.7.0[oracle@raclinux1 ~]$

Questions & AnswersThank you for attendingFill out your evaluationQuestions & Answers