Oracle utilities in 11g R2

1,368
-1

Published on

Oracle 11gR2 utilities

1 Comment
3 Likes
Statistics
Notes
  • Guenadi,

    Remove this at once or you will hear from my attorney for plagiarism and stealing my 2008 Oracle OpenWorld presentation content! You know this is wrong and yet you continue to steal from others.


    Sincerely,
    Ben Prusinski, Oracle ACE Director and Oracle Author
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,368
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide
  • Automatic Diagnostic Repository (ADR)The ADR is a file-based repository for database diagnostic data such as traces, dumps, the alert log, health monitor reports, and more. It has a unified directory structure across multiple instances and multiple products. Beginning with Release 11g, the database, Automatic Storage Management (ASM), and other Oracle products or components store all diagnostic data in the ADR. Each instance of each product stores diagnostic data underneath its own home directory within the ADR. For example, in an Oracle Real Application Clusters environment with shared storage and ASM, each database instance and each ASM instance has an ADR home directory. ADR's unified directory structure, consistent diagnostic data formats across products and instances, and a unified set of tools enable customers and Oracle Support to correlate and analyze diagnostic data across multiple instances.Note:Beginning with Release 11g of Oracle Database, because all diagnostic data, including the alert log, are stored in the ADR, the initialization parameters BACKGROUND_DUMP_DEST and USER_DUMP_DEST are deprecated. They are replaced by the initialization parameter DIAGNOSTIC_DEST, which identifies the location of the ADR.Alert LogThe alert log is an XML file that is a chronological log of database messages and errors. It is stored in the ADR and includes messages about the following:•Critical errors (incidents)•Administrative operations, such as starting up or shutting down the database, recovering the database, creating or dropping a tablespace, and others.•Errors during automatic refresh of a materialized view•Other database eventsYou can view the alert log in text format (with the XML tags stripped) with Enterprise Manager and with the ADRCI utility. There is also a text-formatted version of the alert log stored in the ADR for backward compatibility. However, Oracle recommends that any parsing of the alert log contents be done with the XML-formatted version, because the text format is unstructured and may change from release to release.Trace FilesEach server and background process can write to an associated trace file. Trace files are updated periodically over the life of the process and can contain information on the process environment, status, activities, and errors. In addition, when a process detects a critical error, it writes information about the error to its trace file. The SQL trace facility also creates trace files, which provide performance information on individual SQL statements. You can enable SQL tracing for a session or an instance.Trace file names are platform-dependent. Typically, database background process trace file names contain the Oracle SID, the background process name, and the operating system process number, while server process trace file names contain the Oracle SID, the string "ora", and the operating system process number. The file extension is .trc. An example of a server process trace file name is orcl_ora_344.trc. Trace files are sometimes accompanied by corresponding trace map (.trm) files, which contain structural information about trace files and are used for searching and navigation.DumpsA dump is a specific type of trace file. A dump is typically a one-time output of diagnostic data in response to an event (such as an incident), whereas a trace tends to be continuous output of diagnostic data. When an incident occurs, the database writes one or more dumps to the incident directory created for the incident. Incident dumps also contain the incident number in the file name.Core FilesA core file contains a memory dump, in an all-binary, port-specific format. Core file names include the string "core" and the operating system process ID. Core files are useful to Oracle Support engineers only. Core files are not found on all platforms.Other ADR ContentsIn addition to files mentioned in the previous sections, the ADR contains health monitor reports, data repair records, SQL test cases, incident packages, and more. These components are described later in the chapter.Enterprise Manager Support WorkbenchThe Enterprise Manager Support Workbench (Support Workbench) is a facility that enables you to investigate, report, and in some cases, repair problems (critical errors), all with an easy-to-use graphical interface. The Support Workbench provides a self-service means for you to gather first-failure diagnostic data, obtain a support request number, and upload diagnostic data to Oracle Support with a minimum of effort and in a very short time, thereby reducing time-to-resolution for problems. The Support Workbench also recommends and provides easy access to Oracle advisors that help you repair SQL-related problems, data corruption problems, and more.ADRCI Command-Line UtilityThe ADR Command Interpreter (ADRCI) is a utility that enablesyou to investigate problems, view health check reports, and package and upload first-failure diagnostic data to Oracle Support, all within a command-line environment. ADRCI also enables you to view the names of the trace files in the ADR, and to view the alert log with XML tags stripped, with and without content filtering.New Feature for Oracle 11g: ADR monitoring toolsOracle 11g has introduced a new monitoring tool called the Automatic Diagnostic Repository (ADR). This new tool which is available via both command line and from within Enterprise Manager/Grid Control provides comprehensive reporting and monitoring functions for Oracle trace files, alert log reporting and other features for monitoring the Oracle 11g database environment. Prior to Oracle 11g, trace files were stored by default under the user_dump_dest directory. Before Oracle 11g, the alert.log text based file which contains details on the status of the Oracle database were stored by default under the background dump destination directory before 11g and now is referenced by a single initialization parameter called diagnostic_dest . The previous release parameters for background_dump_dest, core_dump_dest, and user_dump_dest have been deprecated and replaced by the single parameter diagnostic_dest in 11g. Now with Oracle 11g, these files are managed under the ADR directory structure and can be viewed by the ADR tool. The base monitoring file is stored in XML format in a file called log.xml. By default the diagnostic directory structure is tied to the ORACLE_HOME, ORACLE_BASE and ORACLE_SID environment variables. SQL> show parameter diagnostic_destNAME TYPE VALUE------------------------------------ ----------- ------------------------------diagnostic_dest string /u01/app/oracleFigure 1.18: diagnostic_dest parameter with ADR and Oracle 11gThe Oracle 11g ADR uses a home directory to centralize log file maintenance operations. By default, the ADR home directory is located under the directory structure: <ADR_BASE>/diag/rdbms/<db_name>/<instance_id>.To show the home directory details for ADR, we can logon to the command line tool adrci as follows.[oracle@raclinux1 ~]$ adrciADRCI: Release 11.1.0.6.0 - Beta on Tue Jul 29 00:07:57 2008Copyright (c) 1982, 2007, Oracle. All rights reserved.ADR base = "/u01/app/oracle"adrci> show homeADR Homes:diag/rdbms/ora11g/ora11gdiag/rdbms/ora11g/ORA11Gdiag/rdbms/default/ORA11Gdiag/rdbms/unknown/ORA11Gdiag/rdbms/stdby1/stdby1diag/clients/user_oracle/host_3681296775_11diag/clients/user_unknown/host_411310321_11diag/tnslsnr/raclinux1/listenerdiag/tnslsnr/raclinux1/listener1diag/tnslsnr/raclinux1/listener_stdby1Figure 1.19: Using adrci to display ADR home with Oracle 11gIn addition to using the adrci show home command, the location for the ADR configuration can be displayed using the V$DIAG_INFO view as shown in the below figure example.Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL> col name format a20SQL> col value format a50 word_wrappedSQL> select name, value 2 from 3 v$diag_info;NAME VALUE-------------------- --------------------------------------------------Diag Enabled TRUEADR Base /u01/app/oracleADR Home /u01/app/oracle/diag/rdbms/ora11g/ORA11GDiag Trace /u01/app/oracle/diag/rdbms/ora11g/ORA11G/traceDiag Alert /u01/app/oracle/diag/rdbms/ora11g/ORA11G/alertDiag Incident /u01/app/oracle/diag/rdbms/ora11g/ORA11G/incidentDiag Cdump /u01/app/oracle/diag/rdbms/ora11g/ORA11G/cdumpHealth Monitor /u01/app/oracle/diag/rdbms/ora11g/ORA11G/hmDefault Trace File /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA 11G_ora_28603.trcFigure 1.20: Location for diagnostic files in Oracle 11g for ADRThe above query against the V$DIAG_INFO view provides us with the default locations for the diagnostic trace, alert, incident, core dump, ADR home, ADR base, and health monitor files for Oracle 11g.If we examine the log.xml file we notice that the default xml alert log file contains all of the entries that are also contained in the text based alert.log file which is available for backward compatibility with previous releases.SQL> !ls /u01/app/oracle/diag/rdbms/ora11g/ORA11G/alertlog.xmlSQL> !view /u01/app/oracle/diag/rdbms/ora11g/ORA11G/alert/log.xml<msg time='2008-06-09T18:34:25.057-04:00' org_id='oracle' comp_id='rdbms' msg_id='opistr_real:871:3971575317' type='NOTIFICATION' group='startup' level='16' pid='7457' version='1'> <txt>Starting ORACLE instance (normal) </txt></msg><msg time='2008-06-09T18:34:25.840-04:00' org_id='oracle' comp_id='rdbms' msg_id='ksunfy:13399:2937430291' type='NOTIFICATION' group='startup' level='16' pid='7457'> <txt>LICENSE_MAX_SESSION = 0 </txt></msg><msg time='2008-06-09T18:34:25.840-04:00' org_id='oracle' comp_id='rdbms' msg_id='ksunfy:13400:4207019197' type='NOTIFICATION' group='startup' level='16' pid='7457'> <txt>LICENSE_SESSIONS_WARNING = 0 </txt></msg><msg time='2008-06-09T18:34:25.918-04:00' org_id='oracle' comp_id='rdbms' msg_id='kcsnfy:323:968333812' type='NOTIFICATION' group='startup' level='16' pid='7457'> <txt>Picked latch-free SCN scheme 2 </txt></msg><msg time='2008-06-09T18:34:26.494-04:00' org_id='oracle' comp_id='rdbms' msg_id='kcrrdini:15230:1211400554' type='NOTIFICATION' group='startup' level='16' pid='7457'>Figure 1.21: Reviewing the log.xml file for ADR and Oracle 11gSince the format of the log.xml file is not intended for viewing in xml format, it is advised to use the adrci interface to view log files. Returning to the ADR interface via the adrci command tool, we can dump the help contents to provide us with a list of commands available with ADR as shown in the below figure. We can also use the HELP EXTENDED command to display more details.[oracle@raclinux1 ~]$ adrciADRCI: Release 11.1.0.6.0 - Beta on Tue Jul 29 00:29:52 2008Copyright (c) 1982, 2007, Oracle. All rights reserved.ADR base = "/u01/app/oracle"adrci> help extended HELP [topic] Available Topics: BEGIN BACKUP CD DDE DEFINE DESCRIBE END BACKUP LIST DEFINE MERGE ALERT MERGE FILE QUERY SET COLUMN SHOW CATALOG SHOW DUMP SHOW SECTION SHOW TRACE SHOW TRACEMAP SWEEP UNDEFINE VIEWadrci> help HELP [topic] Available Topics: CREATE REPORT ECHO EXIT HELP HOST IPS PURGE RUN SET BASE SET BROWSER SET CONTROL SET ECHO SET EDITOR SET HOMES | HOME | HOMEPATH SET TERMOUT SHOW ALERT SHOW BASE SHOW CONTROL SHOW HM_RUN SHOW HOMES | HOME | HOMEPATH SHOW INCDIR SHOW INCIDENT SHOW PROBLEM SHOW REPORT SHOW TRACEFILE SPOOL There are other commands intended to be used directly by Oracle, type "HELP EXTENDED" to see the listadrci>Figure 1.22: Using help commands with adrci for ADR in Oracle 11gIf we wish to find specific help on a particular topic with ADR, we can issue the command help followed by the topic as shown in the following example.adrci> help dde create incidentUsage: DDE CREATE INCIDENT TYPE <type>Purpose: Create an incident of specified type. If the incident type is associated with an action, the action will be automatically recommended for the new incident. Arguments: <type>: Incident type Notes: The relation DDE_USER_INCIDENT_TYPE shows available incident types. Example: dde create incident type wrong_resultsData Recovery Advisor (DRA)The Data Recovery Advisor (DRA) is a new tool aimed at reducing a user’s time spent analyzing and formulating a suitable recovery plan for a given failure. A ‘failure’ in the context of the DRA, can be a missing, inaccessible, or wrong version of a file (e.g. control file, data file), physical corruptions resulting from I/O errors, or logical block inconsistency. After identifying all current failures, the DRA then recommends the optimal feasible recovery plan, and if the user desires, automatically executes a selected recovery plan. The latest acronym from Oracle 11g is RAT, Real Application testing, an 11g new feature with a really interesting acronym.RAT is like a computerized tape recorder, allowing workload to be captured and replayed for testing and benchmarking purposes. RAT captures transactions against the Oracle database (as made by an application) and allows them to be easily tested. In a sense, RAT has many of the same features as the Quest "Benchmark Factory" product.New performance features in Oracle 11gR1Database Replay The Database Replay feature (sometimes referred to as Workload Replay) in Oracle 11g allows you to reproduce the production database conditions in a testing environment. Database Replay enables you to capture the actual workload on a production system and replay it in a test system. This way, you can analyze the condition of the production database without working on the actual production database. This feature enables you to test the impact of applying changes on a production database. These changes could be database upgrades, switching to RAC, application upgrades, operating system upgrades or storage system changes.SQL performance AnalyzerThe SQL Performance Analyzer (SPA) aims to measure the impact of applying any change of the database on the performance of the SQL statements execution. If it finds performance degradation in one or more SQL statements, it provides you with recommendations on how to improve their performance. This is very useful for a DBA to analyze how a change on the database (including database upgrade) may affect the execution efficiency of SQL statements. Using this tool is explained here because you may consider using it to study the effect of upgrading an Oracle database 10g release 2 to 11g. SPA can be invoked from OEM or by using the DBMS_SQLPA package.Database Health Monitor Oracle 11g introduced a new monitoring framework, the Database Health Monitor. With the Health Monitor, Oracle automatically runs a diagnostic operation check to examine any database corruption or error. Oracle recommends that you frequently invoke the health checks on the database when it is under low usage to catch any corruption or error that may cause damage in the future. The Health Monitor framework provides several types of database health checks. The following are the checks undertaken by the Health Monitor: • Database structure check.• Data block integrity check.• Redo integrity check.• Undo segment integrity check.• Transaction integrity check.• Dictionary integrity check.PL/SQL Native Compilation PL/SQL Native Compilation was introduced in Oracle 9i. In Oracle 11g, the PL/SQL compiler may now generate processor-specific native code directly from the PL/SQL source code without needing to use a third-party compiler and as in Oracle 10g, stores the code in the database catalog. When a unit is needed, the Oracle executable loads the code directly from the catalog into memory, without first staging the unit through a .DLL or .so file. Server Result CacheIn Oracle 11g, there is a new SGA component, Server Result Cache, which is used to cache SQL query and PL/SQL function results. The database serves the results for the executed SQL queries and PL/SQL functions from the cache instead of re-executing the actual query. Of course, the target is to obtain low response time. The cached results stored, become invalid when data in the dependent database objects is modified. As it becomes clear from its concept, result cache is mostly useful in frequently executed queries with rare changes on the retrieved data. Client Side Result CacheIn Oracle 11g, you can configure the client process to cache its query results in the client memory (or in the server). Consider a situation where the client has to call the same data over some slow network link. Although the database can send the result to the client almost instantaneously from its cache, the result must travel over the wire to the client, adding to the overall execution time.SQL Tuning AutomationThe SQL Tuning Advisor is run by default every night during the automated maintenance window in Oracle 11g, in contrast to Oracle 10g when it has to be run manually. The SQL Tuning advisor catches the SQL statements from AWR that are candidate for tuning, (they are called buckets) during four different time periods. It then automatically creates a SQL profile for any poor SQL statement, if that helps. Tuned plans are automatically added to the SQL plan baselines by the automatic SQL tuning task. Enhanced statistics maintenanceStarting with Oracle 11g, when gathering statistics, you have the option to automatically publish the statistics at the end of the gather operation (default behavior), or to have the new statistics saved as pending. Saving the new statistics as pending allows you to validate the new statistics and publish them only if they are satisfactory. SQL Plan ManagementSQL Plan Management (SPM) prevents performance regressions resulting from sudden changes to the execution plan of a SQL statement by providing components for capturing, selecting and evolving SQL plan information. Changes to the execution plan may result from database upgrades, system and data changes, application upgrade or bug fixes. When SPM is enabled, the system maintains a plan history that contains all plans generated by the optimizer and stores them in a component called plan baseline. Among the plan history in the plan baseline, plans that are verified not to cause performance regression are marked as acceptable. The plan baseline is used by the optimizer to decide on the best plan to use when compiling a SQL statement. Repository stored in a data dictionary of plan baselines and statement log maintained by the optimizer is called SQL Management Base (SMB).In-Memory Parallel ExecutionTraditionally, parallel execution has enabled organizations to manage and access large amounts of data by taking full advantage of the I/O capacity of the system. In-Memory Parallel Execution harnesses the aggregated memory in a system to enhance query performance by minimizing or even completely eliminating the physical I/O needed for a parallel operation. Oracle automatically decides if an object is being accessed using parallel execution benefits from being cached in the SGA (buffer cache). The decision to cache an object is based on a well defined set of heuristics including size of the object and the frequency that it is accessed. In an Oracle RAC environment, Oracle maps fragments of the object into each of the buffer caches on the active instances. By creating this mapping, Oracle knows which buffer cache to access to find a specific part or partition of an object to answer a given SQL query. In-memory parallel query harnesses the aggregated memory in a system for parallel operations, enabling it to scale out with the available memory for data caching as the number of nodes in a cluster increases. This new functionality optimizes large parallel operations by minimizing or even completely eliminating the physical I/O needed because the parallel operation can now be satisfied in memory.Auto Degree of Parallelism (DOP) and Queuing When activated, Oracle determines the optimal Degree of Parallelism (DOP) for any given SQL operation based on the size of the objects, the complexity of a statement and the existing hardware resources. The database compensates for wrong or missing user settings for parallel execution, ensuring a more optimal resource consumption and overall system behavior. The DBMS_PARALLEL_EXECUTE package The DBMS_PARALLEL_EXECUTE package provides subprograms to allow a specified INSERT, UPDATE, DELETE, MERGE or anonymous block statement to be applied in parallel chunks. The statement must have two placeholders that define the start and end limit of a chunk. Typically, these are values for the rowid or a surrogate unique key in a large table. But, when an anonymous block is used, the block can interpret the values arbitrarily. The package has subprograms to define ranges that cover the specified table. These include rule-based division of a table's rowid or key range and support user-defined methods. The SQL statement together with the set of chunk ranges defines a task. Another subprogram starts the task. Each task is processed using a scheduler job and automatically commits when it completes. Progress is logged. Untried, successful, and failed chunks are flagged as such on task completion or interruption. Another subprogram allows the task to resume to try untried and failed chunks. Many scenarios require the bulk transformation of a large number of rows. Using an ordinary SQL statement suffers from the all-or-nothing effect. In the common case, where the transformation of one row is independent of that of other rows, it is correct to commit every row that is transformed successfully and to roll back every row where the transformation fails. Some customers have implemented schemes to achieve this from scratch, using the Oracle Scheduler and suitable methods to record progress. This package provides a supported solution and adds database-wide manageability through new catalog views for parallel task metadata. The package is especially useful in online application upgrade scenarios to apply a crossedition trigger to all the rows in the table on which it is defined. Crossedition triggers are intended to fire when database manipulation language (DML) changes are made in a database while an online application that uses the database is being patched or upgraded with edition-based redefinition. The body of a crossedition trigger is designed to handle these DML changes so that they can be appropriately applied after the changes to the application code are completed. Significant performance improvement of On-Commit Fast Refresh On Commit Fast Refresh of a materialized view is now significantly faster due to reducing the time spent on log handling. This provides significantly reduced maintenance time and faster refreshes are possible.
  • Overview Reference for SQL_TRACE, TKProf and Explain Plan [ID 199081.1]     Modified 08-DEC-2008     Type BULLETIN     Status PUBLISHED   Checked for relevance on 22-Feb-2007. Purpose This document gives an overview of some of the tools that can be used to discover more about SQL execution. The article provides information covering SQL_TRACE, TKProf, Explain Plan and Autotrace. Scope & ApplicationFor all DBAs Overview Reference for SQL_TRACE, TKProf and Explain PlanOracle provides extensive tracing and interpretation tools to assist analysts diagnose and resolve sub-optimal performance issues. This article covers the most important of these and points to detailed reference articles.Tracing Related Initialization ParametersPrior to tracing, there are a number of parameters that need to be set so that the trace information is complete. These parameter should be set up in the "init.ora" file for the particular instance (SID) where you wish to use SQL Trace although they can also be set individually at the session level. <<Parameter:TIMED_STATISTICS>> Enable/Disable the collection of timed statistics, such as CPU and elapsed times. TRUE - Enable timingFALSE - Disable timing (Default value).Note that much of the tracing information that can be gathered is rendered useless if TIMED_STATISTICS is set to False and timings are not collected. <<Parameter:MAX_DUMP_FILE_SIZE>> Specifies the maximum size of trace files in operating system blocks. The default value for this was 10000 OS blocks in 8i version, and limited only  by the space available in 9i version. If your trace file is truncated then you will see a message similar to: *** DUMP FILE SIZE IS LIMITED TO 12345 BYTES***and the size of this parameter should be increased. <<Parameter:USER_DUMP_DEST>> Specifies the destination for the trace file. The default value for this parameter is the default destination for oracle dumps on your operating system. These parameters can be dynamically altered using alter system/alter session commands (Note that USER_DUMP_DEST can only be modified at the system level). For example TIMED_STATISTICS can be enabled/disabled dynamically by using the following SQL statement: ALTER SYSTEM/SESSION SET TIMED_STATISTICS = TRUE/FALSE;SQL_TRACESQL_TRACE is the main method for collecting SQL Execution information in Oracle collecting a wide range of information and statistics that can be used to tune SQL operations.Enabling SQL_TRACEThe SQL Trace facility can be enabled/disabled for an individual session or at the instance level. If the initialisation Parameter SQL_TRACE is set to TRUE in the init.ora of an instance, then all sessions will be traced. SQL_TRACE can be set at the instance level by using the initialisation parameter SQL_TRACE: <<Parameter:SQL_TRACE>>: Enable/Disable SQL Trace instance wide. TRUE - Enable statistics to be collected for all sessions.FALSE - Disable statistics to be collected for all sessions.SQL_TRACE can also be enabled/disabled at the system/session by issuing the following SQL statement: ALTER SYSTEM/SESSION SET SQL_TRACE = TRUE/FALSE;Trace can also be enabled/disabled on other sessions (as well as your own) using the DBMS_SUPPORT package. See Note:62160.1 Tracing Sessions in Oracle7/8 for details. For more information on how to setup tracing, refer to: Note:15160.1 Setting SQL Trace in the Oracle Tools. Note:21154.1 EVENT: 10046 "enable SQL statement tracing (including binds/waits).Note 376442.1 Recommended Method for Obtaining 10046 trace for Tuning There is also extensive information in the Oracle9i Database Performance Tuning Guide and Reference Chapter 10: Using SQL Trace and TKProfTrace Files Oracle will generate trace (.trc) files for every session where the value of SQL_TRACE=TRUE and write them to the USER_DUMP_DEST destination. If tracing has been enabled for the instance then individual trace files will be generated for each session, unless otherwise disabled. Note that the generated files may be owned by an operating system user other than your own so the necessary privileges will need to be put in place before they can be formatted with TKProf. Explain PlanThe Explain Plan command generates information that details the execution plan that will be used on a particular query.Note that the EXPLAIN PLAN statement displays execution plans chosen by the Oracle optimizer for SELECT, UPDATE, INSERT, and DELETE statements only. For more details see: Oracle9i Database Performance Tuning Guide and Reference Release 2 (9.2) Part Number A96533-02 It uses a precreated table (PLAN_TABLE) in the current shema to store information about the execution plan chosen by the optimizer. Creating the Plan TableThe plan table is created using the script utlxplan.sql. This script is typically found under the Oracle Home in the rdbms/admin directory. On Unix its location will be: $ORACLE_HOME/rdbms/admin On WindowsNT/2000:         %ORACLE_HOME%
    dbmsadmin This script creates the output table, called PLAN_TABLE, for holding the output of the Explain plan Command. Note that the exact structure of the plan table can change with different release as new features are introduced. Populating the Plan TableThe plan table is populated using the explain plan command: SQL> EXPLAIN PLAN for select * from emp where empno=1000;This command inserts the execution plan of the SQL statement into the plan table. It is also possible to adds the name tag to the explain information by using the set statement_id clause. Displaying the Execution PlanOnce the table has been populated, the explain information needs to be retrieved and formatted. There are a large number of scripts available to format the plan table data. Some of the most popular are noted below: Supplied Scripts:$ORACLE_HOME/rdbms/admin/utlxpls.sql: script to format serial explain plans $ORACLE_HOME/rdbms/admin/utlxplp.sql: script to format parallel explain plansArticles:Note:31101.1 Obtaining Formatted Explain Plan Output Note:39294.1 Formatted Select of PLAN_TABLE for EXPLAIN PLAN command Note:39341.1 Automatic Explain Plan Note:1019631.6 SCRIPT: SCRIPT TO SIMPLIFY THE USE OF EXPLAIN PLANInterpretation of Explain Plan This is a complex topic and is covered in detail in the following article: Note:46234.1 Interpreting Explain PlanThere is also extensive information in the Oracle9i Database Performance Tuning Guide and Reference Chapter 9: Using EXPLAIN PLANAUTOTRACEThe autotrace facility in SQL*Plus allows analysts to view the execution plan and some useful statistics for a SQL statement within a SQL*Plus session. This option was introduced with 7.3 version of Oracle. Autotrace needs to be initiated in the SQL*Plus session prior to executing the statement. The Autotrace command is: SET AUTOTRACE [OPTIONS] [EXPLAIN/STATISTICS]For a detailed explanation of AUTOTRACE functions see: Note:43214.1 AUTOTRACE option in 7.3As with the EXPLAIN PLAN command, to obtain an execution plan the PLAN_TABLE must be created in the user's schema prior to autotracing. Example SQL> set autotrace traceonly explainSQL> select * from dual;Execution Plan ---------------------------------------------------------- 0 SELECT STATEMENT Optimizer=CHOOSE 1 0 TABLE ACCESS (FULL) OF 'DUAL'To enable viewing of STATISTICS data, the autotracing user must have access to dynamic performance tables. To achieve this, grant PLUSTRACE role to the user. The PLUSTRACE role is created by the plustrce.sql script.     On Unix the location is:         $ORACLE_HOME/sqlplus/admin    On WindowsNT/2000:         %ORACLE_HOME%sqlplusadmin This script must be run by the SYS user. A DBA user can then grant the role to the users who wish to use the the AUTOTRACE option. Refer to: Note:1055431.6 ORA-01919 usingAUTOTRACE in SQL*PlusExtensive reference information regarding the autotrace facility can be found in the SQL*Plus Users Guide and Reference Release 9.0.1TKProfThe TKProf facility accepts as input a SQL trace file and produces a formatted output file. For the full syntax of TKProf see the Oracle Server Tuning Manual. If TKProf is invoked with no arguments, an online help is displayed. Basic Syntax of TKProfTKPROF filename_sourcefilename_output EXPLAIN=[username/password] SYS=[YES/No] TABLE=[tablename] filename_source The trace file that needs to be formatted filename_output The output file where the formatted file will be written 'EXPLAIN' This option causes TKProf to generate an execution plan for each SQL statement in the trace file as though it was been parsed at the time the TKProf is being run. TKProf connects as the user [username] and uses the EXPLAIN PLAN command to generate an execution plan for each traced SQL statement. These explain plans are written to the output file. Note that in later releases, the raw trace file automatically records the explain plan that was used at execution time and TKProf will automatically output this plan. If the EXPLAIN option is selected then two explain plans appear in the TKProf output. The first explain plan is the plan that was used at execution time. The second plan is the plan that was generated dynamically at the time the TKProf runs. If session settings etc have changed between the execution time and when the trace file is TKProffed then the first and second plans may differ. 'TABLE' Specifies the schema and name of the table into which TKPROF temporarily places execution plans before writing them to the output file. If the specified table already exists, TKProf deletes its rows then uses it for the EXPLAIN PLAN command and then deletes its rows. If this table does not exist, TKProf creates, uses, then drops it. This parameter is ignored if the EXPLAIN parameter isn't used. 'SYS' Enables and disables the listing of SQL statements issued by the user SYS (recursive SQL statements) into the output file. The default value is YES. For more information on using TKProf see: Note:32951.1 TKPROF Interpretation Note:29012.1 Quick Reference TKPROF usage Note:41634.1 TKPROF and Problem SolvingThere is also extensive information in the TKProf Interpretation (9i and below) [ID 32951.1]  Modified 24-MAR-2009     Type BULLETIN     Status ARCHIVED  PURPOSEThis document gives general advice on the interpretation of TKPROF output andprovides some examples on how this information can be used. SCOPE & APPLICATIONFor users needing advice on what to use TkProf for. For TKProf usage information see:  Note:29012.1 QREF: TKPROF Usage - Quick Reference  Tkprof is an executable that 'parses' Oracle trace files to produce morereadable output. Remember that all the information in TkProf is availablefrom the base trace file. If you have a system that is performing badly, a good way to identify problem SQL statements is to trace a typical user session and then use TkProfto format the output using the sort functions on the tkprof command line.  There are a huge number of sort options that can be accessed by simply typing 'TkProf' at the command prompt. A useful starting point is the 'fchela' sort option which orders the output by elapsed time fetching (rememberthat timing information is only available with timed_statistics set to true in the "init.ora" file). The resultant .prf file will contain the most time consuming SQL statement at the start of the file. Another useful parameter is sys. This can be used to prevent SQL statements run as user SYS from being displayed. This can make the output file much shorter an easier to manage. Remember to always set the TIMED_STATISTICS parameter to TRUE when tracingsessions as otherwise no time based comparisons can be made.  Interpreting TkProf Output Guidelines===================================== Column Meanings=============== call : Statisics for each cursor's activity are divided in to 3 areas:   Parse: statisitics from parsing the cursor. This includes information for plan generation etc. Execute: statisitics for the exection phase of a cursor Fetch : statistics for actually fetching the rows count : number of times we have performed a particular activity on this particular cursor cpu: cpu time used by this cursor elapsed: elapsed time for this cursor disk: This indicates the number of blocks read from disk. Generally you want to see blocks being read from the buffer cache rather than disk.query : This column is incremented if a buffer is read in Consistent mode. A Consistent mode buffer is one that has been generated to give a consistent read snapshot for a long running transaction. The buffer actually contains this status in its header.current: This column is incremented if a buffer found in the buffer cache that is new enough for the current transaction and is in current mode (and it is not a CR buffer). This applies to buffers that have been read in to the cache as well as buffers that already exist in the cache in current mode. rows: Rows retrieved by this step   Explain plan ============ Firstly, we advise that the autotrace feature of SQL*Plus be used on statements rather than using TkProf mainly because the TkProf output can be confusing with regard to whether the Rule or Cost Based optimizer has been used.  Because TkProf explain plan does not show any costs or statistics, it is sometimes not possible to tell definitively which optimizer has been used. That said, the following output from Tkprof explain plan is useful.The Rows column next to the explain plan output shows the number of rows processed by that particular step. The information is gathered from the STAT lines for each cursor in the raw trace output.  Remember that if the cursor is not closed then you will not see any output. Setting SQL_TRACE to false DOES NOT close PL/SQL child cursors. Cursors are closed in SQL*Plus immediately after execution.  TkProf Examples and Discussion============================== Examples: Step 1 - Look at the totals at the end of the tkprof output=========================================================== OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS| call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|------|---------|---------|--------|---------|--------|| Parse | [A] 7 | 1.87 | 4.53 | 385 |[G] 553 | 22 | 0 || Execute | [E] 7 | 0.03 | 0.11 | [P] 0 |[C] 0 | [D] 0 | [F] 0 || Fetch | [E] 6 | 1.39 | 4.21 | [P] 128 |[C] 820 | [D] 3 | [F] 20 |--------------------------------------------------------------------------Misses in library cache during parse: 5Misses in library cache during execute: 1 8 user SQL statements in session. 12 internal SQL statements in session.[B] 54 SQL statements in session. 3 statements EXPLAINed in this session. 1. Compare [A] & [B] to spot over parsing. In this case we have 7 parses for 54 statements which is ok. 2. You can use [P], [C] & [D] to determine the hit ratio. Hit Ratio is logical reads/physical reads: Logical Reads = Consistent Gets + DB Block GetsLogical Reads = query + currentLogical Reads = Sum[C] + Sum[D]Logical Reads = 0+820 + 0+3Logical Reads = 820 + 3Logical Reads = 823 Hit Ratio = 1 - (Physical Reads / Logical Reads)Hit Ratio = 1 - (Sum[P] / Logical Reads)Hit Ratio = 1 - (128 / 823)Hit Ratio = 1 - (0.16)Hit Ratio = 0.84 or 84% 3. We want fetches to be less than the number of rows as this will mean we have done less work (array fetching). To see this we can compare [E] and [F]. [E] = 6 = Number of Fetches[F] = 20 = Number of Rows So we are doing 6 fetches to retrieve 20 rows - not too bad.If arrayfetching was configured then rows could be retrieved with less fetches. Remember that an extra fetch will be done at the end to check thatthe end of fetch has been reached. 4. [G] Shows reads on the Dictionary cache for the statements.  - this should not be a problem on Oracle7. In this case we have done 553 reads from the Library cache. STEP 2 - Examine statements using high resource=============================================== update ...where ... | call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|-----|---------|------|---------|---------|--------|| Parse | 1 | 7 | 122 | 0 | 0 | 0 | 0 || Execute | 1 | 75 | 461 | 5 | [H] 297 | [I] 3 | [J] 1 || Fetch | 0 | 0 | 0 | 0 | 0 | 0 | 0 |----------------------------------------------------------------------- [H] shows that this query is visiting 297 blocks to find the rows to update[I] shows that only 3 blocks are visited performing the update[J] shows that only 1 row is updated. 297 block to update 1 rows is a lot. Possibly there is an index missing?  STEP 3 - Look for over parsing============================== select ... | call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|---------|---------|------|--------|---------|-------|| Parse | [M] 2 | [N] 221 | 329 | 0 | 45 | 0 | 0 || Execute | [O] 3 | [P] 9 | 17 | 0 | 0 | 0 | 0 || Fetch | 3 | 6 | 8 | 0 | [L] 4 | 0 | [K] 1 |------------------------------------------------------------------------- Misses in library cache during parse: 2 [Q] [K] is shows that the query has returned 1 row.[L] shows that we had to read 4 blocks to get this row back.This is fine. [M] show that we are parsing the statement twice - this is not desirable especially as the cpu usage is high [N] in comparison to the execute figures : [O] & [P]. [Q] shows that these parses are hard parses. If [Q] was 1 then the statemnent would have had 1 hard parse followed by a soft parse (which just looks up the already parsed detail in the library cache). See Note:32895.1 for more details.  This is not a particularly bad example since the query has only been executed a few times. However excessive parsing should be avoided as far as possible by: o Ensuring that code is shared:  - use bind variables - make shared pool large enough to hold query definitions in memory long enough to be reused.  NOTE:====We need to ensure, we do not parse a >=9i version 10046 raw trace with an <= 8i tkprof utility.This is because the timings in raw trace file for <=8i versions and >=9i versions are in centiseconds and microseconds respectively.  SEARCH WORDS============ EXPLAIN; PARSE; PLAN; STATEMENT; TKPROF;  QREF: TKPROF Usage - Quick Reference [ID 29012.1]  Modified 25-SEP-2008     Type FAQ     Status PUBLISHED  Quick Reference to Using TKPROF~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  TKPROF allows you to analyse a trace file to determine where time is being spent and what query plans are being used on SQL statements.  1 - Set TIMED_STATISTICS if required at database level. 2 - Get SQL_TRACE output for the session you wish to monitor (See Note:15160.1 for how to set this in various tools)  3 - Find the appropriate trace file (In USER_DUMP_DEST, default $ORACLE_HOME/rdbms/log on Unix). You can find the most recent trace files on Unix with the command: ls -ltr This will list the most recent files LAST  4 - Run tkprof on the trace file thus: tkprof tracefile outfile [explain=user/password] [options...] TKPROF Options~~~~~~~~~~~~~~ print=integer List only the first 'integer' SQL statements. insert=filename List SQL statements and data inside INSERT statements. sys=no TKPROF does not list SQL statements run as user SYS. record=filename Record statements found in the trace file. sort=option Set of zero or more of the following sort options: prscnt number of times parse was calledprscpucpu time parsingprsela elapsed time parsingprsdsk number of disk reads during parseprsqry number of buffers for consistent read during parseprscu number of buffers for current read during parseprsmis number of misses in library cache during parse execnt number of execute was calledexecpucpu time spent executingexeela elapsed time executingexedsk number of disk reads during executeexeqry number of buffers for consistent read during executeexecu number of buffers for current read during executeexerow number of rows processed during executeexemis number of library cache misses during execute fchcnt number of times fetch was calledfchcpucpu time spent fetchingfchela elapsed time fetchingfchdsk number of disk reads during fetchfchqry number of buffers for consistent read during fetchfchcu number of buffers for current read during fetchfchrow number of rows fetched useriduserid of user that parsed the cursor TKProf Basic Overview [ID 41634.1]  Modified 19-SEP-2008     Type REFERENCE     Status PUBLISHED  TKPROF and Problem Solving========================== Purpose-------This document discusses the use of the following: * SQL TRACE * EXPLAIN PLAN * TKPROF  Scope & Application--------------------For users wanting to obtain a query plan and/or execution times for their queries.  Related Documents------------------The majority of the material covered in this document can be found in the following manuals: * Oracle7 Server SQL Reference * Oracle7 Server Tuning  Introduction------------Much information can be gathered about the performance of an Oracledatabase from the dynamic 'V$' views, as these are documented in theOracle7 Server Manual. Oracle also provides additional performancediagnostic tools to assist in monitoring and tuning the server. The reader should be familiar with the concepts of the Oracle Serverinitialization file and startup parameters. Most of the suggestionsdescribed in this document should be conducted by an experienced DBA.  Overview Of SQL TRACE---------------------The diagnostic tool 'sql trace' provides performance information aboutindividual SQL statements and generates the following statistics foreach statement:  * parse, execute, and fetch counts * CPU and elapsed times * physical reads and logical reads * number of rows processed * misses on the library cache  This information is input to a trace (.trc) file and sql trace can beenabled/disabled for a session or an instance.  Setting Initialization Parameters---------------------------------The following parameters need to be set up in the "init<sid>.ora" filefor the particular instance (SID) that you wish to use SQL Trace:  * SQL_TRACE + Enable/Disable SQL Trace for the instance.  Values ------ TRUE Enable statistics to be collected for all sessions. FALSE Disable statistics to be collected for all sessions.  * TIMED_STATISTICS + Enable/Disable the collection of timed statistics, such as CPU and elapsed times.  Values ------ TRUE Enable timing (we usually recommend this) FALSE Default value.  * MAX_DUMP_FILE_SIZE + Specifies the maximum size of trace files operating system blocks.  Values ------ The default value for this is 500 but if your trace file is truncated then increase this value.  * USER_DUMP_DEST + Specifies the destination for the trace file.  Values ------ The default value for this parameter is the default destination for system dumps on your operating system.   Enabling SQL Trace------------------The SQL Trace facility can either be enabled/disabled for an individualsession or the instance.  * To enable the SQL trace facility for your session issue the following SQL statement:  ALTER SESSION SET SQL_TRACE = TRUE;   * To disable the SQL trace facility for your session issue the following SQL statement:  ALTER SESSION SET SQL_TRACE = FALSE;  Enabling TIMED_STATISTICS-------------------------The parameter TIMED_STATISTICS can be enabled/disabled dynamically by using ALTER SYSTEM SET TIMED_STATISTICS = TRUE;  or ALTER SESSION SET TIMED_STATISTICS = TRUE; To disable set TIMED_STATISTICS = FALSE: SQL Trace Facility for an Instance----------------------------------Note that if the initialization parameter SQL_TRACE=TRUE, then statisticswill be gathered for ALL sessions. If the facility has been enabled forthe instance, it may be disabled for an individual session by issuing bythe above SQL statement.  Trace Files-----------Oracle will generate trace (.trc) files for every session where the valueof SQL_TRACE = TRUE and write them to the USER_DUMP_DEST destination.If tracing has been enabled for the instance then individual trace fileswill be generated for each session, unless otherwise disabled (see above).Note, that the generated files may be owned by an operating system userother than your own so you may have to get this user to grant you accessbefore you can use TKPROF to format them.  Using TKPROF------------The TKPROF facility accepts as input an SQL trace file and produces aformatted output file. Note that TKPROF can be run on individual orappended trace files to produce the formatted output file.  * For the full syntax of TKPROF see Oracle7 Server Tuning Manual.  * Invoke TKPROF with no arguments and online help is displayed.  Syntax of TKPROF---------------- TKPROF command ::= >>-- TKPROF filename1 filename2 ----------------------------------------------> | | +- SORT = ---------------------------------+ | | +-- OPTION --+ | | | +---- , ----+ | | V | | |__( OPTION )__| >-----------------------------------------------------------------------------> | | | | | | +-- PRINT = integer --+ +-- INSERT = filname3 --+ +-- SYS = ---------+ | | +- YES -+ | | +- NO --+ >-----------------------------------------------------------------------------> | | +---------------------------------------- EXPLAIN = user/password ------+ | | +---- TABLE = schema.table ----+ >---------------------------------------------------------------------------->< | | +---- RECORD = filname ----+  The Basics Of TKPROF-------------------- For the purpose of this document we will only describe some of thefundamental arguments for TKPROF:  * 'filename1' Specifies the input file, a trace file containing statistics produced by the SQL trace facility. This file can be either a trace file produced for a single session or a file produced by appending together together individual trace files from multiple sessions.  * 'filename2' Specifies the file to which TKPROF writes its formatted output.  * 'EXPLAIN' Determines the execution plan for each SQL statement in in the trace file and writes these execution plans to the output file. TKPROF determines execution plans by issuing the EXPLAIN PLAN command after connecting to Oracle with the user and password specified in this parameter. The specified user must have CREATE SESSION privileges.  * 'TABLE' Specifies the schema and name of the table into which TKPROF temporarily places execution plans before writing them to the output file. If the specified table already exists, TKPROF deletes its rows then uses it for the EXPLAIN PLAN command and then deletes its rows. If this table does not exist, TKPROF creates, uses, then drops it. The specified user must be able to issue INSERT, SELECT, and DELETE statements against the table. If the table does not already exist, the user must also be able to issue CREATE TABLE and DROP TABLE statements. This option allows multiple individuals to run TKPROF concurrently with the same user in the EXPLAIN value. These individuals can specify different TABLE values and avoid destructively interfering with each other's processing on the temporary plan table. If you use the EXPLAIN parameter without the TABLE parameter, TKPROF uses the table PROF$PLAN_TABLE in the schema of the user specified by the EXPLAIN parameter. If you use the TABLE parameter without the EXPLAIN parameter, TKPROF ignores the TABLE parameter.   * 'SYS' Enables and disables the listing of SQL statements issued by the user SYS, or recursive SQL statements into the output file. The default value of YES causes TKPROF to list these statements. The value of NO causes TKPROF to to omit them.  Simple Example-------------- This example shows TKPROF being run to format a trace file named"dsdb2_ora_18468.trc" and writing it to a formatted output filenamed "dsdb2_trace.out".  TKPROF dsdb2_ora_18468.trc dsdb2_trace.out SYS=NO EXPLAIN=SCOTT/TIGER Note that if the command is likely to be longer than a single line on yourterminal screen you may have to use continuation characters.  The Parameters In This Example: ------------------------------- The EXPLAIN value causes TKPROF to connect as the user SCOTT and use the EXPLAIN PLAN command to generate the execution plan for each traced SQL statement. The SYS parameter with the value of NO causes TKPROF to omitrecursive SQL statements from the output file.  Interpretting TKPROF Output=========================== The EXPLAIN PLAN Command------------------------ The EXPLAIN PLAN command displays the execution plan chosen by the Oracle optimizer for SELECT, UPDATE, INSERT, and DELETE statements.A statement's execution plan is the sequence of operations that Oracle performs to execute the statement. By examining the execution plan, you can see exactly how Oracle executes your SQL statement. This information can help you determine whether the SQL statement you have written takes advantage of the indexes available. For the syntax of the EXPLAIN PLAN command, see the Oracle7 Server SQL Reference Manual.   Creating the Output Table------------------------- Before you can issue an EXPLAIN PLAN statement, there must exist a table to hold its output, you do either of the following:  * Run the SQL script "UTLXPLAN.SQL" to create a sample output table called PLAN_TABLE in your schema.  * Issue a CREATE TABLE statement to create an output with any name you choose. You can then issue an EXPLAIN PLAN statement and direct its output to this table. Any table used to store the output of the EXPLAIN PLAN command must have the same column names and datatypes as the PLAN_TABLE.  SQL Trace Facility Statistics----------------------------- TKPROF lists the statistics for a SQL statement returned by the SQL trace facility in rows and columns. Each row corresponds to one ofthree steps of SQL statement processing:   * PARSE This step translates the SQL statement into an execution plan. This includes checks for proper security authorization and checks or the existence of tables, columns, and other referenced objects.  * EXECUTE This step is the actual execution of the statement by Oracle. For INSERT, UPDATE, and DELETE statements, this step modifies the data. For SELECT statements, the step identifies the selected rows.  * FETCHThis step retrieves rows returned by a query. Fetches are only performed for SELECT statements. The step for which each row contains statistics is identified by the value ofthe call column. The other columns of the SQL trace facility output arecombined statistics for all parses, all executes, and all fetches of astatement:   COUNT Number of times a statement was parsed, executed, or fetched.  CPU Total CPU time in seconds for all parse, execute, or fetch calls for the statement.  ELAPSED Total elapsed time in seconds for all parse, execute, or fetch calls for the statement.  DISK Total number of data blocks physically read from the datafiles on disk for all parse, execute, or fetch calls.  QUERY Total number of buffers retrieved in consistent mode for all parse, execute, or fetch calls. Buffers are usually retrieved in consistent mode for queries.  CURRENT Total number of buffers retrieved in current mode. Buffers are often retrieved in current mode for INSERT, UPDATE, and DELETE statements.  + The sum of QUERY & CURRENT is he total number of buffers accessed.  ROWS Total number of rows processed by the SQL statement. This total does not include rows processed by subqueries of the SQL statement. For SELECT statements, the number of rows returned appears for the fetch step.For UPDATE, DELETE, and INSERT statements, the number of rows processed appears for the execute step.  Example Output (Partial) File----------------------------- TKPROF: Release 7.2.2.3.0 - Production on Wed Aug 7 11:29:43 1996 Copyright (c) Oracle Corporation 1979, 1994. All rights reserved. Trace file: <filename>.trcSort options: default ********************************************************************************count = number of times OCI procedure was executedcpu = cpu time in seconds executingelapsed = elapsed time in seconds executingdisk = number of physical reads of buffers from diskquery = number of buffers gotten for consistent readcurrent = number of buffers gotten in current mode (usually for update)rows = number of rows processed by the fetch or execute call******************************************************************************** DELETE FROM RM$HASH_ELMS call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 2 0.00 0.00 0 0 0 0Execute 29 12.04 12.61 6786 6853 108 19Fetch 0 0.00 0.00 0 0 0 0------- ------ -------- ---------- ---------- ---------- ---------- ----------total 31 12.04 12.61 6786 6853 108 19 Misses in library cache during parse: 0Optimizer hint: CHOOSEParsing user id: 9 (DES12A) (recursive depth: 3) Rows Execution Plan------- --------------------------------------------------- 0 DELETE STATEMENT HINT: CHOOSE 16 TABLE ACCESS (FULL) OF 'RM$HASH_ELMS' *****************************END-OF-EXAMPLE-FILE********************************  Resolution of Statistics------------------------ Since timing statistics have a resolution of one hundredth of a second, any operation on a cursor that takes a hundredth of a second or less maynot be timed accurately. Keep this in mind when interpreting statistics.In particular, be careful when interpreting the results from simple queriesthat execute very quickly.  Recursive Calls--------------- Sometimes to execute a SQL statement issued by a user, Oracle must issueadditional statements. Such statements are called 'recursive calls' or'recursive SQL statements'. For example, if you insert a row into a table that does not have enough space to hold that row, Oracle makes recursivecalls to allocate the space dynamically. Recursive calls are also generatedwhen data dictionary information is not available in the data dictionarycache and must be retrieved from disk. If recursive calls occur while the SQL trace facility is enabled, TKPROFproduces statistics for the recursive SQL statements and clearly marks themas recursive SQL statements in the output file. Note that the statistics fora recursive SQL statement are included in the listing for that statement,not in the listing for the SQL statement that caused the recursive call. Sowhen you are calculating the total resources required to process a SQLstatement, you should consider the statistics for that statement as well asthose for recursive calls caused by that statement.  + Note that setting the TKPROF command line parameter to NO suppresses the listing of recursive calls in the output in the output file.  Execution Plan-------------- If you specify the EXPLAIN parameter on the TKPROF command line, TKPROFuses the EXPLAIN PLAN command to generate the execution plan of each SQL statement traced. TKPROF also displays the number of rows processed byeach step of the execution plan. ------------------------------------------------------------------------------- Worldwide Customer Support Services  
  • Overview Reference for SQL_TRACE, TKProf and Explain Plan [ID 199081.1]  Modified 08-DEC-2008     Type BULLETIN     Status PUBLISHED Checked for relevance on 22-Feb-2007. PurposeThis document gives an overview of some of the tools that can be used to discover more about SQL execution. The article provides information covering SQL_TRACE, TKProf, Explain Plan and Autotrace. Scope & ApplicationFor all DBAs Overview Reference for SQL_TRACE, TKProf and Explain PlanOracle provides extensive tracing and interpretation tools to assist analysts diagnose and resolve sub-optimal performance issues. This article covers the most important of these and points to detailed reference articles.Tracing Related Initialization ParametersPrior to tracing, there are a number of parameters that need to be set so that the trace information is complete. These parameter should be set up in the "init.ora" file for the particular instance (SID) where you wish to use SQL Trace although they can also be set individually at the session level. <<Parameter:TIMED_STATISTICS>> Enable/Disable the collection of timed statistics, such as CPU and elapsed times. TRUE - Enable timingFALSE - Disable timing (Default value).Note that much of the tracing information that can be gathered is rendered useless if TIMED_STATISTICS is set to False and timings are not collected. <<Parameter:MAX_DUMP_FILE_SIZE>> Specifies the maximum size of trace files in operating system blocks. The default value for this was 10000 OS blocks in 8i version, and limited only  by the space available in 9i version. If your trace file is truncated then you will see a message similar to: *** DUMP FILE SIZE IS LIMITED TO 12345 BYTES***and the size of this parameter should be increased. <<Parameter:USER_DUMP_DEST>> Specifies the destination for the trace file. The default value for this parameter is the default destination for oracle dumps on your operating system. These parameters can be dynamically altered using alter system/alter session commands (Note that USER_DUMP_DEST can only be modified at the system level). For example TIMED_STATISTICS can be enabled/disabled dynamically by using the following SQL statement: ALTER SYSTEM/SESSION SET TIMED_STATISTICS = TRUE/FALSE;SQL_TRACESQL_TRACE is the main method for collecting SQL Execution information in Oracle collecting a wide range of information and statistics that can be used to tune SQL operations.Enabling SQL_TRACEThe SQL Trace facility can be enabled/disabled for an individual session or at the instance level. If the initialisation Parameter SQL_TRACE is set to TRUE in the init.ora of an instance, then all sessions will be traced. SQL_TRACE can be set at the instance level by using the initialisation parameter SQL_TRACE: <<Parameter:SQL_TRACE>>: Enable/Disable SQL Trace instance wide. TRUE - Enable statistics to be collected for all sessions.FALSE - Disable statistics to be collected for all sessions.SQL_TRACE can also be enabled/disabled at the system/session by issuing the following SQL statement: ALTER SYSTEM/SESSION SET SQL_TRACE = TRUE/FALSE;Trace can also be enabled/disabled on other sessions (as well as your own) using the DBMS_SUPPORT package. See Note:62160.1 Tracing Sessions in Oracle7/8 for details. For more information on how to setup tracing, refer to: Note:15160.1 Setting SQL Trace in the Oracle Tools. Note:21154.1 EVENT: 10046 "enable SQL statement tracing (including binds/waits).Note 376442.1 Recommended Method for Obtaining 10046 trace for Tuning There is also extensive information in the Oracle9i Database Performance Tuning Guide and Reference Chapter 10: Using SQL Trace and TKProfTrace FilesOracle will generate trace (.trc) files for every session where the value of SQL_TRACE=TRUE and write them to the USER_DUMP_DEST destination. If tracing has been enabled for the instance then individual trace files will be generated for each session, unless otherwise disabled. Note that the generated files may be owned by an operating system user other than your own so the necessary privileges will need to be put in place before they can be formatted with TKProf. Explain PlanThe Explain Plan command generates information that details the execution plan that will be used on a particular query.Note that the EXPLAIN PLAN statement displays execution plans chosen by the Oracle optimizer for SELECT, UPDATE, INSERT, and DELETE statements only. For more details see: Oracle9i Database Performance Tuning Guide and Reference Release 2 (9.2) Part Number A96533-02It uses a precreated table (PLAN_TABLE) in the current shema to store information about the execution plan chosen by the optimizer. Creating the Plan TableThe plan table is created using the script utlxplan.sql. This script is typically found under the Oracle Home in the rdbms/admin directory. On Unix its location will be: $ORACLE_HOME/rdbms/admin On WindowsNT/2000:         %ORACLE_HOME%
    dbmsadmin This script creates the output table, called PLAN_TABLE, for holding the output of the Explain plan Command. Note that the exact structure of the plan table can change with different release as new features are introduced. Populating the Plan TableThe plan table is populated using the explain plan command: SQL> EXPLAIN PLAN for select * from emp where empno=1000;This command inserts the execution plan of the SQL statement into the plan table. It is also possible to adds the name tag to the explain information by using the set statement_id clause. Displaying the Execution PlanOnce the table has been populated, the explain information needs to be retrieved and formatted. There are a large number of scripts available to format the plan table data. Some of the most popular are noted below: Supplied Scripts:$ORACLE_HOME/rdbms/admin/utlxpls.sql: script to format serial explain plans $ORACLE_HOME/rdbms/admin/utlxplp.sql: script to format parallel explain plansArticles:Note:31101.1 Obtaining Formatted Explain Plan Output Note:39294.1 Formatted Select of PLAN_TABLE for EXPLAIN PLAN command Note:39341.1 Automatic Explain Plan Note:1019631.6 SCRIPT: SCRIPT TO SIMPLIFY THE USE OF EXPLAIN PLANInterpretation of Explain PlanThis is a complex topic and is covered in detail in the following article: Note:46234.1 Interpreting Explain PlanThere is also extensive information in the Oracle9i Database Performance Tuning Guide and Reference Chapter 9: Using EXPLAIN PLANAUTOTRACEThe autotrace facility in SQL*Plus allows analysts to view the execution plan and some useful statistics for a SQL statement within a SQL*Plus session. This option was introduced with 7.3 version of Oracle. Autotrace needs to be initiated in the SQL*Plus session prior to executing the statement. The Autotrace command is: SET AUTOTRACE [OPTIONS] [EXPLAIN/STATISTICS]For a detailed explanation of AUTOTRACE functions see: Note:43214.1 AUTOTRACE option in 7.3As with the EXPLAIN PLAN command, to obtain an execution plan the PLAN_TABLE must be created in the user's schema prior to autotracing. Example SQL> set autotrace traceonly explainSQL> select * from dual;Execution Plan----------------------------------------------------------0 SELECT STATEMENT Optimizer=CHOOSE1 0 TABLE ACCESS (FULL) OF 'DUAL'To enable viewing of STATISTICS data, the autotracing user must have access to dynamic performance tables. To achieve this, grant PLUSTRACE role to the user. The PLUSTRACE role is created by the plustrce.sql script.     On Unix the location is:         $ORACLE_HOME/sqlplus/admin    On WindowsNT/2000:         %ORACLE_HOME%sqlplusadmin This script must be run by the SYS user. A DBA user can then grant the role to the users who wish to use the the AUTOTRACE option. Refer to: Note:1055431.6 ORA-01919 usingAUTOTRACE in SQL*PlusExtensive reference information regarding the autotrace facility can be found in the SQL*Plus Users Guide and Reference Release 9.0.1TKProfThe TKProf facility accepts as input a SQL trace file and produces a formatted output file. For the full syntax of TKProf see the Oracle Server Tuning Manual. If TKProf is invoked with no arguments, an online help is displayed. Basic Syntax of TKProfTKPROF filename_sourcefilename_output EXPLAIN=[username/password] SYS=[YES/No] TABLE=[tablename] filename_sourceThe trace file that needs to be formattedfilename_outputThe output file where the formatted file will be written'EXPLAIN'This option causes TKProf to generate an execution plan for each SQL statement in the trace file as though it was been parsed at the time the TKProf is being run. TKProf connects as the user [username] and uses the EXPLAIN PLAN command to generate an execution plan for each traced SQL statement. These explain plans are written to the output file. Note that in later releases, the raw trace file automatically records the explain plan that was used at execution time and TKProf will automatically output this plan. If the EXPLAIN option is selected then two explain plans appear in the TKProf output. The first explain plan is the plan that was used at execution time. The second plan is the plan that was generated dynamically at the time the TKProf runs. If session settings etc have changed between the execution time and when the trace file is TKProffed then the first and second plans may differ. 'TABLE'Specifies the schema and name of the table into which TKPROF temporarily places execution plans before writing them to the output file. If the specified table already exists, TKProf deletes its rows then uses it for the EXPLAIN PLAN command and then deletes its rows. If this table does not exist, TKProf creates, uses, then drops it. This parameter is ignored if the EXPLAIN parameter isn't used.'SYS'Enables and disables the listing of SQL statements issued by the user SYS (recursive SQL statements) into the output file. The default value is YES.For more information on using TKProf see: Note:32951.1 TKPROF Interpretation Note:29012.1 Quick Reference TKPROF usage Note:41634.1 TKPROF and Problem SolvingThere is also extensive information in the Oracle9i Database Performance Tuning Guide and Reference Chapter 10: Using SQL Trace and TKProf  TKProf Interpretation (9i and below) [ID 32951.1]  Modified 24-MAR-2009     Type BULLETIN     Status ARCHIVED  PURPOSEThis document gives general advice on the interpretation of TKPROF output andprovides some examples on how this information can be used. SCOPE & APPLICATIONFor users needing advice on what to use TkProf for. For TKProf usage information see:  Note:29012.1 QREF: TKPROF Usage - Quick Reference  Tkprof is an executable that 'parses' Oracle trace files to produce morereadable output. Remember that all the information in TkProf is availablefrom the base trace file. If you have a system that is performing badly, a good way to identify problem SQL statements is to trace a typical user session and then use TkProfto format the output using the sort functions on the tkprof command line.  There are a huge number of sort options that can be accessed by simply typing 'TkProf' at the command prompt. A useful starting point is the 'fchela' sort option which orders the output by elapsed time fetching (rememberthat timing information is only available with timed_statistics set to true in the "init.ora" file). The resultant .prf file will contain the most time consuming SQL statement at the start of the file. Another useful parameter is sys. This can be used to prevent SQL statements run as user SYS from being displayed. This can make the output file much shorter an easier to manage. Remember to always set the TIMED_STATISTICS parameter to TRUE when tracingsessions as otherwise no time based comparisons can be made.  Interpreting TkProf Output Guidelines===================================== Column Meanings=============== call : Statisics for each cursor's activity are divided in to 3 areas:   Parse: statisitics from parsing the cursor. This includes information for plan generation etc. Execute: statisitics for the exection phase of a cursor Fetch : statistics for actually fetching the rows count : number of times we have performed a particular activity on this particular cursor cpu: cpu time used by this cursor elapsed: elapsed time for this cursor disk: This indicates the number of blocks read from disk. Generally you want to see blocks being read from the buffer cache rather than disk.query : This column is incremented if a buffer is read in Consistent mode. A Consistent mode buffer is one that has been generated to give a consistent read snapshot for a long running transaction. The buffer actually contains this status in its header.current: This column is incremented if a buffer found in the buffer cache that is new enough for the current transaction and is in current mode (and it is not a CR buffer). This applies to buffers that have been read in to the cache as well as buffers that already exist in the cache in current mode. rows: Rows retrieved by this step   Explain plan ============ Firstly, we advise that the autotrace feature of SQL*Plus be used on statements rather than using TkProf mainly because the TkProf output can be confusing with regard to whether the Rule or Cost Based optimizer has been used.  Because TkProf explain plan does not show any costs or statistics, it is sometimes not possible to tell definitively which optimizer has been used. That said, the following output from Tkprof explain plan is useful.The Rows column next to the explain plan output shows the number of rows processed by that particular step. The information is gathered from the STAT lines for each cursor in the raw trace output.  Remember that if the cursor is not closed then you will not see any output. Setting SQL_TRACE to false DOES NOT close PL/SQL child cursors. Cursors are closed in SQL*Plus immediately after execution.  TkProf Examples and Discussion============================== Examples: Step 1 - Look at the totals at the end of the tkprof output=========================================================== OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS| call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|------|---------|---------|--------|---------|--------|| Parse | [A] 7 | 1.87 | 4.53 | 385 |[G] 553 | 22 | 0 || Execute | [E] 7 | 0.03 | 0.11 | [P] 0 |[C] 0 | [D] 0 | [F] 0 || Fetch | [E] 6 | 1.39 | 4.21 | [P] 128 |[C] 820 | [D] 3 | [F] 20 |--------------------------------------------------------------------------Misses in library cache during parse: 5Misses in library cache during execute: 1 8 user SQL statements in session. 12 internal SQL statements in session.[B] 54 SQL statements in session. 3 statements EXPLAINed in this session. 1. Compare [A] & [B] to spot over parsing. In this case we have 7 parses for 54 statements which is ok. 2. You can use [P], [C] & [D] to determine the hit ratio. Hit Ratio is logical reads/physical reads: Logical Reads = Consistent Gets + DB Block GetsLogical Reads = query + currentLogical Reads = Sum[C] + Sum[D]Logical Reads = 0+820 + 0+3Logical Reads = 820 + 3Logical Reads = 823 Hit Ratio = 1 - (Physical Reads / Logical Reads)Hit Ratio = 1 - (Sum[P] / Logical Reads)Hit Ratio = 1 - (128 / 823)Hit Ratio = 1 - (0.16)Hit Ratio = 0.84 or 84% 3. We want fetches to be less than the number of rows as this will mean we have done less work (array fetching). To see this we can compare [E] and [F]. [E] = 6 = Number of Fetches[F] = 20 = Number of Rows So we are doing 6 fetches to retrieve 20 rows - not too bad.If arrayfetching was configured then rows could be retrieved with less fetches. Remember that an extra fetch will be done at the end to check thatthe end of fetch has been reached. 4. [G] Shows reads on the Dictionary cache for the statements.  - this should not be a problem on Oracle7. In this case we have done 553 reads from the Library cache. STEP 2 - Examine statements using high resource=============================================== update ...where ... | call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|-----|---------|------|---------|---------|--------|| Parse | 1 | 7 | 122 | 0 | 0 | 0 | 0 || Execute | 1 | 75 | 461 | 5 | [H] 297 | [I] 3 | [J] 1 || Fetch | 0 | 0 | 0 | 0 | 0 | 0 | 0 |----------------------------------------------------------------------- [H] shows that this query is visiting 297 blocks to find the rows to update[I] shows that only 3 blocks are visited performing the update[J] shows that only 1 row is updated. 297 block to update 1 rows is a lot. Possibly there is an index missing?  STEP 3 - Look for over parsing============================== select ... | call | count | cpu | elapsed | disk | query | current | rows ||---------|-------|---------|---------|------|--------|---------|-------|| Parse | [M] 2 | [N] 221 | 329 | 0 | 45 | 0 | 0 || Execute | [O] 3 | [P] 9 | 17 | 0 | 0 | 0 | 0 || Fetch | 3 | 6 | 8 | 0 | [L] 4 | 0 | [K] 1 |------------------------------------------------------------------------- Misses in library cache during parse: 2 [Q] [K] is shows that the query has returned 1 row.[L] shows that we had to read 4 blocks to get this row back.This is fine. [M] show that we are parsing the statement twice - this is not desirable especially as the cpu usage is high [N] in comparison to the execute figures : [O] & [P]. [Q] shows that these parses are hard parses. If [Q] was 1 then the statemnent would have had 1 hard parse followed by a soft parse (which just looks up the already parsed detail in the library cache). See Note:32895.1 for more details.  This is not a particularly bad example since the query has only been executed a few times. However excessive parsing should be avoided as far as possible by: o Ensuring that code is shared:  - use bind variables - make shared pool large enough to hold query definitions in memory long enough to be reused.  NOTE:====We need to ensure, we do not parse a >=9i version 10046 raw trace with an <= 8i tkprof utility.This is because the timings in raw trace file for <=8i versions and >=9i versions are in centiseconds and microseconds respectively.  SEARCH WORDS============ EXPLAIN; PARSE; PLAN; STATEMENT; TKPROF;  QREF: TKPROF Usage - Quick Reference [ID 29012.1]  Modified 25-SEP-2008     Type FAQ     Status PUBLISHED  Quick Reference to Using TKPROF~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  TKPROF allows you to analyse a trace file to determine where time is being spent and what query plans are being used on SQL statements.  1 - Set TIMED_STATISTICS if required at database level. 2 - Get SQL_TRACE output for the session you wish to monitor (See Note:15160.1 for how to set this in various tools)  3 - Find the appropriate trace file (In USER_DUMP_DEST, default $ORACLE_HOME/rdbms/log on Unix). You can find the most recent trace files on Unix with the command: ls -ltr This will list the most recent files LAST  4 - Run tkprof on the trace file thus: tkprof tracefile outfile [explain=user/password] [options...] TKPROF Options~~~~~~~~~~~~~~ print=integer List only the first 'integer' SQL statements. insert=filename List SQL statements and data inside INSERT statements. sys=no TKPROF does not list SQL statements run as user SYS. record=filename Record statements found in the trace file. sort=option Set of zero or more of the following sort options: prscnt number of times parse was calledprscpucpu time parsingprsela elapsed time parsingprsdsk number of disk reads during parseprsqry number of buffers for consistent read during parseprscu number of buffers for current read during parseprsmis number of misses in library cache during parse execnt number of execute was calledexecpucpu time spent executingexeela elapsed time executingexedsk number of disk reads during executeexeqry number of buffers for consistent read during executeexecu number of buffers for current read during executeexerow number of rows processed during executeexemis number of library cache misses during execute fchcnt number of times fetch was calledfchcpucpu time spent fetchingfchela elapsed time fetchingfchdsk number of disk reads during fetchfchqry number of buffers for consistent read during fetchfchcu number of buffers for current read during fetchfchrow number of rows fetched useriduserid of user that parsed the cursor TKProf Basic Overview [ID 41634.1]  Modified 19-SEP-2008     Type REFERENCE     Status PUBLISHED  TKPROF and Problem Solving========================== Purpose-------This document discusses the use of the following: * SQL TRACE * EXPLAIN PLAN * TKPROF  Scope & Application--------------------For users wanting to obtain a query plan and/or execution times for their queries.  Related Documents------------------The majority of the material covered in this document can be found in the following manuals: * Oracle7 Server SQL Reference * Oracle7 Server Tuning  Introduction------------Much information can be gathered about the performance of an Oracledatabase from the dynamic 'V$' views, as these are documented in theOracle7 Server Manual. Oracle also provides additional performancediagnostic tools to assist in monitoring and tuning the server. The reader should be familiar with the concepts of the Oracle Serverinitialization file and startup parameters. Most of the suggestionsdescribed in this document should be conducted by an experienced DBA.  Overview Of SQL TRACE---------------------The diagnostic tool 'sql trace' provides performance information aboutindividual SQL statements and generates the following statistics foreach statement:  * parse, execute, and fetch counts * CPU and elapsed times * physical reads and logical reads * number of rows processed * misses on the library cache  This information is input to a trace (.trc) file and sql trace can beenabled/disabled for a session or an instance.  Setting Initialization Parameters---------------------------------The following parameters need to be set up in the "init<sid>.ora" filefor the particular instance (SID) that you wish to use SQL Trace:  * SQL_TRACE + Enable/Disable SQL Trace for the instance.  Values ------ TRUE Enable statistics to be collected for all sessions. FALSE Disable statistics to be collected for all sessions.  * TIMED_STATISTICS + Enable/Disable the collection of timed statistics, such as CPU and elapsed times.  Values ------ TRUE Enable timing (we usually recommend this) FALSE Default value.  * MAX_DUMP_FILE_SIZE + Specifies the maximum size of trace files operating system blocks.  Values ------ The default value for this is 500 but if your trace file is truncated then increase this value.  * USER_DUMP_DEST + Specifies the destination for the trace file.  Values ------ The default value for this parameter is the default destination for system dumps on your operating system.   Enabling SQL Trace------------------The SQL Trace facility can either be enabled/disabled for an individualsession or the instance.  * To enable the SQL trace facility for your session issue the following SQL statement:  ALTER SESSION SET SQL_TRACE = TRUE;   * To disable the SQL trace facility for your session issue the following SQL statement:  ALTER SESSION SET SQL_TRACE = FALSE;  Enabling TIMED_STATISTICS-------------------------The parameter TIMED_STATISTICS can be enabled/disabled dynamically by using ALTER SYSTEM SET TIMED_STATISTICS = TRUE;  or ALTER SESSION SET TIMED_STATISTICS = TRUE; To disable set TIMED_STATISTICS = FALSE: SQL Trace Facility for an Instance----------------------------------Note that if the initialization parameter SQL_TRACE=TRUE, then statisticswill be gathered for ALL sessions. If the facility has been enabled forthe instance, it may be disabled for an individual session by issuing bythe above SQL statement.  Trace Files-----------Oracle will generate trace (.trc) files for every session where the valueof SQL_TRACE = TRUE and write them to the USER_DUMP_DEST destination.If tracing has been enabled for the instance then individual trace fileswill be generated for each session, unless otherwise disabled (see above).Note, that the generated files may be owned by an operating system userother than your own so you may have to get this user to grant you accessbefore you can use TKPROF to format them.  Using TKPROF------------The TKPROF facility accepts as input an SQL trace file and produces aformatted output file. Note that TKPROF can be run on individual orappended trace files to produce the formatted output file.  * For the full syntax of TKPROF see Oracle7 Server Tuning Manual.  * Invoke TKPROF with no arguments and online help is displayed.  Syntax of TKPROF---------------- TKPROF command ::= >>-- TKPROF filename1 filename2 ----------------------------------------------> | | +- SORT = ---------------------------------+ | | +-- OPTION --+ | | | +---- , ----+ | | V | | |__( OPTION )__| >-----------------------------------------------------------------------------> | | | | | | +-- PRINT = integer --+ +-- INSERT = filname3 --+ +-- SYS = ---------+ | | +- YES -+ | | +- NO --+ >-----------------------------------------------------------------------------> | | +---------------------------------------- EXPLAIN = user/password ------+ | | +---- TABLE = schema.table ----+ >---------------------------------------------------------------------------->< | | +---- RECORD = filname ----+  The Basics Of TKPROF-------------------- For the purpose of this document we will only describe some of thefundamental arguments for TKPROF:  * 'filename1' Specifies the input file, a trace file containing statistics produced by the SQL trace facility. This file can be either a trace file produced for a single session or a file produced by appending together together individual trace files from multiple sessions.  * 'filename2' Specifies the file to which TKPROF writes its formatted output.  * 'EXPLAIN' Determines the execution plan for each SQL statement in in the trace file and writes these execution plans to the output file. TKPROF determines execution plans by issuing the EXPLAIN PLAN command after connecting to Oracle with the user and password specified in this parameter. The specified user must have CREATE SESSION privileges.  * 'TABLE' Specifies the schema and name of the table into which TKPROF temporarily places execution plans before writing them to the output file. If the specified table already exists, TKPROF deletes its rows then uses it for the EXPLAIN PLAN command and then deletes its rows. If this table does not exist, TKPROF creates, uses, then drops it. The specified user must be able to issue INSERT, SELECT, and DELETE statements against the table. If the table does not already exist, the user must also be able to issue CREATE TABLE and DROP TABLE statements. This option allows multiple individuals to run TKPROF concurrently with the same user in the EXPLAIN value. These individuals can specify different TABLE values and avoid destructively interfering with each other's processing on the temporary plan table. If you use the EXPLAIN parameter without the TABLE parameter, TKPROF uses the table PROF$PLAN_TABLE in the schema of the user specified by the EXPLAIN parameter. If you use the TABLE parameter without the EXPLAIN parameter, TKPROF ignores the TABLE parameter.   * 'SYS' Enables and disables the listing of SQL statements issued by the user SYS, or recursive SQL statements into the output file. The default value of YES causes TKPROF to list these statements. The value of NO causes TKPROF to to omit them.  Simple Example-------------- This example shows TKPROF being run to format a trace file named"dsdb2_ora_18468.trc" and writing it to a formatted output filenamed "dsdb2_trace.out".  TKPROF dsdb2_ora_18468.trc dsdb2_trace.out SYS=NO EXPLAIN=SCOTT/TIGER Note that if the command is likely to be longer than a single line on yourterminal screen you may have to use continuation characters.  The Parameters In This Example: ------------------------------- The EXPLAIN value causes TKPROF to connect as the user SCOTT and use the EXPLAIN PLAN command to generate the execution plan for each traced SQL statement. The SYS parameter with the value of NO causes TKPROF to omitrecursive SQL statements from the output file.  Interpretting TKPROF Output=========================== The EXPLAIN PLAN Command------------------------ The EXPLAIN PLAN command displays the execution plan chosen by the Oracle optimizer for SELECT, UPDATE, INSERT, and DELETE statements.A statement's execution plan is the sequence of operations that Oracle performs to execute the statement. By examining the execution plan, you can see exactly how Oracle executes your SQL statement. This information can help you determine whether the SQL statement you have written takes advantage of the indexes available. For the syntax of the EXPLAIN PLAN command, see the Oracle7 Server SQL Reference Manual.   Creating the Output Table------------------------- Before you can issue an EXPLAIN PLAN statement, there must exist a table to hold its output, you do either of the following:  * Run the SQL script "UTLXPLAN.SQL" to create a sample output table called PLAN_TABLE in your schema.  * Issue a CREATE TABLE statement to create an output with any name you choose. You can then issue an EXPLAIN PLAN statement and direct its output to this table. Any table used to store the output of the EXPLAIN PLAN command must have the same column names and datatypes as the PLAN_TABLE.  SQL Trace Facility Statistics----------------------------- TKPROF lists the statistics for a SQL statement returned by the SQL trace facility in rows and columns. Each row corresponds to one ofthree steps of SQL statement processing:   * PARSE This step translates the SQL statement into an execution plan. This includes checks for proper security authorization and checks or the existence of tables, columns, and other referenced objects.  * EXECUTE This step is the actual execution of the statement by Oracle. For INSERT, UPDATE, and DELETE statements, this step modifies the data. For SELECT statements, the step identifies the selected rows.  * FETCH This step retrieves rows returned by a query. Fetches are only performed for SELECT statements. The step for which each row contains statistics is identified by the value ofthe call column. The other columns of the SQL trace facility output arecombined statistics for all parses, all executes, and all fetches of astatement:   COUNT Number of times a statement was parsed, executed, or fetched.  CPU Total CPU time in seconds for all parse, execute, or fetch calls for the statement.  ELAPSED Total elapsed time in seconds for all parse, execute, or fetch calls for the statement.  DISK Total number of data blocks physically read from the datafiles on disk for all parse, execute, or fetch calls.  QUERY Total number of buffers retrieved in consistent mode for all parse, execute, or fetch calls. Buffers are usually retrieved in consistent mode for queries.  CURRENT Total number of buffers retrieved in current mode. Buffers are often retrieved in current mode for INSERT, UPDATE, and DELETE statements.  + The sum of QUERY & CURRENT is he total number of buffers accessed.  ROWS Total number of rows processed by the SQL statement. This total does not include rows processed by subqueries of the SQL statement. For SELECT statements, the number of rows returned appears for the fetch step.For UPDATE, DELETE, and INSERT statements, the number of rows processed appears for the execute step.  Example Output (Partial) File----------------------------- TKPROF: Release 7.2.2.3.0 - Production on Wed Aug 7 11:29:43 1996 Copyright (c) Oracle Corporation 1979, 1994. All rights reserved. Trace file: <filename>.trcSort options: default ********************************************************************************count = number of times OCI procedure was executedcpu = cpu time in seconds executingelapsed = elapsed time in seconds executingdisk = number of physical reads of buffers from diskquery = number of buffers gotten for consistent readcurrent = number of buffers gotten in current mode (usually for update)rows = number of rows processed by the fetch or execute call******************************************************************************** DELETE FROM RM$HASH_ELMS call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 2 0.00 0.00 0 0 0 0Execute 29 12.04 12.61 6786 6853 108 19Fetch 0 0.00 0.00 0 0 0 0------- ------ -------- ---------- ---------- ---------- ---------- ----------total 31 12.04 12.61 6786 6853 108 19 Misses in library cache during parse: 0Optimizer hint: CHOOSEParsing user id: 9 (DES12A) (recursive depth: 3) Rows Execution Plan------- --------------------------------------------------- 0 DELETE STATEMENT HINT: CHOOSE 16 TABLE ACCESS (FULL) OF 'RM$HASH_ELMS' *****************************END-OF-EXAMPLE-FILE********************************  Resolution of Statistics------------------------ Since timing statistics have a resolution of one hundredth of a second, any operation on a cursor that takes a hundredth of a second or less maynot be timed accurately. Keep this in mind when interpreting statistics.In particular, be careful when interpreting the results from simple queriesthat execute very quickly.  Recursive Calls--------------- Sometimes to execute a SQL statement issued by a user, Oracle must issueadditional statements. Such statements are called 'recursive calls' or'recursive SQL statements'. For example, if you insert a row into a table that does not have enough space to hold that row, Oracle makes recursivecalls to allocate the space dynamically. Recursive calls are also generatedwhen data dictionary information is not available in the data dictionarycache and must be retrieved from disk. If recursive calls occur while the SQL trace facility is enabled, TKPROFproduces statistics for the recursive SQL statements and clearly marks themas recursive SQL statements in the output file. Note that the statistics fora recursive SQL statement are included in the listing for that statement,not in the listing for the SQL statement that caused the recursive call. Sowhen you are calculating the total resources required to process a SQLstatement, you should consider the statistics for that statement as well asthose for recursive calls caused by that statement.  + Note that setting the TKPROF command line parameter to NO suppresses the listing of recursive calls in the output in the output file.  Execution Plan-------------- If you specify the EXPLAIN parameter on the TKPROF command line, TKPROFuses the EXPLAIN PLAN command to generate the execution plan of each SQL statement traced. TKPROF also displays the number of rows processed byeach step of the execution plan. ------------------------------------------------------------------------------- Worldwide Customer Support Services  
  • Interpreting HANGANALYZE trace files to diagnose hanging and performance problems [ID 215858.1]  Modified 04-DEC-2008     Type BULLETIN     Status PUBLISHED PURPOSEThis article explains how to use the HANGANALYZE utility to diagnose hanging issues in Oracle Databases.BACKGROUNDWhen a database has severe performance problems, the cause of the problem may be a HANG. Using trace files produced by HANGANALYZE, you will be able to quickly determine if two or more processes are deadlocked and how many other processes may be affected. By using HANGANALYZE, you may also determine if a performance analysis will be required to diagnose the real problem when a HANG is taking place.CONTENTWHY USING HANGANALYZEUSING HANGANALYZEINTERPRETING HANGANALYZE DUMPSHANGANALYZE BY EXAMPLESTATE OF THE NODES DESCRIPTIONLEVEL USAGE RECOMMENDATIONSHANGANALYZE AND OTHER TRACING FACILITIESRELATED DOCUMENTS WHY USING HANGANALYZEA “true” hang in the Oracle database can be defined as an internal deadlock or a cyclical dependency between two or more processes. When dealing with DML locks (i.e., TM), Oracle is able to detect this dependency and rollback one of the processes to break the cyclical condition. On the other hand, when this situation happens with internal kernel-level resources (e.g latches or pins), Oracle is usually unable to automatically detect and resolve the deadlock.A severe performance problem may easily be mistaken for a hang. This usually happens when contention is so bad that it seems like the database is completely hung. Previous to HANGANALYZE (Available since Oracle realease 8.0.6), the only tool available to diagnose this type of problem was the well-known “SYSTEMSTATE dump”. A SYSTEMSTATE dump has the following disadvantages when dealing with hanging issues:  SYSTEMSTATE dump reads the SGA in a “dirty” manner, so it may be inconsistent when the time to dump all the process is long.SYSTEMSTATE dump usually dumps a lot of information (most of which is not needed to determine the source of the hang), which makes difficult to determine the dependencies between processes quickly.SYSTEMSTATE dumps do not identify “interesting” processes on which to perform additional dumps (ERRORSTACK or PROCESS STATE).HANGANALYZE use internal kernel calls to determine if a session is waiting for a resource, and reports the relationships between blockers and waiters. In addition, it determines which processes are “interesting” to be dumped, and may perform automatic PROCESSSTATE dumps and ERRORSTACKS on those processes, based on the level used while executing the HANGANALYZE.Note: HANGANALYZE is not intented to replace a SYSTEMSTATE dump, but may serve as a road map to interpret a SYSTEMSTATE while diagnosing complex issues.  USING HANGANALYZEThe “HANGANALYZE” command is available since Oracle Release 8.1.6. In Oracle9i it was enhanced to provide “cluster wide” information in Real Application Cluster (RAC) environments on a single shot. The meaning of this is that it will generate information for all the sessions in the cluster regardless of the instance that issued the command.HANGANALYZE may be executed using the following syntax:ALTER SESSION SET EVENTS 'immediate trace name HANGANALYZE level <level>';Or when logged in with the “SYSDBA” role,ORADEBUG hanganalyze <level>To perform cluster wide HANGANALYZE use the following syntax:ORADEBUG setmypidORADEBUG setinst allORADEBUG -g def hanganalyze <level>The <level> sets the amount of additional information that will be extracted from the processes found by HANGANALYZE (ERROSTACK dump) based on the “STATE” of the node. See further detail in the “STATE DESCRIPTION” section of this document.The levels are defined as follows:10     Dump all processes (IGN state)5      Level 4 + Dump all processes involved in wait chains (NLEAF state)4      Level 3 + Dump leaf nodes (blockers) in wait chains (LEAF,LEAF_NW,IGN_DMP state)3      Level 2 + Dump only processes thought to be in a hang (IN_HANG state)1-2    Only HANGANALYZE output, no process dump at allINTERPRETING HANGANALYZE DUMPSHANGANALYZE has several sections, which may or not be reported in the trace file. They are only generated when HANGANALYZE finds information related to the section.Sections where session information is reported always have a header that details the information that is extracted. Previous to Oracle9i the information extracted is always consistent, while in Oracle9i, especially when getting “cluster wide” HANGANALYZE trace files, it may gather information related to the operating system only on the node where the HANGANALYZE was issued.Examples of session “headers”:Oracle 8.x chain header:  <sid/sess_srno/proc_ptr/ospid/wait_event>Oracle9i chain header:  <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :Where: sid        = Session ID sess_srno  = Serial# proc_ptr   = Process Pointer ospid      = OS Process Id wait_event = Waitevent cnode      = Node Id (Only available since Oracle9i)State of Nodes headers are explained in the section called: STATE OF THE NODES DESCRIPTIONThe following example shows how to interpret each of the sections of a HANGANALYZE trace file:  *** 2002-05-13 17:02:30.937==============HANG ANALYSIS:==============CYCLES: This section reports the process dependencies between sessions that are in a deadlock condition. Cycles are considered “true” hangs.Cycle 1 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <980/3887/0xe4214964/24065/latch free> -- <2518/352/0xe4216560/24574/latch free> -- <55/10/0xe41236a8/13751/latch free>BLOCKER OF MANY SESSIONS: This section is found when a process is blocking a lot of other sessions. Usually when a process is blocking more that 10 sessions this section will appear in the trace file.Found 21 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <55/10/0xe41236a8/13751/latch free>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <2098/2280/0xe42870d0/3022/db file scattered read>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <1941/1783/0xe41ac9e0/462/No Wait>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <980/3887/0xe4214964/24065/latch free>OPEN CHAINS: This section reports sessions involved on a wait chain. A wait chains means that one session is blocking one or more other sessions.Open chains found:Chain 1 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <2/1/0xe411b0f4/12280/db file parallel write>Chain 2 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <3/1/0xe411b410/12282/No Wait>Chain 6 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <18/1631/0xe4243cf8/25457/db file scattered read> -- <229/1568/0xe422b84c/8460/buffer busy waits>Chain 17 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <56/11/0xe4123ce0/13755/latch free> -- <2384/599/0xe41890dc/22488/latch free> -- <32/2703/0xe41fa284/25693/latch free>OTHER CHAINS: It refers to chains of blockers and waiters related to other sessions identified under “open chains”, but not blocked directly by the process reported on the "open chain". Other chains found:Chain 676 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <20/93/0xe411d644/13597/latch free>Chain 677 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <27/1201/0xe41d3188/15809/latch free>Chain 678 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <36/1532/0xe428be8c/4232/latch free> -- <706/1216/0xe4121aac/23317/latch free>Chain 679 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <43/12/0xe4122d54/13745/latch free>Chain 680 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <80/2/0xe41290d4/13811/library cache pin> -- <1919/1134/0xe421fdbc/3343/enqueue>Additional DUMP detail: HANGANALYZE generates errorstack dumps for individual processes based on: the LEVEL used when the command is issued, and the STATE of the session. This part of the HANGANALYZE trace indicates which processes will be dumped if higher levels are used. It also indicates how many nodes will be dumped so you can measure the impact to the system (if too many errorstack dumps occur simultaneously, the disks serving the trace destination may become saturated). The “STATE DESCRIPTION” section of this document further explains the meaning of the states.Extra information that will be dumped at higher levels:[level  4] :  23 node dumps -- [LEAF] [LEAF_NW] [IGN_DMP][level  5] :  36 node dumps -- [NLEAF][level 10] : 130 node dumps -- [IGN]STATE OF NODES: This section might be considered as the main section of the report. It shows all the sessions connected when the HANGANALYZE trace file was generated. This section essentially describes a dependency graph between nodes(known as an “adjacency list”), where each session is considered a node, and each node may have an “adjacent” node or a “predecessor” node related to it. Depending on the state of the node it may be a blocker, a waiter or a hung node. The “STATE DESCRIPTION” section of this document explains further the meaning of the states.State of nodes([nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessor):[0]/1/1/0xa6f8b0/IGN/1/2//none[1]/2/1/0xa70230/IGN/3/4//none[3]/4/1/0xa71530/IGN/5/6//none[4]/5/1/0xa71eb0/IGN/7/8//none[5]/6/1/0xa72830/IGN/9/10//none[6]/7/1/0xa731b0/IGN/11/12//none[7]/8/1/0xa73b30/IGN/13/14//none[8]/9/1/0xa744b0/IGN_DMP/15/18/[130]/none[9]/10/1/0xa74e30/IGN/19/20//none[10]/11/4202/0xa757b0/IGN/21/22/[130]/none[11]/12/1196/0xa76130/NLEAF/23/28/[49]/none[12]/13/1/0xa76ab0/IGN/29/30/[130]/none[37]/38/37/0xa85830/NLEAF/73/76/[50]/46[46]/47/15/0xa8adb0/NLEAF/91/92/[37][50]/none====================END OF HANG ANALYSIS====================  STATE OF THE NODES DESCRIPTION As described before, HANGANALYZE use the model of “Adjacency Lists” to report the sessions found when the HANGANALYZE command was issued. In Graph Theory, an adjacency list represents a list of nodes that are connected to a given node. For each node, a linked list of nodes connected to it can be set up to represent the neighbor node connected. To be able to interpret a HANGANALYZE trace file it is required to relate the “STATE” of the node with the “adjacent” node to it.A typical entry in the state of the nodes section will be as follow:Oracle 8.x : [nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessorOracle9i: [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessorWhere: Nodenum   = This is secuencial number used by HANGANALYZE to identify each session sid       = Session ID sess_srno = Serial# ospid     = OS Process Id state     = State of the node adjlist   = adjacent node (Usually represents a blocker node) predecessor = predecessor node (Usually represents a waiter node) cnode     = Node number (Only available since Oracle9i)The following describes the important states to be considered:IN_HANG: This might be considered as the most critical STATE. Basically a node in this state is involved in a deadlock, or is hung. Usually there will be another “adjacent node” in the same status. For example: [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/IN_HANG/29/32/[185]/19 [185]/1/16/4966/0x24617270//IN_HANG/30/31/[16]/16In this example the node [16] is waiting for node [185], and the other way around; this is a cyclical condition (deadlock).  LEAF and LEAF_NW: Leaf nodes are considered on top of the wait chain (usually blockers). They are considered “Blockers” when there is another session waiting. This can be easily identified using the “predecesor” field. If there is a node referenced in the ‘prdecessor’ field, the node is considered as “blocker”, otherwise it is considered as a “slow” session waiting for some resource.The difference between LEAF and LEAF_NW is that LEAF nodes are not waiting for something, while LEAF_NW are not waiting or may be using the CPU. A typical representation of these nodes when they are considered blockers is: [ nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/LEAF/29/30//19 [19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186In this example, node [16] is blocking node [19]. Notice that node [16] has node [19] in the predecessor field.Also notice that node [19] has node [16] in the adjacent list.  NLEAF : These sessions are usually considered as “stuck” sessions. It means that there is another session holding a resource needed by the session in this state. By using the adjlist, you can determine which node is the blocker of this process. When many sessions are found in this state, it is likely the database is experiencing a performance problem rather than a hang problem.  IGN and IGN_DMP : Sessions in this state are usually considered as IDLE sessions, unless they reference a node in the “ adjlist” field. In this case, the node is waiting for another node, so it will be considered as a ‘stuck’ session as well.Extending the previous example, [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/LEAF/29/30//19 [19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186 [189]/1/20/36/0x24619830//IGN/95/96/[19]/none [176]/1/7/1/0x24611d80//IGN/75/76//noneYou may notice that node [189] is waiting for node [19] which in turn is waiting for node [16], while node [176] is an IDLE session . This maybe the case when a session has a DML lock but never finished the transaction.  In Oracle9i, two new states were introduced to differentiate between LEAF nodes that have other nodes waiting behind them (i.e., LEAF nodes that are blockers) vs. LEAF nodes that are not affecting other nodes.  SINGLE_NODE and SINGLE_NODE_NW:This can be described the same as LEAF and LEAF_NW, except that they don't have processes depending on them.  LEVEL USAGE RECOMMENDATIONSIt is advisable not to use levels higher than 3 due to the potentially large number of trace files that may be produced (and could overwhelm the I/O subsystem). Since HANGANALYZE will be mostly used to diagnose “true hangs”, a level 3 will dump the processes that are involved in a the hang condition – this usually involves fewer than 4 processes.Level 4 may be used with caution when dealing with performance scenarios where sessions are being stuck for a long period of time, and no other information can be gathered. Be carefull that a HANGANALYZE level 4 will dump all the processes in the following states:[LEAF] [LEAF_NW] [IGN_DMP]. Before doing so, perform a HANGANALYZE level 3, and check the section: “Extra information that will be dumped at higher levels”. This section will tell you exactly how many sessions will be dumped at each level. It is diffucult to know how many dumps are an acceptable number to keep from saturating the disks due to the differences in disk implementations (5 to 10 is probably OK for most systems; larger systems may be able to handle 20 to 40 dumps). Level 5 and 10 are typicaly useless and generate a huge amount of trace files that may severely impact the performance of the database.HANGANALYZE AND OTHER TRACING FACILITIESWhen diagnosing problems using HANGANALYZE it is recommended also to generate a SYSTEMSTATE dump level 2 to be able to see all the session details. HANGANALYZE together with the SYSTEMSTATE dump will help Oracle Support to understand better the status of the database when the problem ocurred and get a detailed view of each of the process connected to the database.RELATED DOCUMENTSNote:175006.1 : Steps to generate HANGANALYZE trace files  Steps to generate HANGANALYZE trace files [ID 175006.1]  Modified 07-MAY-2009     Type BULLETIN     Status PUBLISHED  PURPOSE------- This article explain how to generate HANGANALYZE trace files to diagnostic database HANG issues.When a database has a severe performance response time, the cause of the problem can be a HANG. Using HANGANALYZE trace files you can determine whether an Oracle process is blocking other processes, before starting a performance analysis. Caution: HANGANALYZE run at levels higher that 3 may generate a huge number of trace files for large systems. Do not use levels higher than 3 without discussing their effects with Oracle Support.   STEPS TO GENERATE A HANGANALYZE TRACE FILE:======================================== Use the following set of commands to generate HANGANALYZE trace files.  1- Using SQL*Plus connect as "INTERNAL" (Oracle8i) or "/ AS SYSDBA" (Oracle9i)2- Execute the following commands: SQL> oradebug hanganalyze 3 ... Wait at least 1 minutes to give time to identify process state changes. SQL> oradebug hanganalyze 3  3- Open a separate SQL session and immediately generate a system state dump.  $ sqlplus '/ as sysdba' oradebug setmypid oradebug unlimit oradebug dump systemstate 266 wait 90 seconds oradebug dump systemstate 266 wait 90 seconds oradebug dump systemstate 266 quit  Note: Starting with Oracle 9.2 HANGANALYZE can generate HANGANALYZE cluster wide.Using this feature you can generate session dependencies to all the sessions connected to all the instances of the RAC cluster. Use the following set of commands to generate a RAC cluster wide HANGANALYZE: 1- Using SQL*Plus connect as "/ AS SYSDBA" 2- Execute the following commands: SQL> oradebug setmypid SQL> oradebug setinst all SQL> oradebug -g def hanganalyze 3 ... Wait at least 1 minutes to give time to identify process state changes. SQL> oradebug -g def hanganalyze 3  FYI:==== Due to BUG 1427041 running HANGANALYZE may terminate the instance This is fixed in 8.1.7.3  RELATED DOCUMENTS-----------------Note:215858.1 - Interpreting HANGANALYZE trace files to diagnose hanging and performance problems
  • Interpreting HANGANALYZE trace files to diagnose hanging and performance problems [ID 215858.1]  Modified 04-DEC-2008     Type BULLETIN     Status PUBLISHED PURPOSEThis article explains how to use the HANGANALYZE utility to diagnose hanging issues in Oracle Databases.BACKGROUNDWhen a database has severe performance problems, the cause of the problem may be a HANG. Using trace files produced by HANGANALYZE, you will be able to quickly determine if two or more processes are deadlocked and how many other processes may be affected. By using HANGANALYZE, you may also determine if a performance analysis will be required to diagnose the real problem when a HANG is taking place.CONTENTWHY USING HANGANALYZEUSING HANGANALYZEINTERPRETING HANGANALYZE DUMPSHANGANALYZE BY EXAMPLESTATE OF THE NODES DESCRIPTIONLEVEL USAGE RECOMMENDATIONSHANGANALYZE AND OTHER TRACING FACILITIESRELATED DOCUMENTS WHY USING HANGANALYZEA “true” hang in the Oracle database can be defined as an internal deadlock or a cyclical dependency between two or more processes. When dealing with DML locks (i.e., TM), Oracle is able to detect this dependency and rollback one of the processes to break the cyclical condition. On the other hand, when this situation happens with internal kernel-level resources (e.g latches or pins), Oracle is usually unable to automatically detect and resolve the deadlock.A severe performance problem may easily be mistaken for a hang. This usually happens when contention is so bad that it seems like the database is completely hung. Previous to HANGANALYZE (Available since Oracle realease 8.0.6), the only tool available to diagnose this type of problem was the well-known “SYSTEMSTATE dump”. A SYSTEMSTATE dump has the following disadvantages when dealing with hanging issues:  SYSTEMSTATE dump reads the SGA in a “dirty” manner, so it may be inconsistent when the time to dump all the process is long.SYSTEMSTATE dump usually dumps a lot of information (most of which is not needed to determine the source of the hang), which makes difficult to determine the dependencies between processes quickly.SYSTEMSTATE dumps do not identify “interesting” processes on which to perform additional dumps (ERRORSTACK or PROCESS STATE).HANGANALYZE use internal kernel calls to determine if a session is waiting for a resource, and reports the relationships between blockers and waiters. In addition, it determines which processes are “interesting” to be dumped, and may perform automatic PROCESSSTATE dumps and ERRORSTACKS on those processes, based on the level used while executing the HANGANALYZE.Note: HANGANALYZE is not intented to replace a SYSTEMSTATE dump, but may serve as a road map to interpret a SYSTEMSTATE while diagnosing complex issues.  USING HANGANALYZEThe “HANGANALYZE” command is available since Oracle Release 8.1.6. In Oracle9i it was enhanced to provide “cluster wide” information in Real Application Cluster (RAC) environments on a single shot. The meaning of this is that it will generate information for all the sessions in the cluster regardless of the instance that issued the command.HANGANALYZE may be executed using the following syntax:ALTER SESSION SET EVENTS 'immediate trace name HANGANALYZE level <level>';Or when logged in with the “SYSDBA” role,ORADEBUG hanganalyze <level>To perform cluster wide HANGANALYZE use the following syntax:ORADEBUG setmypidORADEBUG setinst allORADEBUG -g def hanganalyze <level>The <level> sets the amount of additional information that will be extracted from the processes found by HANGANALYZE (ERROSTACK dump) based on the “STATE” of the node. See further detail in the “STATE DESCRIPTION” section of this document.The levels are defined as follows:10     Dump all processes (IGN state)5      Level 4 + Dump all processes involved in wait chains (NLEAF state)4      Level 3 + Dump leaf nodes (blockers) in wait chains (LEAF,LEAF_NW,IGN_DMP state)3      Level 2 + Dump only processes thought to be in a hang (IN_HANG state)1-2    Only HANGANALYZE output, no process dump at allINTERPRETING HANGANALYZE DUMPSHANGANALYZE has several sections, which may or not be reported in the trace file. They are only generated when HANGANALYZE finds information related to the section.Sections where session information is reported always have a header that details the information that is extracted. Previous to Oracle9i the information extracted is always consistent, while in Oracle9i, especially when getting “cluster wide” HANGANALYZE trace files, it may gather information related to the operating system only on the node where the HANGANALYZE was issued.Examples of session “headers”:Oracle 8.x chain header:  <sid/sess_srno/proc_ptr/ospid/wait_event>Oracle9i chain header:  <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :Where: sid        = Session ID sess_srno  = Serial# proc_ptr   = Process Pointer ospid      = OS Process Id wait_event = Waitevent cnode      = Node Id (Only available since Oracle9i)State of Nodes headers are explained in the section called: STATE OF THE NODES DESCRIPTIONThe following example shows how to interpret each of the sections of a HANGANALYZE trace file:  *** 2002-05-13 17:02:30.937==============HANG ANALYSIS:==============CYCLES: This section reports the process dependencies between sessions that are in a deadlock condition. Cycles are considered “true” hangs.Cycle 1 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <980/3887/0xe4214964/24065/latch free> -- <2518/352/0xe4216560/24574/latch free> -- <55/10/0xe41236a8/13751/latch free>BLOCKER OF MANY SESSIONS: This section is found when a process is blocking a lot of other sessions. Usually when a process is blocking more that 10 sessions this section will appear in the trace file.Found 21 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <55/10/0xe41236a8/13751/latch free>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <2098/2280/0xe42870d0/3022/db file scattered read>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <1941/1783/0xe41ac9e0/462/No Wait>Found 12 objects waiting for <sid/sess_srno/proc_ptr/ospid/wait_event> <980/3887/0xe4214964/24065/latch free>OPEN CHAINS: This section reports sessions involved on a wait chain. A wait chains means that one session is blocking one or more other sessions.Open chains found:Chain 1 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <2/1/0xe411b0f4/12280/db file parallel write>Chain 2 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <3/1/0xe411b410/12282/No Wait>Chain 6 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <18/1631/0xe4243cf8/25457/db file scattered read> -- <229/1568/0xe422b84c/8460/buffer busy waits>Chain 17 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <56/11/0xe4123ce0/13755/latch free> -- <2384/599/0xe41890dc/22488/latch free> -- <32/2703/0xe41fa284/25693/latch free>OTHER CHAINS: It refers to chains of blockers and waiters related to other sessions identified under “open chains”, but not blocked directly by the process reported on the "open chain". Other chains found:Chain 676 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <20/93/0xe411d644/13597/latch free>Chain 677 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <27/1201/0xe41d3188/15809/latch free>Chain 678 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <36/1532/0xe428be8c/4232/latch free> -- <706/1216/0xe4121aac/23317/latch free>Chain 679 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <43/12/0xe4122d54/13745/latch free>Chain 680 : <sid/sess_srno/proc_ptr/ospid/wait_event> : <80/2/0xe41290d4/13811/library cache pin> -- <1919/1134/0xe421fdbc/3343/enqueue>Additional DUMP detail: HANGANALYZE generates errorstack dumps for individual processes based on: the LEVEL used when the command is issued, and the STATE of the session. This part of the HANGANALYZE trace indicates which processes will be dumped if higher levels are used. It also indicates how many nodes will be dumped so you can measure the impact to the system (if too many errorstack dumps occur simultaneously, the disks serving the trace destination may become saturated). The “STATE DESCRIPTION” section of this document further explains the meaning of the states.Extra information that will be dumped at higher levels:[level  4] :  23 node dumps -- [LEAF] [LEAF_NW] [IGN_DMP][level  5] :  36 node dumps -- [NLEAF][level 10] : 130 node dumps -- [IGN]STATE OF NODES: This section might be considered as the main section of the report. It shows all the sessions connected when the HANGANALYZE trace file was generated. This section essentially describes a dependency graph between nodes(known as an “adjacency list”), where each session is considered a node, and each node may have an “adjacent” node or a “predecessor” node related to it. Depending on the state of the node it may be a blocker, a waiter or a hung node. The “STATE DESCRIPTION” section of this document explains further the meaning of the states.State of nodes([nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessor):[0]/1/1/0xa6f8b0/IGN/1/2//none[1]/2/1/0xa70230/IGN/3/4//none[3]/4/1/0xa71530/IGN/5/6//none[4]/5/1/0xa71eb0/IGN/7/8//none[5]/6/1/0xa72830/IGN/9/10//none[6]/7/1/0xa731b0/IGN/11/12//none[7]/8/1/0xa73b30/IGN/13/14//none[8]/9/1/0xa744b0/IGN_DMP/15/18/[130]/none[9]/10/1/0xa74e30/IGN/19/20//none[10]/11/4202/0xa757b0/IGN/21/22/[130]/none[11]/12/1196/0xa76130/NLEAF/23/28/[49]/none[12]/13/1/0xa76ab0/IGN/29/30/[130]/none[37]/38/37/0xa85830/NLEAF/73/76/[50]/46[46]/47/15/0xa8adb0/NLEAF/91/92/[37][50]/none====================END OF HANG ANALYSIS====================  STATE OF THE NODES DESCRIPTION As described before, HANGANALYZE use the model of “Adjacency Lists” to report the sessions found when the HANGANALYZE command was issued. In Graph Theory, an adjacency list represents a list of nodes that are connected to a given node. For each node, a linked list of nodes connected to it can be set up to represent the neighbor node connected. To be able to interpret a HANGANALYZE trace file it is required to relate the “STATE” of the node with the “adjacent” node to it.A typical entry in the state of the nodes section will be as follow:Oracle 8.x : [nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessorOracle9i: [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessorWhere: Nodenum   = This is secuencial number used by HANGANALYZE to identify each session sid       = Session ID sess_srno = Serial# ospid     = OS Process Id state     = State of the node adjlist   = adjacent node (Usually represents a blocker node) predecessor = predecessor node (Usually represents a waiter node) cnode     = Node number (Only available since Oracle9i)The following describes the important states to be considered:IN_HANG: This might be considered as the most critical STATE. Basically a node in this state is involved in a deadlock, or is hung. Usually there will be another “adjacent node” in the same status. For example: [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/IN_HANG/29/32/[185]/19 [185]/1/16/4966/0x24617270//IN_HANG/30/31/[16]/16In this example the node [16] is waiting for node [185], and the other way around; this is a cyclical condition (deadlock).  LEAF and LEAF_NW: Leaf nodes are considered on top of the wait chain (usually blockers). They are considered “Blockers” when there is another session waiting. This can be easily identified using the “predecesor” field. If there is a node referenced in the ‘prdecessor’ field, the node is considered as “blocker”, otherwise it is considered as a “slow” session waiting for some resource.The difference between LEAF and LEAF_NW is that LEAF nodes are not waiting for something, while LEAF_NW are not waiting or may be using the CPU. A typical representation of these nodes when they are considered blockers is: [ nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/LEAF/29/30//19 [19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186In this example, node [16] is blocking node [19]. Notice that node [16] has node [19] in the predecessor field.Also notice that node [19] has node [16] in the adjacent list.  NLEAF : These sessions are usually considered as “stuck” sessions. It means that there is another session holding a resource needed by the session in this state. By using the adjlist, you can determine which node is the blocker of this process. When many sessions are found in this state, it is likely the database is experiencing a performance problem rather than a hang problem.  IGN and IGN_DMP : Sessions in this state are usually considered as IDLE sessions, unless they reference a node in the “ adjlist” field. In this case, the node is waiting for another node, so it will be considered as a ‘stuck’ session as well.Extending the previous example, [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor [16]/0/17/154/0x24617be0/26800/LEAF/29/30//19 [19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186 [189]/1/20/36/0x24619830//IGN/95/96/[19]/none [176]/1/7/1/0x24611d80//IGN/75/76//noneYou may notice that node [189] is waiting for node [19] which in turn is waiting for node [16], while node [176] is an IDLE session . This maybe the case when a session has a DML lock but never finished the transaction.  In Oracle9i, two new states were introduced to differentiate between LEAF nodes that have other nodes waiting behind them (i.e., LEAF nodes that are blockers) vs. LEAF nodes that are not affecting other nodes.  SINGLE_NODE and SINGLE_NODE_NW:This can be described the same as LEAF and LEAF_NW, except that they don't have processes depending on them.  LEVEL USAGE RECOMMENDATIONSIt is advisable not to use levels higher than 3 due to the potentially large number of trace files that may be produced (and could overwhelm the I/O subsystem). Since HANGANALYZE will be mostly used to diagnose “true hangs”, a level 3 will dump the processes that are involved in a the hang condition – this usually involves fewer than 4 processes.Level 4 may be used with caution when dealing with performance scenarios where sessions are being stuck for a long period of time, and no other information can be gathered. Be carefull that a HANGANALYZE level 4 will dump all the processes in the following states:[LEAF] [LEAF_NW] [IGN_DMP]. Before doing so, perform a HANGANALYZE level 3, and check the section: “Extra information that will be dumped at higher levels”. This section will tell you exactly how many sessions will be dumped at each level. It is diffucult to know how many dumps are an acceptable number to keep from saturating the disks due to the differences in disk implementations (5 to 10 is probably OK for most systems; larger systems may be able to handle 20 to 40 dumps). Level 5 and 10 are typicaly useless and generate a huge amount of trace files that may severely impact the performance of the database.HANGANALYZE AND OTHER TRACING FACILITIESWhen diagnosing problems using HANGANALYZE it is recommended also to generate a SYSTEMSTATE dump level 2 to be able to see all the session details. HANGANALYZE together with the SYSTEMSTATE dump will help Oracle Support to understand better the status of the database when the problem ocurred and get a detailed view of each of the process connected to the database.RELATED DOCUMENTSNote:175006.1 : Steps to generate HANGANALYZE trace files  Steps to generate HANGANALYZE trace files [ID 175006.1]  Modified 07-MAY-2009     Type BULLETIN     Status PUBLISHED  PURPOSE------- This article explain how to generate HANGANALYZE trace files to diagnostic database HANG issues.When a database has a severe performance response time, the cause of the problem can be a HANG. Using HANGANALYZE trace files you can determine whether an Oracle process is blocking other processes, before starting a performance analysis. Caution: HANGANALYZE run at levels higher that 3 may generate a huge number of trace files for large systems. Do not use levels higher than 3 without discussing their effects with Oracle Support.   STEPS TO GENERATE A HANGANALYZE TRACE FILE:======================================== Use the following set of commands to generate HANGANALYZE trace files.  1- Using SQL*Plus connect as "INTERNAL" (Oracle8i) or "/ AS SYSDBA" (Oracle9i)2- Execute the following commands: SQL> oradebug hanganalyze 3 ... Wait at least 1 minutes to give time to identify process state changes. SQL> oradebug hanganalyze 3  3- Open a separate SQL session and immediately generate a system state dump.  $ sqlplus '/ as sysdba' oradebug setmypid oradebug unlimit oradebug dump systemstate 266 wait 90 seconds oradebug dump systemstate 266 wait 90 seconds oradebug dump systemstate 266 quit  Note: Starting with Oracle 9.2 HANGANALYZE can generate HANGANALYZE cluster wide.Using this feature you can generate session dependencies to all the sessions connected to all the instances of the RAC cluster. Use the following set of commands to generate a RAC cluster wide HANGANALYZE: 1- Using SQL*Plus connect as "/ AS SYSDBA" 2- Execute the following commands: SQL> oradebug setmypid SQL> oradebug setinst all SQL> oradebug -g def hanganalyze 3 ... Wait at least 1 minutes to give time to identify process state changes. SQL> oradebug -g def hanganalyze 3  FYI:==== Due to BUG 1427041 running HANGANALYZE may terminate the instance This is fixed in 8.1.7.3  RELATED DOCUMENTS-----------------Note:215858.1 - Interpreting HANGANALYZE trace files to diagnose hanging and performance problems
  • To start diagnosing a problem with the process size, such as a suspected leak,a heapdump of the offending process is required:  $sqlplus SQL> connect '/ as sysdba' SQL> oradebug setospid <pid> SQL> oradebug unlimit SQL> oradebug dump heapdump 5 --this dumps PGA and UGA heaps  The following information should be provided to Oracle Support Services:To start diagnosing a problem with the process size, such as a suspected leak,a heapdump of the offending process is required:  $sqlplus SQL> connect '/ as sysdba' SQL> oradebug setospid <pid> SQL> oradebug unlimit SQL> oradebug dump heapdump 5 --this dumps PGA and UGA heaps  The following information should be provided to Oracle Support Services: To List All The Named Events Set For A Database [ID 436036.1]  Modified 09-MAR-2009     Type HOWTO     Status PUBLISHED In this Document  Goal  Solution  ReferencesApplies to: Oracle Server Enterprise Edition - Version: 8.0.3.0 to 11.2.0.1.0Information in this document applies to any platform.Oracle Server - Enterprise Edition - Version: 8.0 to 11.2 GoalOracle Internal Events, both named event (eg: ERRORSTACK) as numeric events (eg 10046) can be set at the session level. This document outlines how DataBase Administrators can verify which events have been set in a specific session or in the entire system. SolutionThis can be achieved through the ORADEBUG functionality. The exact syntax to use depends on the Oracle version you are using. If you want to retrieve the events set in a specific session, then use the following oradebug commands to identify the session:1. identify the session     SQL> select p.pid, p.spid, s.username          from v$process p, v$session s          where p.addr = s.paddr;  2. Attach to the desired session     SQL> connect / as sysdba       SQL> oradebug setorapid <pid>    --OR--     SQL> oradebug setospid <spid>    ...If you want to retrieve the events set in your current SESSION or at the SYSTEM level, you can use the following commands:SQL> connect / as sysdbaSQL> oradebug setmypid ... Oracle 8i / 9i / 10gSQL> oradebug dump events <level> Level 1 - Dump session group's event settings  Level 2 - Dump process group's event settings  Level 4 - Dump system group's event settings (Ie the instance wide events)  Oracle 10g / 11g and aboveSQL> oradebug eventdump <level> SESSION - Dump session group's event settings  PROCESS - Dump process group's event settings  SYSTEM  - Dump system group's event settings (Ie the instance wide events)             -- OR --SQL> alter session set events 'immediate eventdump(<level>)'; SESSION - Dump session group's event settings PROCESS - Dump process group's event settings   SYSTEM  - Dump system group's event settings (Ie the instance wide events) Example 1:  Oracle 9i - system eventsThis example illustrates how to retrieve the events set at the system level in an Oracle 9i database.SQL> alter system set events '942 trace name errorstack level 3'; Session altered.After the event is set we dump the events informationSQL> oradebug setmypid Statement processed. SQL> oradebug dump events 4; Statement processed. SQL> oradebug tracefile_name/u01/oracle/product/9.2.0/admin/ORCL/udump/ORCL_ora_2620.trcThe following information will be dumped in the in the trace file ORCL_ora_2620.trc.1. *** 2007-06-13 20:12:11.421 2. Dump event group for level SYSTEM 3. TC AddrEvt#(b10) Action TR Addr Arm Life 4. 78B6584 942 1 78b65c4 0 0 5. TR Name TR level TR address TR arm TR life TR type 6. ERRORSTACK 3 0 1 2 1961960 Notice the line 2 "Dump event group for level SYSTEM" which tells you which level was used to dump the information. In our case level 1 was used to dump the session information.In line 4 the second column value tells us the event that was set, in our case 942.In line 6 the first column tells us the name of the event, in our case the named event "ERRORSTACK" appears. On the same line check the second column, the determines the level at which the event was set, in our case "3"Hence from the above information was can conclude that the event "942 trace name ERRORSTACK level 3" was set at the system level Example 2: Oracle 11g - events set at session user SCOTTThis example illustrates how to retrieve the events user SCOTT has set at his session in an Oracle 11g database: SQL> select p.pid, p.spid, s.username           from v$process p, v$session s            where p.addr = s.paddr; For example: PID SPID USERNAME ---- ------- --------- 15 28390 SCOTT Attach to the above identified session:SQL> connect / as sysdbaSQL> oradebug setorapid 15 --OR-- SQL> oradebug setospid 28390 Unix process pid: 32493, image: oracle@jschelfh-pc (TNS V1-V3)Get the event information:SQL> oradebug eventdump session 942 trace name ERRORSTACK level 3  Example 3: Oracle 11g - events set at your current session This example illustrates how to retrieve the events set in your current session environment in an Oracle 11g database: SQL> alter system set events '942 trace name errorstack level 3'; Session altered. SQL> oradebug tracefile_name/oracle/admin/L111/udump/l112_ora_32493.trcThe following information will be dumped in the in the trace file l112_ora_32493.trc*** ACTION NAME:() 2009-03-06 12:48:52.533 Dumping Event (group=SYSTEM) 942 trace name errorstack level 3 Comparison with DBMS_SYSTEM.READ_EVWe can also use the prodecureREAD_EV from the package DBMS_SYSTEM to get the events set in the database. But the main difference between the usage would be that the DBMS_SYSTEM.READ_EV will generate information only on the numeric events like 10235, 10046, etc., that are set in the database. But, checking the events with the help of this named event "EVENTS", we can check for both the named and numerical events that are set. Also we can get additional information regarding each event that is set, from the trace file generated.  For example, ERRORSTACK, HEAPDUMP in following format cannot be checked through DBMS_SYSTEM.READ_EV,   ALTER SYSTEM SET EVENTS '4031 TRACE NAME HEAPDUMP LEVEL 2';  ALTER SYSTEM SET EVENTS '942 TRACE NAME ERRORSTACK LEVEL 3';but can be checked through the named event 'EVENTS'.ReferencesNOTE:28446.1 - How to Determine What Events/Parameters are Set in the Current Session    
  • Debug Core Dumps / Core Files on UNIX [ID 16534.1]  Modified 10-AUG-2009     Type FAQ     Status PUBLISHED  Checked for relevance on 11-Aug-2009   Creating a Stack Trace From a Core File    Anyone who has worked with UNIX for any reasonable period of time islikely to have run into a 'core dump' situation. When an application runs intosomething that the process should be aware of, it generates some signals.Signals are software interrupts. They provide a way of handling asynchronousevents, e.g. a user request to stop a process. For every signal there is adefault action. The application that generates the signals can choose to ignoreor catch those signals or let their default action apply. When the defaultaction is labeled 'terminate w/core' the application terminates with a 'coredump'. The core file is left in the current working directory of the process.It contains a memory image of the process at the time of the termination. (Thefact that the file is named core shows how long this feature has been part ofUNIX.) This file can then be used with most Unix debuggers to examine the stateof the process at the time it is terminated. The core file will not begenerated if  (a) the process was set-user-ID and the current user is not the owner of the program file, or (b) the process was set-group-ID and the current user is not the group owner of the file, or (c) the user does not have permission to write in the current working directory, or (d) the file is to big.  A 'core dump' can occur under the following circumstances:  (a) Dividing by zero or (SIGFPE) other arithmetic exceptions vary by port. (b) Dereferencing invalid address. (SIGSEGV) (c) Dereferencing "impossible" address (SIGBUS) (d) Executing illegal instructions (SIGILL) The permissions of the core file are usually user-read, user-write, group-read,and other-read,  $ ls -l core -rw-r--r-- 1 bclinton 8483248 Dec 31 12:00 core  The Unix debuggers that can be used to extract more information from acore file include: dbx, adb, crash, xdb, sdb, kadb, etc. The most common onesused are dbx and adb. This bulletin will explain how dbx can be used.  Suppose Forms40 runtime f40runm produced a core dump. To analyse the corefile with dbx, use the following command,  C shell: dbx $ORACLE_HOME/bin/f40runm core | tee stack.trc  Bourne or Korn shell: dbx $ORACLE_HOME/bin/f40runm core 2>&1 | tee stack.trc It is very important to include the full path of the executable. It is alsorecommended(but not necessary) to have read priviledge on the executable. Inthis case only Oracle can use chmod to alter the file protection on f??runm.  Once the command is issued, dbx will print out several lines ofinformationand then return to a dbx prompt(dbx). Type 'where' at the prompt(no quotes whenactually entering it) dbx will print out the stack trace. Type 'quit' at theprompt to exit out of dbx. All the output should be captured in a file calledstack.trc. Now contact Oracle Support with this stack trace file.  An alternative method to capture the stack trace output from a Unixdebugger is to use the 'script' command. Running 'script' without any argumentswill start a new shell and save all terminal output into a file named'typescript' in the current directory. Be sure to 'exit' from the 'script'command after leaving the debugger.  With other UNIX debuggers, the commands to retrieve stack trace areslightly different. For example,  Debugger Trace Command Exit Command -------- ------------- ------------ adb $c Ctrl-D sdb t q xdb t 100 quit  On Windows, the equivalent of a 'core dump' is a GPF(Generic ProtectionFault). A Windows application called drwatson.exe can be used to capture anddiagnose a GPF. It resides under c:windows.  On VMS, a similar application termination will produce a 'stack dump'.Unlike a 'core dump', a 'stack dump' will print out all the useful informationabout the application's status at the point of termination on the screen.Therefore, there is no need for using any outside debuggers.  Reference:----------- Note:1007808.6 HOW TO HANDLE CORE DUMPS ON UNIX Hung Programs/Processes [ID 106958.1]  Modified 16-NOV-2008     Type BULLETIN     Status PUBLISHED  Checked for relevance on 17-Nov-2008. PURPOSE------- This article documents a means of obtaining a stack trace from a hung applicationprogram such as OCI, precompiler program, or Java application. SCOPE & APPLICATION------------------- This article is intended to aid analysts and developers in debugginghung processes. The method documented here should be used as a last effortin obtaining meaningful diagnostic information from the hung process. This approach is not recommended for production systems.  Debugging Hung Programs/Processes--------------------------------- Hung processes are frequently the result of process contention for locking of some resource. The resource could be a system, database, or applicationresource. The contention for one of these resources can sometimes result in adeadlock between competing processes for a resource where exclusive control isrequired. When such deadlocks occur, it will result in one or more processes appearing to hang. Debugging the cause of the hang can some times be very difficult, particularly in cases where the behavior is sporadic. The diagnosticsof such behavior can be aided through the use of a good source or symbolic debugger, or inclusion of strategicly placed print/display statements in the program. In some cases, these approaches may not provide much useful information,or they may not be feasable. When confronted with this type of situation, it maybe necessary to take a more drastic debugging approach. It may be that the program is hanging in deeply nested calls of some library to which the developeror analyst does not have access to the source code for debugging. In these cases,it is possible to obtain a call stack from the hung process by sending it a signal which causes the process to dump core. On UNIX systems, the kill command can be used to signal the process. If the process is in an uniterruptable state at the time the signal is sent, the processwill frequently dump core, thereby allowing the use of a debugger to obtain a stack trace. See note:1812.1 for further information on obtain stack tracesfrom core files. The following stack trace was obtained from a hung multi-threadedOCI program that was hanging in a pthread_lock call. The core file was obtain bysending kill -6 (SIGIOT or SIGABRT) to the process id of the hung process. The following stack trace shows that the process had hung on a call topthread_lock. -----------------------stack trace starts here--------------------------------- #0 0x4057258b in __sigsuspend (set=0xbf7ff1b8)  at ../sysdeps/unix/sysv/linux/sigsuspend.c:48 48 ../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory. (gdb) bt #0 0x4057258b in __sigsuspend (set=0xbf7ff1b8)  at ../sysdeps/unix/sysv/linux/sigsuspend.c:48 #1 0x4054b211 in __pthread_lock (lock=0x8061c60, self=0xbf7ffe60)  at restart.h:32 #2 0x4054899a in __pthread_mutex_lock (mutex=0x8061c50) at mutex.c:84 #3 0x4041cd50 in sltsmna ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #4 0x402e7819 in sntpreap ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #5 0x4041db02 in sslsshandler ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #6 0x4054a522 in pthread_sighandler (signo=17, ctx={gs = 0, __gsh = 0,  fs = 0, __fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0,  edi = 3212833984, esi = 3212836448, ebp = 3212833948, esp = 3212833920,  ebx = 3212833984, edx = 10, ecx = 8, eax = 4294967292, trapno = 16,   err = 0, eip = 1079453067, cs = 35, __csh = 0, eflags = 646,  esp_at_signal = 3212833920, ss = 43, __ssh = 0, fpstate = 0x0,  oldmask = 2147483648, cr2 = 0}) at signals.c:91 #7 0x40572408 in __restore ()  at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127 #8 0x4054b211 in __pthread_lock (lock=0x8061c60, self=0xbf7ffe60)  at restart.h:32 #9 0x4054899a in __pthread_mutex_lock (mutex=0x8061c50) at mutex.c:84 ---Type <return> to continue, or q <return> to quit--- #10 0x4041cd50 in sltsmna ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #11 0x402e8d9e in sntpclose ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #12 0x402e7626 in ntpdisc ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #13 0x401fd6f2 in nsclose ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #14 0x40265c7f in nsdisc ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #15 0x4024085c in nioqds ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #16 0x4018e98e in upidhs ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #17 0x40144d26 in kpudtch ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #18 0x401859a1 in OCIServerDetach ()  from /u01/app/oracle/product/815/lib/libclntsh.so.8.0 #19 0x804a2ea in ora_close_db () #20 0x80494d4 in thread_run () #21 0x40547eca in pthread_start_thread (arg=0xbf7ffe60) at manager.c:213 (gdb) q ------------------------stack trace ends here----------------------------------- RELATED DOCUMENTS----------------- Note:16534.1 Debug CORE Dumps on UNIX Note:1812.1 Getting a Stack Trace From a CORE File  How To Get a Stack Trace [ID 557409.1]  Modified 27-MAR-2008     Type HOWTO     Status PUBLISHED In this Document  Goal  Solution  ReferencesApplies to: Oracle Server - Enterprise Edition - Version: 8.1.7.4 to 11.1Information in this document applies to any platform.GoalSometimes, when an ORA-7445 error is signalled, the trace file written to report the error does not contain a valid stack trace. A normal stack will be found at the beginning of the trace file and will start with the dump functions, followed by the functions accessed by that process, in reverse order, and will end with sou2o. Where the stack trace has failed there may be no stack trace at all or the stack trace may only contain the dump functions such as kse*, or OS functions.A valid stack trace is essential to diagnosis of an ORA-7445 error. The purpose of this document is to give guidance on getting a stack trace where none is printed in the trace file.The intended audience is database administrators and support engineers.Solution1. Check the value of the initialization parameter 'shadow_core_dump'. This parameter controls whether the SGA will be included in the core file for client processes. The possible values are:FULL - include the SGA in core dumpsPARTIAL - the SGA is not includedNONE - do not produce core files If the parameter is set to NONE then it should be changed to FULL or PARTIAL2. If on a Unix platform, check the ulimit value for coredumps - it may be set to zero. To do this, at the OS prompt of the Unix user who owns oracle do:ulimit -aIf the parameter representing the size of core dumps is set to zero then set it back to the default of 2Gb. The name of the parameter varies between platforms but will be called something similar to 'coredumpfilesize'. You may need to involve your system administrator to set the parameter at a global level.You should also ensure that the limits for file and stack are high.3. Check whether a stack trace is written if you force a trace file:    SQL>alter session set events '942 trace name errorstack level 1';    SQL>select * from dummy_table;    The trace will be written to the user_dump_destination.4. If a stack trace is written then set the following in your initialization file:    event="7445 trace name errorstack level 3"5. If a stack is not written then check whether there is a core file. This will usually be written to the location indicated by the initialization parameter 'core_dump_dest'. If this is not set then the core file should be written to $ORACLE_HOME/dbs.If a core file has been written then use Note 1812.1 'TECH: Getting a Stack Trace from a CORE file' to try to extract the stack using a debugger.You could also try the STACKX utility to extract the stack. See Note 362791.1 'STACKX User Guide'6.If you're using Windows then check whether a <SID>core.log has been written. This will contain the stack trace towards the beginning of the file and should be present in the CDUMP destination for the instance.  It is possible however that the <SID>core.log file could be written to  $ORACLE_HOME/database.7. If a core file is not written confirm that there is enough space in the core_dump_dest using df -k.8. Check that the unix processes can write core files by executing kill -11 against an unimportant process such the one created by 'clock &'. A core file should be written to the current directory.9. If none of the above resolves the problem then check with support for known bugs which prevent stacks being written, such as hidden Bug 5671740 on Linux or Bug 5674899 on HP Tru64.10. There may also be an OS issue - ensure that you are at the latest patch level on your operating system.11. If all else has been checked then a new bug should be raised. ReferencesBUG:5671740 - INCOMPLETE CALL STACK FOR LINUX 64-BIT WITH 2.6.9 KERNELBUG:5674899 - A LOT OF TRACE FILES CONTAINING: ST_DUMP_FRAME: NO PROC ENTRY FORNOTE:1007808.6 - HOW TO HANDLE CORE DUMPS ON UNIXNOTE:16534.1 - Debug Core Dumps / Core Files on UNIXNOTE:1812.1 - TECH: Getting a Stack Trace from a CORE fileNOTE:30858.1 - Init.ora Parameter "SHADOW_CORE_DUMP" Reference NoteNOTE:362791.1 - STACKX User GuideNOTE:3722628.8 - Bug 3722628 - Linux: Oracle Call Stack may be incomplete in trace files  STACKX User Guide [ID 362791.1]  Modified 30-NOV-2007     Type SCRIPT     Status PUBLISHED STACKXCore / Stack Trace Extraction ToolVersion 1.2Roger Snowden, Center of ExpertiseContentsIntroductionSupported PlatformsDownload STACKXDebuggers RecognizedInstalling STACKXUninstalling STACKXRunning STACKXThread SupportBug FixesOpen Issues and LimitationsReporting Feedback Introductionstackx is a shell script utility for extracting diagnostic stack trace information from core file on UNIX/LINUX platforms. Core files are typically created when an application program fails in such a way it cannot continue operation. Often this is from an attempt by the application program to access a memory location outside its range permitted by the operating system. When this happens, the operating system sends a signal to the program, which in turn may produce a disk file containing information about its operation, prior to the program terminating execution. This file is the "core" file, and contains information on program functions called immediately prior to program failure. This information is called a "stack trace" or "back trace" and is embedded within the binary core file, which is not human readable. In order to extract the useful information and put it into human readable form, some effort is required. stackx assists in that effort.Stack trace information normally consists of a list of the current program function (at time of termination) as well as the "stack" of functions called prior to the current function. This information can be instrumental in determining the cause of a program failure, if it is extracted properly.Extracting stack trace information from a core is not difficult, but this operation requires an operating system utility known as a debugger, or other similar utility, and the specific technique required varies with the debugging utility used. To simplify this effort, stackx was created to provide a simpler and consistent command line interface for automating this operation.stackx works as follows:Chooses a debugger from a list of supported utilities in the current environment path, or a user-specified debugger passed as a command line argument.Finds the core-generating application program name from inside the core file.Runs the debugger and passes appropriate commands to that debugger.Output is text, normally displayed to the users screen. Output can be redirected to a trace file by passing an appropriate redirect operator and output filename, in the conventional UNIX manner. This is shown in the RUNNING STACKX section below.Back to Contents Supported PlatformsCore stack trace extraction is supported on the following platforms:IBM AIXIntel Linux (RedHat and SuSE)HP-UX (10.* and 11.*)HP Tru64Sun Solaris (2.5 - 2.10)Back to Contents Download STACKXCurrent stackx Version: 1.2.0   August, 2007 Click here to download the file (use the name  stackx1_2.tar   when saving).If a file download dialog box does not appear when clicking on the above link, you may need to clear your web browser's cache and/or restart your web browser. If you are still unable to download the file, you may request that we email you a copy: Roger.Snowden@oracle.comBack to Contents Debuggers RecognizedDebuggers recognized and discovered by stackx are (in order of preference):pstack (Solaris only)gdbadbdbxsdbpstack is not actually a debugger, but a specialized tool supported by Solaris as part of their /proc tool suite, which quickly extracts a stack trace from a core file. As such, it is the preferred extraction tool on Solaris platforms and will be invoked if stackx can find it in the path and a specific debugger is not passed as a command line argument to stackx.When pstack is available, both pmap and pflags utilities will also be invoked if possible. The pmap utility prints information about the address space of a process. The pflags utility prints tracing flags, pending and held signals for a process.NOTE: Linux also has a utility with the name "pstack", however the Linux pstack is only useful for extracting a stack trace on a running program, not a core file. Linux pstack is not applicable to stackx, as stackx is only used for core file stack trace extraction.If explicitly specified by command line argument, on Solaris platforms, the mdb debugger will also be used.Back to Contents Installing STACKXstackx is distributed as a tar archive, stackx.tar. To install stackx, Create a new directory on the target machine where you would like stackx to be installedCopy the distribution file to the new directory on the target machine.  Untar the resulting uncompressed file in the following manner:$ tar -xvf stackx.tarThree files will be extracted:stackx -- the "run" scriptREADME.txt -- a guide on using stackx in text formatREADME.htm -- a guide on using stackx in html formatBack to Contents Uninstalling STACKXTo uninstall stackx remove the files in the stackx directory created during the install.Back to Contents Running STACKXTo run stackx simply invoke the stackx script, passing either a local core filename or the fully qualified core filename as an argument. In the following examples a core file named core.1234 is used. $ stackx core.1234Optionally, if a particular debugger is desired, or no debugger is available in the path, a fully qualified debugger name can be passed as an optional first argument. If the desired debugger is in the current path, then it does not have to be fully qualified with path name. However, a fully-qualified name will eliminate issues of ambiguity where more than one otherwise-identical debugger executable is installed on a machine: $ stackx /usr/bin/adb core.1234If you want output sent to a text file: $ stackx core.1234 > stacktrace.txtor$ stackx /usr/bin/adb core.1234 > stacktrace.txtNOTE: The core file does not have to be in the local directory. A fully qualified filename can be accessed by stackx, but only if read permission is available to the executing session on that file. It may be necessary to execute stackx as root, or to change permissions on the core file. For a core file named core.1234, you can set global read permission this way: $ chmod +r core.1234Back to Contents Thread SupportProcesses that run multiple threads present a special case for diagnostic efforts. In order to be useful for diagnosing problems, a stack trace for each thread must be generated. stackx will invoke appropriate commands for debuggers to extract stack traces for all threads in a core, if the underlying debugger supports this type of operation. Such utilities include:pstack (Solaris only)gdbadbdbxmdb (Solaris only)However, such support is not universally available on each debugger. In particular, older versions of adb and dbx do not support multi-threaded cores. Unfortunately, there is no reliable way to determine the version of a debugger prior to running it. As such, stackx may not operate properly if a multi-threaded core is generated and a non multi-threaded debugger is used. It is recommended to install a debugging utility that supports multi-threaded programs and cores. The gdb debugger is generally available as an open-source utility and can be freely downloaded and installed for most platforms. Current versions of gdb support multi-threaded cores.stackx may also be run on platforms not listed in this document, but support is nevertheless restricted to those debuggers listed here. To determine if a debugger is available on your platform, do the following for each debugger of interest:$ which <debuggername>If a debugger is available, it will be displayed with the fully qualified pathname. However, if a particular debugger is not found, it may simply be that the current path does not reflect its location. For further assistance, consult the administrator of the machine of interest.Oracle Support recommends testing unlisted platforms on a non-production server first, as behavior may be unpredictable.Back to ContentsBug Fixes and Enhancements, release 1.2 (August 22, 2007)Corrected comparator operator to -gtAdded register info (all registers), gdb onlyCorrected for AIX, 'file' command output variation.Improved output tagging, to more uniform xml-like presentation to ease parsing for support tools.For dbx, added redundant debug command syntax to accommodate syntax variations among non-detectable versions of the dbx debugger. Some spurious output will be generated, but probability of stack extraction is greatly improved.The two character limitation on detected program name no longer applies. stackx is expected to reliably find the name of all core-generating executables.Corrected expression syntax in determining execCountBug Fixes and Enhancements, release 1.1AIX has differences in output from 'which' command, corrections made to accommodate this platform issue.AIX has differences in output from 'file' command, corrections made to accommodate this platform issue.dbx now given priority over adb as default debugger selection.Documentation clarifies need to run stackx on same machine on which corefile was created.Back to ContentsOpen Issues and LimitationsMulti-threaded cores may not be supported by all debuggers, particularly on older platforms, as noted above. The remedy is to install a more contemporary debugger, such as gdb. There may be some cases where the executable application program name cannot be determined reliably, regardless of program name length. In such cases, stackx will not extract a stack trace for that core file, but will report this fact in its output. There is no workaround for this limitation. The stack trace extraction should be performed on the machine on which the core was created. If the core file is moved to a different machine, and then stackx run on that new machine, two major problems can occur:  The core file may not be binary-compatible with the new machine, in which case the debugging operation will likely fail. An example of this would be a core file from a HP-UX machine, and an extraction attempt from a Solaris machine. Please run stackx on the machine on which the corefile was created. The executable file may not be located in the actual path location as on the original machine. Since stackx gets the name of the executable that created the core file, that executable path name is also extracted from the core file. If the executable is in a different path from the machine on which the corefile was created, then the debugger will not be able to extract a stack trace for that core file.Back to Contents  Reporting FeedbackDo NOT log a service request for stackx.  If you encounter problems running stackx or would like to provide feedback, please send email to roger.snowden@oracle.com.Please include a detailed description of the problem, symptom or usage in question. Also, include any output generated by the script.Back to ContentsLegal Notices and Terms of UseSTACKXCore / Stack Trace Extraction ToolVersion 1.2Roger Snowden, Center of ExpertiseContentsIntroductionSupported PlatformsDownload STACKXDebuggers RecognizedInstalling STACKXUninstalling STACKXRunning STACKXThread SupportBug FixesOpen Issues and LimitationsReporting Feedback Introductionstackx is a shell script utility for extracting diagnostic stack trace information from core file on UNIX/LINUX platforms. Core files are typically created when an application program fails in such a way it cannot continue operation. Often this is from an attempt by the application program to access a memory location outside its range permitted by the operating system. When this happens, the operating system sends a signal to the program, which in turn may produce a disk file containing information about its operation, prior to the program terminating execution. This file is the "core" file, and contains information on program functions called immediately prior to program failure. This information is called a "stack trace" or "back trace" and is embedded within the binary core file, which is not human readable. In order to extract the useful information and put it into human readable form, some effort is required. stackx assists in that effort.Stack trace information normally consists of a list of the current program function (at time of termination) as well as the "stack" of functions called prior to the current function. This information can be instrumental in determining the cause of a program failure, if it is extracted properly.Extracting stack trace information from a core is not difficult, but this operation requires an operating system utility known as a debugger, or other similar utility, and the specific technique required varies with the debugging utility used. To simplify this effort, stackx was created to provide a simpler and consistent command line interface for automating this operation.stackx works as follows:Chooses a debugger from a list of supported utilities in the current environment path, or a user-specified debugger passed as a command line argument.Finds the core-generating application program name from inside the core file.Runs the debugger and passes appropriate commands to that debugger.Output is text, normally displayed to the users screen. Output can be redirected to a trace file by passing an appropriate redirect operator and output filename, in the conventional UNIX manner. This is shown in the RUNNING STACKX section below.Back to Contents Supported PlatformsCore stack trace extraction is supported on the following platforms:IBM AIXIntel Linux (RedHat and SuSE)HP-UX (10.* and 11.*)HP Tru64Sun Solaris (2.5 - 2.10)Back to Contents Download STACKXCurrent stackx Version: 1.1.0   September, 2006 Click here to download the file (use the name  stackx.tar   when saving).If a file download dialog box does not appear when clicking on the above link, you may need to clear your web browser's cache and/or restart your web browser. If you are still unable to download the file, you may request that we email you a copy: Roger.Snowden@oracle.comBack to Contents Debuggers RecognizedDebuggers recognized and discovered by stackx are (in order of preference):pstack (Solaris only)gdbadbdbxsdbpstack is not actually a debugger, but a specialized tool supported by Solaris as part of their /proc tool suite, which quickly extracts a stack trace from a core file. As such, it is the preferred extraction tool on Solaris platforms and will be invoked if stackx can find it in the path and a specific debugger is not passed as a command line argument to stackx.When pstack is available, both pmap and pflags utilities will also be invoked if possible. The pmap utility prints information about the address space of a process. The pflags utility prints tracing flags, pending and held signals for a process.NOTE: Linux also has a utility with the name "pstack", however the Linux pstack is only useful for extracting a stack trace on a running program, not a core file. Linux pstack is not applicable to stackx, as stackx is only used for core file stack trace extraction.If explicitly specified by command line argument, on Solaris platforms, the mdb debugger will also be used.Back to Contents Installing STACKXstackx is distributed as a tar archive, stackx.tar. To install stackx, Create a new directory on the target machine where you would like stackx to be installedCopy the distribution file to the new directory on the target machine.  Untar the resulting uncompressed file in the following manner:$ tar -xvf stackx.tarThree files will be extracted:stackx -- the "run" scriptREADME.txt -- a guide on using stackx in text formatREADME.htm -- a guide on using stackx in html formatBack to Contents Uninstalling STACKXTo uninstall stackx remove the files in the stackx directory created during the install.Back to Contents Running STACKXTo run stackx simply invoke the stackx script, passing either a local core filename or the fully qualified core filename as an argument. In the following examples a core file named core.1234 is used. $ stackx core.1234Optionally, if a particular debugger is desired, or no debugger is available in the path, a fully qualified debugger name can be passed as an optional first argument. If the desired debugger is in the current path, then it does not have to be fully qualified with path name. However, a fully-qualified name will eliminate issues of ambiguity where more than one otherwise-identical debugger executable is installed on a machine: $ stackx /usr/bin/adb core.1234If you want output sent to a text file: $ stackx core.1234 > stacktrace.txtor$ stackx /usr/bin/adb core.1234 > stacktrace.txtNOTE: The core file does not have to be in the local directory. A fully qualified filename can be accessed by stackx, but only if read permission is available to the executing session on that file. It may be necessary to execute stackx as root, or to change permissions on the core file. For a core file named core.1234, you can set global read permission this way: $ chmod +r core.1234Back to Contents Thread SupportProcesses that run multiple threads present a special case for diagnostic efforts. In order to be useful for diagnosing problems, a stack trace for each thread must be generated. stackx will invoke appropriate commands for debuggers to extract stack traces for all threads in a core, if the underlying debugger supports this type of operation. Such utilities include:pstack (Solaris only)gdbadbdbxmdb (Solaris only)However, such support is not universally available on each debugger. In particular, older versions of adb and dbx do not support multi-threaded cores. Unfortunately, there is no reliable way to determine the version of a debugger prior to running it. As such, stackx may not operate properly if a multi-threaded core is generated and a non multi-threaded debugger is used. It is recommended to install a debugging utility that supports multi-threaded programs and cores. The gdb debugger is generally available as an open-source utility and can be freely downloaded and installed for most platforms. Current versions of gdb support multi-threaded cores.stackx may also be run on platforms not listed in this document, but support is nevertheless restricted to those debuggers listed here. To determine if a debugger is available on your platform, do the following for each debugger of interest:$ which <debuggername>If a debugger is available, it will be displayed with the fully qualified pathname. However, if a particular debugger is not found, it may simply be that the current path does not reflect its location. For further assistance, consult the administrator of the machine of interest.Oracle Support recommends testing unlisted platforms on a non-production server first, as behavior may be unpredictable.Back to ContentsBug Fixes and Enhancements, release 1.2 (August 22, 2007)Corrected comparator operator to -gtAdded register info (all registers), gdb onlyCorrected for AIX, 'file' command output variation.Improved output tagging, to more uniform xml-like presentation to ease parsing for support tools.For dbx, added redundant debug command syntax to accommodate syntax variations among non-detectable versions of the dbx debugger. Some spurious output will be generated, but probability of stack extraction is greatly improved.The two character limitation on detected program name no longer applies. stackx is expected to reliably find the name of all core-generating executables.Corrected expression syntax in determining execCountBug Fixes and Enhancements, release 1.1AIX has differences in output from 'which' command, corrections made to accommodate this platform issue.AIX has differences in output from 'file' command, corrections made to accommodate this platform issue.dbx now given priority over adb as default debugger selection.Documentation clarifies need to run stackx on same machine on which corefile was created.Back to ContentsOpen Issues and LimitationsMulti-threaded cores may not be supported by all debuggers, particularly on older platforms, as noted above. The remedy is to install a more contemporary debugger, such as gdb. There may be some cases where the executable application program name cannot be determined reliably, regardless of program name length. In such cases, stackx will not extract a stack trace for that core file, but will report this fact in its output. There is no workaround for this limitation. The stack trace extraction should be performed on the machine on which the core was created. If the core file is moved to a different machine, and then stackx run on that new machine, two major problems can occur:  The core file may not be binary-compatible with the new machine, in which case the debugging operation will likely fail. An example of this would be a core file from a HP-UX machine, and an extraction attempt from a Solaris machine. Please run stackx on the machine on which the corefile was created. The executable file may not be located in the actual path location as on the original machine. Since stackx gets the name of the executable that created the core file, that executable path name is also extracted from the core file. If the executable is in a different path from the machine on which the corefile was created, then the debugger will not be able to extract a stack trace for that core file.Back to Contents  Reporting FeedbackDo NOT log a service request for stackx.  If you encounter problems running stackx or would like to provide feedback, please send email to roger.snowden@oracle.com.Please include a detailed description of the problem, symptom or usage in question. Also, include any output generated by the script.Back to ContentsLegal Notices and Terms of UseAttachmentsstackx1_2.tar (50 KB)HOW TO HANDLE CORE DUMPS ON UNIX [ID 1007808.6]  Modified 25-OCT-2005     Type PROBLEM     Status PUBLISHED  Problem Description: ==================== This article will discuss what a core dump is, and what should be done if you receive a core dump from an Oracle product. Problem Explanation: ==================== A core dump is an image copy of the state of a process, which is produced if the process attempts to do something not allowed by the Operating System. The process information is written to a file named "core", which is usually created in the current working directory of the process. Some of the things a program might attempt to do that would cause a core dump include: trying to execute an illegal/unknown instruction, trying to access memory outside the allowed range, or trying to use an unavailable resource. A core file contains much information to aid in determining why a process crashed. One of the things contained in a core file is a called stack trace, which identifies what commands a process has, or was attempting to execute. Sometimes a trace file is automatically produced by an Oracle process, and is normally indicated with an ORA-07445 error. When this occurs, a file with the extension ".trc" will be produced. To find this file, check your init<SID>.ora for the parameters "user_dump_dest" and "background_dump_dest". The trace file will typically be located in one of these directories. A trace file is not always produced automatically, and the solution of this article will discuss how to manually produce a call stack trace. Search Words: ============= segmentation fault ORA-7445 dbx gdb dde adb sdb debug debugger  Solution Description: ===================== Before attempting to manually create a stack trace from a core file, please verify the following: * Your swap space is 2 to 3 times the amount of physical memory (RAM) installed on the system, or whichever is greater of the following 2 items: 1GB or 2 times the size of all SGAs combined. You can approximate the size of an SGA with this formula: ((db_block_size * db_block_buffers)+shared_pool+log_buffers)/.9 For help with how to check swap space, refer to PRE 1016233.6. * You are using a supported compiler/linker. Check the Installation Guide for Operating System requirements. NOTE: On Solaris systems, make sure that "/usr/ccs/bin" is before "/usr/ucb" in the PATH environment variable. You can confirm this using the following command: % which ld The path returned should be "/usr/ccs/bin/ld". * You have at least 50MB of free space available in "/tmp". Creating a Stack Trace from a Core File ======================================= To create a stack trace from a core file, perform the following steps: 1. Determine which executable program caused the core dump: To determine which executable caused a core dump, change to the directory where the core file is located, and use the "file" command. For example, if you had a core file in your home directory, "/home/user", use the following commands: % cd /home/user % file core core: ELF 32-bit MSB core file SPARC Version 1, from 'sqlplus' The "from" information indicates that 'sqlplus' caused the core dump. (a) If your system does not display "from" information when using the "file" command, and your are NOT using SCO UNIX, try the following command instead: % cd /home/user % strings core | head -10 CORE CORE CORE sqlplus <<----- first executable program sqlplus -h system-23_getm TERM=xterms console ORACLE_HOME guicommon/tk2 /SQLPlus/mesg/SP2US.msb This will display the first 10 text entries in the core file. Start at the first line and look for an entry which is an executable program. In the example above, 'sqlplus' caused the core dump. (b) If you are using SCO UNIX, use the following command instead: % cd /home/user % strings core | head -75 | tail -25 2. Determine which debugger exists on your system. There are many different debuggers in the UNIX world. Look at the list below of the most common debuggers, then check your system for one of them. Common debuggers ================ dbx, gdb, dde, adb, sdb, debug If these debuggers exist on your system, they would most likely be located in one of the following directories: /bin, /usr/bin, /opt/langtools/bin, /usr/ccs/bin, /usr/ucb To search your system for a debugger, you can use the "which" command, which will search only the directories in your current PATH, or the "find" command, which can search the entire system. For example, to search for the dbx debugger: % which dbx no dbx ... % find / -name "dbx" -print /usr/ccs/bin/dbx3. Now that you have located a debugger, you are ready to use it to produce the stack trace. The Unix command "script" will capture the output of a terminal session and write it to a file. You will use "script" to capture the output of the debugger as shown below. First change directories to the location where the core file is located. 4. The next step depends upon which debugger you are using: (a) If you are NOT using the "dde" or "debug" debugger, do this: ------------------------------------------------------------ % script mystack.trc % <debugger> $ORACLE_HOME/bin/<executable> core where <debugger> is the debugger you are using, and <executable> is the executable program which caused the core dump. (b) If you are using the "dde" debugger, use this command: ------------------------------------------------------ % script mystack.trc % dde -ui line core $ORACLE_HOME/bin/<executable> (c) If you are using the "debug" debugger, use this command: -------------------------------------------------------- % script mystack.trc % debug <display option> -c core $ORACLE_HOME/bin/<executable> where <display option> is not entered, unless "debug" starts an X window session, in which case <display option> equals "-i c". 5. Now look at the table below and find your debugger in the first column, then enter the command in the second column to produce the stack trace. debugger command to produce stack ======== ======================== dbx where gdbbt dde tb adb $c sdb t debug stack 6. Exit the debugger program: - if you are using the "adb" debugger, exit using <Ctrl>-D - if you are using any of the other debuggers, exit with the command "q" 7. After exiting the debugger, issue the command "exit" at the shell prompt. You should receive a "Script Done" message. You now have a stack trace in the file named "mystack.trc", and you can send this file to Oracle Support for further analysis. Solution Explanation: ===================== The call stack trace in the "mystack.trc" file can be useful to Oracle Support in determining why a core dump from an Oracle product occurred.   
  • Windows NT: Killing an Oracle Thread [ID 69882.1]  Modified 12-JUN-2007     Type FAQ     Status PUBLISHED   PURPOSE:-------- To discuss the ORAKILL utility and its application on an Oracle database on Windows NT.  SCOPE & APPLICATION:-------------------- You have an Oracle process that is using a lot of Windows NT resources and you want to determine if a user session is causing it. If it is a user session, you want to kill the corresponding user thread without crashing the instance. This is useful for DBAs who want to avoid shutting down the entire instance because of a single errant user session. Windows NT is a thread-based operating system (OS) rather than a process-based OS like UNIX. The Oracle background processes and user sessions are threads within the ORACLE.EXE process running in Windows NT. Since you don't have multiple processes, but one process with multiple threads, killing the ORACLE.EXE process (using NT Task Manager, for example) would crash the entire instance.  Since Windows NT does not provide a tool to kill individual threads, Oracle ships the ORAKILL utility with RDBMS versions 7.3.3.6 & up. This utility has the same effect of a KILL -9 command for killing a shadow process or background process in UNIX.  ORAKILL is run from the Command Prompt in the following syntax:  C:> orakill sid thread  where: sid is the Oracle instance to target thread is the thread id of the thread to kill Note: You can display the syntax of most Oracle executables by typing the executable name at the command prompt with no options.  To find the thread id consuming most of the CPU, the NT Resource Kit offers QuickSlice (QSLICE.EXE) which shows individual threads within a given process. (Please refer to the NT Resource Kit for instructions on how to use QSLICE.) If you do not have QSLICE, you can run the following query within a SQL*Plus session connected as a DBA user:  select p.spid THREADID, s.osuser, s.program from v$process p, v$session s where p.addr=s.paddr;  The result will be similar to this: THREADID OSUSER PROGRAM--------- ----------------------- -----------------------------169 SYSTEM ORACLE.EXE215 SYSTEM ORACLE.EXE280 SYSTEM ORACLE.EXE267 SYSTEM ORACLE.EXE287 SYSTEM ORACLE.EXE288 SYSTEM ORACLE.EXE271 SYSTEM ORACLE.EXE 282 SYSTEM ORACLE.EXE 266269239 PROD_NTdjones SVRMGRL.EXE281 SSMITH-PCssmith SQLPLUSW.EXE 12 rows selected.SVRMGR> WARNING: Killing a background thread (DBWR, LGWR, SMON or PMON) will crash the instance or put it in a non-usable state. Usually the first processes listed inthe above query (with OSUSER as SYSTEM and PROGRAM as ORACLE.EXE) are the background processes. However, to be certain, you can use the following query to find out the thread ID of the background process:   select vb.name NOME, vp.program PROCESSNAME, vp.spid THREADID, vs.sid SID from v$sessionvs, v$processvp, v$bgprocessvb where vb.paddr <> '00' and vb.paddr = vp.addr and vp.addr = vs.paddr;   The result will be similar to this:  NOME PROCESSNAME THREADID SID----- ----------------------------------- --------- ------PMON ORACLE.EXE 169 1DBW0 ORACLE.EXE 215 2LGWR ORACLE.EXE 280 3CKPT ORACLE.EXE 267 4SMON ORACLE.EXE 287 5RECO ORACLE.EXE 288 6SNP0 ORACLE.EXE 271 7SNP1 ORACLE.EXE 282 8 8 rows selected. RelatedProductsOracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition orakillThe orakill utility is provided only with Oracle databases on Windows platforms. The executable (orakill.exe) is available to DBAs to kill Oracle sessions directly from the DOS command line without requiring any connection to the database.  In the UNIX world, a DBA can kill a shadow process by issuing the kill –9 command from the UNIX prompt. UNIX is able to provide this capability given that the UNIX operating system is based on processes that fork other processes. All processes can be listed by using the ps UNIX command. The Oracle background processes will be listed separately from all of the Oracle sessions since they have their own process. Unlike the UNIX operating system, Windows systems are thread-based. For each instance, the background processes and sessions are all contained within the oracle.exe executable. These processes are not listed in the “Processes” tab of Windows Task Manager. Each session creates its own thread within oracle.exe and therefore, is not exposed to the Windows user. Killing the oracle.exe process in Windows would crash the entire database.  The orakill utility serves the same purpose as kill –9 in UNIX . The command requires the instance and the SPID of the thread to kill. The utility will display exactly how to obtain the SPID in the event the command was entered without parameters: C:oracle9iin>orakill Usage: orakill sid thread  where sid = the Oracle instance to target thread = the thread id of the thread to kill  The thread id should be retrieved from the spid column of a query such as:  select spid, osuser, s.program from v$process p, v$session s where p.addr=s.paddr If the statement suggested by Oracle (above) to retrieve the Thread ID is executed, the results below are displayed:  select a.username, a.osuser, b.spid from v$session a, v$process b where a.paddr = b.addr and a.username is not null;  USERNAME OSUSER SPID------------------------------ ------------------------------ -----SCOTT Scott 3116AMOORE Alex 4760DMOORE Dave 768 With the SPID for each user listed above, the session for any user can be killed.  C:oracle9iin>orakill ORCL92 4760 Kill of thread id 4760 in instance ORCL92 successfully signalled.  SQL> select a.username, a.osuser, b.spid 2 from v$session a, v$process b 3 where a.paddr = b.addr 4 and a.username is not null;  USERNAME OSUSER SPID------------------------------ ------------------------------ -----SCOTT Scott 3116DMOORE Dave 768 2 rows selected. Notice that SPID 4760, user AMOORE is gone.  Why does Oracle provide a utility to kill sessions from the DOS prompt, when a DBA could kill a user session from within Oracle? The following command will also kill the user session:  alter system kill session(sid, serial#);  The sid (session ID) and serial# above can be obtained from the v$session view. There are a couple of reasons a DBA might use orakill instead of the alter system kill session command.  1.   The alter system statement will not clear any locks that exist. Instead, the session will remain connected until it times out, then the session is killed and the locks are released. The orakill command will kill the thread and the locks instantly. 2.   A DBA may be unable to gain access to a SQL prompt due to a runaway query consuming all database resources. In this case, the session can be killed without ever logging in to the database. These are good reasons to kill threads directly from the DOS prompt, but they do not address how to get the required kill information from the database if access is unavailable. How can a DBA obtain the SPID?  One way to obtain the Oracle SPID is to use a tool like QuickSlice from Microsoft (free download) that will display Windows threads and their IDs. The QuickSlice main screen (Figure 3.1) displays each executable that is active in the operating system.  The entries listed in QuickSlice match those displayed in the Windows Task Manager. QuickSlice also displays the amount of the CPU being used by an executable, as does the Task Manager.   Figure 3.1 – QuickSlice Main ScreenThe DBA can use the main QuickSlice screen to determine the CPU-intensive executables, and drill down into the threads for more information.  The next QuickSlice window (Figure 3.2) displays the threads for a given executable (oracle.exe). In the example below, TID 300 is using the CPU.    Figure 3.2 – QuickSlice Thread WindowThe DBA can quickly identify the most consumptive thread and decide what to do. The TID column (Thread ID) in QuickSliceis a Hex value and matches the decimal value for the spid column from v$session. In this case, the TID consuming the CPU is 300 (Hex), which equals Session spid 768 (Decimal). Therefore, the command to kill this session would be:  C:oracle9iin>orakill ORCL92 768 In this example, the thread (Oracle session) was killed in the operating system without ever logging into the database. Before killing the session, the DBA may decide to view the SQL being executed by the session. This can be obtained by using the TID above (300) in the following SQL statement:  select b.username, a.sql_text from v$sqltext_with_newlines a, v$session b, v$process cwhere c.spid = to_number('300', 'xxx') and c.addr = b.paddr and b.sql_address = a.address; Tips for using orakill·     The orakill utility should be used as a last resort only. If the session cannot be killed more gracefully (via alter system kill session), or the instance is inaccessible via SQL, then orakill should be used to terminate the offending session. ·     Background processes should not be terminated, mainly user sessions. Killing a background process like SMON or PMON can cause serious Oracle errors and can bring the database down. To confirm that it is not a background session being killed, the following query can be used to identify the SPID for each background process:select c.name, b.spid, a.sid from v$session a, v$process b, v$bgprocess cwhere c.paddr <> '00' and c.paddr = b.addr and b.addr = a.paddr;    NAME SPID SID----- ------------ ----------PMON 1680 1DBW0 1828 2LGWR 1844 3CKPT 1852 4SMON 1848 5RECO 2060 6CJQ0 2064 7QMN0 2072 8 ·     Access to the Windows machine containing the database must be secure. Any user with access to the box could access orakill or the Windows Task Manager and damage database processes. ·     The DBA may decide to use operating system utilities to kill sessions and monitor Oracle CPU utilization. Starting at the operating system level is an effective technique employed by many seasoned DBAs.
  • What is an ORA-600 Internal Error? [ID 146580.1]  Modified 21-OCT-2008     Type REFERENCE     Status PUBLISHED In this Document  Purpose  What is an ORA-600 Internal Error?  ReferencesApplies to: Oracle Server - Enterprise EditionOracle Server - Personal EditionOracle Server - Standard EditionInformation in this document applies to any platform.PurposeThis article provides a brief description of ORA-600 Internal Errors. What is an ORA-600 Internal Error?ORA-600 errors are raised from the kernel code of the Oracle RDBMS software when an internal inconsistency is detected or an unexpected condition is met. This situation is not necessarily a bug as it might be caused by problems with the Operating System, lack of resources, hardware failures, etc. With the ORA-600 error comes a list of arguments in square brackets. The first of these arguments tells us from where in the code the error was caught and thus is the key information in identifying the problem. This argument is either a number or a character string. The remaining arguments are used to supply further information eg. values of internal variables etc. Whenever an ORA-600 error is raised a trace file is generated and an entry written to the alert.log with details of the trace file location. For Oracle11g an incident will be created - for earlier versions the trace file will be written to either USER_DUMP_DEST (if the error was caught in a user process) or BACKGROUND_DUMP_DEST (if the error was caught in a background process like PMON or SMON). The trace file contains vital information about what led to the error condition. ReferencesOERR: ORA 600 "internal error code, arguments: [%s],[%s],[%s], [%s], [%s], [ID 18485.1]  Modified 17-JUL-2006     Type REFERENCE     Status PUBLISHED  Error: ORA 600Text: internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s]-------------------------------------------------------------------------------Cause: This is the generic internal error number for Oracle program exceptions. This indicates that a process has encountered an exceptional condition.Action: Report as a bug - the first argument is the internal error number*** Important: The notes below are for experienced users - See Note:22080.1  Explanation: Any ORA-600 error indicates Oracle has detected an internal inconsistency or a problem which it doesnt know how best to address. These are *NOT* necessarily bugs and can occur for reasons such as Operating System IO problems, running out of some resource etc.. Diagnosis: Oracle support need more information to determine what caused the error. The most useful items are:  a) The alert log for the instance (and any other instances in a Parallel Server environment) These should be in BACKGROUND_DUMP_DEST.  b) The trace file for the failing session and any other relevant trace files. c) Information on what action was being performed.  d) Whether the problem is reproducible or was a one offoccurance.  e) Anything that has recently been changed in the system environment. Eg: Has the OS been upgraded, Have any database files just been restored, etc... Articles: What information to collect for ORA 600 <Bul:105787.757> ORA-600/ORA-7445 Troubleshooter Note: 153788.1     RelatedProductsOracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Personal Edition Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Standard Edition Keywords  
  • ×