Cloug Undocumented Secrets Black Magic


Published on

My encore session from OOW and CLOUG on Oracle 11g Tips

1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Cloug Undocumented Secrets Black Magic

  1. 1. Session : Undocumented Oracle DBA Utilities: Black Magic for the Oracle Expert Ben Prusinski Ben Prusinski and Associates [email_address] CLOUG/ Santiago, Chile Monday 13 April 2009
  2. 2. Speaker Qualifications Ben Prusinski <ul><li>Oracle ACE and Oracle Certified Professional with 14 plus years of real world experience with Oracle since version 7.3.4 </li></ul><ul><li>Oracle Author of two books on Oracle database technology </li></ul>
  3. 3. Agenda: Undocumented Oracle Utilities <ul><li>Top new features in Oracle 11g </li></ul><ul><li>Performance Tools: </li></ul><ul><li>Autotrace, SQL Trace, TKPROF, DBMS_STATS, SQL Tune </li></ul><ul><li>Oradebug </li></ul><ul><li>Hidden Oracle 11g parameters </li></ul><ul><li>Useful V$ and X$ tables for Oracle 11g </li></ul><ul><li>Power System Tools: dbx, trace, orakill </li></ul><ul><li>ORA-0600 Tips and Tricks </li></ul>
  4. 4. Top Features for Oracle 11g <ul><li>ADR for monitoring </li></ul><ul><li>Data Recovery Advisor with RMAN for Oracle Backup and Recovery with Oracle 11g </li></ul><ul><li>Real Application Testing (RAT) features with 11g </li></ul><ul><li>New partitioning features for Oracle 11g Data Partitioning </li></ul><ul><li>Enhancements for Oracle 11g Performance Tuning </li></ul><ul><li>Invisible Indexes new feature for Oracle 11g </li></ul>
  5. 5. What is ADR for Oracle 11g? <ul><li>ADR is the new Automatic Diagnostic Monitor with 11g release. Many new monitoring and reporting features. </li></ul><ul><li>Completely new paradigm for Oracle DBA monitoring </li></ul><ul><li>Available in both command line interface (adrci) or from Oracle 11g Grid/Database Control </li></ul><ul><li>Replaces alert.log, trace files text based monitoring in releases before 11g. </li></ul><ul><li>Alert.log still available but recommend you learn ADR </li></ul>
  6. 6. Introduction to Oracle 11g ADR <ul><li>The new feature Oracle 11g Automatic Diagnostic Repository (ADR) has following new features: </li></ul><ul><li>One stop shop for monitoring and reporting on Oracle 11g </li></ul><ul><li>New feature for incident reporting and packaging (IPS) </li></ul><ul><li>Available with Oracle 11g Database/Grid Control GUI as well as from command line utility (adrci) </li></ul><ul><li>Provides access to all of the Oracle 11g trace files for networking, alert.log, etc. </li></ul>
  7. 7. Inside Automatic Diagnostic Monitor (ADR) for Oracle 11g <ul><li>ADR stores main configuration details in log.xml file </li></ul><ul><li>Default ADR directory set Oracle 11g initialization diagnostic_dest parameter </li></ul><ul><li>SQL> show parameter diagnostic_dest </li></ul><ul><li>NAME TYPE VALUE </li></ul><ul><li>--------------------------------- ----------- ------ </li></ul><ul><li>diagnostic_dest string /u01/app/oracle </li></ul>
  8. 8. Inside the ADR for Oracle 11g <ul><li>ADR home directory located under default: </li></ul><ul><li><ADR_BASE>/diag/rdbms/<db_name>/<instance_id > </li></ul><ul><li>[oracle@raclinux1 ~]$ adrci </li></ul><ul><li>ADRCI: Release - Beta on Tue Jul 29 00:07:57 2008 </li></ul><ul><li>Copyright (c) 1982, 2007, Oracle. All rights reserved. </li></ul><ul><li>ADR base = &quot;/u01/app/oracle&quot; </li></ul><ul><li>adrci> show home </li></ul><ul><li>ADR Homes: </li></ul><ul><li>diag/rdbms/ora11g/ora11g </li></ul>
  9. 9. Using the Automatic Diagnostic Repository (ADR) <ul><li>Command line utility (adrci) from Oracle 11g Grid Control </li></ul><ul><li>Help Commands with ADR: help and extended help </li></ul><ul><li>adrci> help </li></ul><ul><li>HELP [topic] </li></ul><ul><li>Available Topics: </li></ul><ul><li>CREATE REPORT </li></ul><ul><li>ECHO </li></ul><ul><li>EXIT </li></ul><ul><li>HELP </li></ul><ul><li>HOST </li></ul><ul><li>IPS </li></ul><ul><li>PURGE </li></ul><ul><li>RUN </li></ul><ul><li>SET BASE </li></ul><ul><li>SET BROWSER </li></ul><ul><li>SET CONTROL </li></ul><ul><li>SET ECHO </li></ul><ul><li>SET EDITOR </li></ul><ul><li>SET HOMES | HOME | HOMEPATH </li></ul><ul><li>SET TERMOUT </li></ul><ul><li>SHOW ALERT </li></ul><ul><li>SHOW BASE </li></ul><ul><li>SHOW CONTROL </li></ul><ul><li>SHOW HM_RUN </li></ul><ul><li>SHOW HOMES | HOME | HOMEPATH </li></ul><ul><li>SHOW INCDIR </li></ul><ul><li>SHOW INCIDENT </li></ul>
  10. 10. Using extended help with Oracle 11g ADR <ul><li>[oracle@raclinux1 ~]$ adrci </li></ul><ul><li>ADRCI: Release - Beta on Tue Jul 29 00:29:52 2008 </li></ul><ul><li>Copyright (c) 1982, 2007, Oracle. All rights reserved. </li></ul><ul><li>ADR base = &quot;/u01/app/oracle&quot; </li></ul><ul><li>adrci> help extended </li></ul><ul><li>HELP [topic] </li></ul><ul><li>Available Topics: </li></ul><ul><li>BEGIN BACKUP </li></ul><ul><li>CD </li></ul><ul><li>DDE </li></ul><ul><li>DEFINE </li></ul><ul><li>DESCRIBE </li></ul><ul><li>END BACKUP </li></ul><ul><li>LIST DEFINE </li></ul><ul><li>MERGE ALERT </li></ul><ul><li>MERGE FILE </li></ul><ul><li>QUERY </li></ul><ul><li>SET COLUMN </li></ul><ul><li>SHOW CATALOG </li></ul><ul><li>SHOW DUMP </li></ul><ul><li>SHOW SECTION </li></ul><ul><li>SHOW TRACE </li></ul><ul><li>SHOW TRACEMAP </li></ul><ul><li>SWEEP </li></ul><ul><li>UNDEFINE </li></ul><ul><li>VIEW </li></ul>
  11. 11. Viewing alert.log and trace files with ADR for Oracle 11g <ul><li>Using ADR to view alert.log: </li></ul><ul><li>Use the show alert command with ADR: </li></ul><ul><li>[oracle@raclinux1 ~]$ adrci </li></ul><ul><li>ADRCI: Release - Beta on Sun Aug 17 21:17:08 2008 </li></ul><ul><li>Copyright (c) 1982, 2007, Oracle. All rights reserved. </li></ul><ul><li>ADR base = &quot;/u01/app/oracle&quot; </li></ul><ul><li>adrci> show alert </li></ul><ul><li>Choose the alert log from the following homes to view: </li></ul><ul><li>1: diag/rdbms/ora11g/ora11g </li></ul><ul><li>2: diag/rdbms/ora11g/ORA11G </li></ul><ul><li>3: diag/rdbms/default/ORA11G </li></ul><ul><li>4: diag/rdbms/unknown/ORA11G </li></ul><ul><li>5: diag/clients/user_oracle/host_3681296775_11 </li></ul><ul><li>6: diag/tnslsnr/raclinux1/listener </li></ul><ul><li>Q: to quit </li></ul><ul><li>Please select option: 1 </li></ul>
  12. 12. Viewing the Oracle 11g alert.log with ADR <ul><li>2008-07-31 15:23:53.534000 -04:00 </li></ul><ul><li>Starting ORACLE instance (normal) </li></ul><ul><li>2008-07-31 15:23:55.226000 -04:00 </li></ul><ul><li>LICENSE_MAX_SESSION = 0 </li></ul><ul><li>LICENSE_SESSIONS_WARNING = 0 </li></ul><ul><li>Shared memory segment for instance monitoring created </li></ul><ul><li>Picked latch-free SCN scheme 2 </li></ul><ul><li>Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/11g/dbs/arch </li></ul><ul><li>Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST </li></ul><ul><li>Autotune of undo retention is turned on. </li></ul>
  13. 13. Viewing trace files with ADR for Oracle 11g <ul><li>Use the show tracefile command for ADR with Oracle 11g </li></ul><ul><li>adrci> show tracefile </li></ul><ul><li>diag/rdbms/ora11g/ora11g/trace/alert_ora11g.log </li></ul><ul><li>diag/rdbms/ora11g/ora11g/trace/ora11g_ora_21555.trc </li></ul><ul><li>diag/rdbms/ora11g/ora11g/trace/ora11g_vktm_21688.trc </li></ul>
  14. 14. Incident Reporting with ADR for Oracle 11g <ul><li>ADR uses incidents for critical errors with Oracle 11g </li></ul><ul><li>These incidents can be packaged by ADR and sent to Oracle support for analysis with Oracle 11g </li></ul><ul><li>Useful new feature with Oracle 11g for problem resolution and reporting </li></ul><ul><li>Performs many of the tasks used by the RDA (Remote Diagnostic Assistant) for releases before Oracle 11g. </li></ul>
  15. 15. Show Incident with ADR and Oracle 11g <ul><li>Critical errors appear as incidents: can use show incident command for ADR with Oracle 11g </li></ul><ul><li>adrci> set homepath diag/rdbms/ora11g/ORA11G </li></ul><ul><li>adrci> show incident </li></ul><ul><li>ADR Home = /u01/app/oracle/diag/rdbms/ora11g/ORA11G: </li></ul><ul><li>************************************************************************* </li></ul><ul><li>INCIDENT_ID PROBLEM_KEY CREATE_TIME </li></ul><ul><li>-------------------- ----------------------------------------------------------- ---------------------------------------- </li></ul><ul><li>11009 ORA 600 [kssadpm: null parent] 2008-08-15 19:08:30.464549 -04:00 </li></ul><ul><li>1 rows fetched </li></ul>
  16. 16. Show incident detail for ADR and Oracle 11g <ul><li>Detail view for incidents with ADR and Oracle 11g: </li></ul><ul><li>adrci> show incident -mode detail -p &quot;incident_id=11009&quot; </li></ul><ul><li>ADR Home = /u01/app/oracle/diag/rdbms/ora11g/ORA11G: </li></ul><ul><li>************************************************************************* </li></ul><ul><li>********************************************************** </li></ul><ul><li>INCIDENT INFO RECORD 1 </li></ul><ul><li>********************************************************** </li></ul><ul><li>INCIDENT_ID 11009 </li></ul><ul><li>STATUS ready </li></ul><ul><li>CREATE_TIME 2008-08-15 19:08:30.464549 -04:00 </li></ul><ul><li>PROBLEM_ID 1 </li></ul>
  17. 17. Packaging Incidents for Oracle 11g Support <ul><li>ADR provides the following commands for managing incidents: </li></ul><ul><li>IPS CREATE PACKAGE for new incident with Oracle 11g </li></ul><ul><li>IPS ADD INCIDENT <incident_id> package <package_id> </li></ul><ul><li>IPS GENERATE PACKAGE <package_id> in <directory> </li></ul><ul><li>This procedure creates a ZIP file with the trace diagnostics for Oracle support to analyze for critical errors. </li></ul><ul><li>Use the IPS SHOW CONFIGURATION command to check configuration for ADR incidents. </li></ul>
  18. 18. Example: Incident packages for ADR and Oracle 11g <ul><li>First we need to create a new package for incident: </li></ul><ul><li>adrci> ips create package </li></ul><ul><li>Created package 1 without any contents, correlation level typical </li></ul><ul><li>Then we need to add the incident to the new package for ADR: </li></ul><ul><li>adrci> ips add incident 11009 package 1; </li></ul><ul><li>Added incident 11009 to package 1 </li></ul><ul><li>Finally, we need to generate the package with data: </li></ul><ul><li>adrci> ips generate package 1 in /u01/app/oracle/adrci/support </li></ul><ul><li>Generated package 1 in file /u01/app/oracle/adrci/support/, mode complete </li></ul><ul><li>The new file is generated as a zip file in the directory for support. </li></ul>
  19. 19. Performance Toolkit for Oracle 11g <ul><li>Every DBA should use these tools for tuning Oracle! </li></ul><ul><li>Autotrace </li></ul><ul><li>SQL Trace </li></ul><ul><li>TKPROF </li></ul><ul><li>DBMS_STATS </li></ul><ul><li>DBMS_SQLTUNE packages </li></ul><ul><li>ORADEBUG </li></ul>
  20. 20. Using Autotrace with Oracle 11g <ul><li>Autotrace useful quick and dirty tool for tracing with Oracle 11g. </li></ul><ul><li>Provides many tracing and tuning options: </li></ul><ul><li>explain plan </li></ul><ul><li>costs associated with SQL queries </li></ul><ul><li>tuning statistics </li></ul><ul><li>Used with SQL*PLUS interface </li></ul><ul><li>Excellent starting point for tuning SQL queries </li></ul>
  21. 21. Using Autotrace with Oracle 11g <ul><li>Use the set autotrace on command in SQL*Plus to enable Autotrace and set autotrace off to disable autotrace tool. </li></ul><ul><li>Syntax for autotrace: </li></ul><ul><li>SQL> set autotrace </li></ul><ul><li>Usage: SET AUTOT[RACE] {OFF | ON | TRACE[ONLY]} [EXP[LAIN]] [STAT[ISTICS]] </li></ul><ul><li>SQL> </li></ul>
  22. 22. Autotrace Example for Oracle 11g <ul><li>SQL> set autotrace on </li></ul><ul><li>SQL> select prod_name, prod_list_price </li></ul><ul><li>2 from sh.products </li></ul><ul><li>3 where prod_list_price>7.99 </li></ul><ul><li>4 and </li></ul><ul><li>prod_list_price<69.99; </li></ul>
  23. 23. Autotrace with Oracle 11g <ul><li>Execution Plan </li></ul><ul><li>---------------------------------------------------------- </li></ul><ul><li>Plan hash value: 1954719464 </li></ul><ul><li>------------------------------------------------------------------------------ </li></ul><ul><li>| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | </li></ul><ul><li>------------------------------------------------------------------------------ </li></ul><ul><li>| 0 | SELECT STATEMENT | | 3 | 105 | 3 (0)| 00:00:01 | </li></ul><ul><li>|* 1 | TABLE ACCESS FULL| PRODUCTS | 3 | 105 | 3 (0)| 00:00:01 | </li></ul><ul><li>------------------------------------------------------------------------------ </li></ul><ul><li>Predicate Information (identified by operation id): </li></ul><ul><li>--------------------------------------------------- </li></ul><ul><li>1 - filter(&quot;PROD_LIST_PRICE&quot;<69.99 AND &quot;PROD_LIST_PRICE&quot;>7.99) </li></ul><ul><li>Statistics </li></ul><ul><li>---------------------------------------------------------- </li></ul><ul><li>8 recursive calls </li></ul><ul><li>0 db block gets </li></ul><ul><li>10 consistent gets </li></ul><ul><li>0 physical reads </li></ul><ul><li>0 redo size </li></ul><ul><li>2548 bytes sent via SQL*Net to client </li></ul><ul><li>453 bytes received via SQL*Net from client </li></ul><ul><li>5 SQL*Net roundtrips to/from client </li></ul><ul><li>0 sorts (memory) </li></ul><ul><li>0 sorts (disk) </li></ul><ul><li>52 rows processed </li></ul>
  24. 24. How to Use SQL Trace with Oracle 11g <ul><li>SQL Trace powerful tool for the serious Oracle Performance Expert </li></ul><ul><li>Event based: 10046 (SQL Trace), 10053 (Cost Based Optimizer- CBO) </li></ul><ul><li>Can tune instance, database or session level </li></ul><ul><li>Use DBMS_SUPPORT, ORADEBUG, or SQL*PLUS commands to enable tracing activities with Oracle 11g. </li></ul>
  25. 25. How to Perform SQL Trace for Oracle 11g <ul><li>Oracle Metalink Note: 199081.1 </li></ul><ul><li>Provides Good Overview Reference for SQL Trace </li></ul><ul><li>Using SQL*PLUS commands for SQL trace at session/instance levels: </li></ul><ul><li>ALTER SYSTEM/SESSION SET SQL_TRACE=TRUE; </li></ul><ul><li>ALTER SYSTEM/SESSION SET TIMED_STATISTICS=TRUE; </li></ul><ul><li>Optional for SQL Timing tracing details: </li></ul><ul><li>ALTER SYSTEM/SESSION SET EVENTS '10046 trace name context forever, level <level>'; </li></ul><ul><li>Outputs SQL trace files to user_dump_dest as .trc file. </li></ul><ul><li>SHOW PARAMETER USER_DUMP_DEST </li></ul>
  26. 26. SQL Trace Example for Oracle 11g <ul><li>Should use SYS enabled account to run the SQL trace </li></ul><ul><li>Need to have explain plan enabled (ie: utlxplan.sql) </li></ul><ul><li>Enable parameters for SQL tracing: timed_statistics, sql_trace </li></ul><ul><li>Recommend use ALTER SESSION as ALTER SYSTEM can place load on Oracle 11g system and impact performance. </li></ul><ul><li>SQL> alter session set timed_statistics=true; </li></ul><ul><li>Session altered. </li></ul><ul><li>SQL> alter session set sql_trace=true; </li></ul><ul><li>Session altered. </li></ul><ul><li>SQL> show parameter user_dump_dest </li></ul><ul><li>NAME TYPE VALUE </li></ul><ul><li>------------------------------------ ----------- ------------------------------ </li></ul><ul><li>user_dump_dest string /u01/app/oracle/diag/rdbms/ora </li></ul><ul><li>11g/ORA11G/trace </li></ul>
  27. 27. Contents of SQL trace file <ul><li>Use the ALTER SESSION SET TRACEFILE_IDENTIFIER command to name trace file for easy identification </li></ul><ul><li>SQL> alter session set tracefile_identifier='products_sqltune'; </li></ul><ul><li>Session altered. </li></ul><ul><li>SQL> show parameter tracefile_identifier </li></ul><ul><li>NAME TYPE VALUE </li></ul><ul><li>------------------------------------ ----------- ------------------------------ </li></ul><ul><li>tracefile_identifier string products_sqltune </li></ul><ul><li>Contents of sample SQL Trace file: </li></ul><ul><li>PARSING IN CURSOR #2 len=104 dep=0 uid=0 oct=3 lid=0 tim=1219028054543774 hv=4040144915 ad='269bdafc' sqlid='9cwk38msczf0m' </li></ul><ul><li>select prod_name, prod_list_price </li></ul><ul><li>from sh.products </li></ul><ul><li>where prod_list_price>11.99 </li></ul><ul><li>and </li></ul><ul><li>prod_list_price<99.99 </li></ul><ul><li>END OF STMT </li></ul><ul><li>PARSE #2:c=20000,e=29286,p=0,cr=2,cu=0,mis=1,r=0,dep=0,og=1,tim=1219028054543755 </li></ul><ul><li>EXEC #2:c=0,e=46,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1219028054543990 </li></ul><ul><li>FETCH #2:c=0,e=195,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,tim=1219028054544342 </li></ul><ul><li>FETCH #2:c=0,e=60,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,tim=1219028054544999 </li></ul><ul><li>FETCH #2:c=0,e=115,p=0,cr=2,cu=0,mis=0,r=15,dep=0,og=1,tim=1219028054561250 </li></ul><ul><li>FETCH #2:c=0,e=89,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,tim=1219028054617141 </li></ul><ul><li>FETCH #2:c=0,e=31,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1219028054628186 </li></ul><ul><li>STAT #2 id=1 cnt=46 pid=0 pos=1 obj=70464 op='TABLE ACCESS FULL PRODUCTS (cr=7 pr=0 pw=0 time=0 us cost=3 size=175 card=5)' </li></ul>
  28. 28. What is the TKPROF Utility? <ul><li>TKPROF is the Trace Kernel Profiler </li></ul><ul><li>Useful for formatting trace files for tuning sessions </li></ul><ul><li>Flexible options for performance report statistics </li></ul><ul><li>Cleans up the raw SQL trace files into human readable tuning reports. </li></ul><ul><li>Oracle Metalink Note:41634.1 TKPROF Simplistic Overview </li></ul>
  29. 29. How to Use TKPROF <ul><li>TKPROF takes the trace file and output report file plus performance specific parameters. </li></ul><ul><li>Many parameter options for using tkprof </li></ul><ul><li>Syntax for TKPROF: </li></ul><ul><li>tkprof trace_file.trc output_file.rpt [waits={yes|no}] [sort={yes|no}] [print=option] [aggregate={yes|no}] [insert={filename}] [sys={yes|no}] [table={schema.table}] [explain={schema/password}] [record=filename] [width=n] </li></ul>
  30. 30. Getting Help with TKPROF <ul><li>TKPROF Help: </li></ul><ul><li>[oracle@raclinux1 trace]$ tkprof </li></ul><ul><li>Usage: tkprof tracefile outputfile [explain= ] [table= ] </li></ul><ul><li>[print= ] [insert= ] [sys= ] [sort= ] </li></ul><ul><li>table=schema.tablename Use 'schema.tablename' with 'explain=' option. </li></ul><ul><li>explain=user/password Connect to ORACLE and issue EXPLAIN PLAN. </li></ul><ul><li>print=integer List only the first 'integer' SQL statements. </li></ul><ul><li>aggregate=yes|no </li></ul><ul><li>insert=filename List SQL statements and data inside INSERT statements. </li></ul><ul><li>sys=no TKPROF does not list SQL statements run as user SYS. </li></ul><ul><li>record=filename Record non-recursive statements found in the trace file. </li></ul><ul><li>waits=yes|no Record summary for any wait events found in the trace file. </li></ul><ul><li>sort=option Set of zero or more of the following sort options: </li></ul><ul><li>prscnt number of times parse was called </li></ul><ul><li>prscpu cpu time parsing </li></ul><ul><li>prsela elapsed time parsing </li></ul><ul><li>prsdsk number of disk reads during parse </li></ul><ul><li>prsqry number of buffers for consistent read during parse </li></ul>
  31. 31. Formatting SQL trace file with TKPROF <ul><li>Example: </li></ul><ul><li>[oracle@raclinux1 trace]$ tkprof ORA11G_ora_9163_products_sqltune.trc tune_sql.rpt waits=yes aggregate=yes sys=yes explain=sys/oracle </li></ul><ul><li>TKPROF: Release - Production on Sun Aug 17 23:52:36 2008 </li></ul><ul><li>Copyright (c) 1982, 2007, Oracle. All rights reserved. </li></ul>
  32. 32. Output Report for SQL Trace with TKPROF <ul><li>SQL ID : 1mpqj84y8wd72 </li></ul><ul><li>select * </li></ul><ul><li>from </li></ul><ul><li>oe.products </li></ul><ul><li>call count cpu elapsed disk query current rows </li></ul><ul><li>------- ------ -------- ---------- ---------- ---------- ---------- ---------- </li></ul><ul><li>Parse 1 0.01 0.01 0 0 0 0 </li></ul><ul><li>Execute 1 0.00 0.00 0 0 0 0 </li></ul><ul><li>Fetch 21 0.08 0.33 385 410 0 288 </li></ul><ul><li>------- ------ -------- ---------- ---------- ---------- ---------- ---------- </li></ul><ul><li>total 23 0.09 0.35 385 410 0 288 </li></ul><ul><li>Misses in library cache during parse: 1 </li></ul><ul><li>Optimizer mode: ALL_ROWS </li></ul><ul><li>Parsing user id: SYS </li></ul><ul><li>Rows Row Source Operation </li></ul><ul><li>------- --------------------------------------------------- </li></ul><ul><li>288 HASH JOIN OUTER (cr=410 pr=385 pw=385 time=18571 us cost=108 size=144864 card=288) </li></ul><ul><li>288 TABLE ACCESS FULL PRODUCT_INFORMATION (cr=16 pr=14 pw=14 time=25509 us cost=5 size=63360 card=288) </li></ul><ul><li>288 TABLE ACCESS FULL PRODUCT_DESCRIPTIONS (cr=394 pr=371 pw=371 time=14095 us cost=102 size=81504 card=288) </li></ul><ul><li>Elapsed times include waiting on following events: </li></ul><ul><li>Elapsed times include waiting on following events: </li></ul><ul><li>Event waited on Times Max. Wait Total Waited </li></ul><ul><li>---------------------------------------- Waited ---------- ------------ </li></ul><ul><li>SQL*Net message to client 21 0.00 0.00 </li></ul><ul><li>db file sequential read 2 0.00 0.00 </li></ul><ul><li>db file scattered read 22 0.06 0.21 </li></ul><ul><li>SQL*Net message from client 21 148.92 155.68 </li></ul><ul><li>SQL*Net more data to client 4 0.01 0.02 </li></ul><ul><li>******************************************************************************** </li></ul>
  33. 33. New Features for DBMS_STATS and Oracle 11g <ul><li>DBMS_STATS provides collection and reporting of database and system level statistics for Oracle 11g. </li></ul><ul><li>New feature for DBMS_STATS with Oracle 11g is the ability to collect/gather column level statistics </li></ul><ul><li>Extended statistics collection for Oracle 11g and DBMS_STATS </li></ul><ul><li>Public and Private statistics collection </li></ul>
  34. 34. Extended Statistics with DBMS_STATS and Oracle 11g <ul><li>First check default settings for DBMS_STATS and Oracle 11g using DBMS_STATS.GET_PARAM and DBMS_STATS.GET_PREFS </li></ul><ul><li>SQL> select dbms_stats.get_param('PUBLISH') from dual; </li></ul><ul><li>DBMS_STATS.GET_PARAM('PUBLISH') </li></ul><ul><li>--------------------------------------------------------------------------------TRUE </li></ul><ul><li>SQL> select dbms_stats.get_prefs('stale_percent','SH','products') </li></ul><ul><li>2 stale_percent from dual; </li></ul><ul><li>STALE_PERCENT </li></ul><ul><li>----------------------------------------------------------------------------- </li></ul><ul><li>10 </li></ul>
  35. 35. Collecting Extended Statistics for Oracle 11g <ul><li>Create column group for DBMS_STATS and Oracle 11g: </li></ul><ul><li>SQL> var ret_col varchar2(2000) </li></ul><ul><li>SQL> exec :ret_col := dbms_stats.create_extended_stats('SH','PRODUCTS','(PROD_ID, PROD_NAME)'); </li></ul><ul><li>PL/SQL procedure successfully completed. </li></ul><ul><li>SQL> print :ret_col </li></ul><ul><li>RET_COL </li></ul><ul><li>Gather column statistics for Oracle 11g SH.PRODUCTS table </li></ul><ul><li>SQL> begin </li></ul><ul><li>2 dbms_stats.gather_table_stats( </li></ul><ul><li>3 ownname=>'SH', </li></ul><ul><li>4 tabname=>'PRODUCTS', </li></ul><ul><li>5 estimate_percent=>100, </li></ul><ul><li>6 method_opt=>'FOR ALL COLUMNS SIZE SKEWONLY FOR COLUMNS (PROD_ID, PROD_NAME)', </li></ul><ul><li>7 cascade=> true </li></ul><ul><li>8 ); </li></ul><ul><li>9 end; </li></ul><ul><li>10 / </li></ul><ul><li>SQL> select table_name, extension </li></ul><ul><li>2 from dba_stat_extensions </li></ul><ul><li>3 where owner='SH'; </li></ul><ul><li>TABLE_NAME </li></ul><ul><li>------------------------------ </li></ul><ul><li>EXTENSION </li></ul><ul><li>-------------------------------------------------------------------------------- </li></ul><ul><li>PRODUCTS </li></ul><ul><li>(&quot;PROD_ID&quot;,&quot;PROD_NAME&quot;) </li></ul>
  36. 36. DBMS_SQLTune Package <ul><li>What is the DBMS_SQL Tune package? </li></ul><ul><li>Automates SQL tuning by using profiles to tune SQL. </li></ul><ul><li>Uses DBMS_SQLTUNE package as part of the SQL Tune Advisors (STA) with Oracle 11g </li></ul>
  37. 37. Using DBMS_SQLTUNE for Oracle 11g <ul><li>Example: </li></ul><ul><li>SQL> DECLARE </li></ul><ul><li>2 sql_task_name VARCHAR2(30); </li></ul><ul><li>3 tune_sqltext CLOB; </li></ul><ul><li>4 BEGIN </li></ul><ul><li>5 tune_sqltext :=' SELECT * ' || </li></ul><ul><li>6 'FROM sales '|| 'WHERE prod_id=20 AND '|| 'cust_id=200 '; </li></ul><ul><li>7 sql_task_name := DBMS_SQLTUNE.CREATE_TUNING_TASK( </li></ul><ul><li>8 sql_text=>tune_sqltext, </li></ul><ul><li>9 user_name=>'SH', </li></ul><ul><li>10 scope=>'COMPREHENSIVE', </li></ul><ul><li>11 time_limit=>60, </li></ul><ul><li>12 task_name=>'TUNE_SH_TASK', </li></ul><ul><li>13 description=>'New task for tuning SH query'); </li></ul><ul><li>14 END; </li></ul><ul><li>15 / </li></ul><ul><li>PL/SQL procedure successfully completed. </li></ul>
  38. 38. Using DBMS_SQLTUNE for Oracle 11g (Continued) <ul><li>DBMS_SQLTUNE uses the CREATE_TUNING_TASK function as part of SQL Tuning Advisor (STA) with 11g </li></ul><ul><li>Once we have created the tuning task, we need to execute it with the DBMS_SQLTUNE.EXECUTE_TUNING_TASK: </li></ul><ul><li>SQL> begin </li></ul><ul><li>2 DBMS_SQLTUNE.EXECUTE_TUNING_TASK(task_name=>' TUNE_SH_TASK'); </li></ul><ul><li>3 end; </li></ul><ul><li>4 / </li></ul><ul><li>PL/SQL procedure successfully completed. </li></ul>
  39. 39. Viewing SQL Tuning Advisor Report for Oracle 11g <ul><li>To view the SQL Tuning report, query the DBMS_SQLTUNE.REPORT_TUNING_TASK: </li></ul><ul><li>SQL> select dbms_sqltune.report_tuning_task('TUNE_SH_TASK') </li></ul><ul><li>2 from dual; </li></ul><ul><li>DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK') </li></ul><ul><li>---------------------------------------------------------------------------------------------------- </li></ul><ul><li>GENERAL INFORMATION SECTION </li></ul><ul><li>------------------------------------------------------------------------------- </li></ul><ul><li>Tuning Task Name : TUNE_SH_TASK </li></ul><ul><li>Tuning Task Owner : SH </li></ul><ul><li>Workload Type : Single SQL Statement </li></ul><ul><li>Scope : COMPREHENSIVE </li></ul><ul><li>Time Limit(seconds) : 60 </li></ul><ul><li>Completion Status : COMPLETED </li></ul><ul><li>Started at : 08/14/2008 13:46:22 </li></ul><ul><li>Completed at : 08/14/2008 13:46:23 </li></ul><ul><li>DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK') </li></ul><ul><li>---------------------------------------------------------------------------------------------------- </li></ul><ul><li>------------------------------------------------------------------------------- </li></ul><ul><li>Schema Name: SH </li></ul><ul><li>SQL ID : dbq1k7xwc387x </li></ul><ul><li>SQL Text : SELECT * FROM sales WHERE prod_id=20 AND cust_id=200 </li></ul><ul><li>------------------------------------------------------------------------------- </li></ul><ul><li>FINDINGS SECTION (1 finding) </li></ul><ul><li>------------------------------------------------------------------------------- </li></ul><ul><li>1- Index Finding (see explain plans section below) </li></ul>
  40. 40. Introducing Oradebug <ul><li>What is Oradebug? </li></ul><ul><li>Why should you use it? </li></ul><ul><li>Performance Tuning </li></ul><ul><li>Hang Analysis </li></ul><ul><li>Dump Analysis for Oracle 11g Memory </li></ul><ul><li>Suspend and Resume Oracle processes </li></ul>
  41. 41. Inside Oradebug <ul><li>Oradebug introduced in version 7.x release of Oracle </li></ul><ul><li>Poorly documented utility </li></ul><ul><li>Recommend caution when using oradebug </li></ul><ul><li>Syntax for Oradebug </li></ul>
  42. 42. Help options for Oradebug <ul><li>Oradebug has MANY options!!!! </li></ul><ul><li>Use the oradebug help command to list all the main options available: </li></ul><ul><li>SQL> oradebug help </li></ul><ul><li>HELP [command] Describe one or all commands </li></ul><ul><li>SETMYPID Debug current process </li></ul><ul><li>SETOSPID <ospid> Set OS pid of process to debug </li></ul><ul><li>SETORAPID <orapid> ['force'] Set Oracle pid of process to debug </li></ul><ul><li>SETORAPNAME <orapname> Set Oracle process name to debug </li></ul><ul><li>SHORT_STACK Get abridged OS stack </li></ul><ul><li>CURRENT_SQL Get current SQL </li></ul><ul><li>DUMP <dump_name> <lvl> [addr] Invoke named dump </li></ul><ul><li>DUMPSGA [bytes] Dump fixed SGA </li></ul><ul><li>DUMPLIST Print a list of available dumps </li></ul><ul><li>EVENT <text> Set trace event in process </li></ul><ul><li>SESSION_EVENT <text> Set trace event in session </li></ul><ul><li>DUMPVAR <p|s|uga> <name> [level] Print/dump a fixed PGA/SGA/UGA variableDUMPTYPE <address> <type> <count> Print/dump an address with type info </li></ul><ul><li>SETVAR <p|s|uga> <name> <value> Modify a fixed PGA/SGA/UGA variabl </li></ul>
  43. 43. How to Use Oradebug- Tracing a Session <ul><li>First, we need to obtain the SPID from V$SESSION and V$PROCESS: </li></ul><ul><li>SQL> select username, sid, serial#, paddr </li></ul><ul><li>2 from </li></ul><ul><li>3 v$session </li></ul><ul><li>4 where username='SH'; </li></ul><ul><li>USERNAME SID SERIAL# PADDR </li></ul><ul><li>------------------------------ ---------- ---------- -------- </li></ul><ul><li>SH 127 573 37A56AF0 </li></ul><ul><li>SQL> select addr, pid, spid </li></ul><ul><li>2 from v$process </li></ul><ul><li>3 where addr='37A56AF0'; </li></ul><ul><li>ADDR PID SPID </li></ul><ul><li>-------- ---------- ------------------------ </li></ul><ul><li>37A56AF0 18 14298 </li></ul>
  44. 44. Tracing a session with Oradebug <ul><li>Then we plug in the SPID to oradebug to trace session: </li></ul><ul><li>SQL> oradebug setospid 14298 </li></ul><ul><li>Oracle pid: 18, Unix process pid: 14298, image: (TNS V1-V3) </li></ul><ul><li>We then set the unlimit parameter for the trace file with Oradebug: </li></ul><ul><li>SQL> oradebug unlimit </li></ul><ul><li>Statement processed. </li></ul><ul><li>Now we are ready to start the trace for our session with Oradebug: </li></ul><ul><li>SQL> oradebug event 10046 trace name context forever, level 4 </li></ul><ul><li>Statement processed. </li></ul>
  45. 45. Tracing SQL for Session with Oradebug <ul><li>Oradebug uses Events for tracing options. </li></ul><ul><li>Some common trace events: </li></ul><ul><li>10053- Cost Based Optimizer (CBO) trace </li></ul><ul><li>10046- SQL Trace </li></ul><ul><li>To start tracing with Oradebug: </li></ul><ul><li>SQL> oradebug event 10046 trace name context forever, level 4 </li></ul><ul><li>Statement processed. </li></ul><ul><li>To stop and turn off tracing for Oradebug: </li></ul><ul><li>SQL> oradebug event 10046 trace name context off </li></ul><ul><li>Statement processed. </li></ul>
  46. 46. Locate the Session Trace file for Oradebug <ul><li>Trace file output based on user_dump_dest parameter </li></ul><ul><li>Naming convention for trace files: </li></ul><ul><li>sid_ora_pid_traceid.trc </li></ul><ul><li>Easy way to find trace file is to set tracefile_identifier parameter: </li></ul><ul><li>alter session set tracefile_identifier=‘mytrace’; </li></ul><ul><li>Useful query to find trace file: </li></ul><ul><li>select u_dump.value || '/' || db_name.value || '_ora_' || v$process.spid || nvl2(v$process.traceid, '_' || v$process.traceid, null ) || '.trc' &quot;Trace File&quot; from v$parameter u_dump cross join v$parameter db_name cross join v$process join v$session on v$process.addr = v$session.paddr where = 'user_dump_dest' and = 'db_name' and v$session.audsid= sys_context ('userenv','sessionid'); </li></ul>
  47. 47. Viewing the Oradebug trace file for user session <ul><li>Locate the trace file for oradebug </li></ul><ul><li>Format it with tkprof </li></ul><ul><li>[oracle@raclinux1 trace]$ tkprof ORA11G_ora_14298.trc trace_sh_session_sql.rpt waits=yes aggregate=yes sys=no explain=sh/sh </li></ul><ul><li>TKPROF: Release - Production on Sat Aug 16 03:47:15 2008 </li></ul><ul><li>Copyright (c) 1982, 2007, Oracle. All rights reserved. </li></ul><ul><li>[oracle@raclinux1 trace]$ view trace_sh_session_sql.rpt </li></ul><ul><li>Open the trace file with text editor </li></ul><ul><li>$ view trace_sh_session_sql.rpt </li></ul>
  48. 48. Using Oradebug for Hang Analysis <ul><li>Oradebug is excellent tool for Solving Database Hang Issues. </li></ul><ul><li>Use the HANGANALYZE option with Oradebug </li></ul><ul><li>Syntax: </li></ul><ul><li>oradebug hanganalyze <level> </li></ul><ul><li>For Oracle RAC environment the syntax is: </li></ul><ul><li>oradebug –g all hanganalyze <level> </li></ul><ul><li>Level indicates amount of details to collect for a hang. </li></ul>
  49. 49. Oracle Database Hang Analysis with Oradebug <ul><li>The debug levels defined for HANGANLYZE with oradebug are as follows: </li></ul><ul><li>Level 10: Dumps all processes ( IGN state ) </li></ul><ul><li>Level 5: Level 4 + Dump all processes involved wait chain ( NLEAF ) </li></ul><ul><li>Level 4: Level 3 + Dump leaf nodes ( blockers ) in wait chains ( LEAF, LEAF_NW, IGN_DMP state) </li></ul><ul><li>Level 3: Level 2 + Dump only processes considered in hang condition ( IN_HANG state ) </li></ul><ul><li>Levels 1-2: Generate only HANGANALYZE output with no process dump </li></ul>
  50. 50. Hang Analysis Example for Oracle 11g <ul><li>Set Environment for Hang Analysis: </li></ul><ul><li>SQL> oradebug setmypid </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug unlimit </li></ul><ul><li>Statement processed. </li></ul><ul><li>Perform Hang Analysis: </li></ul><ul><li>SQL> oradebug hanganalyze 3 </li></ul><ul><li>Hang Analysis in /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora_10509.trc </li></ul><ul><li>SQL> exit </li></ul>
  51. 51. Review Hang Analysis Oracle 11g <ul><li>Review output hang analysis from trace file: </li></ul><ul><li>[oracle@raclinux1 trace]$ view ORA11G_ora_10509.trc </li></ul><ul><li>*** 2008-08-16 02:50:45.862 </li></ul><ul><li>Processing Oradebug command 'hanganalyze 3' </li></ul><ul><li>*** 2008-08-16 02:50:45.917 </li></ul><ul><li>=============================================================================== </li></ul><ul><li>HANG ANALYSIS: </li></ul><ul><li>instances (db_name.oracle_sid): ora11g.ora11g </li></ul><ul><li>oradebug_node_dump_level: 3 </li></ul><ul><li>analysis initiated by oradebug </li></ul><ul><li>=============================================================================== </li></ul><ul><li>Chains most likely to have caused the hang: </li></ul><ul><li><no chains found> </li></ul><ul><li>=============================================================================== </li></ul><ul><li>No chains found. </li></ul><ul><li>=============================================================================== </li></ul><ul><li>Extra information that will be dumped at higher levels: </li></ul><ul><li>State of ALL nodes </li></ul><ul><li>([nodenum]/cnode/sid/sess_srno/session/ospid/state/[adjlist]): </li></ul><ul><li>*** 2008-08-16 02:50:45.918 </li></ul><ul><li>=============================================================================== </li></ul><ul><li>END OF HANG ANALYSIS </li></ul><ul><li>=============================================================================== </li></ul>
  52. 52. Dump Analysis for Oracle 11g Memory <ul><li>Use Oradebug to dump contents of SGA and PGA memory </li></ul><ul><li>Useful for debugging memory leak issues </li></ul><ul><li>Dump Analysis for Oracle 11g Memory </li></ul>
  53. 53. Dumping the SGA with Oradebug for Oracle 11g <ul><li>Use the oradebug dumpsga command: </li></ul><ul><li>SQL> oradebug setmypid </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug unlimit </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug dumpsga </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug tracefile_name </li></ul><ul><li>/u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora_21292.trc </li></ul><ul><li>SQL> exit </li></ul>
  54. 54. Viewing SGA memory dump for Oracle 11g <ul><li>Contents of SGA dump in oradebug trace file: </li></ul><ul><li>----- Fixed Areas Dump (level=2) ----- </li></ul><ul><li>----- Dump of the Fixed SGA ----- </li></ul><ul><li>ksmsgft ksmsgf_ [20000000, 20001000) = 00000000 00000000 00000000 00000000 ... </li></ul><ul><li>Dump of memory from 0x20000010 to 0x20001000 </li></ul><ul><li>20000010 00000000 00000000 00000000 00000000 [................] </li></ul><ul><li>Repeat 254 times </li></ul><ul><li>kywmr * kywmrsga_ [20001000, 20001004) = 37834020 </li></ul><ul><li>ksllt kywmll_ [20001004, 20001068) = 00000000 00000000 00000000 00000000 ... </li></ul><ul><li>Dump of memory from 0x20001014 to 0x20001068 </li></ul><ul><li>20001010 00000000 00000000 00000000 [............] </li></ul><ul><li>20001020 00000000 00000000 00000000 00000000 [................] </li></ul><ul><li>Repeat 1 times </li></ul><ul><li>20001040 20001040 20001040 20001048 20001048 [@.. @.. H.. H.. ] </li></ul><ul><li>20001050 00000000 00000000 00000000 00000000 [................] </li></ul><ul><li>20001060 00000000 00000000 [........] </li></ul><ul><li>ub4 kywmpleq_ [20001068, 2000106C) = 00000001 </li></ul><ul><li>ub4 kywmpleq1_ [2000106C, 20001070) = 00000001 </li></ul>
  55. 55. Suspend and Resume Runaway Oracle Processes <ul><li>Oradebug useful for stopping and restarting Oracle sessions </li></ul><ul><li>Be careful when using oradebug in Production Environments! </li></ul><ul><li>Syntax for Suspend Process with Oradebug: </li></ul><ul><li>oradebug suspend </li></ul><ul><li>To resume a process that has been suspended: </li></ul><ul><li>oradebug resume </li></ul><ul><li>Useful when need to delay batch process for more critical job. </li></ul>
  56. 56. Find the Session to Suspend <ul><li>Find the session information to suspend: </li></ul><ul><li>SQL> select s.username, s.sid, p.spid </li></ul><ul><li>2 from </li></ul><ul><li>3 v$session s, v$process p </li></ul><ul><li>4 where s.paddr=p.addr </li></ul><ul><li>5 and </li></ul><ul><li>6 s.username<>'SYS'; </li></ul><ul><li>USERNAME SID SPID </li></ul><ul><li>------------------------------ ---------- ------------------------ </li></ul><ul><li>SH 140 18335 </li></ul>
  57. 57. Suspend the Session with Oradebug <ul><li>oradebug suspend: </li></ul><ul><li>SQL> oradebug setospid 18335 </li></ul><ul><li>Oracle pid: 26, Unix process pid: 18335, image: (TNS V1-V3) </li></ul><ul><li>SQL> oradebug suspend </li></ul><ul><li>Statement processed. </li></ul>
  58. 58. Resume Process for Oracle 11g with Oradebug <ul><li>oradebug resume </li></ul><ul><li>SQL> oradebug resume </li></ul><ul><li>Statement processed. </li></ul>
  59. 59. Using Oradebug for Oracle RAC events <ul><li>Oradebug use to trace issues for IPC with interconnect and RAC </li></ul><ul><li>Also possible to trace Clusterware processes for Oracle RAC with oradebug </li></ul><ul><li>Syntax for IPC memory dump with Oracle RAC: </li></ul><ul><li>oradebug ipc </li></ul>
  60. 60. Oracle RAC tracing with Oradebug for IPC <ul><li>Example: </li></ul><ul><li>SQL> oradebug setmypid </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug unlimit </li></ul><ul><li>Statement processed. </li></ul><ul><li>SQL> oradebug ipc </li></ul><ul><li>Information written to trace file. </li></ul><ul><li>SQL> oradebug tracefile_name </li></ul><ul><li>/u01/app/oracle/admin/RACDB/udump/racdb1_ora_6391.trc </li></ul><ul><li>SQL> exit </li></ul>
  61. 61. Oracle 11g RAC Trace with Oradebug <ul><li>Review trace file for interconnect trace with Oradebug for RAC: </li></ul><ul><li>SSKGXPT 0xcc75e9c flags SSKGXPT_READPENDING info for network 0 </li></ul><ul><li>socket no 7 IP UDP 2247 </li></ul><ul><li>sflags SSKGXPT_UP </li></ul><ul><li>info for network 1 </li></ul><ul><li>socket no 0 IP UDP 0 </li></ul><ul><li>sflags SSKGXPT_DOWN </li></ul><ul><li>active 0 actcnt 1 </li></ul><ul><li>context timestamp 0 </li></ul><ul><li>no ports </li></ul><ul><li>sconno accono ertt state seq# sent async sync rtrans acks </li></ul>
  62. 62. Hidden Oracle 11g Parameters <ul><li>Oracle 11g has many new additional hidden parameters with many of the new additions! </li></ul><ul><li>There are more than 1500 hidden parameters for Oracle 11g! </li></ul><ul><li>SQL> select count(*) from x$ksppi </li></ul><ul><li>2 where substr(ksppinm,1,1)='_'; </li></ul><ul><li>COUNT(*) </li></ul><ul><li>---------- </li></ul><ul><li>1631 </li></ul><ul><li>Query to find these hidden Oracle 11g parameters: </li></ul><ul><li>select ksppinm from x$ksppi </li></ul><ul><li>where substr(ksppinm,1,1)='_‘; </li></ul>
  63. 63. Some new hidden parameters for Oracle 11g <ul><li>Some hidden parameters for 11g new features: </li></ul><ul><li>Hang Manager Hidden Parameters for Oracle 11g: </li></ul><ul><li>_hm_analysis_output_disk </li></ul><ul><li>_hm_analysis_oradebug_node_dump_level </li></ul><ul><li>_hm_analysis_oradebug_sys_dump_level </li></ul><ul><li>_hm_verification_interval </li></ul><ul><li>_hm_log_incidents </li></ul><ul><li>Be careful when using these. </li></ul><ul><li>Only change in test and sandbox </li></ul><ul><li>Best to work with Oracle support since these are undocumented parameters! </li></ul>
  64. 64. Oracle 11g V$ and X$ tables for troubleshooting and tuning <ul><li>Each release of Oracle adds many new V$ and X$ performance view tables. </li></ul><ul><li>More than 700 different X$ tables in Oracle 11g! </li></ul><ul><li>There are over 450 different V$ dynamic performance views in Oracle 11g! </li></ul><ul><li>Query to find the V$ performance views for Oracle 11g: </li></ul><ul><li>select name, type </li></ul><ul><li>from v$fixed_table </li></ul><ul><li>where name like ‘V$%‘; </li></ul><ul><li>Query to find the list of X$ tables for Oracle 11g: </li></ul><ul><li>select name, type </li></ul><ul><li>from v$fixed_table </li></ul><ul><li>where name like 'X$%‘; </li></ul>
  65. 65. Useful V$ and X$ Tables for Oracle 11g <ul><li>Some interesting new V$ Dynamic Performance Views : </li></ul><ul><li>V$MEMORY_DYNAMIC_COMPONENTS </li></ul><ul><li>V$MEMORY_TARGET_ADVICE </li></ul><ul><li>V$ALERT_TYPES </li></ul><ul><li>V$IR_FAILURE </li></ul><ul><li>V$IR_REPAIR </li></ul><ul><li>V$IR_MANUAL_CHECKLIST </li></ul><ul><li>V$IR_FAILURE_SET </li></ul><ul><li>V$HM_CHECK </li></ul><ul><li>V$HM_CHECK_PARAM </li></ul><ul><li>V$HM_RUN </li></ul><ul><li>V$HM_FINDING </li></ul><ul><li>V$HM_RECOMMENDATION </li></ul><ul><li>V$HM_INFO </li></ul><ul><li>Perform a describe on each of these V$ dynamic performance views for detail information. </li></ul>
  66. 66. Some Useful X$ tables for Oracle 11g <ul><li>These are some useful new X$ tables for Oracle 11g: </li></ul><ul><li>X$DIAG_INFO </li></ul><ul><li>X$IR_WORKING_FAILURE_SET </li></ul><ul><li>X$IR_WORKING_REPAIR_SET </li></ul><ul><li>X$IR_REPAIR_OPTION </li></ul><ul><li>X$IR_MANUAL_OPTION </li></ul><ul><li>X$IR_REPAIR_STEP </li></ul>
  67. 67. Power tools for UNIX and Windows <ul><li>When Oracle tools are not enough, these OS tools are essential: </li></ul><ul><li>dbx (Solaris and Linux) </li></ul><ul><li>truss (Solaris) </li></ul><ul><li>strace (Linux) </li></ul><ul><li>orakill (Windows) </li></ul><ul><li>Useful References on Metalink for Core Debug Analysis: </li></ul><ul><li>Oracle Metalink Note:16534.1 “Debug Core Dumps / Core Files on UNIX” </li></ul><ul><li>Oracle Metalink Note:1007808.6 “HOW TO HANDLE CORE DUMPS ON UNIX” </li></ul>
  68. 68. DBX -Debugging Solaris and Linux <ul><li>DBX is trace debugger for Solaris and also available for free download from as part of Solaris Developer Studio. </li></ul><ul><li>Provides comprehensive debugger level commands for Solaris and Linux OS system calls and Oracle 11g instance and network level internal processes. </li></ul><ul><li>DBX is an advanced complex tool with complex syntax </li></ul><ul><li>Complete reference guide from SUN: </li></ul><ul><li>Sun Studio 12: Debugging a Program With dbx: </li></ul><ul><li> </li></ul>
  69. 69. Using dbx for Oracle 11g Troubleshooting <ul><li>DBX can be used to trace Oracle 11g processes </li></ul><ul><li>Getting help with dbx: </li></ul><ul><li>[oracle@suntest bin]$ pwd </li></ul><ul><li>/opt/sun/sunstudio12/bin </li></ul><ul><li>[oracle@suntest bin]$ ./dbx </li></ul><ul><li>For information about new features see `help changes' </li></ul><ul><li>(dbx) help </li></ul><ul><li>command summary </li></ul><ul><li>Execution and Tracing </li></ul><ul><li>cancel catch clear cont delete </li></ul><ul><li>fix fixed handler ignore intercept </li></ul><ul><li>next pop replay rerun restore </li></ul><ul><li>run runargs save status step </li></ul><ul><li>stop trace unintercept when whocatches </li></ul><ul><li>(dbx) </li></ul>
  70. 70. Using dbx for Oracle 11g <ul><li>Find the PID for Oracle process </li></ul><ul><li>[oracle@raclinux1 bin]$ ps -ef|grep oracle </li></ul><ul><li>oracle 19994 19993 0 00:31 ? 00:00:00 oracleORA11G (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) </li></ul><ul><li>oracle 19993 3834 0 00:31 pts/2 00:00:00 sqlplus </li></ul><ul><li>Now we need to get a trace for Oracle </li></ul><ul><li>oracle@raclinux1 bin]$ ./dbx $ORACLE_HOME/bin/oracle 19994 </li></ul><ul><li>Reading </li></ul><ul><li>Reading </li></ul><ul><li>Reading </li></ul><ul><li>Reading </li></ul><ul><li>Attached to process 19994 </li></ul><ul><li>t@3086931648 (l@19994) stopped in _dl_sysinfo_int80 at 0x90a7a2 </li></ul><ul><li>0x0090a7a2: _dl_sysinfo_int80+0x0002: ret </li></ul><ul><li>(dbx) </li></ul><ul><li>Enable Collection Process and Continue Trace </li></ul><ul><li>(dbx) collector enable </li></ul><ul><li>Back to our DBX trace session: </li></ul><ul><li>(dbx) cont </li></ul>
  71. 71. Continued- Tracing Oracle 11g with dbx <ul><li>Use the dbx thread commands to look at Oracle 11g memory process stack </li></ul><ul><li>Use the dbx thread commands </li></ul><ul><li>(dbx) thread t@3086931648 </li></ul><ul><li>t@3086931648 (l@24976) stopped in _dl_sysinfo_int80 at 0x90a7a2 </li></ul><ul><li>0x0090a7a2: _dl_sysinfo_int80+0x0002: ret </li></ul><ul><li>(dbx) where </li></ul><ul><li>current thread: t@3086931648 </li></ul><ul><li>=>[1] _dl_sysinfo_int80(0xf493504, 0xe5b652e, 0x9, 0xf5a684e, 0x2010, 0xf493504) , at 0x90a7a2 </li></ul><ul><li>(dbx) thread -info </li></ul><ul><li>Thread t@3086931648 (0x0) at priority 0 </li></ul><ul><li>state: active on l@3107 </li></ul><ul><li>base function: 0x0: 0x00000000() stack: 0x0[0] </li></ul><ul><li>flags: (none) </li></ul><ul><li>masked signals: (none) </li></ul><ul><li>Currently active in _dl_sysinfo_int80 </li></ul>
  72. 72. Debug Oracle 11g with dbx <ul><li>The debug function with dbx is useful for Oracle 11g: </li></ul><ul><li>(dbx) debug -f </li></ul><ul><li>Debugging: oracle (process id 24976) </li></ul>
  73. 73. Truss: Solaris power tool for Tracing Oracle 11g <ul><li>Truss provides ability to walk through trace at step by step level for Oracle 11g internal processes. </li></ul><ul><li>Syntax: </li></ul><ul><ul><li>truss [-fcaeil] [-[tvx] [!] syscall... ] [-s [!] signal... ] [-m [!] fault... ] [-[rw] [!] fd... ] [-o outfile ] command | -p pid </li></ul></ul><ul><li>Typical options for truss: </li></ul><ul><li>-o output to file </li></ul><ul><li>-f trace child processes </li></ul><ul><li>-c count system calls </li></ul><ul><li>-p trace calls based on Unix PID </li></ul>
  74. 74. How to Use Truss with Solaris for Oracle 11g <ul><li>Example: </li></ul><ul><li>solaris02$ truss -cp 3316 </li></ul><ul><li>^C </li></ul><ul><li>Syscall seconds calls errors </li></ul><ul><li>read .000 2 </li></ul><ul><li>Write .000 2 </li></ul><ul><li>times .000 33 </li></ul><ul><li>yield .000 226 </li></ul>
  75. 75. Strace tool for Linux and Oracle 11g tracing <ul><li>Strace is similar to truss for Linux O/S tracing with Oracle 11g </li></ul><ul><li>Powerful tool with command line syntax: </li></ul><ul><li>usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file] </li></ul><ul><li>[-p pid] ... [-s strsize] [-u username] [-E var=val] ... </li></ul><ul><li>[command [arg ...]] </li></ul><ul><li>or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ... </li></ul><ul><li>[command [arg ...]] </li></ul>
  76. 76. How to Use Strace with Linux and Oracle 11g <ul><li>Options available for strace: </li></ul><ul><li>[oracle@raclinux1 ~]$ strace </li></ul><ul><li>-c -- count time, calls, and errors for each syscall and report summary </li></ul><ul><li>-f -- follow forks, -ff -- with output into separate files </li></ul><ul><li>-F -- attempt to follow vforks, -h -- print help message </li></ul><ul><li>-i -- print instruction pointer at time of syscall </li></ul><ul><li>-q -- suppress messages about attaching, detaching, etc. </li></ul><ul><li>-r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs </li></ul><ul><li>-T -- print time spent in each syscall, -V -- print version </li></ul><ul><li>-v -- verbose mode: print unabbreviated argv, stat, termio[s], etc. args </li></ul><ul><li>-x -- print non-ascii strings in hex, -xx -- print all strings in hex </li></ul><ul><li>-a column -- alignment COLUMN for printing syscall results (default 40) </li></ul><ul><li>-e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]... </li></ul><ul><li>options: trace, abbrev, verbose, raw, signal, read, or write </li></ul><ul><li>-o file -- send trace output to FILE instead of stderr </li></ul><ul><li>-O overhead -- set overhead for tracing syscalls to OVERHEAD usecs </li></ul><ul><li>-p pid -- trace process with process id PID, may be repeated </li></ul><ul><li>-s strsize -- limit length of print strings to STRSIZE chars (default 32) </li></ul><ul><li>-S sortby -- sort syscall counts by: time, calls, name, nothing (default time) </li></ul><ul><li>-u username -- run command as username handling setuid and/or setgid </li></ul><ul><li>-E var=val -- put var=val in the environment for command </li></ul><ul><li>-E var -- remove var from the environment for command </li></ul>
  77. 77. Strace for Oracle 11g: Worked Example <ul><li>Perform system call trace for Oracle 11g: </li></ul><ul><li>[oracle@raclinux1 ~]$ strace oracle </li></ul><ul><li>execve(&quot;/u01/app/oracle/product/11.1.0/11g/bin/oracle&quot;, [&quot;oracle&quot;], [/* 41 vars */]) = 0 </li></ul><ul><li>uname({sys=&quot;Linux&quot;, node=&quot;;, ...}) = 0 </li></ul><ul><li>brk(0) = 0xf56b000 </li></ul>
  78. 78. Tracing System Calls with STrace for Oracle 11g <ul><li>Similar to truss: </li></ul><ul><li>Check for System Call Errors for Oracle 11g with Strace: </li></ul><ul><li>[oracle@raclinux1 bin]$ strace -c oracle </li></ul><ul><li>% time seconds usecs/call calls errors syscall </li></ul><ul><li>------ ----------- ----------- --------- --------- ---------------- </li></ul><ul><li>4.19 0.865232 41202 21 write </li></ul><ul><li>4.02 0.831352 207838 4 lstat64 </li></ul><ul><li>4.02 0.831329 207832 4 _llseek </li></ul><ul><li>4.02 0.831327 207832 4 times </li></ul><ul><li>3.42 0.707903 707903 1 io_setup </li></ul><ul><li>3.42 0.707867 707867 1 _sysctl </li></ul><ul><li>3.42 0.707864 707864 1 socket </li></ul><ul><li>3.42 0.707843 707843 1 bind </li></ul><ul><li>3.42 0.707839 707839 1 set_thread_area </li></ul><ul><li>3.42 0.707837 707837 1 futex </li></ul><ul><li>3.42 0.707836 707836 1 dup </li></ul><ul><li>3.42 0.707836 707836 1 set_tid_address </li></ul><ul><li>3.42 0.707835 707835 1 gettid </li></ul><ul><li>3.42 0.707833 707833 1 fcntl64 </li></ul><ul><li>3.42 0.707831 707831 1 getuid32 </li></ul><ul><li>1.52 -1.1313491 2986 105 munmap </li></ul><ul><li>------ ----------- ----------- --------- --------- ---------------- </li></ul><ul><li>100.00 20.669398 1134 108 total </li></ul><ul><li>*Note: shortened for clarity </li></ul>
  79. 79. Tracing System Calls for Oracle 11g DBWR <ul><li>Example: Using STrace to find System calls for Oracle 11g DBWR process </li></ul><ul><li>[oracle@raclinux1 ~]$ strace -c -p 3873 </li></ul><ul><li>Process 3873 attached - interrupt to quit </li></ul><ul><li>Process 3873 detached </li></ul><ul><li>% time seconds usecs/call calls errors syscall </li></ul><ul><li>------ ----------- ----------- --------- --------- ---------------- </li></ul><ul><li>82.95 0.003289 12 282 gettimeofday </li></ul><ul><li>7.92 0.000314 8 40 times </li></ul><ul><li>5.42 0.000215 8 26 getrusage </li></ul><ul><li>1.99 0.000079 6 13 13 semtimedop </li></ul><ul><li>0.76 0.000030 30 1 open </li></ul><ul><li>0.68 0.000027 27 1 read </li></ul><ul><li>0.28 0.000011 11 1 close </li></ul><ul><li>------ ----------- ----------- --------- --------- ---------------- </li></ul><ul><li>100.00 0.003965 364 13 total </li></ul>
  80. 80. Using orakill to trace Windows and Oracle 11g <ul><li>Useful tool available for Windows and Oracle 11g called orakill </li></ul><ul><li>Oracle Metalink Notes: 69882.1 </li></ul><ul><li>Useful for killing runaway processes on Windows and Oracle 11g as well as for zombie and hung sessions </li></ul><ul><li>Syntax: </li></ul><ul><li>orakill <sid> <thread> </li></ul><ul><li>Can use QSLICE.EXE to find threads (available on Windows NT Resource Kit from Microsoft) </li></ul>
  81. 81. How to Use Orakill for Windows and Oracle 11g <ul><li>Query if not using QSLICE.EXE for details: </li></ul><ul><li>C:>orakill </li></ul><ul><li>Usage: orakill sid thread </li></ul><ul><li>where sid = the Oracle instance to target </li></ul><ul><li>thread = the thread id of the thread to kill </li></ul><ul><li>The thread id should be retrieved from the spid column of a query such as: </li></ul><ul><li>select spid, osuser, s.program from </li></ul><ul><li>v$process p, v$session s where p.addr=s.paddr </li></ul>
  82. 82. Example: Using Orakill for Windows and Oracle 11g <ul><li>Use QSlice.exe to find the Oracle process: </li></ul><ul><li>Then use orakill to end process that is hung. </li></ul><ul><li>C:>orakill test 958 </li></ul><ul><li>Kill of thread id </li></ul><ul><li>958 in instance test successfully </li></ul><ul><li>signalled. </li></ul>
  83. 83. ORA-0600: Tips and Tricks <ul><li>ORA-0600: Internal errors </li></ul><ul><li>Call Oracle support first </li></ul><ul><li>Leverage Metalink ORA-0600 support matrix </li></ul><ul><li>Collect dumps with oradebug if possible </li></ul>
  84. 84. What is an ORA-0600 Error? <ul><li>Internal Oracle error results in kernel memory dump </li></ul><ul><li>Dumps core file and trace files </li></ul><ul><li>Useful Metalink Notes: 146580.1 , 146581.1 </li></ul><ul><li>Use the Oracle Metalink ORA-0600 Lookup Tool </li></ul><ul><li>for addressing these internal errors </li></ul><ul><li>Note 153788.1 - Troubleshoot an ORA-600 Error using the ORA-600 Argument Lookup Tool </li></ul>
  85. 85. Using the ORA-0600 Lookup Tool <ul><li>From Oracle Support- see Note 153788.1 </li></ul><ul><li>Troubleshoot an ORA-600 Error using the ORA-600 Argument Lookup Tool </li></ul><ul><li>Also can be used for review and analyze ORA-7445 errors </li></ul>
  86. 86. Resolving ORA-0600 Errors <ul><li>Work closely with Oracle Support </li></ul><ul><li>Usually caused by bug such as memory leak </li></ul><ul><li>Test recent changes between development, test, QA and Production to ensure continuity of issue. </li></ul><ul><li>Ideal to have same release of Oracle to be able to reproduce the error for all environments and platforms. </li></ul>
  87. 87. ORA-0600 Troubleshooting <ul><li>Use ADR to view incident and associated trace file: </li></ul><ul><li>Dump continued from file: /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora_12562.trc </li></ul><ul><li>ORA-00600: internal error code, arguments: [kssadpm: null parent], [], [], [], [], [], [], [] </li></ul><ul><li>Plug in the arguments from trace file to ORA-0600 Metalink troubleshooting lookup tool. </li></ul><ul><li>The related solution will then be displayed: </li></ul><ul><li>Known Issues: Bug# 5605370 </li></ul><ul><li>* See Note 5605370.8 * This bug is alerted in Note 454464.1 Various dumps / instance crash possible Fixed:, </li></ul>
  88. 88. Preguntas? <ul><li>Hay algunas preguntas? </li></ul><ul><li>I’ll also be available in the Oracle ACE lodge </li></ul>
  89. 89. Conclusion <ul><li>Muchas gracias! </li></ul><ul><li>Please complete your evaluation form </li></ul><ul><li>Ben Prusinski </li></ul><ul><li>Undocumented Oracle DBA Utilities: Black Magic for the Oracle Expert </li></ul><ul><li>My company- Ben Prusinski and Associates </li></ul><ul><li> </li></ul><ul><li>Oracle Blog </li></ul><ul><li> </li></ul>