• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Sql Performance Tuning with ASH & AWR: Real World Use Cases
 

Sql Performance Tuning with ASH & AWR: Real World Use Cases

on

  • 2,804 views

Rapid SQL Performance Tuning with ASH and AWR without the limitations of a GUI.

Rapid SQL Performance Tuning with ASH and AWR without the limitations of a GUI.

Statistics

Views

Total Views
2,804
Views on SlideShare
2,794
Embed Views
10

Actions

Likes
4
Downloads
0
Comments
0

2 Embeds 10

http://www.linkedin.com 8
https://www.linkedin.com 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • How many of you know how to write SQL?How many of you know the difference between a Physical I/O and a Logical I/O?
  • Collects database performance statistics stored in memory and in database used by ADDM and Advisors (SQL Tuning, Segment, Undo)Show expensive SQL and its elapsed and CPU times…point that remaining is waitsShow that same SQLs others statsPoint that it could be IO and show wait events sectionAlso show ‘segment statistics’ section for objects of this SQLMention that AWR does NOT give wait or segment statistics by SQL or sessionTime to gather more data specific to our SQL or Session…Select * from dict where table_name like ‘DBA_HIST%SQL%’;TABLE_NAME COMMENTS------------------------------ --------------------------------------------------------------------------------DBA_HIST_COLORED_SQL Marked SQLs for snapshotsDBA_HIST_SQLBIND SQL Bind InformationDBA_HIST_SQLCOMMAND_NAME Sql command typesDBA_HIST_SQLSTAT SQL Historical Statistics InformationDBA_HIST_SQLTEXT SQL TextDBA_HIST_SQL_BIND_METADATA SQL Bind Metadata InformationDBA_HIST_SQL_PLAN SQL Plan InformationDBA_HIST_SQL_SUMMARY Summary of SQL StatisticsDBA_HIST_SQL_WORKAREA_HSTGRM SQL Workarea Histogram Historyselect distinct stat_name from V$SYS_TIME_MODEL ;STAT_NAME----------------------------------------------------------------DB CPUbackground elapsed timehard parse elapsed timeRMAN cpu time (backup/restore)DB timehard parse (bind mismatch) elapsed timeinbound PL/SQL rpc elapsed timerepeated bind elapsed timebackground cpu timeconnection management call elapsed timefailed parse elapsed timehard parse (sharing criteria) elapsed timePL/SQL compilation elapsed timefailed parse (out of shared memory) elapsed timeJava execution elapsed timeparse time elapsedsequence load elapsed timesql execute elapsed timePL/SQL execution elapsed time19 rows selected.
  • 5min max
  • 11g has SQL_PLAN_LINE_ID, SQL_PLAN_OPERATION to further drilldown an SQL performance issue
  • Collects database performance statistics stored in memory and in database used by ADDM and Advisors (SQL Tuning, Segment, Undo)Show expensive SQL and its elapsed and CPU times…point that remaining is waitsShow that same SQLs others statsPoint that it could be IO and show wait events sectionAlso show ‘segment statistics’ section for objects of this SQLMention that AWR does NOT give wait or segment statistics by SQL or sessionTime to gather more data specific to our SQL or Session…Select * from dict where table_name like ‘DBA_HIST%SQL%’;TABLE_NAME COMMENTS------------------------------ --------------------------------------------------------------------------------DBA_HIST_COLORED_SQL Marked SQLs for snapshotsDBA_HIST_SQLBIND SQL Bind InformationDBA_HIST_SQLCOMMAND_NAME Sql command typesDBA_HIST_SQLSTAT SQL Historical Statistics InformationDBA_HIST_SQLTEXT SQL TextDBA_HIST_SQL_BIND_METADATA SQL Bind Metadata InformationDBA_HIST_SQL_PLAN SQL Plan InformationDBA_HIST_SQL_SUMMARY Summary of SQL StatisticsDBA_HIST_SQL_WORKAREA_HSTGRM SQL Workarea Histogram Historyselect distinct stat_name from V$SYS_TIME_MODEL ;STAT_NAME----------------------------------------------------------------DB CPUbackground elapsed timehard parse elapsed timeRMAN cpu time (backup/restore)DB timehard parse (bind mismatch) elapsed timeinbound PL/SQL rpc elapsed timerepeated bind elapsed timebackground cpu timeconnection management call elapsed timefailed parse elapsed timehard parse (sharing criteria) elapsed timePL/SQL compilation elapsed timefailed parse (out of shared memory) elapsed timeJava execution elapsed timeparse time elapsedsequence load elapsed timesql execute elapsed timePL/SQL execution elapsed time19 rows selected.
  • OEM performance pageAvailability – tools repository is down or the laptop which has the tool installed crashed
  • OEM performance pageAvailability – tools repository is down or the laptop which has the tool installed crashed
  • better explanation for using begin_interval_time one snapshot before
  • 11g has SQL_PLAN_LINE_ID, SQL_PLAN_OPERATION to further drilldown an SQL performance issue
  • PIO - number of times the db went to disk to retrieve the dataPotential solutions:Increase memory- upgrade disk susbsytemredudce PIO
  • Standard approach pre-10g - trace
  • 92% in just 3 steps
  • We also see that ASH data confirms AWR that I/O is the bottleneck (99% AWR vs 98% ASH)
  • CBO trace (event 10053) – lots of work, option of last resortDo we have bad stats - E-Rows vs A-Rows (statistics_level=all,gather_plan_statistics, SQL trace (event 10046)Usually bad stats =>CBO underestimates the cardinality/timeIn our case it overestimates.optimizer settings => CBO trace
  • PIO - number of times the db went to disk to retrieve the data
  • optimizer_index_cost_adj = 10 (default is 100). index access cost = 10% of the normal cost.
  • CBO trace (event 10053) – lots of work, option of last resortDo we have bad stats - E-Rows vs A-Rows (statistics_level=all,gather_plan_statistics, SQL trace (event 10046)Usually bad stats =>CBO underestimates the cardinality/timeIn our case it overestimates.optimizer settings => CBO trace
  • Not spilling to temp because no column indicating it
  • 99% on IO, but static PIO and large variance in time => large variance in PIO latencyDisk subsytem problem?Contact SAs? Check if query needs that much PIO first.
  • Plan not in memory anymoreExplain plan_line_id =2 when display_awr shows only 0 & 1. I will explain it later.
  • Plan not in memory anymoreExplain plan_line_id =2 when display_awr shows only 0 & 1. I will explain it later.
  • Historical bind variable not available through OEM (at least 10g)
  • Display_awr does not show the load table step, but display_cursor does.Plan hash values are the same!

Sql Performance Tuning with ASH & AWR: Real World Use Cases Sql Performance Tuning with ASH & AWR: Real World Use Cases Presentation Transcript

  • 1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. All the views expressed in this presentation are the opinion of the authors and do not necessarily reflect the views of Oracle. The queries are provided "as is", without warranty of any kind, expressed or implied.2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • SQL Performance Tuning with Oracle AWR & ASH Real World Use Cases3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • About myself • Vlado Barun, M.Sc. – Sr. Principal Performance Engineer with Oracle Cloud Services – OCP, OCE SQL, MCP – 15 years in the Database Arena (Oracle DB - 10 years)4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Objective • Enable you to start exploring the rich repository of performance metrics in ASH & AWR independent of any specific GUI and related limitations • Provide list of resources that enable you to “hit the ground running”5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Agenda • Introduction to ASH & AWR (very brief) • Review 2 real world production use cases – step-by-step from data gathering to resolution – write ASH/AWR queries from scratch – use pre-existing free scripts – Note: check license requirements • Resources6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Automatic Workload Repository (AWR)• Collects database performance statistics – Objects (access and usage statistics) – SQL Statement statistics – Wait events statistics – System statistics – ASH Statistics (DBA_HIST_ACTIVE_SESS_HISTORY) – Time Model Statistics based on time usage for activities – Every 1 hr by default• Stored in DBA_HIST% views• How to access? – $ORACLE_HOME/rdbms/admin/awrrpt.sql, SQLT, custom scripts,… – OEM, SQL Developer – Other tools7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Active Session History (ASH) Introduction v$active_session_history dba_hist_active_sess_history (1 second intervals) (10 second intervals) 3 ASH Samples in AWR Browse Read Reviews Add to Cart Checkout Books For One Book 21 ASH Samples TIME SQL Trace (Microsecond level) Application Logging (custom level) = time spent in database8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH - Dimensions When Row Source Object SAMPLE_ID SQL_PLAN_LINE_ID CURRENT_OBJ# SAMPLE_TIME SQL_PLAN_OPERATION CURRENT_FILE# CURRENT_BLOCK# CURRENT_ROW# Session Waits SESSION_ID EVENT SESSION_SERIAL# EVENT_ID Application SESSION_TYPE EVENT# SERVICE_HASH SESSION_STATE SEQ# PROGRAM USER_ID P1TEXT MODULE QC_INSTANCE_ID P1 ACTION QC_SESSION_ID P2TEXT MACHINE QC_SESSION_SERIAL# P2 CLIENT_ID P3TEXT IN and IS columns P3 WAIT_CLASS Statistics WAIT_CLASS_ID TM_DELTA_TIME SQL WAIT_TIME TM_DELTA_CPU_TIME SQL_ID TIME_WAITED TM_DELTA_DB_TIME IS_SQLID_CURRENT CURRENT_OBJ# DELTA_TIME SQL_CHILD_NUMBER CURRENT_FILE# DELTA_READ_IO_REQUESTS SQL_OPCODE CURRENT_BLOCK# DELTA_WRITE_IO_REQUESTS SQL_OPNAME CURRENT_ROW# DELTA_READ_IO_BYTES TOP_LEVEL_SQL_ID DELTA_WRITE_IO_BYTES TOP_LEVEL_SQL_OPCODE DELTA_INTERCONNECT_IO_BYTES SQL_EXEC_ID Blocking Session PGA_ALLOCATED SQL_EXEC_START BLOCKING_SESSION_STATUS TEMP_SPACE_ALLOCATED SQL_PLAN_HASH_VALUE BLOCKING_SESSION SQL_PLAN_OPTIONS BLOCKING_SESSION_SERIAL# BLOCKING_INST_ID BLOCKING_HANGCHAIN_INFO9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH - Summary • Samples the current state of all active session – 1 second samples – Stores various attributes (dimensions) of a session • Stored in – V$ACTIVE_SESSION_HISTORY • 1-second sampling rate – DBA_HIST_ACTIVE_SESS_HISTORY - Sub-sampling to disk into AWR Snapshot - 1-in-10 samples • How to access? - $ORACLE_HOME/rdbms/admin/ashrpt.sql, ashrpti.sql, SQLT, custom scripts,… - OEM, SQL Developer, Real Time SQL Monitoring, … - Other tools10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • GUI or command line• It is about knowing the right tool for the right situation• Understand the strengths and weaknesses of each• GUI tools strengths => cmd line weaknesses – Easy to use – Good visualization of data – Provides concise summary of system activity Performance – etc… data• GUI tools weaknesses=> cmd line strength Performance data – Slow - click…wait…click…wait… exposed by GUI – Availability – GUI infrastructure is down – Limited exposure – subset of available data – etc… 11 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH dimensions growthSeehttps://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history 12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Understand the issue What Identify (app is slow) Initial Data How Much Quantify collection (takes 1m) AWR/ASH, Trace, Apps Logs/Tables, When Timeline OS/IO stats etc (during peak hours) Where Component (Web Tier, App Tier, DB , Network, Storage, Analyze etc.) How Flow chart Additional Data SQL/Object Stats, Collection and Root Cause Execution Plans, Why (Stats not updated) etc Analysis13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 1What Slow ETLHow Much 20+ hoursWhen Once a month Informatica + DBWhere Extract, Transform, LoadHow14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Find slow SQL (ASH)SQL> SELECT sql_id 2 , COUNT(*) as sample_count 3 FROM v$active_session_history 4 WHERE sample_time BETWEEN TO_DATE(2011-06-29 07:57, yyyy-mm-dd hh24:mi) 5 AND TO_DATE(2011-06-30 04:30, yyyy-mm-dd hh24:mi) 6 AND user_id = 61 7 AND program like pmdtm@% 8 GROUP BY sql_id 9 ORDER BY 2 DESC 10 ;no rows selected SQL> SELECT min(sample_time) 2 FROM v$active_session_history 3 ; MIN(SAMPLE_TIME) ---------------------------------------- 2011-07-01 19:14:48 1 row selected. 15 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Find slow SQL (ASH in AWR) SELECT sql_id , COUNT(*) as sample_count FROM dba_hist_active_sess_history v$active_session_history JOIN dba_hist_snapshot USING (dbid, instance_number, snap_id) WHERE sample_time BETWEEN TO_DATE(2011-06-29 07:57, yyyy-mm-dd hh24:mi) AND TO_DATE(2011-06-30 04:30, yyyy-mm-dd hh24:mi) AND user_id = 61 AND program like pmdtm@%‘GROUP AND BY begin_interval_time BETWEEN TO_DATE(2011-06-29 07:00,yyyy-mm-dd hh24:mi) sql_idORDER BY 2 DESC AND TO_DATE(2011-06-30 05:00, yyyy-mm-dd hh24:mi) ; SQL_ID SAMPLE_COUNT ------------- ------------ dkwucgaxxhm2z 7362 grksgbxxuacaf 6041 … 24 rows selected. 16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Find slow SQL (ASH in AWR) - readableSQL>SELECT sql_id, sql_type 2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct 3 , round(sample_cnt_sql /6/60,1) hours 4 FROM(SELECT DISTINCT sql_id 5 , aa.name AS sql_type 6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql 7 , COUNT(*) over () AS sample_cnt_total 8 FROM dba_hist_active_sess_history ash 9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id) 10 JOIN audit_actions aa on (ash.sql_opcode = aa.action) 11 WHERE begin_interval_time BETWEEN… snip … 18 ORDER BY 3 desc) 19 WHERE rownum <= 3;SQL_ID SQL_TYPE SAMPLE_PCT HOURS------------- -------- ---------- -----dkwucgaxxhm2z SELECT 25 20.4grksgbxxuacaf SELECT 21 16.8gh6w2fm3avn3j SELECT 12 10.13 rows selected. 17 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Active Session History (ASH) - Dimensions When Row Source Object SAMPLE_ID SQL_PLAN_LINE_ID CURRENT_OBJ# SAMPLE_TIME SQL_PLAN_OPERATION CURRENT_FILE# CURRENT_BLOCK# CURRENT_ROW# Session Waits SESSION_ID EVENT SESSION_SERIAL# EVENT_ID Application SESSION_TYPE EVENT# SERVICE_HASH SESSION_STATE SEQ# PROGRAM USER_ID P1TEXT MODULE QC_INSTANCE_ID P1 ACTION QC_SESSION_ID P2TEXT MACHINE QC_SESSION_SERIAL# P2 CLIENT_ID P3TEXT IN and IS columns P3 WAIT_CLASS Statistics WAIT_CLASS_ID TM_DELTA_TIME SQL WAIT_TIME TM_DELTA_CPU_TIME SQL_ID TIME_WAITED TM_DELTA_DB_TIME IS_SQLID_CURRENT CURRENT_OBJ# DELTA_TIME SQL_CHILD_NUMBER CURRENT_FILE# DELTA_READ_IO_REQUESTS SQL_OPCODE CURRENT_BLOCK# DELTA_WRITE_IO_REQUESTS SQL_OPNAME CURRENT_ROW# DELTA_READ_IO_BYTES TOP_LEVEL_SQL_ID DELTA_WRITE_IO_BYTES TOP_LEVEL_SQL_OPCODE DELTA_INTERCONNECT_IO_BYTES SQL_EXEC_ID Blocking Session PGA_ALLOCATED SQL_EXEC_START BLOCKING_SESSION_STATUS TEMP_SPACE_ALLOCATED SQL_PLAN_HASH_VALUE BLOCKING_SESSION SQL_PLAN_OPTIONS BLOCKING_SESSION_SERIAL# BLOCKING_INST_ID BLOCKING_HANGCHAIN_INFO18 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL statsSQL> SELECT PLAN_HASH_VALUE, 2 , sum(EXECUTIONS_DELTA) as executions 3 , sum(ROWS_PROCESSED_DELTA) as rows_processed 4 , sum(ELAPSED_TIME_DELTA) as elapsed_time 5 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time 6 , sum(DIRECT_WRITES_DELTA) as direct_writes 7 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO 8 FROM dba_hist_sqlstat 9 WHERE snap_id between 132042 and 132124 10 AND sql_id = dkwucgaxxhm2z 11 GROUP BY sql_id, PLAN_HASH_VALUE, OPTIMIZER_ENV_HASH_VALUE; Plan IO Hash Rows Elapsed CPU WAIT Direct Value EXECUTIONS Processed Time TIME TIME Writes LIO PIO ---------- ---------- ---------- ---------- ---------- ----------- -------- -------- ------- 1700726234 1 5997889 7.3806E+10 903837577 73047955458 219418 36152215 6427679 1 row selected. 19 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL stats - readableSQL> SELECT PLAN_HASH_VALUE, executions 2 , round(elapsed_time /1000000/60/60,1) as elapsed_hours 3 , rows_processed 4 , round(cpu_time /elapsed_time * 100) as cpu_time_pct 5 , round(iowait_time /elapsed_time * 100) as iowait_pct 6 , LIO, PIO, direct_writes 7 , round(iowait_time /PIO/1000,1) as mili_sec_PIO 8 FROM(SELECT PLAN_HASH_VALUE 9 , sum(EXECUTIONS_DELTA) as executions 10 , sum(ROWS_PROCESSED_DELTA) as rows_processed 11 , sum(ELAPSED_TIME_DELTA) as elapsed_time 12 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time 13 , sum(DIRECT_WRITES_DELTA) as direct_writes 14 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO 15 FROM dba_hist_sqlstat 16 WHERE snap_id between 132042 and 132124 17 AND sql_id = dkwucgaxxhm2z 18 GROUP BY sql_id, PLAN_HASH_VALUE); Plan CPU IO Milisec Hash Elapsed Rows TIME WAIT Direct / Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes Pio---------- ---------- -------- ---------- ---- ----- ---------- ---------- -------- -------1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4 20 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL planSQL>select * from table(dbms_xplan.display_awr(dkwucgaxxhm2z, 1700726234, null, ALL LAST)); ---------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | TempSpc| Cost (%CPU)| Time | ---------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 9553K(100)| | | 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 | | 2 | NESTED LOOPS OUTER | | 8011K| 45G| | 7148K (1)| 23:49:40 | | 3 | NESTED LOOPS OUTER | | 8011K| 45G| | 4743K (1)| 15:48:44 | | 4 | NESTED LOOPS OUTER | | 8011K| 45G| | 3941K (1)| 13:08:24 | | 5 | NESTED LOOPS OUTER | | 8011K| 44G| | 3140K (1)| 10:28:03 | | 6 | HASH JOIN RIGHT OUTER | | 8011K| 44G| | 2338K (1)| 07:47:43 | | 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 | | 8 | HASH JOIN | | | | | | | | 9 | HASH JOIN | | | | | | | | 10 | INDEX FAST FULL SCAN | S_SRC_M10 | 6633 | 285K| | 10 (0)| 00:00:01 | | 11 | INDEX FAST FULL SCAN | S_SRC_M12 | 6633 | 285K| | 10 (0)| 00:00:01 | | 12 | INDEX FAST FULL SCAN | S_SRC_P1 | 6633 | 285K| | 8 (0)| 00:00:01 | | 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| 228M| 2338K (1)| 07:47:42 | | 14 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 7488K| 142M| | 38163 (2)| 00:07:38 | | 15 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51545 (1)| 00:10:19 | | 16 | TABLE ACCESS FULL | S_INDUST_LANG | 4023 | 113K| | 199 (0)| 00:00:03 | | 17 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51289 (1)| 00:10:16 | | 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8 (0)| 00:00:01 | | 19 | NESTED LOOPS OUTER | | 8011K| 42G| | 51224 (1)| 00:10:15 | | 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 | | 21 | NESTED LOOPS | | | | | | | | 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 | | 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | 1 (0)| 00:00:01 | | 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35 (3)| 00:00:01 | | 25 | TABLE ACCESS BY INDEX ROWID| S_ORG_EXT | 421K| 419M| | 2693 (1)| 00:00:33 | | 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | 1 (0)| 00:00:01 | | 27 | INDEX UNIQUE SCAN | S_OU_PRTNR_TYPE_P1 | 1 | | | 1 (0)| 00:00:01 | | 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 | | 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 | | 30 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 | | 31 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 | | 32 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 | | 33 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 | | 34 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 | | 35 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 | | 36 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 | | 37 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 | ---------------------------------------------------------------------------------------------------------------------- … 21 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Find slow SQL (ASH in AWR)SQL>SELECT sql_id, sql_type 2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct 3 , round(sample_cnt_sql /6/60,1) hours 4 FROM(SELECT DISTINCT sql_id 5 , aa.name AS sql_type 6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql 7 , COUNT(*) over () AS sample_cnt_total 8 FROM dba_hist_active_sess_history ash 9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id) 10 JOIN audit_actions aa on (ash.sql_opcode = aa.action) 11 WHERE begin_interval_time BETWEEN… snip … 18 ORDER BY 3 desc) 19 WHERE rownum <= 3;SQL_ID SQL_TYPE SAMPLE_PCT HOURS------------- -------- ---------- -----dkwucgaxxhm2z SELECT 25 20.4grksgbxxuacaf SELECT 21 16.8gh6w2fm3avn3j SELECT 12 10.13 rows selected. 22 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH rowsource data (plan line id)SQL>SELECT sql_id, sql_plan_line_id 2 , round(sample_cnt_plan_line/sample_cnt_total*100) AS sql_plan_line_pct 3 , round(sample_cnt_plan_line/6) minutes 4 FROM( 5 SELECT DISTINCT sql_id, sql_plan_line_id 6 , COUNT(*) over (PARTITION BY sql_plan_line_id) AS sample_cnt_plan_line 7 , COUNT(*) over () AS sample_cnt_total 8 FROM dba_hist_active_sess_history ash 9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id) 10 WHERE sql_id = dkwucgaxxhm2z‘ and begin_interval_time BETWEEN… snip …SQL_ID SQL_PLAN_LINE_ID SQL_PLAN_LINE_PCT MINUTES------------- ---------------- ----------------- ----------dkwucgaxxhm2z 34 48 593dkwucgaxxhm2z 35 27 331dkwucgaxxhm2z 25 17 213… 23 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH rowsource (plan line id + event)SQL>SELECT sql_id, sql_plan_line_id, event 2 , round(sample_cnt_plan_line_event/sample_cnt_total*100) as plan_line_event_pct 3 , round(sample_cnt_plan_line_event/6) minutes 4 FROM (SELECT DISTINCT sql_id 5 , sql_plan_line_id, nvl(event, CPU) as event 6, COUNT(*) over (PARTITION BY sql_plan_line_id,event)AS sample_cnt_plan_line_event 7, COUNT(*) over () AS sample_cnt_total 8 FROM dba_hist_active_sess_history ash 9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id) 10 WHERE sql_id = dkwucgaxxhm2z and begin_interval_time BETWEEN …… snip …SQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES------------- ---------------- ------------------------- ------------------- ----------dkwucgaxxhm2z 34 db file sequential read 48 591dkwucgaxxhm2z 35 db file sequential read 27 328dkwucgaxxhm2z 25 db file sequential read 17 212… snip …22 rows selected. 24 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • ASH rowsource (10g) (object + event)SQL>SELECT sql_id, object_name, event 2 , round(sample_cnt_object_event/sample_cnt_total*100) as object_event_pct 3 , round(sample_cnt_object_event/6) minutes 4 FROM(SELECT DISTINCT sql_id, object_name, event 5 , count(*) over (partition by object_name,event) as sample_cnt_object_event 6 , count(*) over () as sample_cnt_total 7 FROM ( SELECT sql_id, nvl(event,CPU) as event 8 , case 9 when wait_class_id in (3871361733,1740759767,427450380,3875070507) 10 then object_name else N/A‘ end as object_name 11 FROM dba_hist_active_sess_history ash… snip …SQL_ID OBJECT_NAME EVENT OBJECT_EVENT_PCT MINUTES------------- -------------------- ------------------------- ---------------- ----------dkwucgaxxhm2z S_ADDR_PER db file sequential read 52 634dkwucgaxxhm2z S_ADDR_PER_P1 db file sequential read 28 347dkwucgaxxhm2z S_ORG_EXT db file sequential read 16 195… snip … 25 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL plan----------------------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |----------------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 9553K(100)| || 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 || 2 | NESTED LOOPS OUTER | | 8011K| 45G| | 7148K (1)| 23:49:40 || 3 | NESTED LOOPS OUTER | | 8011K| 45G| | 4743K (1)| 15:48:44 || 4 | NESTED LOOPS OUTER | | 8011K| 45G| | 3941K (1)| 13:08:24 || 5 | NESTED LOOPS OUTER | | 8011K| 44G| | 3140K (1)| 10:28:03 || 6 | HASH JOIN RIGHT OUTER | | 8011K| 44G| | 2338K (1)| 07:47:43 || 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 || 8 | HASH JOIN | | | | | | || 9 | HASH JOIN | | | | | • IO is 99% of elapsed time | || 10 | INDEX FAST FULL SCAN | S_SRC_M10 | 6633 | 285K| | 10 (0)| 00:00:01 || 11 | INDEX FAST FULL SCAN | S_SRC_M12 | 6633 | 285K| | • 75% of time in nested loop 10 (0)| 00:00:01 || 12 | INDEX FAST FULL SCAN | S_SRC_P1 | 6633 | 285K| | 8 (0)| 00:00:01 || 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| (2,34,35) 228M| 2338K (1)| 07:47:42 || 14 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 7488K| 142M| | 38163 (2)| 00:07:38 || 15 || 16 | HASH JOIN RIGHT OUTER TABLE ACCESS FULL | | S_INDUST_LANG | 8011K| | 4023 | 43G| 113K| | • Query is processing a large | 51545 199 (1)| 00:10:19 | (0)| 00:00:03 || 17 | HASH JOIN RIGHT OUTER | | 8011K| 43G| data set | 51289 (1)| 00:10:16 || 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8 (0)| 00:00:01 || 19 | NESTED LOOPS OUTER | | 8011K| 42G| | 51224 (1)| 00:10:15 || 20 | VIEW | | 8011K| 42G| • Nested loop is not an efficient | 51169 (1)| 00:10:15 || 21 | NESTED LOOPS | | | | | | | method for joining large data| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 || 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | sets 1 (0)| 00:00:01 || 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35 (3)| 00:00:01 || 25 | TABLE ACCESS BY INDEX ROWID| S_ORG_EXT | 421K| 419M| | 2693 (1)| 00:00:33 || 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | • Why does the CBO choose this 1 (0)| 00:00:01 || 27 | INDEX UNIQUE SCAN | S_OU_PRTNR_TYPE_P1 | 1 | | | 1 (0)| 00:00:01 | inefficient plan ?!| 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 || 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 – (0)| 00:00:01 | Bad stats?| 30 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 || 31 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 – (0)| 00:00:01 | Incorrect optimizer settings?| 32 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 | – Or something else?| 33 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 || 34 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 || 35 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 || 36 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 || 37 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 |----------------------------------------------------------------------------------------------------------------------… 26 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL statsSQL> SELECT PLAN_HASH_VALUE, executions 2 , round(elapsed_time /1000000/60/60,1) as elapsed_hours 3 , rows_processed 4 , round(cpu_time /elapsed_time * 100) as cpu_time_pct 5 , round(iowait_time /elapsed_time * 100) as iowait_pct 6 , LIO, PIO, direct_writes 7 , round(iowait_time /PIO/1000,1) as mili_sec_PIO 8 FROM(SELECT PLAN_HASH_VALUE 9 , sum(EXECUTIONS_DELTA) as executions 10 , sum(ROWS_PROCESSED_DELTA) as rows_processed 11 , sum(ELAPSED_TIME_DELTA) as elapsed_time 12 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time 13 , sum(DIRECT_WRITES_DELTA) as direct_writes 14 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO 15 FROM dba_hist_sqlstat 16 WHERE snap_id between 132042 and 132124 17 AND sql_id = dkwucgaxxhm2z 18 GROUP BY sql_id, PLAN_HASH_VALUE); Plan CPU IO Milisec Hash Elapsed Rows TIME WAIT Direct / Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes Pio---------- ---------- -------- ---------- ----- ---- ---------- ---------- -------- -------1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4 27 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL Optimizer Env valuesSQL>select * from table(dbms_xplan.display_awr(dkwucgaxxhm2z, 1700726234, null, ALL LAST +OUTLINE)); --------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | --------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 9553K(100)| | | 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 |..snip…Outline Data-------------/*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE(11.1.0.7) DB_VERSION(11.1.0.7) OPT_PARAM(_b_tree_bitmap_plans false) OPT_PARAM(_optim_peek_user_binds false) OPT_PARAM(optimizer_index_cost_adj 10) ALL_ROWS OUTLINE_LEAF(@"SEL$335DD26A") MERGE(@"SEL$3")..snip… 28 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • AWR SQL plan----------------------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |----------------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 9553K(100)| || 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 || 2 | NESTED LOOPS OUTER | | 8011K| 45G| | 7148K (1)| 23:49:40 || 3 | NESTED LOOPS OUTER | | 8011K| 45G| | 4743K (1)| 15:48:44 || 4 | NESTED LOOPS OUTER | | 8011K| 45G| | 3941K (1)| 13:08:24 || 5 | NESTED LOOPS OUTER | | 8011K| 44G| | 3140K (1)| 10:28:03 || 6 | HASH JOIN RIGHT OUTER | | 8011K| 44G| | 2338K (1)| 07:47:43 || 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 || 8 | HASH JOIN | | | | | | || 9 | HASH JOIN | | | | | | || 10 | INDEX FAST FULL SCAN | S_SRC_M10 | 6633 | 285K| | 10 (0)| 00:00:01 || 11 | INDEX FAST FULL SCAN | S_SRC_M12 | 6633 | 285K| | 10 (0)| 00:00:01 || 12 | INDEX FAST FULL SCAN | S_SRC_P1 | 6633 | 285K| | 8 (0)| 00:00:01 || 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| 228M| 2338K (1)| 07:47:42 || 14 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 7488K| 142M| | 38163 (2)| 00:07:38 || 15 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51545 (1)| 00:10:19 || 16 | TABLE ACCESS FULL | S_INDUST_LANG | 4023 | 113K| | 199 (0)| 00:00:03 || 17 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51289 (1)| 00:10:16 || 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8Direct (0)| 00:00:01 || 19 | NESTED LOOPS OUTER | | 8011K| 42G| | 51224 (1)| 00:10:15 || 20 | VIEW | | 8011K| 42G| | 51169Writes (1)| 00:10:15 || 21 | NESTED LOOPS | | | | | | || 22 | NESTED LOOPS | | 8011K| 8091M| | 51169-------- (1)| 00:10:15 || 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | 1 (0)| 00:00:01 || 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35219,418 (3)| 00:00:01 || 25 | TABLE ACCESS BY INDEX ROWID| S_ORG_EXT | 421K| 419M| | 2693 (1)| 00:00:33 || 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | 1 (0)| 00:00:01 || 27 | INDEX UNIQUE SCAN | S_OU_PRTNR_TYPE_P1 | 1 | | | 1 (0)| 00:00:01 || 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1workarea_size_policy=manual (0)| 00:00:01 || 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 || 30 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1hash_area_size=2147483647 (0)| 00:00:01 || 31 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 || 32 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 || 33 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 || 34 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 || 35 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 || 36 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 || 37 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 |----------------------------------------------------------------------------------------------------------------------… 29 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • New SQL Planselect * from table(dbms_xplan.display_cursor(null, null, ALLSTATS LAST));--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| Id | Operation | Name | Starts | E -Rows |E-Bytes| Cost (%CPU)| A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | | | 674K(100)| 0 |00:11:40.62 | 16M| 3031K| | | ||* 1 | HASH JOIN RIGHT OUTER | | 1 | 17M| 98G| 674K (2)| 5352K|01:20:42.29 | 16M| 3031K| 1143K| 1143K| 74M (0)|| 2 | VIEW | index$_join$_010 | 1 | 6633 | 285K| 383 (1)| 6707 |00:00:01.93 | 749 | 725 | | | ||* 3 | HASH JOIN | | 1 | | | | 6707 |00:00:01.93 | 749 | 725 | 1143K| 1143K| 73M (0)||* 4 | HASH JOIN | | 1 | | | | 6707 |00:00:01.07 | 680 | 659 | 1117K| 1117K| 73M (0)|| 5 | INDEX FAST FULL SCAN | S_SRC_M10 | 1 | 6633 | 285K| 103 (0)| 6707 |00:00:00.08 | 338 | 328 | | | || 6 | INDEX FAST FULL SCAN | S_SRC_M12 | 1 | 6633 | 285K| 100 (0)| 6707 |00:00:00.17 | 342 | 331 | | | || 7 | INDEX FAST FULL SCAN | S_SRC_P1 | 1 | 6633 | 285K| 76 (0)| 6707 |00:00:00.04 | 69 | 66 | | | ||* 8 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 41G| 674K (2)| 5352K|01:20:08.10 | 16M| 3030K| 1208M| 33M| 1455M (0)|| 9 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:01:53.38 | 392K| 392K| | | ||* 10 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 41G| 566K (2)| 5352K|01:17:16.78 | 16M| 2638K| 1208M| 33M| 1455M (0)|| 11 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:02:21.69 | 392K| 392K| | | ||* 12 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 41G| 458K (2)| 5352K|01:14:39.07 | 15M| 2246K| 285M| 10M| 459M (0)|| 13 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 1 | 7488K| 142M| 38163 (2)| 6777K|00:01:55.31 | 173K| 138K| | | ||* 14 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 420K (2)| 5352K|01:12:09.86 | 15M| 2107K| 445M| 18M| 606M (0)|| 15 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:04:56.19 | 154K| 102K| | | ||* 16 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 393K (2)| 5352K|01:07:28.03 | 15M| 2005K| 445M| 18M| 606M (0)|| 17 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:48.23 | 102K| 19057 | | | ||* 18 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 367K (2)| 5352K|01:05:45.34 | 15M| 1986K| 445M| 18M| 606M (0)|| 19 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:00.01 | 102K| 0 | | | ||* 20 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 340K (2)| 5352K|01:05:35.55 | 15M| 1986K| 915K| 915K| 74M (0)|| 21 | TABLE ACCESS FULL | S_INDUST_LANG | 1 | 4023 | 113K| 199 (0)| 4023 |00:00:00.04 | 713 | 709 | | | ||* 22 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 39G| 340K (2)| 5352K|01:05:23.59 | 15M| 1985K| 899K| 899K| 74M (0)|| 23 | TABLE ACCESS FULL | S_ORG_PRTNR | 1 | 324 | 43740 | 8 (0)| 319 |00:00:00.02 | 23 | 21 | | | ||* 24 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 38G| 340K (2)| 5352K|01:05:23.46 | 15M| 1985K| 1593K| 1593K| 64M (0)|| 25 | VIEW | index$_join$_014 | 1 | 1 | 94 | 1 (100)| 0 |00:00:00.06 | 3 | 2 | | | ||* 26 | HASH JOIN | | 1 | | | | 0 |00:00:00.06 | 3 | 2 | 1269K| 1269K| 64M (0)|| 27 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_P1 | 1 | 1 | 94 | 0 (0)| 0 |00:00:00.02 | 3 | 2 | | | || 28 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_U1 | 0 | 1 | 94 | 0 (0)| 0 |00:00:00.01 | 0 | 0 | | | || 29 | VIEW | | 1 | 7460K| 38G| 340K (2)| 5352K|01:05:12.63 | 15M| 1985K| | | ||* 30 | HASH JOIN | | 1 | 7460K| 7527M| 340K (2)| 5352K|01:05:12.59 | 15M| 1985K| 1078K| 1078K| 72M (0)|| 31 | INDEX FULL SCAN | SO_ANA_BU_U2 | 1 | 19 | 285 | 1 (0)| 19 |00:00:00.02 | 1 | 1 | | | || 32 | TABLE ACCESS FULL | S_ORG_EXT | 1 | 7068K| 7030M| 339K (2)| 6887K|01:05:18.93 | 15M| 1985K| | | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Evaluating the changesBefore: Plan CPU IO Milisec Hash Elapsed Rows TIME WAIT Direct / Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4After: Plan CPU IO Milisec Hash Elapsed Rows TIME WAIT Direct / Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------2491926841 1 1.3 5,352,381 5 96 13,134,981 2,651,076 0 2  Reduced elapsed time by 94%, Logical IO by 63% and Physical IO by 59%  But increased IOPS by 6x => ensure that disk subsystem can handle it !!! 31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – Find sql & statsTop 3 SQL for the time periodSQL_ID SQL_TYPE SAMPLE_PCT HOURS------------- ---------- ---------- ----------dvaxm0c9tdj80 INSERT 11 3.977uh2gt7ytnm9 SELECT 9 3.212dy8wkk6x5b5 INSERT 7 2.6Stats for SQL dvaxm0c9tdj80 Plan CPU IO milisec Hash Elapsed Rows time WAIT Direct /DAY Value executions Minutes processed PCT PCT LIO PIO Writes PIO---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 172011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 352011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 132011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 212011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32 32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – SQL detailsSQL PLanSQL>select * from table(dbms_xplan.display_awr(dvaxm0c9tdj80, 4061473922, null, ALL LAST’));-------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |-------------------------------------------------------------------------------| 0 | INSERT STATEMENT | | | | 24155 (100)| || 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |-------------------------------------------------------------------------------SQL TextINSERT /*+APPEND*/ INTO S_ETL_I_IMG_70 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)SELECT ROW_ID, 1, I, LAST_UPDFROM S_PARTYWHERE S_PARTY.LAST_UPD > :1; 33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – SQL rowsourceSQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES------------- ---------------- ------------------------- ------------------- ----------dvaxm0c9tdj80 2 db file sequential read 100 235dvaxm0c9tdj80 2 CPU 0 1Stats for SQL dvaxm0c9tdj80 Plan CPU IO milisec Hash Elapsed Rows time WAIT Direct /DAY Value executions Minutes processed PCT PCT LIO PIO Writes PIO---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 172011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 352011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 132011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 212011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32 34 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – Index Structure*SQL> @i S_PARTYType Unq? Part? Index name STATUS Column name DATA_TYPE------ ---- ------ ----------- -------- -------------- ---------NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2 S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR… snip … NO NO S_PARTY_W1 VALID LAST_UPD DATE VALID ROW_ID VARCHAR2*Output based on modified version of Tim Gorman’s script at www.evdbt.com/tool.htm  IO is 99% of elapsed timeSQL Text  Using perfect index, but wrong access methodINSERT /*+APPEND*/ INTO S_ETL_I_IMG_70  Why does the CBO choose this (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD) inefficient plan ?!SELECT ROW_ID, 1, I, LAST_UPDFROM S_PARTY  Bad stats?WHERE S_PARTY.LAST_UPD > :1;  Incorrect optimizer settings?  Or something else?INDEX FULL SCAN | S_PARTY_W1 35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – Bad stats?SQL>select * from table(dbms_xplan.display_awr(dvaxm0c9tdj80, 4061473922, null, ALL LAST’));-------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |-------------------------------------------------------------------------------| 0 | INSERT STATEMENT | | | | 24155 (100)| || 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |------------------------------------------------------------------------------- 36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – Replicate take 1SQL>explain plan for 2 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70 3 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD) 4 SELECT ROW_ID, 1, I, LAST_UPD 5 FROM S_PARTY 6 WHERE S_PARTY.LAST_UPD > :x;Explained.SQL>select * from table(dbms_xplan.display(null, null, ALLSTATS LAST));… snip …------------------------------------------------------------| Id | Operation | Name | E-Rows |------------------------------------------------------------| 0 | INSERT STATEMENT | | 913K|| 1 | LOAD TABLE CONVENTIONAL | S_ETL_I_IMG_70 | ||* 2 | INDEX RANGE SCAN | S_PARTY_W1 | 913K|------------------------------------------------------------Predicate Information (identified by operation id):--------------------------------------------------- 2 - access("S_PARTY"."LAST_UPD">:X)Object Statistics History  WRI$_OPTSTAT_TAB_HISTORY  WRI$_OPTSTAT_IND_HISTORY  Etc. 37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – bind data typeSQL>select distinct name, was_captured, last_captured 2 , sys.anydata.gettypename(value_anydata) as data_type 3 , sys.anydata.accesstimestamp(value_anydata) as data_value 4 from DBA_HIST_SQLBIND where sql_id = dvaxm0c9tdj80’ order by last_captured; wasNAME captured LAST_CAPTURED DATA_TYPE DATA_VALUE----- -------- ------------------------- -------------------- --------------------:1 YES 2011-06-29 17:51:44 SYS.TIMESTAMP 2011-06-28 08:12:58:1 YES 2011-06-30 18:16:03 SYS.TIMESTAMP 2011-06-29 17:01:20:1 YES 2011-07-01 03:21:51 SYS.TIMESTAMP 2011-06-30 17:01:11:1 YES 2011-07-02 17:59:52 SYS.TIMESTAMP 2011-07-01 03:03:19:1 YES 2011-07-03 17:59:08 SYS.TIMESTAMP 2011-07-02 17:01:38Type Unq? Part? Index name STATUS Column name DATA_TYPE------ ---- ------ ----------- -------- -------------- ---------NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2 S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR… snip … NO NO S_PARTY_W1 VALID LAST_UPD DATE VALID ROW_ID VARCHAR2 38 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2 – Replicate (2nd try)SQL>declare 2 x timestamp; 3 begin 4 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70 5 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD) 6 SELECT ROW_ID, 1, I, LAST_UPD 7 FROM S_PARTY 8 WHERE S_PARTY.LAST_UPD > :x; 9 end; 10 / ---------------------------------------------------------- | Id | Operation | Name | E-Rows | ---------------------------------------------------------- | 0 | INSERT STATEMENT | | | | 1 | LOAD TABLE CONVENTIONAL | | | |* 2 | INDEX FULL SCAN | S_PARTY_W1 | 913K| ---------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter(INTERNAL_FUNCTION("S_PARTY"."LAST_UPD")>:B1) 39 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Use Case 2: Visualizing the improvement IMG Process Duration 10 9 8 7Elapsed Time (Hours) 6 5 4 3 2 1 0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 ETL Day 40 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Free script collections • Tanel Poder • tech.e2sn.com/oracle-scripts-and-tools • Kerry Osborne - • kerryosborne.oracle-guy.com/papers/my_favorite_scripts_2010.zip • Tim Gorman • evdbt.com/tools.htm • Adrian Billington • oracle-developer.net/utilities.php • Jonathan Lewis • jonathanlewis.wordpress.com/2009/12/18/simple-scripts/41 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Resources SQL Tuning Forms Tuning – Trace files – Forms Tracing (MOS Doc : 373548.1) – SQLT output (MOS Doc: 215187.1) – Generic MOS Doc : 438652.1 – Trace Analyzer (MOS Doc : 224270.1) Reports Tracing – AWR Report (MOS Doc : 748642.1) – MOS Doc: 111311.1 – AWR SQL Report (awrsqrpt.sql) – 11g Real-Time SQL Monitoring Database Tuning – SQL Tuning Advisor – AWR Report (MOS Doc : 748642.1) – ADDM report (MOS Doc : 250655.1) PL/SQL Tuning – Active Session History (ASH) Report – Product logs – LTOM output (MOS Doc : 352363.1) – PL/SQL Profiler (MOS Doc : 808005.1) OS – OSWatcher (MOS Doc : 301137.1) Middletier Tuning – JVM Logs – Third Party Tools42 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • To learn even more • Performance Fundamentals for Oracle Database 10g and 11g by Graham Wood and John Beresniewicz (http://www.oracle.com/technetwork/database/focus-areas/manageability/db-perf-tuning-ow08-131582.pdf) • ASH website by Kyle Hailey (https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history) • Expert Oracle Database Architecture: Oracle Database 9i, 10g, and 11g Programming Techniques and Solutions by Thomas Kyte • Troubleshooting Oracle Performance by Christian Antognini • Blogs – Tanel Poder, Kerry Osborne, Jonathan Lewis, Cary Millsap, Greg Rahn, Randolf Geist, etc.43 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Summary • Ask questions • Understand the database internals • Use multiple tools depending on the situation • Combine the best of both (i.e. PerfSheet from Tanel Poder)44 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • Q&A45 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 46 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 47 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.