Barun_Practical_and_Efficient_SQL_Performance_Tuning

2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The following is not intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract. It is not a commitment to
deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions. The development,
release, and timing of any features or functionality described for
Oracle’s products remains at the sole discretion of Oracle.
All the views expressed in this presentation are the opinion of the
authors and do not necessarily reflect the views of Oracle.
The queries are provided "as is", without warranty of any kind,
expressed or implied.

Practical & Efficient SQL Performance Tuning
Real World SQL Performance Tuning Use Cases

Objective
• Rapid root cause analysis
• Enable you to start exploring the rich repository of
performance metrics in ASH & AWR independent of any
specific GUI and related limitations
• SQL Performance Tuning vs System-wide tuning vs
Capacity Planning

About myself
• Vlado Barun, M.Sc.
– Real World Performance Team at Oracle
– Previous Roles: Developer, DBA, Architect, Manager
– OCP, OCE SQL, MCP
– 17+ years in the Database Arena (Oracle DB - 13 years)
– linkedin.com/in/vbarun

Agenda
• Introduction to SQL Execution Plans, SQLMON, ASH &
AWR (very brief)
• Review 2 real world production use cases
– step-by-step from data gathering to resolution
– write ASH/AWR queries from scratch
– use pre-existing free scripts
– Note: check license requirements
• Resources

The universal experience of programmers
who have been using measurement tools
has been that their intuitive guesses fail.
- Donald Knuth
Why do we need tools like SQLMON,
AWR and ASH?

It is a capital mistake to theorize before one
has data. Insensibly one begins to twist facts
to suit theories, instead of theories to suit
facts.
― Sherlock Holmes,
“The Adventures of Sherlock Holmes”
by Sir Arthur Conan Doyle
Why do we need tools like SQLMON,
AWR and ASH?

Dbms_xplan.display_cursor
select * from table(dbms_xplan.display_cursor('&SQL_ID',null,'ALLSTATS LAST'));
SQL_ID 0s5pdsggxtc54, child number 0
-------------------------------------
Plan hash value: 2832870632
----------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 152K(100)| |
| 1 | SORT AGGREGATE | | 1 | 2 | | |
|* 2 | TABLE ACCESS FULL| TAB_A | 21M| 41M| 152K (5)| 00:00:03 |
----------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("COSTED_FLAG" IS NULL)
Note
-----
- Warning: basic plan statistics not available. These are only collected when:
* hint 'gather_plan_statistics' is used for the statement or
* parameter 'statistics_level' is set to 'ALL', at session or system level
SQL_ID 0s5pdsggxtc54, child number 1
-------------------------------------
Plan hash value: 2832870632
--------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:01.73 | 435K|
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:01.73 | 435K|
|* 2 | TABLE ACCESS FULL| TAB_A | 1 | 1351K| 2001K|00:00:01.64 | 435K|
--------------------------------------------------------------------------------------
Estimates with
Statistics_level = Typical
Actual runtime stats with
Statistics_level = ALL
or with
gather_plan_statistics hint

Dbms_xplan.display_awr
select * from table(dbms_xplan.display_awr('&SQL_ID', null, null, 'ALL LAST'));
SQL_ID 0s5pdsggxtc54
--------------------
----------------------------------------------------------------------------
----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 22217 (100)| |
| 1 | SORT AGGREGATE | | 1 | 2 | | |
| 2 | TABLE ACCESS FULL| TAB_A | 1351K| 2639K| 22217 (1)| 00:04:27 |
----------------------------------------------------------------------------

SQL Execution Plans
• DBMS_XPLAIN.DISPLAY_CURSOR
• Plan is from memory
• Predicates and outline data
• Actual stats when ‘statistics_level’ set to ALL or
with gather_plan_statistics hint
• DBMS_XPLAIN.DISPLAY_AWR
• Plan is from AWR (DBA_HIST% views)
• Runtime (actual) stats are not displayed

Real-Time SQL Monitoring
• Monitors SQLs while they are running
• Automatically monitors parallel SQLs and SQLs taking more than 5
CPU seconds
• Plan is from memory
• Predicates
• CPU and Elapsed Time
• No. of reads and writes
• Waits by plan step!
• Stored in V$SQL_MONITOR and V$SQL_PLAN_MONITOR views
• How to access?
• DBMS_SQLTUNE.REPORT_SQL_MONITOR
• Enterprise Manager
• Custom scripts

Challenge #2
• How to identify the SQL ID that needs to be optimized?
– Somebody tells you the SQL ID
– You figure it out yourself
• OEM
• ADDM
• AWR
• ASH
• Session/Module level SQL Trace
• Application logs
• etc

Automatic Workload Repository (AWR)
• Collects database performance statistics
– Objects (access and usage statistics)
– SQL Statement statistics
– Wait events statistics
– System statistics
– Time Model Statistics based on time usage for activities
– Every 1 hr by default
• Stored in DBA_HIST% views
• How to access?
– $ORACLE_HOME/rdbms/admin/awrrpt.sql, SQLT, custom
scripts,…
– OEM, SQL Developer
– Other tools

Active Session History (ASH)
• Samples the current state of all active session
– 1 second samples
– Stores various attributes (dimensions) of a session

= time spent in database
SQL Trace (Microsecond level)
Application Logging (custom level)
Browse
Books
Read Reviews
For One Book
Add to
Cart
Checkout
TIME
v$active_session_history
(1 second intervals)
dba_hist_active_sess_history
(10 second intervals)
51 ASH Samples
5 ASH Samples in
AWR
Active Session History (ASH) Introduction

When
SAMPLE_ID
SAMPLE_TIME
Session
SESSION_ID
SESSION_SERIAL#
SESSION_TYPE
SESSION_STATE
USER_ID
QC_INSTANCE_ID
QC_SESSION_ID
QC_SESSION_SERIAL#
CLIENT_ID
IN and IS columns
SQL
SQL_ID
IS_SQLID_CURRENT
SQL_CHILD_NUMBER
SQL_OPCODE
SQL_OPNAME
TOP_LEVEL_SQL_ID
TOP_LEVEL_SQL_OPCODE
SQL_EXEC_ID
SQL_EXEC_START
SQL_PLAN_HASH_VALUE
SQL_PLAN_OPTIONS
Row Source
SQL_PLAN_LINE_ID
SQL_PLAN_OPERATION
Waits
EVENT
EVENT_ID
EVENT#
SEQ#
P1TEXT
P1
P2TEXT
P2
P3TEXT
P3
WAIT_CLASS
WAIT_CLASS_ID
WAIT_TIME
TIME_WAITED
CURRENT_OBJ#
CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Blocking Session
BLOCKING_SESSION_STATUS
BLOCKING_SESSION
BLOCKING_SESSION_SERIAL#
BLOCKING_INST_ID
BLOCKING_HANGCHAIN_INFO
Object
CURRENT_OBJ#
CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Application
SERVICE_HASH
PROGRAM
MODULE
ACTION
MACHINE
Statistics
TM_DELTA_TIME
TM_DELTA_CPU_TIME
TM_DELTA_DB_TIME
DELTA_TIME
DELTA_READ_IO_REQUESTS
DELTA_WRITE_IO_REQUESTS
DELTA_READ_IO_BYTES
DELTA_WRITE_IO_BYTES
DELTA_INTERCONNECT_IO_BYTES
PGA_ALLOCATED
TEMP_SPACE_ALLOCATED
ASH columns (subset)

ASH column growth
See
https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history

Active Session History (ASH)
• Samples the current state of all active session
– 1 second samples
– Stores various attributes (dimensions) of a session
• Stored in
– V$ACTIVE_SESSION_HISTORY
• 1-second sampling rate
– DBA_HIST_ACTIVE_SESS_HISTORY
- Sub-sampling to disk into AWR Snapshot
- 1-in-10 samples
• How to access?
- $ORACLE_HOME/rdbms/admin/ashrpt.sql, ashrpti.sql, SQLT, custom
scripts,…
- OEM, SQL Developer, …
- Other tools

ASH – further reading
• Performance Fundamentals for Oracle Database 10g and
11g by Graham Wood, Uri Shaft and John Beresniewicz
(http://www.oracle.com/technetwork/database/focus-areas/manageability/db-perf-tuning-ow08-131582.pdf)
• ASH website by Kyle Hailey
(https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history)
• Grahams’s talk

GUI command line
• It is about knowing the right tool for the right situation
• Understand the strengths and weaknesses of each
• GUI tools strengths
– Easy to use
– Good visualization of data
– Provides concise summary of system activity
– etc…
• GUI tools weaknesses
– Slow - click…wait…click…wait…
– Availability – GUI infrastructure is down
– Limited exposure – subset of available data
– etc…
Performance
data
Performance data
exposed by GUI
or
=> cmd line strength
=> cmd line weaknesses

What Identify
(app is slow)
Quantify
(takes 1m)
Timeline
(during peak hours)
Component
(Web Tier, App Tier,
DB , Network, Storage,
etc.)
Flow chart
Root Cause
(i.e. non-representative
statistics)
AWR/ASH, Trace,
Apps Logs/Tables,
OS/IO stats etc
SQL/Object Stats,
Execution Plans,
etc
Initial
Data
collectionHow Much
When
Where
How
Why
Analyze
Additional
Data
Collection
and
Analysis
Understand the issue

What Slow ETL
20+ hours
Once a month
Informatica + DB
How Much
When
Where
How Extract, Transform, Load
Use Case 1

Find slow SQL (ASH)
SQL> SELECT sql_id
2 , COUNT(*) as sample_count
3 FROM v$active_session_history
4 WHERE sample_time BETWEEN TO_DATE('2011-06-29 07:57', 'yyyy-mm-dd hh24:mi')
5 AND TO_DATE('2011-06-30 04:30', 'yyyy-mm-dd hh24:mi')
6 AND user_id = 61
7 AND program like 'pmdtm@%'
8 GROUP BY sql_id
9 ORDER BY 2 DESC
10 ;
no rows selected
SQL> SELECT min(sample_time)
2 FROM v$active_session_history
3 ;
MIN(SAMPLE_TIME)
----------------------------------------
2011-07-01 19:14:48
1 row selected.

Find slow SQL (ASH in AWR)
SELECT sql_id
, COUNT(*) as sample_count
FROM dba_hist_active_sess_historyv$active_session_history
WHERE sample_time BETWEEN TO_DATE('2011-06-29 07:57', 'yyyy-mm-dd hh24:mi')
AND TO_DATE('2011-06-30 04:30', 'yyyy-mm-dd hh24:mi')
AND user_id = 61
AND program like 'pmdtm@%‘
GROUP BY sql_id
ORDER BY 2 DESC
;
JOIN dba_hist_snapshot USING (dbid, instance_number, snap_id)
AND begin_interval_time BETWEEN TO_DATE('2011-06-29 07:00','yyyy-mm-dd hh24:mi')
AND TO_DATE('2011-06-30 05:00', 'yyyy-mm-dd hh24:mi')
SQL_ID SAMPLE_COUNT
------------- ------------
dkwucgaxxhm2z 7362
grksgbxxuacaf 6041
…
24 rows selected.

Find slow SQL (ASH in AWR) - readable
SQL>SELECT sql_id, sql_type
2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct
3 , round(sample_cnt_sql /6/60,1) hours
4 FROM(SELECT DISTINCT sql_id
5 , aa.name AS sql_type
6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql
7 , COUNT(*) over () AS sample_cnt_total
8 FROM dba_hist_active_sess_history ash
9 JOIN dba_hist_snapshot s USING (dbid, INSTANCE_NUMBER, snap_id)
10 JOIN audit_actions aa on (ash.sql_opcode = aa.action)
11 WHERE begin_interval_time BETWEEN
… snip …
18 ORDER BY 3 desc)
19 WHERE rownum <= 3;
SQL_ID SQL_TYPE SAMPLE_PCT HOURS
------------- -------- ---------- -----
dkwucgaxxhm2z SELECT 25 20.4
grksgbxxuacaf SELECT 21 16.8
gh6w2fm3avn3j SELECT 12 10.1
3 rows selected.

When
SAMPLE_ID
SAMPLE_TIME
Session
SESSION_ID
SESSION_SERIAL#
SESSION_TYPE
SESSION_STATE
USER_ID
QC_INSTANCE_ID
QC_SESSION_ID
QC_SESSION_SERIAL#
CLIENT_ID
IN and IS columns
SQL
SQL_ID
IS_SQLID_CURRENT
SQL_CHILD_NUMBER
SQL_OPCODE
SQL_OPNAME
TOP_LEVEL_SQL_ID
TOP_LEVEL_SQL_OPCODE
SQL_EXEC_ID
SQL_EXEC_START
SQL_PLAN_HASH_VALUE
SQL_PLAN_OPTIONS
Row Source
SQL_PLAN_LINE_ID
SQL_PLAN_OPERATION
Waits
EVENT
EVENT_ID
EVENT#
SEQ#
P1TEXT
P1
P2TEXT
P2
P3TEXT
P3
WAIT_CLASS
WAIT_CLASS_ID
WAIT_TIME
TIME_WAITED
CURRENT_OBJ#
CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Blocking Session
BLOCKING_SESSION_STATUS
BLOCKING_SESSION
BLOCKING_SESSION_SERIAL#
BLOCKING_INST_ID
BLOCKING_HANGCHAIN_INFO
Object
CURRENT_OBJ#
CURRENT_FILE#
CURRENT_BLOCK#
CURRENT_ROW#
Application
SERVICE_HASH
PROGRAM
MODULE
ACTION
MACHINE
Statistics
TM_DELTA_TIME
TM_DELTA_CPU_TIME
TM_DELTA_DB_TIME
DELTA_TIME
DELTA_READ_IO_REQUESTS
DELTA_WRITE_IO_REQUESTS
DELTA_READ_IO_BYTES
DELTA_WRITE_IO_BYTES
DELTA_INTERCONNECT_IO_BYTES
PGA_ALLOCATED
TEMP_SPACE_ALLOCATED
Active Session History (ASH) - Dimensions

AWR SQL stats
SQL> SELECT PLAN_HASH_VALUE,
2 , sum(EXECUTIONS_DELTA) as executions
3 , sum(ROWS_PROCESSED_DELTA) as rows_processed
4 , sum(ELAPSED_TIME_DELTA) as elapsed_time
5 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time
6 , sum(DIRECT_WRITES_DELTA) as direct_writes
7 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO
8 FROM dba_hist_sqlstat
9 WHERE snap_id between 132042 and 132124
10 AND sql_id = 'dkwucgaxxhm2z'
11 GROUP BY sql_id, PLAN_HASH_VALUE, OPTIMIZER_ENV_HASH_VALUE;
Plan IO
Hash Rows Elapsed CPU WAIT Direct
Value EXECUTIONS Processed Time TIME TIME Writes LIO PIO
---------- ---------- ---------- ---------- ---------- ----------- -------- -------- -------
1700726234 1 5997889 7.3806E+10 903837577 73047955458 219418 36152215 6427679
1 row selected.

AWR SQL stats - readable
SQL> SELECT PLAN_HASH_VALUE, executions
2 , round(elapsed_time /1000000/60/60,1) as elapsed_hours
3 , rows_processed
4 , round(cpu_time /elapsed_time * 100) as cpu_time_pct
5 , round(iowait_time /elapsed_time * 100) as iowait_pct
6 , LIO, PIO, direct_writes
7 , round(iowait_time /PIO/1000,1) as mili_sec_PIO
8 FROM(SELECT PLAN_HASH_VALUE
9 , sum(EXECUTIONS_DELTA) as executions
10 , sum(ROWS_PROCESSED_DELTA) as rows_processed
11 , sum(ELAPSED_TIME_DELTA) as elapsed_time
12 , sum(CPU_TIME_DELTA) as cpu_time, sum(IOWAIT_DELTA) as iowait_time
13 , sum(DIRECT_WRITES_DELTA) as direct_writes
14 , sum(BUFFER_GETS_DELTA) as LIO, sum(DISK_READS_DELTA) as PIO
15 FROM dba_hist_sqlstat
16 WHERE snap_id between 132042 and 132124
17 AND sql_id = 'dkwucgaxxhm2z'
18 GROUP BY sql_id, PLAN_HASH_VALUE);
Plan CPU IO Milisec
Hash Elapsed Rows TIME WAIT Direct /
Value Executions Hours Processed PCT PCT LIO PIO Writes Pio
---------- ---------- -------- ---------- ---- ----- ---------- ---------- -------- -------
1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4

AWR SQL plan
SQL>select * from table(dbms_xplan.display_awr('dkwucgaxxhm2z', 1700726234, null,
'ALL LAST'));
----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | | 9553K(100)| |
| 1 | NESTED LOOPS OUTER | | 8011K| 46G| | 9553K (1)| 31:50:39 |
| 6 | HASH JOIN RIGHT OUTER | | 8011K| 44G| | 2338K (1)| 07:47:43 |
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 10 | INDEX FAST FULL SCAN | S_SRC_M10 | 6633 | 285K| | 10 (0)| 00:00:01 |
| 12 | INDEX FAST FULL SCAN | S_SRC_P1 | 6633 | 285K| | 8 (0)| 00:00:01 |
| 13 | HASH JOIN RIGHT OUTER | | 8011K| 44G| 228M| 2338K (1)| 07:47:42 |
| 14 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 7488K| 142M| | 38163 (2)| 00:07:38 |
| 15 | HASH JOIN RIGHT OUTER | | 8011K| 43G| | 51545 (1)| 00:10:19 |
| 16 | TABLE ACCESS FULL | S_INDUST_LANG | 4023 | 113K| | 199 (0)| 00:00:03 |
| 18 | TABLE ACCESS FULL | S_ORG_PRTNR | 324 | 43740 | | 8 (0)| 00:00:01 |
| 19 | NESTED LOOPS OUTER | | 8011K| 42G| | 51224 (1)| 00:10:15 |
| 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
| 23 | INDEX FULL SCAN | SO_ANA_BU_U2 | 19 | 285 | | 1 (0)| 00:00:01 |
| 24 | INDEX RANGE SCAN | S_ORG_EXT_M91 | 74493 | | | 35 (3)| 00:00:01 |
| 25 | TABLE ACCESS BY INDEX ROWID| S_ORG_EXT | 421K| 419M| | 2693 (1)| 00:00:33 |
| 26 | TABLE ACCESS BY INDEX ROWID | S_OU_PRTNR_TYPE | 1 | 94 | | 1 (0)| 00:00:01 |
| 27 | INDEX UNIQUE SCAN | S_OU_PRTNR_TYPE_P1 | 1 | | | 1 (0)| 00:00:01 |
| 28 | TABLE ACCESS BY INDEX ROWID | S_ORG_EXT | 1 | 42 | | 1 (0)| 00:00:01 |
| 29 | INDEX UNIQUE SCAN | S_ORG_EXT_P1 | 1 | | | 1 (0)| 00:00:01 |
| 34 | TABLE ACCESS BY INDEX ROWID | S_ADDR_PER | 1 | 59 | | 1 (0)| 00:00:01 |
| 35 | INDEX UNIQUE SCAN | S_ADDR_PER_P1 | 1 | | | 1 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------------
…

Find slow SQL (ASH in AWR)
SQL>SELECT sql_id, sql_type
2 , round(sample_cnt_sql/sample_cnt_total*100) as sample_pct
3 , round(sample_cnt_sql /6/60,1) hours
4 FROM(SELECT DISTINCT sql_id
5 , aa.name AS sql_type
6 , COUNT(*) over (PARTITION BY sql_id) AS sample_cnt_sql
10 JOIN audit_actions aa on (ash.sql_opcode = aa.action)
11 WHERE begin_interval_time BETWEEN
… snip …
18 ORDER BY 3 desc)
19 WHERE rownum <= 3;
------------- -------- ---------- -----
dkwucgaxxhm2z SELECT 25 20.4
grksgbxxuacaf SELECT 21 16.8
gh6w2fm3avn3j SELECT 12 10.1
3 rows selected.

ASH rowsource (10g) (object + event)
SQL>SELECT sql_id, object_name, event
2 , round(sample_cnt_object_event/sample_cnt_total*100) as object_event_pct
3 , round(sample_cnt_object_event/6) minutes
4 FROM(SELECT DISTINCT sql_id, object_name, event
5 , count(*) over (partition by object_name,event) as sample_cnt_object_event
6 , count(*) over () as sample_cnt_total
7 FROM ( SELECT sql_id, nvl(event,'CPU') as event
8 , case
9 when wait_class_id in (3871361733,1740759767,427450380,3875070507)
10 then object_name else 'N/A‘ end as object_name
… snip …
SQL_ID OBJECT_NAME EVENT OBJECT_EVENT_PCT MINUTES
------------- -------------------- ------------------------- ---------------- ----------
dkwucgaxxhm2z S_ADDR_PER db file sequential read 52 634
dkwucgaxxhm2z S_ADDR_PER_P1 db file sequential read 28 347
dkwucgaxxhm2z S_ORG_EXT db file sequential read 16 195
… snip …

AWR SQL plan
'ALL LAST'));
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
----------------------------------------------------------------------------------------------------------------------
…

ASH rowsource data (plan line id)
SQL>SELECT sql_id, sql_plan_line_id
2 , round(sample_cnt_plan_line/sample_cnt_total*100) AS sql_plan_line_pct
3 , round(sample_cnt_plan_line/6) minutes
4 FROM(
5 SELECT DISTINCT sql_id, sql_plan_line_id
6 , COUNT(*) over (PARTITION BY sql_plan_line_id) AS sample_cnt_plan_line
10 WHERE sql_id = 'dkwucgaxxhm2z‘ and begin_interval_time BETWEEN
… snip …
SQL_ID SQL_PLAN_LINE_ID SQL_PLAN_LINE_PCT MINUTES
------------- ---------------- ----------------- ----------
dkwucgaxxhm2z 34 48 593
…

ASH rowsource (plan line id + event)
SQL>SELECT sql_id, sql_plan_line_id, event
2 , round(sample_cnt_plan_line_event/sample_cnt_total*100) as plan_line_event_pct
3 , round(sample_cnt_plan_line_event/6) minutes
4 FROM (SELECT DISTINCT sql_id
5 , sql_plan_line_id, nvl(event, 'CPU') as event
6, COUNT(*) over (PARTITION BY sql_plan_line_id,event)AS sample_cnt_plan_line_event
7, COUNT(*) over () AS sample_cnt_total
10 WHERE sql_id = 'dkwucgaxxhm2z' and begin_interval_time BETWEEN …
… snip …
SQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES
------------- ---------------- ------------------------- ------------------- ----------
dkwucgaxxhm2z 34 db file sequential read 48 591
… snip …
22 rows selected.

AWR SQL plan
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
----------------------------------------------------------------------------------------------------------------------
…
• IO is 99% of elapsed time
• 75% of time in one nested loop
(plan lines 2,34,35)
• Query is processing a large
data set
• Nested loop is not an efficient
method for joining large data
sets
• Why does the CBO choose this
inefficient plan ?!
– Non-representative stats?
– Incorrect optimizer settings?
– Or something else?
Rows
Processed
----------
5,997,889

AWR SQL Optimizer Env values
'ALL LAST +OUTLINE'));
---------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
..snip…
Outline Data
-------------
/*+
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('11.1.0.7')
DB_VERSION('11.1.0.7')
OPT_PARAM('_b_tree_bitmap_plans' 'false')
OPT_PARAM('_optim_peek_user_binds' 'false')
OPT_PARAM('optimizer_index_cost_adj' 10)
ALL_ROWS
OUTLINE_LEAF(@"SEL$335DD26A")
MERGE(@"SEL$3")
..snip…
Alter session set optimizer_index_cost_adj=100;
or
Select /*+ OPT_PARAM('optimizer_index_cost_adj' 100) */
or
…

AWR SQL plan
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| 7 | VIEW | index$_join$_009 | 6633 | 285K| | 39 (3)| 00:00:01 |
| 8 | HASH JOIN | | | | | | |
| 9 | HASH JOIN | | | | | | |
| 20 | VIEW | | 8011K| 42G| | 51169 (1)| 00:10:15 |
| 21 | NESTED LOOPS | | | | | | |
| 22 | NESTED LOOPS | | 8011K| 8091M| | 51169 (1)| 00:10:15 |
----------------------------------------------------------------------------------------------------------------------
…
Direct
Writes
--------
219,418
Alter session set workarea_size_policy=manual;
Alter session set hash_area_size=2147483647;
or
adjust pga_aggregate_target
and/or
adjust _pga_max_size and smm_max_size
or
…

New SQL Plan
select * from table(dbms_xplan.display_cursor(null, null, 'ALLSTATS LAST'));
----------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------
| Id | Operation | Name | Starts | E-Rows |E-Bytes| Cost (%CPU)| A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
----------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | | 674K(100)| 0 |00:11:40.62 | 16M| 3031K| | | |
|* 1 | HASH JOIN RIGHT OUTER | | 1 | 17M| 98G| 674K (2)| 5352K|01:20:42.29 | 16M| 3031K| 1143K| 1143K| 74M (0)|
| 2 | VIEW | index$_join$_010 | 1 | 6633 | 285K| 383 (1)| 6707 |00:00:01.93 | 749 | 725 | | | |
|* 3 | HASH JOIN | | 1 | | | | 6707 |00:00:01.93 | 749 | 725 | 1143K| 1143K| 73M (0)|
|* 4 | HASH JOIN | | 1 | | | | 6707 |00:00:01.07 | 680 | 659 | 1117K| 1117K| 73M (0)|
| 5 | INDEX FAST FULL SCAN | S_SRC_M10 | 1 | 6633 | 285K| 103 (0)| 6707 |00:00:00.08 | 338 | 328 | | | |
| 6 | INDEX FAST FULL SCAN | S_SRC_M12 | 1 | 6633 | 285K| 100 (0)| 6707 |00:00:00.17 | 342 | 331 | | | |
| 7 | INDEX FAST FULL SCAN | S_SRC_P1 | 1 | 6633 | 285K| 76 (0)| 6707 |00:00:00.04 | 69 | 66 | | | |
|* 8 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 41G| 674K (2)| 5352K|01:20:08.10 | 16M| 3030K| 1208M| 33M| 1455M (0)|
| 9 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:01:53.38 | 392K| 392K| | | |
| 11 | TABLE ACCESS FULL | S_ADDR_PER | 1 | 13M| 744M| 107K (2)| 14M|00:02:21.69 | 392K| 392K| | | |
| 13 | TABLE ACCESS FULL | S_ORG_EXT_LSX | 1 | 7488K| 142M| 38163 (2)| 6777K|00:01:55.31 | 173K| 138K| | | |
| 15 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:04:56.19 | 154K| 102K| | | |
| 17 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:48.23 | 102K| 19057 | | | |
| 19 | INDEX FAST FULL SCAN | S_ORG_EXT_W3 | 1 | 7068K| 276M| 26545 (1)| 6887K|00:00:00.01 | 102K| 0 | | | |
|* 20 | HASH JOIN RIGHT OUTER | | 1 | 7460K| 40G| 340K (2)| 5352K|01:05:35.55 | 15M| 1986K| 915K| 915K| 74M (0)|
| 21 | TABLE ACCESS FULL | S_INDUST_LANG | 1 | 4023 | 113K| 199 (0)| 4023 |00:00:00.04 | 713 | 709 | | | |
| 23 | TABLE ACCESS FULL | S_ORG_PRTNR | 1 | 324 | 43740 | 8 (0)| 319 |00:00:00.02 | 23 | 21 | | | |
| 25 | VIEW | index$_join$_014 | 1 | 1 | 94 | 1 (100)| 0 |00:00:00.06 | 3 | 2 | | | |
|* 26 | HASH JOIN | | 1 | | | | 0 |00:00:00.06 | 3 | 2 | 1269K| 1269K| 64M (0)|
| 27 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_P1 | 1 | 1 | 94 | 0 (0)| 0 |00:00:00.02 | 3 | 2 | | | |
| 28 | INDEX FAST FULL SCAN| S_OU_PRTNR_TYPE_U1 | 0 | 1 | 94 | 0 (0)| 0 |00:00:00.01 | 0 | 0 | | | |
| 29 | VIEW | | 1 | 7460K| 38G| 340K (2)| 5352K|01:05:12.63 | 15M| 1985K| | | |
|* 30 | HASH JOIN | | 1 | 7460K| 7527M| 340K (2)| 5352K|01:05:12.59 | 15M| 1985K| 1078K| 1078K| 72M (0)|
| 31 | INDEX FULL SCAN | SO_ANA_BU_U2 | 1 | 19 | 285 | 1 (0)| 19 |00:00:00.02 | 1 | 1 | | | |
| 32 | TABLE ACCESS FULL | S_ORG_EXT | 1 | 7068K| 7030M| 339K (2)| 6887K|01:05:18.93 | 15M| 1985K| | | |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------Select /*+ gather_plan_statistics */
…
Or
Alter sessions set statistics_level=all;
Or
…

Evaluating the changes
Before:
Plan CPU IO Milisec
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------
1700726234 1 20.5 5,997,889 1 99 36,152,215 6,427,679 219,418 11.4
After:
Plan CPU IO Milisec
Value EXECUTIONS Hours Processed PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- ---------- ---- ---- ----------- ---------- -------- ----------
2491926841 1 1.3 5,352,381 5 96 13,134,981 2,651,076 0 2
 Reduced elapsed time by 94%, Logical IO by 63% and Physical IO by 59%
 But increased IOPS by 6x => ensure that disk subsystem can handle it !!!

Use Case 2 – Find sql & stats
Top 3 SQL for the time period
------------- ---------- ---------- ----------
dvaxm0c9tdj80 INSERT 11 3.9
77uh2gt7ytnm9 SELECT 9 3.2
12dy8wkk6x5b5 INSERT 7 2.6
Stats for SQL dvaxm0c9tdj80
Plan CPU IO Milisec
Hash Elapsed Rows time WAIT Direct /
DAY Value Executions Minutes processed* PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------
2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 17
2011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 35
2011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 13
2011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 21
2011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32
* Should be 0

Use Case 2 – SQL details
SQL Plan
SQL>select * from table(dbms_xplan.display_awr('dvaxm0c9tdj80', 4061473922, null, 'ALL
LAST’));
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 24155 (100)| |
| 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |
-------------------------------------------------------------------------------
SQL Text
INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
(ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
SELECT ROW_ID, 1, 'I', LAST_UPD
FROM S_PARTY
WHERE S_PARTY.LAST_UPD > :1;
SQL Rowsource Stats
SQL_ID SQL_PLAN_LINE_ID EVENT PLAN_LINE_EVENT_PCT MINUTES
------------- ---------------- ------------------------- ------------------- ----------
dvaxm0c9tdj80 2 db file sequential read 100 235
dvaxm0c9tdj80 2 CPU 0 1

Use Case 2 – Index Structure
SQL Text
INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
(ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
SELECT ROW_ID, 1, 'I', LAST_UPD
FROM S_PARTY
WHERE S_PARTY.LAST_UPD > :1;
INDEX FULL SCAN | S_PARTY_W1
 IO is 99% of elapsed time
 Using perfect index, but wrong
access method
 Why does the CBO choose this
inefficient plan ?!
 Non-representative stats?
 Incorrect optimizer settings?
 Or something else?
*SQL> @i S_PARTY
Type Unq? Part? Index name STATUS Column name DATA_TYPE
------ ---- ------ ----------- -------- -------------- ---------
NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2
S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR
… snip …
NO NO S_PARTY_W1 VALID LAST_UPD DATE
VALID ROW_ID VARCHAR2
*Output based on modified version of Tim Gorman’s script at www.evdbt.com/tool.htm

Use Case 2 – Bad stats?
SQL>select * from table(dbms_xplan.display_awr('dvaxm0c9tdj80', 4061473922, null,
'ALL LAST’));
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 24155 (100)| |
| 1 | INDEX FULL SCAN | S_PARTY_W1 | 911K| 17M| 24155 (1)| 00:04:50 |
-------------------------------------------------------------------------------
Stats for SQL dvaxm0c9tdj80
Plan CPU IO milisec
Hash Elapsed Rows time WAIT Direct /
DAY Value Executions Minutes processed* PCT PCT LIO PIO Writes PIO
---------- ---------- ---------- -------- ---------- ---- ---- ----------- ---------- -------- --------
2011-06-29 4061473922 1 118 9,289 1 99 449,940 406,585 0 17
2011-06-30 4061473922 1 236 6,754 0 100 408,245 406,481 0 35
2011-07-01 4061473922 1 89 352 1 99 406,564 406,349 0 13
2011-07-02 4061473922 1 145 7,383 1 99 410,586 406,358 0 21
2011-07-03 4061473922 1 220 13,944 1 100 413,684 406,027 0 32
* Should be 0

Test Case Builder
• Reliplcate objects in non-prod

Use Case 2 – Replicate (1st attempt)
SQL>explain plan for
2 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
3 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
4 SELECT ROW_ID, 1, 'I', LAST_UPD
5 FROM S_PARTY
6 WHERE S_PARTY.LAST_UPD > :x;
Explained.
SQL>select * from table(dbms_xplan.display(null, null, 'ALLSTATS LAST'));
… snip …
------------------------------------------------------------
| Id | Operation | Name | E-Rows |
------------------------------------------------------------
| 0 | INSERT STATEMENT | | 913K|
| 1 | LOAD TABLE CONVENTIONAL | S_ETL_I_IMG_70 | |
|* 2 | INDEX RANGE SCAN | S_PARTY_W1 | 913K|
------------------------------------------------------------
---------------------------------------------------
2 - access("S_PARTY"."LAST_UPD">:X)
Object Statistics History
 WRI$_OPTSTAT_TAB_HISTORY
 WRI$_OPTSTAT_IND_HISTORY
 Etc.

Use Case 2 – bind data type
SQL>select distinct name, was_captured, last_captured
2 , sys.anydata.gettypename(value_anydata) as data_type
3 , sys.anydata.accesstimestamp(value_anydata) as data_value
4 from DBA_HIST_SQLBIND where sql_id = 'dvaxm0c9tdj80’ order by last_captured;
was
NAME captured LAST_CAPTURED DATA_TYPE DATA_VALUE
----- -------- ------------------------- -------------------- --------------------
:1 YES 2011-06-29 17:51:44 SYS.TIMESTAMP 2011-06-28 08:12:58
Type Unq? Part? Index name STATUS Column name DATA_TYPE
------ ---- ------ ----------- -------- -------------- ---------
NORMAL NO NO S_PARTY_F1 VALID PAR_PARTY_ID VARCHAR2
S_PARTY_M1 VALID ROOT_PARTY_FLG CHAR
… snip …
NO NO S_PARTY_W1 VALID LAST_UPD DATE
VALID ROW_ID VARCHAR2

Use Case 2 – Replicate (2nd attempt)
SQL>declare
2 x timestamp;
3 begin
4 INSERT /*+APPEND*/ INTO S_ETL_I_IMG_70
5 (ROW_ID, MODIFICATION_NUM, OPERATION, LAST_UPD)
6 SELECT ROW_ID, 1, 'I', LAST_UPD
7 FROM S_PARTY
8 WHERE S_PARTY.LAST_UPD > :x;
9 end;
10 /
----------------------------------------------------------
| Id | Operation | Name | E-Rows |
----------------------------------------------------------
| 0 | INSERT STATEMENT | | |
| 1 | LOAD TABLE CONVENTIONAL | | |
|* 2 | INDEX FULL SCAN | S_PARTY_W1 | 913K|
----------------------------------------------------------
---------------------------------------------------
2 - filter(INTERNAL_FUNCTION("S_PARTY"."LAST_UPD")>:B1)

Use Case 2: Visualizing the
improvement
0
1
2
3
4
5
6
7
8
9
10
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
ElapsedTime(Hours)
ETL Day
IMG Process Duration

Free script collections
• Tanel Poder
• tech.e2sn.com/oracle-scripts-and-tools
• Kerry Osborne -
• kerryosborne.oracle-guy.com/papers/my_favorite_scripts_2010.zip
• Tim Gorman
• evdbt.com/tools.htm
• Adrian Billington
• oracle-developer.net/utilities.php
• Jonathan Lewis
• jonathanlewis.wordpress.com/2009/12/18/simple-scripts/

SQL Tuning
– Trace files
– SQLT output (MOS Doc: 215187.1)
– Trace Analyzer (MOS Doc :
224270.1)
– AWR Report (MOS Doc : 748642.1)
– AWR SQL Report (awrsqrpt.sql)
– 11g Real-Time SQL Monitoring
– SQL Tuning Advisor
PL/SQL Tuning
– Product logs
– PL/SQL Profiler (MOS Doc :
808005.1)
Middletier Tuning
– JVM Logs
– Third Party Tools
Forms Tuning
– Forms Tracing (MOS Doc : 373548.1)
– Generic MOS Doc : 438652.1
Reports Tracing
– MOS Doc: 111311.1
Database Tuning
– AWR Report (MOS Doc : 748642.1)
– ADDM report (MOS Doc : 250655.1)
– Active Session History (ASH) Report
– LTOM output (MOS Doc : 352363.1)
OS
– OSWatcher (MOS Doc : 301137.1)
Resources

To learn even more
• Performance Fundamentals for Oracle Database 10g and 11g
by Graham Wood and John Beresniewicz
(http://www.oracle.com/technetwork/database/focus-areas/manageability/db-perf-tuning-ow08-131582.pdf)
• ASH website by Kyle Hailey
(https://sites.google.com/site/embtdbo/wait-event-documentation/ash---active-session-history)
• Expert Oracle Database Architecture: Oracle Database 9i, 10g,
and 11g Programming Techniques and Solutions by Thomas
Kyte
• Troubleshooting Oracle Performance by Christian Antognini
• Blogs
– Tanel Poder, Kerry Osborne, Jonathan Lewis, Cary Millsap, Greg
Rahn, Randolf Geist, Craig Shallahamer, Maria Colgan, etc.

Summary
• Ask questions
• Understand the database internals
• Use multiple tools depending on the situation
• Combine the best of cmd line and GUI (i.e. PerfSheet
from Tanel Poder)
• Don’t forget the human/organizational aspects

Q&A

Barun_Practical_and_Efficient_SQL_Performance_Tuning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Barun_Practical_and_Efficient_SQL_Performance_Tuning

Similar to Barun_Practical_and_Efficient_SQL_Performance_Tuning (20)

Barun_Practical_and_Efficient_SQL_Performance_Tuning