The Practical Use of Active Session History

/*Go-Faster*/ Consultancy
PRACTICAL USE OF
ACTIVE SESSION
HISTORY
THE
ASH MAN
COMETH!

/*Go-Faster*/ Consultancy
PRACTICAL USE
OF ACTIVE
SESSION HISTORY
DAVID KURTZ
UKOUG2024 /*+Go-Faster*/ Consultancy

David Kurtz, /*+Go-Faster Consultancy*/ © 2024 www.go-faster.co.uk
WHO AM I
Go-Faster Consultancy
• Performance tuning
• Oracle RDBMS
• PeopleSoft ERP
• www.go-faster.co.uk
• blog.go-faster.co.uk
• blog.psftdba.com
Oak Table
@go-faster.co.uk

Performance
ASH Fundamentals
• What is it?
• How does it work?
Practical Examples
• Curated from real-life
AGENDA
Mining the ASH data
• ASH in Performance Tools
• Instrumentation
• Profiling DB Time
• Adaptive Plans
• Literals & Binds
• RAC
• Blocking Lock Analysis
• Temporary Tablespace/PGA Usage
• Index Usage
• Plan Stability
• Transactions & Executions
• Effective Parallelism
• Sources of physical I/O
• Things that can go wrong.

FOLLOW THE MONEY
TIME = MONEY

FOLLOW THE TIME
TIME = MONEY

How much time did I spent
doing what?
What can I do about it?
THERE IS ONLY ONE QUESTION IN
PERFORMANCE OPTIMISATION
Often, the answer is to stop
doing it

ASH FUNDAMENTALS
“Sir Charles had evidently stood there for five or ten minutes."
"How do you know that?"
"Because the ash had twice dropped from his cigar."
"Excellent! This is a colleague, Watson, after our own heart.”

A number of slides in this opening section have been
lovingly ripped off from various presentations by
Graham Wood on the subject of ASH.
• Sifting through the ASHes (2004)
https://www.oracle.com/technetwork/database/manag
eability/ppt-active-session-history-129612.pdf
• ASH Architecture and Advanced Usage (2019) https://
youtu.be/rxQkvXIY7X0
• Database Time-Based Performance Tuning: From
Theory to Practice (2015) https://
youtu.be/Aknb_iZrPbc
John Beresniewicz
• Database Time Fundamentals (2020) https://
youtu.be/jKZXGWuLe-k
• DB Time basics (2020) https://youtu.be/xZViQXZPzx8
FURTHER READING

ACTIVE SESSION HISTORY IS A SAMPLING
TECHNOLOGY
Sampling technology introduced in 10g
• Built into the database, so it is always on
• Low overhead, so still works well if system heavily loaded
• No locking or latching
• Only sessions that are currently active are collected by the
sampling process.
• Samples collected in memory
• 1 in 10 samples persisted to AWR

PROCESS STATES
time
Query Fetch Update Commit
Client Process
6s
10.25s
4.25s

ASH IS A SAMPLING TECHNOLOGY
time
sessions
4.5s
4.5s
.5s .5s .5s .5s .5s .5s .5s .5s .5s
2s 2s
2s 1.5s
1.5s
1.25s 1.25s
5s
5s
6s
6s
4s
1 3 4 3 2 2 1 2 1 3 4 0
0 1 2 3 4 5 6 7 8 9 10 11 12
0
1
2
3
4
5
6
A.A.S.
0 1 2 3 4 5 6 7 8 9 10 11 12
0
1
2
3
4
5
6
A.A.S.

26 ASH Samples ≈ 26s DB Time
Actually 24s
These discrepancies are not
unusual in a small sample set.
They become less significant as
the number of samples increases.
ASH SAMPLING  APPROXIMATION
time
sessions
4.5s
4.5s
.5s .5s .5s .5s .5s .5s .5s .5s .5s
2s 2s
2s 1.5s
1.5s
1.25s 1.25s
5s
5s
6s
6s
4s
1 3 4 3 2 2 1 2 1 3 4 0

= Active Sessions
The number of active sessions at any time is the rate
of change of the “DB Time function” at that time.
ASH CALCULUS
DB Time is the integral of the Active Sessions
function
As →

Where
• Δt = duration between samples
• S is the set of ASH samples
DB Time is approximately the interval of time
between each sample multiplied by the number of
samples
ASH MATHEMATICS
Where
• Δt = duration between ASH samples
In v$active_session_history each ASH sample
represents about 1 second.
In DBA_HIST_ACTIVE_SESS_HISTORY each ASH
sample represents about 10 seconds.

Recent
SELECT …
, SUM(1) ash_secs
FROM v$active_session_history
WHERE …
GROUP BY …
ORDER BY ash_secs DESC, …
OLD ASH SQL
Historical
SELECT …
, SUM(10) ash_secs
FROM dba_hist_active_sess_history
WHERE …
GROUP BY …

Recent
SELECT …
, SUM(usecs_per_row)/1e6 ash_secs
FROM v$active_session_history
WHERE …
GROUP BY …
usecs_per_row = µs of database time per row
~1.02s/row
(documented from 21c)
ASH SQL
NEW COLUMN FROM ORACLE 12.2
Historical
SELECT …
WHERE …
GROUP BY …
~10.2s/row

Circular buffer
in SGA
2Mb / CPU
ASH ARCHITECTURE
AWR
Session
State
Objects
Session
State
Objects
Session
State
Objects
MMON lite
(MMNL)
v$session
v$session_wait
v$active_session_history DBA_HIST_ACTIVE_SESS_HISTORY
AWR
snapshot
Readers
unlatched
Indexed
on time
Writer goes in
one direction
Readers go in
other direction
1 in 10 samples on AWR
snapshot or out-of-space
Variable length
records

ASH is licenced as a part of the Diagnostics Pack
Only available on Enterprise Edition
• Database: $47,500/processor*
• Diagnostics: $7,500/processor*
• ⪅16% of license
What if you don’t have the licence?
What if you have Standard Edition
*Oracle Technology Global Price List, June 13, 2024
ASH LICENCING
Conceptually, ASH is sampling and storing
v$session
OraSASH or Oracle Simulation ASH
• Open source project
• http://pioro.github.io/orasash/
• Originated by Kyle Hailey
• Maintained by Marcin Przepiorowski
• PL/SQL procedure samples x$ksuse
• Run from database scheduler
• Writes persisted data across database link to
another database to minimize measurement
intrusion effect.
• Query the data in much the same way as real
ASH (as if on Oracle 10g).

ASH
• Licensed
• Always there
• No marginal cost
• Real Time, Proactive
• Statistical data
• Who is blocking me?
• SQL and plan if captured by AWR
• Estimate of duration
• Per wait event
• Per Operation
ASH –V- SQL*TRACE
SQL*Trace
• Free
• Enable, Collect file, Profile
• Run-time overhead
• Reactive
• Every SQL & database call
• Being Blocked
• Actual SQL and execution plan
• Exact duration
• Individual waits on events
• Operations in Plan

ASH can be used to resolve most of your
performance issues.
Sometimes, you will still need SQL*Trace
ASH –V- SQL*TRACE
From 12.2 you can also just query your trace
files.
• v$diag_trace_file
• v$diag_trace_file_contents
See also
• Tim Hall’s Oracle-Base blog: SQL trace, 10046,
trcsess and tkprof in Oracle
• https://
oracle-base.com/articles/misc/sql-trace-10046-
trcsess-and-tkprof#query_contents_of_trace_fi
les

ASH FUNDEMENTALS
QUESTIONS?

ASH: A DATA WAREHOUSE OF PERFORMANCE
METRICS

• Session
• Session ID and serial
• query coordinator
• Container
• Consumer Group ID
• in flags
connection, parse, exec, bind, PL/SQL, Java,
close, sequence, inmemory, tablespace
encryption
• Wait
• Class, event id, name and parameters
• Blocking session ID, instance and status
ASH DIMENSIONS & ATTRIBUTES
• Application
• program, machine, port,
• module, action, client_id, ECID …
• SQL
• SQL_ID, Op Code, plan hash, plan line id,
top level SQL ID, full plan hash value,
exec ID
• Transaction ID
• force matching signature
• PGA and Temp usage
• Object
• object, file, block and row number

WHERE IS ASH REPORTED?
Enterprise Manager / Enterprise Manager Express
AWR Report
• including ASH Report within AWR Report
ASH Report
eDB360 / SQLD360

Tooltips with
mouse over
Active Sessions
graph by attribute
Top attribute
profile
Choose Time
Window
Top Application
Modules from
Instrumentation

ASH Report in
AWR Report
For pair of AWR
snapshots

Specific time
window

ASH DATA IN OEM

ASH DATA IN OEM
You can run ASH reports via EM
Filter by
Module

These processes were responsible for
86% of total DB activity
Average 14.8 active sessions (out 32
processes)
If I go on I get SQL statements
But I don’t get execution plans.
EXAMPLE ASH REPORT

Open Source Tool
• Carlos Sierra & Mauro Pagano
• https://carlos-sierra.net/category/sqld360/
• https://mauro-pagano.com/
• https://github.com/sqldb360/sqldb360
Zero install footprint
• All SQL scripts -> HTML & Google Charts
ASH also extracted to CSV
• External table: https://
blog.go-faster.co.uk/2019/10/reading-active-sess
ion-history.html
EDB360/SQLD360

Keep it
• 8-42 days locally
• Copy to central repository
• Retain for 400 days
• awrextr.sql / awrload.sql
• OEM 12c4 manage this
• Compare different databases
• Eg. artefacts common to multiple databases
might indicate an infrastucture issue?
DATA WAREHOUSE OF PERFORMANCE
METRICS
Multi-tenant database
• By default AWR snapshots run in CDB and
flushes ASH for all PDBS
• Can configure AWR snapshots in individual
PDBs
• if higher snapshot frequency required.
• Double capture of data.
• In PDB
• Can only see ASH from own PDB
• In CDB
• Can see ASH for all PDBs

INSTRUMENTATION, INSTRUMENTATION,
INSTRUMENTATION.

APPLICATION INSTRUMENTATION
It is essential to be able to match
• database sessions
• application processes
DBMS_APPLICATION_INFO
• set_module, set_action
• Set session module and action attributes
• Picked up by ASH, AWR and session trace
• Put calls in application
• Few packaged application vendors do this

PL/SQL INSTRUMENTATION
k_module CONSTANT VARCHAR2(64) := $$PLSQL_UNIT;
…
PROCEDURE my_procedure IS
l_module VARCHAR2(64);
l_action VARCHAR2(64);
BEGIN
dbms_application_info.read_module(module_name=>l_module
,action_name=>l_action);
dbms_application_info.set_module(module_name=>k_module
,action_name=>'MY_PROCEDURE');
…
…
…
dbms_application_info.set_module(module_name=>l_module
EXCEPTION
WHEN … THEN
dbms_application_info.set_module(module_name=>l_module
RAISE / EXIT
…
END my_procedure;

DIY INSTRUMENTATION
You may need to be creative!
• PeopleTools: When a process starts (and sets its own status), I
have a trigger that calls a PL/SQL API that sets module and action
CREATE OR REPLACE TRIGGER sysadm.psftapi_store_prcsinstance
BEFORE UPDATE OF runstatus ON sysadm.psprcsrqst FOR EACH ROW
WHEN ((new.runstatus IN('3','7','8','9','10') OR old.runstatus IN('7','8')) AND
new.prcstype != 'PSJob')
BEGIN
…
psftapi.set_action(p_prcsinstance=>:new.prcsinstance
,p_runstatus=>:new.runstatus, p_prcsname=>:new.prcsname);
…
EXCEPTION WHEN OTHERS THEN NULL; --do not crash the scheduler
END;
/

NO INSTRUMENTATION!

WITH INSTRUMENTATION!

Map Session Attributes to Resource Consumer
Group
• Combinations of
• SERVICE, MODULE, ACTION,
• Also
• Oracle User, Client Program, OS User, Client ID
Consumer Group ID recorded on ASH record
RESOURCE MANAGER
MODULE / ACTION
• Session  ASH  AWR
• SQL Stats  AWR
• Consumer Group Mapping
• Automatically Enable/Disable SQL*Trace

INSTRUMENTATION, INSTRUMENTATION,
INSTRUMENTATION.
QUESTIONS

ASH MINING

WHAT ARE YOU LOOKING FOR?
You need a clear idea of the question you are asking of the
ASH data.
What are you interested in?
• Time Window
• Recent –v- Historical
• Single Session / Group of Sessions / Whole Database
• All ASH Data / One Event / One SQL ID / One Plan
• Related ASH data (sessions blocked by lock)

PROFILE DB TIME BY DIFFERENT ATTRIBUTE
TO FIND TOP SQLS
• SQL_ID
• FORCE_MATCHING_SIGNATURE – to treat literals as binds
• TOP_LEVEL_SQL_ID – PL/SQL call
• SQL_PLAN_HASH_VALUE
• SQL_PLAN_LINE_ID
• SQL_FULL_PLAN_HASH_VALUE – adaptive plan
• EVENT_CLASS
• EVENT
• IN_PARSE, IN_HARD_PARSE, IN_BIND, IN_SQL_EXECUTION, IN_PLSQL_EXECUTION,
IN_CURSOR_CLOSE

QUERYING ASH REPOSITORY
DBA_HIST_ACTIVE_SESS_HISTORY
• WRH$_ACTIVE_SESSION_HISTORY partitioned on DBID and SNAP_ID
DBA_HIST_SNAPSHOT
• WRM$_SNAPSHOT

QUERYING ASH REPOSITORY
SELECT /*+LEADING(x) USE_NL(h)*/
…
FROM dba_hist_active_sess_history h
, dba_hist_snapshot x
WHERE x.snap_id = h.snap_id
AND x.dbid = h.dbid
AND x.instance_number = h.instance_number
AND x.end_interval_time >= …
AND x.begin_interval_time <= …
AND h.sample_time BETWEEN … AND …
AND …
GROUP BY …

EXAMPLE
SELECT /*+LEADING(x h) USE_NL(h)*/
h.sql_id
h.sql_plan_hash_value
FROM dba_hist_snapshot x
, dba_hist_active_sess_history h
WHERE x.end_interval_time
>= TO_DATE('202411010730','yyyymmddhh24mi')
AND x.begin_interval_time
<= TO_DATE('202411010830','yyyymmddhh24mi')
AND h.sample_time BETWEEN TO_DATE('202411010730','yyyymmddhh24mi')
AND TO_DATE('202411010830','yyyymmddhh24mi')
AND h.SNAP_id = X.SNAP_id
AND h.dbid = x.dbid
AND h.instance_number = x.instance_number
AND h.module like 'PSAPPSRV%'
GROUP BY h.sql_id, h.sql_plan_hash_value
ORDER BY ash_secs DESC
/
Application not
instrumented
Can only filter by Module
Snapshots that intersect with the
period for which process was running
ASH Data for period for
which process was running

TOP SQL
SQL Plan
SQL_ID Hash Value ASH_SECS
------------- ---------- ----------
7hvaxp65s70qw 1051046890 1360
fdukyw87n6prc 313261966 760
8d56bz2qxwy6j 2399544943 720
876mfmryd8yv7 156976114 710
bphpwrud1q83t 3575267335 690
…

GET EXECUTION PLAN FROM AWR
SELECT * from table(
dbms_xplan.display_workload_repository(
'7hvaxp65s70qw', 1051046890, 'ADVANCED')
);

GET EXECUTION PLAN FROM LIBRARY CACHE
dbms_xplan.display_cursor(
'7hvaxp65s70qw', 0, 'ADVANCED')
);

USE SQL QUERY TO GENERATE
CODE TO OBTAIN EXECUTION PLAN
SELECT DISTINCT
'SELECT * from table( dbms_xplan.display_cursor('''
||sql_id ||''',' ||sql_child_number ||',''ADVANCED''));'
FROM (
…
)

WHICH PART OF EXECUTION PLAN
CONSUMED THE MOST TIME?
SELECT …
, h.sql_plan_line_id
, sum(usecs_per_row)/1e6 ash_secs
…
WHERE …
AND h.sql_id = 'a47fb0x1b23jn'
GROUP BY …
, h.sql_plan_line_id
ORDER BY ASH_SECS DESC

PRCSINSTANCE SQL_PLAN_HASH_VALUE SQL_PLAN_LINE_ID ASH_SECS
------------ ------------------- ---------------- ----------
4945802 483167840 25 2410
483167840 24 1190
483167840 26 210
483167840 20 190
483167840 21 30
483167840 16 20
483167840 23 10
483167840 22 10
483167840 18 10
483167840 10
483167840 7 10

Plan hash value: 483167840
---------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------------------------------
…
| 14 | NESTED LOOPS | | | | | | Q1,04 | PCWP | |
| 15 | NESTED LOOPS | | 3988 | 669K| 113K (1)| 00:06:08 | Q1,04 | PCWP | |
| 16 | HASH JOIN SEMI | | 3851 | 481K| 112K (1)| 00:06:05 | Q1,04 | PCWP | |
| 17 | PX RECEIVE | | 3771K| 233M| 61175 (1)| 00:03:19 | Q1,04 | PCWP | |
| 18 | PX SEND HASH | :TQ10003 | 3771K| 233M| 61175 (1)| 00:03:19 | Q1,03 | P->P | HASH |
| 19 | PX BLOCK ITERATOR | | 3771K| 233M| 61175 (1)| 00:03:19 | Q1,03 | PCWC | |
| 20 | TABLE ACCESS FULL | PS_CM_DEPLETE | 3771K| 233M| 61175 (1)| 00:03:19 | Q1,03 | PCWP | |
| 21 | BUFFER SORT | | | | | | Q1,04 | PCWC | |
| 22 | PX RECEIVE | | 6058K| 364M| 50906 (1)| 00:02:46 | Q1,04 | PCWP | |
| 23 | PX SEND HASH | :TQ10001 | 6058K| 364M| 50906 (1)| 00:02:46 | | S->P | HASH |
| 24 | INDEX FULL SCAN | PS_CM_DEPLETE_COST | 6058K| 364M| 50906 (1)| 00:02:46 | | | |
| 25 | INDEX UNIQUE SCAN | PS_TRANSACTION_INV | 1 | | 1 (0)| 00:00:01 | Q1,04 | PCWP | |
| 26 | TABLE ACCESS BY INDEX ROWID| PS_TRANSACTION_INV | 1 | 44 | 1 (0)| 00:00:01 | Q1,04 | PCWP | |
…
---------------------------------------------------------------------------------------------------------------------------------------

PROFILE BY LINE AND EVENT
--------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 318 (100)| |
| 1 | LOAD TABLE CONVENTIONAL | | | | | |
| 2 | FILTER | | | | | |
| 3 | TABLE ACCESS BY INDEX ROWID| PS_PROJ_RESOURCE | 724 | 337K| 314 (0)| 00:00:04 |
| 4 | INDEX RANGE SCAN | PSEPROJ_RESOURCE | 724 | | 9 (0)| 00:00:01 |
| 5 | INDEX RANGE SCAN | PS_PRJRE_SBF_TMP | 1 | 6 | 4 (0)| 00:00:01 |
--------------------------------------------------------------------------------------------------
ASH
SQL_PLAN_LINE_ID EVENT Secs
---------------- ------------------------------------------ --------
3 db file sequential read 40320
3 read by other session 12280
1 10680
3 2800
db file sequential read 1560
520
4 240
4 read by other session 200
read by other session 80
--------
sum 74200

ASH MINING
QUESTIONS?

LITERALS –V – BIND VARIABLES
NON-SHARABLE SQL

BIND VARIABLES
Developers Should Use Them!
Unfortunately…

Bind Variables
• Frequently executed SQL statement
• Similar (small) amount of work
• Same execution plan usually suitable for all.
• Overhead of SQL optimization is significant
compared to that of execution
• Share Child Cursor
• Typical for OLTP
• In general, preferred approach
• Same SQL_ID
• https://jonathanlewis.wordpress.com/2009/05/06/
philosophy-1/
"HISTOGRAMS & BIND VARIABLES EXIST FOR
DIAMETRICALLY OPPOSED REASONS"*
Histograms
• Large and highly variable amounts of work
• Need different execution plans
• Overhead of SQL optimisation is small
compared to execution.
• Histograms may be beneficial
• Using literals values (rather than bind
variables) might be preferable.
• Different SQL_IDs
• Creates additional ASH challenges

NON-SHAREABLE OLTP SQL
SQL_ID djqf1zcypm5fm
--------------------
SELECT …
FROM ps_tl_exception a,
ps_personal_data b,
ps_perall_sec_qry b1,
…
WHERE b.emplid = b1.emplid
AND b1.oprid = '12345678'
…

MANY SQL_IDS
SAME EXECUTION PLAN
PRCSINSTANCE SQL_ID SQL_PLAN_HASH_VALUE ASH_SECS
------------ ------------- ------------------- ----------
50002824 0 50
50002824 2ybtak62vmx58 2262951047 20
50002824 ck3av6cnquwfc 2262951047 20
50002824 gvys6kd9fqn7u 2262951047 20
50002824 7ymcbn6q8utj8 2262951047 10
50002824 9qud2n3qq7nzr 2262951047 10
50002824 6pxvns97m1fua 2262951047 10
50002824 5ngqj5zg8vbz8 2262951047 10
50002824 9zp6nndfvn66b 2262951047 10
50002824 15kfs3c3005xm 2262951047 10
50002824 4qvhpygc7cq2t 2262951047 10
50002824 23yc8dcz9z4yj 2262951047 10
50002824 bn8xczrvs2hpr 2262951047 10
50002824 9g6k9dnrjap08 2262951047 10
50002824 1art8dhzbvpwt 2262951047 10
50002824 6gqj337xnr5y4 2262951047 10
…

FORCE_MATCHING_SIGNATURE
• Treat literals as binds
GROUP BY ?
SQL_PLAN_HASH_VALUE
• But sometimes you get differences in the
SQL that produce different FMS but result
in same execution plan
• Different number of terms in IN()
• Additional criteria, usually on
unindexed columns
• If two statements are similar enough to
produce the same execution plan,
• Then they are likely to experience the
same challenges and benefit from the
same optimisations.

AGGREGATE BY
SQL_PLAN_HASH_VALUE
PRCSINSTANCE SQL_PLAN_HASH_VALUE ASH_SECS
------------ ------------------- ----------
50002824 2262951047 2300
50002824 0 60
50002824 3085938243 20
50002824 563410926 10
50002824 1068931976 10

CAN YOU FIND THE SQL IN AWR?
Now Find the SQL with that plan.
If it was captured by AWR
• Lots of parsing causes statements to be aged out of library cache before they get stored
in AWR by a snapshot
• Only Top-n statements are captured.

SELECT h.sql_id, h.sql_plan_hash_value
, sum(10)/1e6 ash_secs
, 10*COUNT(t.sql_id) awr_secs
FROM dba_hist_snapshot X
LEFT OUTER JOIN dba_hist_sqltext t
ON t.sql_id = h.sql_id
WHERE …
GROUP BY h.sql_id
, h.sql_plan_hash_value
, AVG(usecs_per_row)/1e6
*COUNT(t.sql_id) awr_secs
LEFT OUTER JOIN dba_hist_sqltext t
ON t.sql_id = h.sql_id
WHERE …
GROUP BY h.sql_id

WITH x AS (
, avg(usecs_per_row)/1e6*count(t.sql_id) awr_secs
LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id
WHERE …
), y as (
SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value
ORDER BY x.awr_secs DESC, x.ash_secs DESC) ranking
, x.sql_id, x.sql_plan_hash_value
, SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) ash
, SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) awr
, COUNT(DISTINCT sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids
FROM x
)
SELECT * FROM y
WHERE y.ranking = 1
ORDER BY tot_ash_secs desc, ranking

STATEMENTS NOT IN AWR
REPOSITORY
Longest SQL Plan
RANKING SQL_ID Hash Value TOT_ASH_SECS TOT_AWR_SECS SQL_IDS
---------- ------------- ---------- ------------ ------------ ----------
1 1wfhpn9k2x3hq 0 7960 4600 13
1 2wsan9j1pk3j2 1061502179 4230 4230 1
1 bnxddum0rrvyh 918066299 2640 1200 179
1 02cymzmyt4mdh 508527075 2070 0 45
1 5m0xbf7vn8490 2783301143 1700 0 49
1 0jfp0g054cb3n 4135405048 1500 0 47
1 11bygm2nyqh0s 3700906241 1370 0 27
1. Special case. There is no plan because it’s the dbms_stats function. There were 13 statements, but
in reality they were all totally different. PHV=0 indicates PL/SQL or INSERT…VALUES.
2. One SQL, one plan. Either this is a shareable SQL_ID, or it just executed once.
3. This is many statements with the same plan, at least 179, possibly more. 264 samples, representing
2640 seconds of SQL, but only 120 samples for SQLs captured by AWR.
4. Sometimes we don’t capture any of the SQL statements
1
2
3
4

LITERALS –V- BIND VARIABLES
NON-SHAREABLE SQL
QUESTIONS?

ADAPTIVE PLANS

ADAPTIVE EXECUTION PLAN
SELECT sql_full_plan_hash_value, sql_plan_hash_value, sql_adaptive_plan_resolved
, COUNT(DISTINCT sql_exec_id) execs
FROM dba_hist_active_Sess_history
WHERE sql_id = '4dszd9dysry0c'
AND sql_plan_hash_value > 0
GROUP BY dbid, sql_plan_hash_value, sql_full_plan_hash_value, sql_adaptive_plan_resolved
SQL_FULL_PLAN_HASH_VALUE SQL_PLAN_HASH_VALUE SQL_ADAPTIVE_PLAN_RESOLVED EXECS ASH_SECS
------------------------ ------------------- -------------------------- ---------- ----------
4059585501 4114868852 1 274 3050
4059585501 3412983073 1 387 3980
Adaptive Plan
One SQL statement,
two plan hash values,
same full plan hash value

select * from table(dbms_xplan.display('ASH_PLAN_TABLE','4dszd9dysry0c',null ,'dbid=2783210685 and plan_hash_value = 3412983073'));
---------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 13 (100)| |
| 1 | SORT ORDER BY | | 9 | 747 | 13 (8)| 00:00:01 |
| 2 | HASH JOIN | | 9 | 747 | 12 (0)| 00:00:01 |
| 3 | JOIN FILTER CREATE | :BF0000 | 9 | 747 | 12 (0)| 00:00:01 |
| 4 | TABLE ACCESS BY INDEX ROWID BATCHED| PGRELS | 9 | 585 | 5 (0)| 00:00:01 |
| 5 | INDEX RANGE SCAN | LINKSOURCE_201 | 9 | | 3 (0)| 00:00:01 |
| 6 | JOIN FILTER USE | :BF0000 | 122 | 2196 | 7 (0)| 00:00:01 |
| 7 | TABLE ACCESS STORAGE FULL | USERGROUPS | 122 | 2196 | 7 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------
Note
-----
- this is an adaptive plan

FROM 12C: ALWAYS SPECIFY +ADAPTIVE
dbms_xplan.display_workload_repository(
'7hvaxp65s70qw', 1051046890,
'ADVANCED +ADAPTIVE')
);
From 12c, always use +ADAPTIVE so
that plan line numbers match
SQL_PLAN_LINE_ID
From 19c, +ADAPTIVE also generated a
Query Block Registry section

select * from table(dbms_xplan.display('ASH_PLAN_TABLE','4dszd9dysry0c','+ADAPTIVE' ,'dbid=2783210685 and plan_hash_value = 3412983073'));
------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
| 1 | SORT ORDER BY | | 9 | 747 | 13 (8)| 00:00:01 |
| 2 | HASH JOIN | | 9 | 747 | 12 (0)| 00:00:01 |
|- 4 | NESTED LOOPS | | 9 | 747 | 12 (0)| 00:00:01 |
|- 5 | NESTED LOOPS | | | | | |
|- 6 | STATISTICS COLLECTOR | | | | | |
|- 9 | INDEX UNIQUE SCAN | SYS_C008784 | | | | |
|- 10 | TABLE ACCESS BY INDEX ROWID | USERGROUPS | 1 | 18 | 7 (0)| 00:00:01 |
| 11 | JOIN FILTER USE | :BF0000 | 122 | 2196 | 7 (0)| 00:00:01 |
| 12 | TABLE ACCESS STORAGE FULL | USERGROUPS | 122 | 2196 | 7 (0)| 00:00:01 |
------------------------------------------------------------------------------------------------------------
Note
-----
- this is an adaptive plan (rows marked '-' are inactive)

select * from table(dbms_xplan.display('ASH_PLAN_TABLE','4dszd9dysry0c','+ADAPTIVE' ,'dbid=2783210685 and plan_hash_value = 4114868852'));
------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
| 1 | SORT ORDER BY | | 8 | 664 | 13 (8)| 00:00:01 |
|- 2 | HASH JOIN | | 8 | 664 | 12 (0)| 00:00:01 |
| 4 | NESTED LOOPS | | 8 | 664 | 12 (0)| 00:00:01 |
| 5 | NESTED LOOPS | | | | | |
|- 6 | STATISTICS COLLECTOR | | | | | |
| 9 | INDEX UNIQUE SCAN | SYS_C008784 | | | | |
| 10 | TABLE ACCESS BY INDEX ROWID | USERGROUPS | 1 | 18 | 7 (0)| 00:00:01 |
|- 11 | JOIN FILTER USE | :BF0000 | 122 | 2196 | 7 (0)| 00:00:01 |
|- 12 | TABLE ACCESS STORAGE FULL | USERGROUPS | 122 | 2196 | 7 (0)| 00:00:01 |
------------------------------------------------------------------------------------------------------------
Note
-----
- this is an adaptive plan (rows marked '-' are inactive)

PROFILING ADAPTIVE PLANS
SELECT sql_plan_hash_value
, sql_full_plan_hash_value
, sql_plan_line_id
, sql_adaptive_plan_resolved
WHERE sql_id = '4dszd9dysry0c'
GROUP BY
sql_plan_hash_value
, sql_full_plan_hash_value
, sql_plan_line_id
, sql_adaptive_plan_resolved
ORDER BY 1,2,4
/

PROFILING ADAPTIVE PLANS
SQL_PLAN_HASH_VALUE SQL_FULL_PLAN_HASH_VALUE SQL_PLAN_LINE_ID SQL_ADAPTIVE_PLAN_RESOLVED ASH_SECS
------------------- ------------------------ ---------------- -------------------------- ----------
3412983073 4059585501 1 1 60
3412983073 4059585501 2 1 1540
3412983073 4059585501 3 1 80
3412983073 4059585501 7 1 50
3412983073 4059585501 8 1 50
3412983073 4059585501 12 1 750
3412983073 4059585501 1 560
4114868852 4059585501 0 10
4114868852 4059585501 1 1 230
4114868852 4059585501 2 1 20
4114868852 4059585501 3 1 200
4114868852 4059585501 4 1 40
4114868852 4059585501 7 1 140
4114868852 4059585501 8 1 150
4114868852 4059585501 9 1 70
4114868852 4059585501 10 1 800
4114868852 4059585501 1 1180
SQL Plan Line IDs
correspond to
+ADAPTIVE plan

FROM 19C: USE
DBMS_XPLAN.DISPLAY_WORKLOAD_REPOSITORY
Officially, DISPLAY_AWR is deprecated in 19c
dbms_xplan.display_workload_repository
('7hvaxp65s70qw', 1051046890, 'ADVANCED +ADAPTIVE');

ADAPTIVE PLANS
QUESTIONS?

ASH & RAC

Each RAC instance
• has its own ASH buffer
• manages its own ASH sampling
• does its own AWR snapshots
ASH & RAC
Consequences
• Different instances start/stop at different times
• SAMPLE_ID does not match across RAC
instances - it is not even close
• ASH samples occur at slightly different times
on different instances
• SAMPLE_TIME does not exactly match
across instances
• microsecond accuracy timestamp
• round it off to sample interval (1s or 10s)
• match within sample interval +/- 0.5s or 5s
• SNAP_ID does match because AWR is run on a
schedule common to all instances

ROUND TIMESTAMP WITH
PL/SQL FUNCTION IN WITH CLAUSE
WITH FUNCTION tsround(p_in IN TIMESTAMP, p_len INTEGER) RETURN timestamp IS
l_date VARCHAR2(20);
l_secs NUMBER;
l_date_fmt VARCHAR2(20) := 'J';
l_secs_fmt VARCHAR2(20) := 'SSSSS.FF9';
BEGIN
l_date := TO_CHAR(p_in,l_date_fmt);
l_secs := ROUND(TO_NUMBER(TO_CHAR(p_in,l_secs_fmt)),p_len);
IF l_secs >= 86400 THEN
l_secs := l_secs - 86400;
l_date := l_date + 1;
END IF;
RETURN TO_TIMESTAMP(l_date||l_secs,l_date_fmt||l_secs_fmt);
END;
x as (
select …
, tsround(sample_time,-1) sample_time
…
…
GROUP BY … tsround(sample_time,-1) …
)
select …
from x
/

LOCK ANALYSIS WITH ASH

WHAT DO YOU DO WHEN
AWR DATA INDICATES LOCKING

WHERE DID WE WAIT ON A LOCK?
h.sql_id
WHERE x.end_interval_time >=TO_DATE('201901261100','yyyymmddhh24mi')
AND x.begin_interval_time <=
TO_DATE('201901261300','yyyymmddhh24mi')
AND h.snap_id = x.snap_id
AND h.dbid = x.dbid
AND h.event = 'enq: TX - row lock contention'
ORDER BY ash_secs desc
/

WHERE DID WE WAIT ON A LOCK?
SQL Plan
------------- ---------- ----------
7qxdrwcn4yzhh 3723363341 26030
652mx4tffq415 1888029394 11230
c9jjtvk0qf649 3605988889 6090
artqgxug4z0f1 8450529 240
gtj7zuzy2b4g6 2565837323 100

STATEMENTS BLOCKED BY TX LOCKS
SQL_ID 7qxdrwcn4yzhh
--------------------
UPDATE PSIBQUEUEINST SET QUEUESEQID=QUEUESEQID+:1
WHERE QUEUENAME=:2
SQL_ID 652mx4tffq415
--------------------
UPDATE PSAPMSGPUBSYNC SET LASTUPDDTTM=SYSDATE
WHERE QUEUENAME=:1
SQL_ID c9jjtvk0qf649
--------------------
UPDATE PSAPMSGSUBCSYNC SET LASTUPDDTTM=SYSDATE
WHERE QUEUENAME=:1

THE REAL QUESTION ABOUT LOCKING:
What is the session that is holding the lock doing while it is holding the lock?
• and can I do something about that?
Home-made sequences are not scalable. Should really be using an Oracle Sequence.
• Not possible in a PeopleSoft Application

RESOLVE THE LOCK CHAIN
Navigating the lock chain works across RAC
instances from 11g.
There may not be any ASH data for the ultimate
blocking session (C) because it is not active on
the database.
Session A
SESSION_ID
SESSION_SERIAL#
BLOCKING_SESSION
BLOCKING_SESSION_SERIAL#
Session C
SESSION_ID
SESSION_SERIAL#
Session B
SESSION_ID
SESSION_SERIAL#
BLOCKING_SESSION
BLOCKING_SESSION_SERIAL#

EXTRACT ASH DATA FOR PERIOD IN
QUESTION
CREATE TABLE my_ash AS
SELECT /*+LEADING(x) USE_NL(h)*/ h.*
WHERE x.end_interval_time >= TO_DATE('201901261100','yyyymmddhh24mi')
AND x.begin_interval_time <= TO_DATE('201901261300','yyyymmddhh24mi')
AND h.snap_id = X.snap_id
AND h.dbid = x.dbid
AND h.instance_number = x.instance_number;
CREATE UNIQUE INDEX my_ash ON my_ash (dbid, instance_number, snap_id, sample_id, session_id, sample_time,
session_serial#) COMPRESS 4;
CREATE INDEX my_ash2 ON my_ash (event, dbid, instance_number, snap_id) COMPRESS 3;
Repeatedly update the blocking instance/session information to the next session along the chain
• Stop when number of updates does not decrease.
• See https://www2.go-faster.co.uk/docs/Practical_ASH.pdf

WHAT ARE THE BLOCKING SESSIONS
DOING (SAME INSTANCE)?
SELECT /*+LEADING(x w) USE_NL(h w)*/
h.sql_id
FROM my_ash w
LEFT OUTER JOIN my_ash h
ON h.snap_id = w.snap_id
AND h.dbid = w.dbid
AND w.blocking_inst_id = w.instance_number
AND h.sample_id = w.sample_id
AND h.sample_time = w.sample_time
AND h.instance_number = w.blocking_inst_id
AND h.session_id = w.blocking_session
AND h.session_serial# = w.blocking_session_serial#
WHERE w.event = 'enq: TX - row lock contention'
/

WHAT ARE THE BLOCKING SESSIONS
DOING (CROSS-INSTANCE)?
SELECT /*+LEADING(x w) USE_NL(h w)*/
h.sql_id
FROM my_ash w
LEFT OUTER JOIN my_ash h
ON h.snap_id = w.snap_id
AND h.dbid = w.dbid
AND h.instance_number = w.blocking_inst_id
AND h.session_id = w.blocking_session
AND h.session_serial# = w.blocking_sessio_serial#
AND (( w.blocking_inst_id = w.instance_number
AND h.sample_id = w.sample_id
AND h.sample_time = w.sample_time)
OR ( w.blocking_inst_id != w.instance_number
AND h.sample_time >= w.sample_time-5/86400
AND h.sample_time < w.sample_time+5/86400))
WHERE w.event = 'enq: TX - row lock contention'
/
Each instance is responsible for it’s own
ASH samples.
Therefore, while the SNAP_IDs are
consistent between different RAC
instances, SAMPLE_IDs will usually be
different.
Every 10th
sample is persisted to AWR,
therefore SAMPLE_TIME could be up to ±5
seconds apart across any 2 instances.
Just because the waiting session was
blocked on it’s instance when sampled, it
doesn’t mean the blocking instance was
still blocking or active on its instance when
sampled!

WHAT ARE THE BLOCKING
SESSIONS DOING?
SQL_ID SQL_PLAN_HASH_VALUE ASH_SECS
------------- ------------------- ----------
292101
5st32un4a2y92 2494504609 106702
652mx4tffq415 1888029394 7030
artqgxug4z0f1 8450529 580
7qxdrwcn4yzhh 3723363341 270
1. This SQL_ID is blank. May not be able to find ASH sample for blocking session because it
is idle – busy on the client not the database.
2. This statement is running while the session holds a lock that is blocking another session.

AWR PLAN COST TRAP

EXECUTION PLAN CAPTURED BY AWR:
CORRECT PLAN, OLD COSTS, OLD BINDS
DISPLAY_AWR()
SQL_ID 5st32un4a2y92
--------------------
SELECT 'X' FROM PS_CDM_LIST WHERE CONTENTID = :1
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
| 1 | INDEX FAST FULL SCAN| PS_CDM_LIST | 1 | 5 | 22 (10)| 00:00:01 |
------------------------------------------------------------------------------------
Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
1 - SEL$1 / PS_CDM_LIST@SEL$1
Peeked Binds (identified by position):
--------------------------------------
1 - :1 (NUMBER): 17776

FRESH EXECUTION PLAN GENERATED BY
EXECUTE EXPLAIN PLAN
Note increase of cost of full scan.
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 6 | 3178 (9)| 00:00:05 |
|* 1 | INDEX FAST FULL SCAN| PS_CDM_LIST | 1 | 6 | 3178 (9)| 00:00:05 |
------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("CONTENTID"=TO_NUMBER(:1))

EXECUTION PLAN CAPTURED BY AWR:
CORRECT PLAN, OLD COSTS, OLD BINDS
The captured plan is a valid guide to the path of execution
Do not rely on costs or number of rows as an indication of the amount of work done.
See also
• http://blog.go-faster.co.uk/2019/09/working-with-awr-old-statistics-in.html
• http://blog.go-faster.co.uk/2019/10/purging-sql-statements-and-execution.html

IS YOUR PARALLELISM EFFECTIVE?

ASH DATA TO RELATE PQ:QC
• Not all PQ processes are active all the time during a parallel operation
• ASH data can relate the Parallel Query Server to Query Coordinator
• Each active PQ and QC process is sampled by ASH
• Therefore, I can see what proportion of PQ process are active.
SQL> desc dba_hist_active_Sess_history
Name Null? Type
----------------------------------------- -------- ----------------------------
…
INSTANCE_NUMBER NOT NULL NUMBER
SAMPLE_ID NOT NULL NUMBER
SAMPLE_TIME NOT NULL TIMESTAMP(3)
SESSION_ID NOT NULL NUMBER
SESSION_SERIAL# NUMBER
SQL_ID VARCHAR2(13)
SQL_EXEC_ID NUMBER
QC_INSTANCE_ID NUMBER
QC_SESSION_ID NUMBER
QC_SESSION_SERIAL# NUMBER
…

RATIO OF DB TIME TO ELAPSED TIME
 EFFECTIVE PARALLELISM
…
END;
x AS (
SELECT sql_id, sql_plan_hash_value
, qc_instance_id, qc_session_id, qc_Session_serial#, sql_exec_id
, avg(usecs_per_row)/1e6*count((DISTINCT tsround(sample_time,-1)) elap_secs
, COUNT(DISTINCT instance_number||session_id||session_serial#) pq_processes
…
WHERE qc_session_id IS NOT NULL
AND sql_exec_id IS NOT NULL
GROUP BY dbid, sql_id, sql_plan_hash_value
, qc_instance_id, qc_session_id, qc_Session_serial#, sql_exec_id)
SELECT x.*
, ash_secs/elap_secs average_parallelism
FROM x
ORDER BY 1,2,3,4;
/

EFFECTIVE PARALLELISM
 HOW MANY PQ PROCESSES ARE BUSY
QC QC QC
SQL Plan Inst Session Session SQL Exec ASH Elap PQ Average
SQL_ID Hash Value ID ID Serial# ID Secs Secs Procs Parallelism
------------- ---------- ---- ------- ------- ---------- ------- ------- ------ -----------
...
24wzn598zadua 140529908 1 3568 9539 16777216 3330 600 33 5.6
2494336853 1 485 3867 16777216 3510 780 33 4.5
2494336853 1 638 24214 16777216 3480 610 33 5.7
2494336853 1 1347 60198 16777216 3590 680 33 5.3
2494336853 1 3199 47602 16777216 3260 600 33 5.4
2494336853 1 5804 45996 16777216 3620 630 33 5.7
...
2nvh42ardn5gg 1436363544 1 713 40067 16777216 3410 30 129 113.7
1436363544 1 1655 36375 16777216 2340 20 128 117.0
1436363544 1 3571 2760 16777216 2760 30 125 92.0
1436363544 1 3682 29887 16777216 3050 30 128 101.7
1436363544 1 6063 36763 16777216 2540 30 123 84.7
...

PL/SQL

SQL_ID –V- TOP_LEVEL_SQL_ID
TOP_LEVEL_SQL_ID is the SQL_ID of the ultimate calling command.
• That may be a PL/SQL call that calls other SQL_IDs
• Including in other PL/SQL functions/procedures
• If just SQL then TOP_LEVEL_SQL_ID = SQL_ID
SELECT /*+leading(r q x h) use_nl(h)*/
NULLIF(h.top_level_sql_id, h.sql_id) top_level_sql_id
, h.sql_id
...
...

SQL_ID –V- TOP_LEVEL_SQL_ID
TOP_LEVEL_SQL SQL_ID Hash Value SQL_IDS PLAN_EXECS PLAN_ASH_SECS
------------- ------------- ---------- ---------- ---------- -------------
0 0 0 210
6np8gdbrmj8s4 2609910643 8 12 160
105xa4pfkv2jz 1dtnz2z7ujv23 3901024798 2 14 140
3m3ubmf7529mh 2188542943 2 13 140
g21xv51r09w4j 2905535923 1 10 100

INVESTIGATING I/O
“Detection is, or ought to be, an exact science and should be treated
in the same cold and unemotional manner”.

ASH FOR SINGLE WAIT EVENT
h.sql_id
WHERE x.end_interval_time <=TO_DATE('202001261100','yyyymmddhh24mi')
AND x.begin_interval_time >=TO_DATE('202001261300','yyyymmddhh24mi')
AND h.dbid = x.dbid
AND h.event = 'db file sequential read'
/

STATEMENTS WITH HIGHEST SINGLE BLOCK
I/O
SQL Plan
------------- ---------- ----------
90pp7bcnmz68r 2961772154 2490
81gz2rtabaa8n 1919624473 2450
7hvaxp65s70qw 1051046890 1320
7fk8raq16ch0u 3950826368 890
9dzpwkff7zycg 2020614776 840
…

WHAT KIND OF SINGLE BLOCK READ?
For I/O wait events ASH reports
• File number
• Block number
• Object number
• Row number (since 11g)
Only valid on physical I/O (mostly db file%) events.
• p1 = file#, p2 = block#
• Not accurate on other events because session information not cleared from previous file
operation.

CATEGORISE TABLESPACES/DATA
FILES
CREATE TABLE my_data_files as
SELECT tablespace_name
, file_id
, CASE
WHEN f.tablespace_name LIKE 'SYS%' THEN 'SYSTEM'
WHEN f.tablespace_name LIKE 'UNDO%' THEN 'UNDO'
WHEN f.tablespace_name LIKE '%IDX%' THEN 'INDEX'
WHEN f.tablespace_name LIKE '%INDEX%' THEN 'INDEX'
ELSE 'TABLE'
END AS tablespace_type
FROM dba_data_files f
/
Materialised DBA_DATA_FILES to working storage table

ASH DATA BY TABLESPACE TYPE
SELECT /*+LEADING(x h) USE_NL(h f)*/
f.tablespace_type
, dmk_data_files f
WHERE x.end_interval_time <=TO_DATE('202002161300','yyyymmddhh24mi')
AND x.begin_interval_time >=TO_DATE('202002161100','yyyymmddhh24mi')
AND h.dbid = x.dbid
AND h.event LIKE 'db file%'
AND h.p1text = 'file#'
AND h.p2text = 'block#'
AND f.file_id(+) = h.p1
GROUP BY f.tablespace_type
/

TABLES ASH_SECS
------ ----------
INDEX 30860
TABLE 26970
UNDO 1370
SYSTEM 490
ASH DATA BY TABLESPACE TYPE
Most time spent on index read
• Includes index maintenance during DML
Not much undo, so not much consistent read.

Need own copy of DBA_OBJECTS
WHICH TABLES ACCOUNT FOR THE
I/O?
CREATE TABLE my_objects
(object_id NUMBER NOT NULL
,owner VARCHAR2(30) NOT NULL
,object_name VARCHAR2(128) NOT NULL
,subobject_name VARCHAR2(30)
,PRIMARY KEY (OBJECT_ID))
/
INSERT INTO my_objects
SELECT object_id, owner, object_name,
subobject_name
FROM dba_objects
WHERE object_type LIKE 'TABLE%'
UNION ALL
SELECT o.object_id, i.table_owner, i.table_name,
o.subobject_name
FROM dba_objects o, dba_indexes i
WHERE o.object_type like 'INDEX%'
AND i.owner = o.owner
AND i.index_name = o.object_name
/

WHICH OBJECTS ARE USED?
o.owner, o.object_name
, dmk_objects o
WHERE x.end_interval_time >= SYSDATE-7
AND x.begin_interval_time <= SYSDATE
AND h.sample_time >= SYSDATE-7
AND h.sample_time <= SYSDATE
AND h.dbid = x.dbid
AND h.current_obj# = o.object_id(+)
GROUP BY o.owner, o.object_name
HAVING SUM(10) >= 3600

WHICH OBJECTS ARE USED?
ASH
OWNER OBJECT_NAME Secs
-------- -------------------- -------
SYSADM PS_TL_RPTD_TIME 800510
SYSADM PS_TL_PAYABLE_TIME 327280
SYSADM PS_GP_RSLT_ACUM 287870
SYSADM PS_SCH_DEFN_DTL 161690
SYSADM PS_SCH_DEFN_TBL 128070
SYSADM PS_GP_RSLT_PIN 124560
SYSADM PS_GP_PYE_PRC_STAT 92410
SYSADM PS_SCH_ADHOC_DTL 88810
…

WHICH PROCESSES READ THIS
TABLE?
SELECT /*+LEADING(x) USE_NL(h)*/
o.owner, o.object_name
, h.module
, dmk_objects o
WHERE x.end_interval_time >= SYSDATE-7
AND x.begin_interval_time <= SYSDATE
AND h.sample_time >= SYSDATE-7
AND h.sample_time <= SYSDATE
AND h.snap_id = X.snap_id
AND h.dbid = x.dbid
AND h.current_obj# = o.object_id
AND o.object_name = 'PS_GP_RSLT_ACUM'
GROUP BY o.owner, o.object_name, h.module
HAVING SUM(10) >= 900

WHICH PROCESSES READ THIS
TABLE?
ASH
OWNER OBJECT_NAME MODULE Secs
-------- -------------------- ---------------- -------
SYSADM PS_GP_RSLT_ACUM XXX_HOL_MGMT 79680
SYSADM PS_GP_RSLT_ACUM DBMS_SCHEDULER 37810
SYSADM PS_GP_RSLT_ACUM SQL*Plus 37060
SYSADM PS_GP_RSLT_ACUM GPGBHLE 30710
SYSADM PS_GP_RSLT_ACUM GPPDPRUN 27440
SYSADM PS_GP_RSLT_ACUM XXX_AE_AB007 21440
SYSADM PS_GP_RSLT_ACUM SQL Developer 11210
SYSADM PS_GP_RSLT_ACUM GPGBEPTD 7240
SYSADM PS_GP_RSLT_ACUM XXX_CAPITA 5850
SYSADM PS_GP_RSLT_ACUM GPGB_PSLIP_X 5030
SYSADM PS_GP_RSLT_ACUM GPGB_EDI 4880
People are writing
ad-hoc queries in
SQL*Plus
Statistics Gathering
Top Application
Process
Payroll Calculation

INVESTIGATING I/O
QUESTIONS

MONITORING TEMPORARY SPACE AND PGA
USAGE WITH ASH

TEMP_SPACE_ALLOCATED
• "Amount of TEMP memory (in bytes) consumed by
this session at the time this sample was taken"
• The manual has always said it is memory, but it
includes physical temp tablespace usage.
USAGE WITH ASH
PGA_ALLOCATED
• "Amount of PGA memory (in bytes) consumed by
this session at the time this sample was taken"

Problem Statement:
• In a Financials system, part of the overnight batch is
split into 4 roughly equally sized processes.
• Every night:
• 2 succeed, 2 fail with:
• "ORA-01652 Unable to extend temp segment by
in tablespace"
• But not the same 2 every time!?
• Rather than simply extended the temporary
tablespace, need to find out what is consuming
temporary table.
WHY SHOULD I BE INTERESTED IN TEMP/PGA?
Instantaneous temporary utilisation recorded in ASH
• DBA_HIST_ACTIVE_SESS_HISTORY.
TEMP_SPACE_ALLOCATED
Profile maximum temporary utilisation
• For particular processes
• link through instrumentation (MODULE/ACTION)
• Within time window of batch
• by SQL statement (SQL_ID)
• by Execution Plan (SQL_PLAN_HASH_VALUE)
Found first two always succeed, last two always fail
• Because first two filled up the temporary
tablespace!

TEMPORARY TABLESPACE USAGE
SELECT /*+leading(r x h) use_nl(h)*/
h.sql_id
, COUNT(DISTINCT sql_exec_id) num_execs
, avg(usecs_per_row)/1e6*count(DISTINCT sample_id) elap_secs
, ROUND(MAX(temp_space_allocated)/1024/1024,0) tempMb
, COUNT(distinct r.prcsinstance) PIs
, sysadm.psprcsrqst r
…
WHERE …
Joined ASH to Application
Scheduler table
through instrumentation

Can see temporary usage of individual SQL statements
SQL_ID SQL_PLAN_HASH_VALUE NUM_EXECS ASH_SECS ELAP_SECS TEMPMB PIS
------------- ------------------- ---------- ---------- ---------- ---------- ----------
a47fb0x1b23jn 483167840 3 6280 910 132 3
cbw2bztjyztnq 544286790 4 5920 390 4
fcrxxp8f0c8cg 2119221636 2 4480 280 2
8h7ga9g761naj 4127129594 1 3980 3980 1
8cypfzadbub4k 4127129594 1 3450 3450 1
3gz46jhw7b5x8 3643021188 8 3190 200 4
a47fb0x1b23jn 3805993318 1 2610 1120 132 1
dxqkbuynhqk09 2119221636 1 2240 140 1
c75jcr5s71s2h 2119221636 1 2240 140 1

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Cost (%CPU)| E-Time | OMem | 1Mem | Used-Mem | Used-Tmp|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | | 1583K(100)| | | | | |
| 1 | SORT ORDER BY | | 471K| 4121M| 3680M| 1583K (1)| 00:01:02 | 7469M| 27M| 97M (1)| 6640K|
|* 2 | HASH JOIN RIGHT OUTER | | 471K| 4121M| | 684K (1)| 00:00:27 | 883K| 883K| 1336K (0)| |
| 3 | TABLE ACCESS FULL | TBL_P************ | 382 | 25976 | | 7 (0)| 00:00:01 | | | | |
| 4 | NESTED LOOPS OUTER | | 471K| 4091M| | 684K (1)| 00:00:27 | | | | |
| 6 | TABLE ACCESS FULL | TBL_A****************** | 5467 | 336K| | 272 (0)| 00:00:01 | | | | |
|* 7 | HASH JOIN RIGHT OUTER | | 471K| 4031M| 53M| 671K (1)| 00:00:27 | 137M| 15M| 148M (0)| |
| 8 | TABLE ACCESS FULL | TBL_C**************** | 2535K| 24M| | 1285 (2)| 00:00:01 | | | | |
| 10 | TABLE ACCESS FULL | TBL_M************** | 3220 | 100K| | 7 (0)| 00:00:01 | | | | |
| 12 | VIEW | index$_join$_016 | 484 | 3388 | | 2 (0)| 00:00:01 | | | | |
|* 13 | HASH JOIN | | | | | | | 1519K| 1519K| 1588K (0)| |
| 14 | INDEX FAST FULL SCAN | IDX_S**************** | 484 | 3388 | | 1 (0)| 00:00:01 | | | | |
| 15 | INDEX FAST FULL SCAN | SYS_C00266671 | 484 | 3388 | | 1 (0)| 00:00:01 | | | | |
| 17 | TABLE ACCESS FULL | TBL_P************* | 7 | 917 | | 3 (0)| 00:00:01 | | | | |
|* 18 | HASH JOIN | | 471K| 3950M| 3296K| 467K (1)| 00:00:19 | 4421K| 1814K| 4629K (0)| |
| 19 | TABLE ACCESS FULL | TBL_P******* | 1151 | 3278K| | 136 (0)| 00:00:01 | | | | |
|* 20 | HASH JOIN | | 166K| 935M| 695M| 420K (1)| 00:00:17 | 1006M| 28M| 361M (1)| 2660K|
| 21 | NESTED LOOPS | | 917K| 685M| | 29731 (1)| 00:00:02 | | | | |
|* 22 | TABLE ACCESS BY INDEX ROWID BATCHED| TBL_U*** | 9780 | 2253K| | 381 (0)| 00:00:01 | | | | |
|* 23 | INDEX RANGE SCAN | IDX_U******** | 9827 | | | 30 (0)| 00:00:01 | | | | |
|* 24 | TABLE ACCESS BY INDEX ROWID BATCHED| TBL_A****** | 94 | 51418 | | 3 (0)| 00:00:01 | | | | |
|* 25 | INDEX RANGE SCAN | FK_A******** | 2 | | | 2 (0)| 00:00:01 | | | | |
|* 26 | TABLE ACCESS FULL | TBL_P******* | 443K| 2154M| | 249K (1)| 00:00:10 | | | | |
| 27 | TABLE ACCESS BY INDEX ROWID | TBL_A******************** | 1 | 68 | | 1 (0)| 00:00:01 | | | | |
|* 28 | INDEX UNIQUE SCAN | SYS_C00251697 | 1 | | | 0 (0)| | | | | |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
EXECUTION PLAN FROM MEMORY
SHOWING MEMORY UTILISATIONS
Estimated amount of memory needed to perform
the operation in memory only. This is also called
the optimal execution
Estimated amount of memory needed to perform the
operation in one pass (write to and read from disk
(temp) only once). This is called a one-pass execution.
Used-Mem - The amount of memory actually used for this operation.
The number in brackets shows number of passes.
0=optimal execution, used only memory and no temporary space.
1=one-pass execution.
>1=multi-pass execution, shows number of passes.
Used-Tmp - The amount of temporary
space used for this operation.

WITH x AS (
SELECT instance_number, session_id, session_serial#,
force_matching_signature
, sql_id, sql_plan_hash_value
, ROUND(MAX(temp_space_allocated)/1024/1024 ,0)
MAXtempMb
, ROUND(AVG(temp_space_allocated)/1024/1024 ,0)
AVGtempMb
, COUNT(DISTINCT sql_exec_id) num_execs
, sum(usecs_per_row)/1e6
*count(DISTINCT sample_id) elap_secs
…
GROUP BY instance_number, session_id, session_serial#,
force_matching_signature,
sql_id, sql_plan_hash_value, sql_exec_id
)
, y AS (
SELECT force_matching_signature
, sql_id, sql_plan_hash_value
, MAX(maxtempmb) maxtempmb
, AVG(maxtempmb) avgtempmb
, SUM(num_execs) num_execs
, SUM(ash_secs) ash_secs
, SUM(elap_secs) elap_secs
FROM x
WHERE maxtempmb>0
GROUP BY force_matching_signature,
sql_id, sql_plan_hash_value
)
SELECT y.*
, y.ash_secs/NULLIF(y.num_execs,0) avg_secs
FROM y
ORDER BY MAXtempMB DESC
FETCH FIRST 20 ROWS ONLY
/

E.G.: PROFILE TOP STATEMENTS/PLANS BY
MAXIMUM TEMPORARY SPACE USAGE
Force Matching SQL Plan Maximum Average Num ASH Elap Average
Signature SQL_ID Hash Value Temp MB Temp MB Execs Secs Secs ASH Secs
--------------------- ------------- ---------- ------- ------- ------ -------- -------- --------
5122026579641258320 830b19784ysxj 2966117505 8862 8514 12 9750 9750 813
5122026579641258320 830b19784ysxj 709075361 8832 6803 298 288330 288330 968
5122026579641258320 830b19784ysxj 2974823484 8768 8404 31 26450 26450 853
5122026579641258320 830b19784ysxj 1972274835 8639 6921 369 376110 376110 1019
5122026579641258320 830b19784ysxj 1077118775 8505 7951 4 3480 3480 870
5122026579641258320 830b19784ysxj 1272495438 8091 5264 11 2960 2960 269
5122026579641258320 830b19784ysxj 3100072734 8009 5721 34 13790 13790 406
5122026579641258320 830b19784ysxj 243927102 8005 7067 6 3680 3680 613
5122026579641258320 830b19784ysxj 2247842993 7458 6377 5 2520 2520 504
5122026579641258320 830b19784ysxj 3127766361 7425 6619 25 23850 23850 954
5122026579641258320 830b19784ysxj 1850679095 7354 6884 23 21410 21410 931
5122026579641258320 830b19784ysxj 3092175162 7282 6757 34 33810 33810 994
5122026579641258320 830b19784ysxj 1231072535 6291 5101 5 2940 2940 588
5122026579641258320 830b19784ysxj 1568053671 5617 4232 112 126740 126740 1132
5122026579641258320 830b19784ysxj 3131581979 5339 5338 5 5680 5680 1136
0 0 5239 131 0 21100 21100
11888108257230087975 869p596a8sxr8 1410581471 5239 3737 93 1170 1170 13
11502273137232072687 9tsc7ud8yy5g4 2807295339 5239 168 190 2340 2340 12
8909297011825569365 3b1ajxpzxqkr0 1351482720 5239 3690 47 550 550 12
7988435406548533980 4f568h63hd1mg 2565855517 5239 102 92 1130 1130 12

EXAMINE TEMP/PGA FOR ONE EXECUTION
ALTER SESSION SET nls_timestamp_format='DD/MM/RR HH24:MI:SS';
SELECT instance_number, session_id, session_serial#, sample_id, sql_exec_id,
sample_time, temp_space_allocated, pga_allocated, sql_plan_line_id
FROM dba_active_session_history
WHERE sql_id = '830b19784ysxj'
AND sql_plan_hash_value = 709075361
AND instance_number = 2
AND session_id = 1008
AND session_serial# = 15439
ORDER BY 1,2,3,4,5
INSTANCE_NUMBER SESSION_ID SESSION_SERIAL# SAMPLE_ID SQL_EXEC_ID SAMPLE_TIME TEMP_SPACE_ALLOCATED PGA_ALLOCATED SQL_PLAN_LINE_ID
--------------- ---------- --------------- ---------- ----------- -------------------- -------------------- ------------- ----------------
2 1008 15439 242021011 30/03/20 21:39:44 9269248
2 1008 15439 242021021 33566122 30/03/20 21:39:54 524288000 138637312 20
2 1008 15439 242021031 33566122 30/03/20 21:40:04 896532480 673738752 1
2 1008 15439 242021041 33566122 30/03/20 21:40:14 1241513984 635334656 1
2 1008 15439 242021051 33566122 30/03/20 21:40:24 1572864000 641495040 1
…
2 1008 15439 242021181 33566122 30/03/20 21:42:35 7191134208 668102656 28
2 1008 15439 242021191 33566122 30/03/20 21:42:45 5451546624 74805248
…

Note gaps in ASH. Temp & PGA clearly still in use.
Occurs because session is idle on the database
while result of query is fetched to app server.
No gaps during build up of data

MONITOR PGA AND TEMP USAGE ACROSS
DATABASE
Cannot simply aggregate Temporary & PGA Allocations group by SAMPLE_ID. You will
miss inactive sessions.
• But I can interpolate the allocations between two points where I don't have samples.
• I need a temporary working storage table with one row
• for every known sample ID
• for each RAC instance.
• I need to round the sample times off to match across RAC nodes
• Then I will iterate though the statements that consumed PGA and Temp and build up a
total.
• Finally I can report from the working table (and aggregate across RAC nodes)

CREATE A WORKING STORAGE TABLE
WITH EVERY KNOWN SAMPLE ID
CREATE TABLE scott.my_allocated AS
l_date VARCHAR2(20);
l_secs NUMBER;
l_date_fmt VARCHAR2(20) := 'J';
l_secs_fmt VARCHAR2(20) := 'SSSSS.FF9';
BEGIN
l_date := TO_CHAR(p_in,l_date_fmt);
l_secs := ROUND(TO_NUMBER(TO_CHAR(p_in,l_secs_fmt)),p_len);
IF l_secs >= 86400 THEN
l_secs := l_secs - 86400;
l_date := l_date + 1;
END IF;
RETURN TO_TIMESTAMP(l_date||l_secs,l_date_fmt||l_secs_fmt);
END;
SELECT dbid, instance_number, sample_id, tsround(sample_time,-1) sample_time
, count(*) active_sessions
, 0 temp_space_allocated
, 0 temp_sessions
, 0 pga_allocated
, 0 pga_sessions
FROM dba_hist_Active_Sess_history
…
GROUP BY dbid, instance_number, sample_id, tsround(sample_time,-1);

ITERATE THROUGH ASH
FOR STATEMENTS THAT CONSUME TEMP/PGA
BEGIN
FOR s IN (
WITH x as (
SELECT dbid, instance_number, sample_id, sample_time
, temp_space_allocated
, pga_allocated,
, LEAD(sample_id,1)
OVER (PARTITION BY dbid, instance_number, session_id, session_serial#, sql_id, sql_plan_hash_value, sql_exec_id
ORDER BY sample_id) as next_sample_id
, LEAD(temp_space_allocated,1) OVER (…) as next_temp_space_allocated
, LEAD(pga_allocated,1) OVER (…) as next_pga_allocated
, MAX(temp_space_allocated) OVER (…) max_temp_space_allocated
, MAX(pga_allocated) OVER (…) max_pga_allocated
…
)
SELECT x.dbid, x.instance_number, x.sample_id, x.next_sample_id /*linear interpolation*/
, x.temp_space_allocated, (x.next_temp_space_allocated-x.temp_space_allocated)/(x.next_sample_id-x.sample_id) m_temp_space_allocated
, x.pga_allocated , (x.next_pga_allocated -x.pga_allocated )/(x.next_sample_id-x.sample_id) m_pga_allocated
FROM x
WHERE max_temp_space_allocated > 0 OR max_pga_allocated > 0 /*restrict to queries that use temp or pga*/
) LOOP
…

FOR EACH SAMPLE ID
COMPUTE A RUNNING TOTAL OF TEMP/PGA
…
) LOOP
UPDATE scott.my_allocated u
SET u.temp_space_allocated = u.temp_space_allocated
+ s.temp_space_allocated
+ s.m_temp_space_allocated * (u.sample_id-s.sample_id)
, u.pga_allocated = u.pga_allocated
+ s.pga_allocated
+ s.m_pga_allocated * (u.sample_id-s.sample_id)
WHERE u.dbid = s.dbid
AND u.instance_number = s.instance_number
AND u.sample_id >= s.sample_id
AND u.sample_id <= NVL(s.next_sample_id-1,s.sample_id);
END LOOP;
END;
/
This deals with gaps due to inactive sessions still
consuming resources

USAGE WITH ASH
QUESTIONS?

INDEX USAGE

WHO IS USING THIS INDEX?
CURRENT_OBJ# has been suggested as a way to identify index usage.
• It only identifies index physical read
• So it also includes index maintenance during DML
• Doesn’t work if the index has been rebuilt since the ASH sample was taken
• and therefore has a new object number

SQL PLANS CAPTURED BY AWR
SQL statements and plans captured during AWR snapshot
• Top N by Elapsed Time, CPU Time, Parse Calls, Shareable Memory, Version Count
DBA_HIST_SQL_PLAN
• OBJECT_ID
• OBJECT_OWNER
• OBJECT_TYPE
• OBJECT_NAME

WHO IS USING THIS INDEX?
Join plans that reference an index to ASH data
• Join by SQL_PLAN_HASH_VALUE
• Do not join by SQL_ID
Filter out
• SQL*Plus, SQL Developer, Ad-Hoc query tools
• Statistics collection

EXTRACT ASH FOR STATEMENTS THAT USE
SPECIFIED INDEXES
CREATE TABLE my_ash COMPRESS AS
WITH p AS (
SELECT DISTINCT p.plan_hash_value
, p.object_owner, p.object_type, p.object_name
FROM dba_hist_sql_plan p
WHERE p.object_name LIKE 'PS_PROJ_RESOURCE'
AND p.object_type LIKE 'INDEX%'
AND p.object_owner = 'SYSADM')
SELECT p.object# object_id
, p.object_owner, p.object_type, p.object_name
, h.*
, p
WHERE h.sql_plan_hash_value = p.plan_hash_value

PROFILE THE ASH EXTRACTED
WITH h AS (
SELECT object_name
, CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program,'[^.@]+',1,1)
WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module,'[^.]+',1,2)
ELSE REGEXP_SUBSTR(h.program,'[^.@]+',1,1)
END as module
, CASE WHEN h.action LIKE 'PI=%' THEN NULL
ELSE h.action
END as action
, CAST(sample_time AS DATE) sample_time
, sql_id, sql_plan_hash_value, sql_exec_id
FROM my_ash h
)
SELECT object_name, module, action
, COUNT(DISTINCT sql_plan_hash_value) sql_plans
, COUNT(DISTINCT sql_id||sql_plan_hash_value||sql_exec_id) sql_execs
, MAX(sample_time) max_sample_time
FROM h
WHERE NOT lower(module) IN('oracle','toad','sqlplus','sqlplusw')
AND NOT lower(module) LIKE 'sql%'
GROUP BY object_name, module, action
ORDER BY object_name, object_name, ash_secs desc

PROFILE THE ASH EXTRACTED
ASH SQL SQL Last
OBJECT_NAME MODULE ACTION Secs Plans Execs Sample
------------------ -------------------- -------------------------------- ------- ----- ------ -------------------
…
PSMPROJ_RESOURCE PC_TL_TO_PC GF_PBINT_AE.XxBiEDM.Step07.S 60 2 6 18:35:18 20/05/2020
****************** -------
sum 60
PSNPROJ_RESOURCE PC_TL_TO_PC GF_PBINT_AE.CallmeA.Step26.S 6720 1 49 18:53:58 26/05/2020
PC_TL_TO_PC GF_PBINT_AE.CallmeA.Step30.S 3460 1 60 06:33:27 27/05/2020
GF_OA_CMSN GF_OA_CMSN.01INIT.Step01.S 2660 1 47 19:19:40 26/05/2020
PC_TL_TO_PC GF_PBINT_AE.CallmeG.Step01.S 1740 1 61 06:34:17 27/05/2020
PC_PRICING GF_PBINT_AE.CallmeG.Step01.S 200 1 10 08:09:55 22/05/2020
PC_AP_TO_PC GF_PBINT_AE.CallmeH.Step00A.S 170 1 17 21:53:26 21/05/2020
PC_PRICING GF_PBINT_AE.CallmeA.Step36.S 20 1 1 08:02:46 05/05/2020
PC_TL_TO_PC GF_PBINT_AE.CallmeA.Pseudo.S 20 1 1 19:45:11 06/05/2020
****************** -------
sum 21770
…
This index used widely.
Probably can’t drop it.
This index used lightly.
Perhaps we don’t really need it.
Indexes that do not appear
may not be used.

LIMITATIONS OF METHOD
AWR doesn’t capture all SQLs/Plans
• A very effective index that is only used occasionally might not be captured.
• Results are only indicative, not absolute.
ASH data purged after 8 days (by default)
• An index only be used for annual/monthly process might not be detected, but it might be
essential for that process
• Consider keeping ASH longer in production database. As long as typical cycle for
application.
• Consider establishing longer term repository, retaining perhaps 400 days.
• Consider how Automatic Indexing in 19c behaves
• By default, it waits 373 days before dropping an index

BEFORE I CAN DROP AN INDEX…
Need to do a lot more analysis looking at whether this query needs this index.
Recommendation
• First make index invisible
• Drop it later if no negative consequences

INDEX USAGE
QUESTIONS?

MONITORING EXECUTION PLAN CHANGE AND
PLAN STABILITY

DID MY EXECUTION PLAN CHANGE?
Can see change in execution plan and performance
• (same SQL, different literals, same force matching signature)
SQL_PLAN
PRCSINSTANCE BEGINDTTM SQL_ID HASH_VALUE EXEC_SECS ASH_SECS
------------ ------------------- ------------- ---------- --------- ---------
1964975 08:30:52 22/01/2020 46smbgcfcrb8d 2602481067 20379 20080
1965250 09:08:51 22/01/2020 fpftdx2405zyq 2602481067 20983 20690
1968443 16:42:51 22/01/2020 3rxad5z3ccusv 3398716340 105 80
1968469 16:47:21 22/01/2020 3rxad5z3ccusv 3398716340 90 70
1968485 16:50:19 22/01/2020 3rxad5z3ccusv 3398716340 62 40
1968698 17:40:01 22/01/2020 0ku8f514k3nt0 3398716340 76 50
1968866 18:19:19 22/01/2020 cbmyvpsxzyf5n 3398716340 139 120
1968966 18:34:24 22/01/2020 5jb1sgmjc7436 3398716340 187 170

Problem Statement:
• Large Payroll
• Most of the employees
• Small Payroll
• Weekly payroll
• Different legislature
• Single employee recalculations
Poor payroll performance
• Because plans from small payroll used in large.
• We didn't get new child cursors
EFFECTIVENESS OF PLAN STABILITY
Stabilise plans in all payrolls with profiles based
on large payroll.
Three scenarios
1. Large payroll – collecting profiles, this is
the baseline
2. Small payroll – no profiles
3. Small payroll – with profiles applied

EFFECT OF PLAN STABILITY
SELECT /*+ LEADING(@q1 r1@q1 x1@q1 h1@q1) USE_NL(h1@q1)
LEADING(@q2 r2@q2 x2@q2 h2@q2) USE_NL(h2@q2)
LEADING(@q3 r3@q3 x3@q3 h3@q3) USE_NL(h3@q3) */
q1.sql_id
, q1.sql_plan_hash_value, q1.ash_secs
, DECODE(q1.sql_plan_hash_value,q2.sql_plan_hash_value,'**SAME**',
q2.sql_plan_hash_value) sql_plan_hash_value2, q2.ash_secs
, DECODE(q1.sql_plan_hash_value,q3.sql_plan_hash_value,'**SAME**',
q3.sql_plan_hash_value) sql_plan_hash_value3, q3.ash_secs
FROM (…) Q1
LEFT OUTER JOIN (…) Q2 ON q1.sql_id = q2.sql_id
INNER JOIN (…) Q3 ON q1.sql_id = q3.sql_id
ORDER BY q3.ash_secs desc, q1.sql_id
/
Usual Query to profile DB Time by SQL_ID
and Plan Hash Value in each of three
in-line views, for different scenarios

EFFECT OF PLAN STABILITY
SQL_ID SCENARIO 1 ASH_SECS SCENARIO 2 ASH_SECS SCENARIO 3 ASH_SECS
------------- ---------- -------- ---------- -------- ---------- --------
4uzmzh74rdrnz 2514155560 280 3829487612 >28750 **SAME** 50231
4n482cm7r9qyn 1595742310 680 869376931 140 **SAME** 8892
2f66y2u54ru1v 1145975676 630 **SAME** 531
1n2dfvb3jrn2m 1293172177 150 **SAME** 150
652y9682bqqvp 3325291917 30 **SAME** 110
d8gxmqp2zydta 1716202706 10 678016679 10 **SAME** 32
2np47twhd5nga 3496258537 10 **SAME** 27
4ru0618dswz3y 2621940820 10 539127764 223
4ru0618dswz3y 539127764 100 **SAME** 22
4ru0618dswz3y 3325291917 10 539127764 22
4ru0618dswz3y 1403673054 110 539127764 22
gnnu2hfkjm2yd 1559321680 80 **SAME** 19
1. Better with profile, but not great, but it did run
2. A little worse
3. 4 execution plans, now just 1, much better.

TRANSACTIONS & EXECUTIONS

WHEN DID THE TRANSACTION BEGIN?
SELECT /*+leading(r h) use_nl(h)*/
h.sql_id, h.sql_plan_hash_value, h.xid
, SUM(1) ash_secs
, (NVL(r.enddttm,SYSDATE)-r.begindttm)*86400 exec_secs
, MIN(sample_time) first_sample_time
--, MAX(sample_time) last_sample_time
FROM gv$active_session_history h
, sysadm.psprcsrqst r
WHERE h.sample_time BETWEEN r.begindttm AND NVL(r.enddttm,SYSDATE)
AND h.module LIKE r.prcsname
AND h.action LIKE 'PI='||r.prcsinstance||'%'
AND r.prcsinstance = 10026580
AND h.sql_id = 'dungu07axr0z5'
GROUP BY r.prcsinstance, r.prcsname, r.begindttm, r.enddttm
, h.sql_id, h.sql_plan_hash_value, h.xid
ORDER BY last_sample_time, ash_secs desc
/

SQL Plan ASH Exec
SQL_ID Hash Value XID Secs Secs First Running
------------- ---------- ---------------- ------ ------ -------------------------
7uj72ad03k13k 3087414546 82 1124 28-APR-20 04.42.48.662 PM
7uj72ad03k13k 3087414546 000A001400044C6D 1 1124 28-APR-20 04.44.11.672 PM
1ng9qkc0zspkh 3423396304 104 1124 28-APR-20 04.44.12.682 PM
1ng9qkc0zspkh 3423396304 0007002D0004116E 5 1124 28-APR-20 04.45.57.971 PM
Transaction ID only created when first row is modified

DELETE /*GPPCANCL_D_ERNDGRP*/
FROM PS_GP_RSLT_ERN_DED
WHERE EMPLID BETWEEN :1 AND :2
AND CAL_RUN_ID= :3
AND EMPLID IN (SELECT EMPLID FROM PS_GP_GRP_LIST_RUN
WHERE RUN_CNTL_ID=:4 AND OPRID=:5)
AND EXISTS (SELECT 'X' FROM PS_GP_PYE_RCLC_WRK RW
WHERE RW.CAL_ID = PS_GP_RSLT_ERN_DED.CAL_ID
AND RW.CAL_RUN_ID = PS_GP_RSLT_ERN_DED.CAL_RUN_ID
AND RW.GP_PAYGROUP = PS_GP_RSLT_ERN_DED.GP_PAYGROUP
AND RW.EMPLID BETWEEN :6 AND :7
AND RW.CAL_RUN_ID = :8
AND RW.EMPLID = PS_GP_RSLT_ERN_DED.EMPLID
AND RW.EMPL_RCD = PS_GP_RSLT_ERN_DED.EMPL_RCD)
Depending on data, it could be a while before this statement found something to delete.

HOW MANY EXECUTIONS
SELECT /*+LEADING(x h) USE_NL(h)*/ h.program
, h.sql_id, h.sql_plan_hash_value
, COUNT(DISTINCT h.instance_number||h.session_id||h.session_serial#||h.sql_exec_id) execs
, COUNT(DISTINCT xid) XIDs
WHERE x.end_interval_time >= …
AND x.begin_interval_time <= …
AND h.sample_time >= …
AND h.sample_time <= …
AND h.dbid = x.dbid
GROUP BY h.program, h.sql_id, h.sql_plan_hash_value
/

HOW MANY EXECUTIONS?
SQL Plan ASH
PROGRAM SQL_ID Hash Value Secs EXECS XIDS USERS
----------- ------------- ---------- ------ ------- ----- ------
t_async.exe 7q90ra0vmd9xx 2723153562 3020 297 0 20
t_async.exe 6mw25bgbh1stj 1229059401 320 32 0 17
…
Samples ≈ Executions
• Based on DBA_HIST_ACTIVE_SESS_HISTORY
• 1 sample / 10 seconds.
• Each sample is worth 10 seconds.
• Probably underestimates number of executions.

LOTS OF SHORT-LIVED SQL
STATEMENTS
SQL_PLAN
PRCSINSTANCE NUM_SQL_ID HASH_VALUE EXEC_SECS ASH_SECS
------------ ---------- ---------- --------- ---------
50007687 169 953836181 3170 1690
50007687 50 807301148 3170 500
50007687 22 4034059499 3170 220
50007687 14 2504475139 3170 140
50007687 2 0 3170 70
50007687 1 1309703960 3170 20
50007687 1 3230852326 3170 10
…
We sampled 169 different statements with the same execution plan.
Probably more than 169 statements that took about 1690 seconds, but we only sampled 169.
169 SQLs 1690 Secs

APPLICATION REPORTS LOTS OF COMPILES

PROCESS STATES
Except there is actually 64224 database calls, not just 4!
time
1690s
2596s

THINGS THAT CAN GO WRONG

THINGS THAT CAN GO WRONG
WITH DBMS_XPLAN
Statement not in Library Cache
• Only Some Statements in Library Cache
• Lots of short-lived non-shareable SQL
DISPLAY_AWR
• ORA-6502 – very large SQL
• ORA-44002 – short-lived objects(?)
• ORA-1422 – duplicate SQL from cloning
SQL_PLAN_LINE_ID doesn't match the line number DBMS_XPLAN

STATEMENT NOT IN LIBRARY CACHE
SELECT * FROM
table(dbms_xplan.display_cursor('gpdwr389mg61h',0,'ADVANCED'));
PLAN_TABLE_OUTPUT
-----------------------------------------------
SQL_ID: gpdwr389mg61h, child number: 0 cannot be found

ORA-06502
This seems to be associated with very large SQL statements
SQL_ID 9vnan5kqsh1aq
--------------------
An uncaught error happened in prepare_sql_statement : ORA-06502: PL/SQL: numeric or value error
---------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
| 1 | HASH GROUP BY | | 1 | 164 | 1 (100)| 00:00:01 |
…

ORA-44002
I have seen this with Global Temporary Tables and with direct path mode (the APPEND hint).
PLAN_TABLE_OUTPUT
------------------------------------------------------
ERROR: cannot get definition for table 'BZTNCMUX31XP5'
ORA-44002: invalid object name

ORA-01422
This happens on a database that has been cloned, often from production to test.
• Consider recreating AWR repository on clone?
An uncaught error happened in prepare_sql_statement :
ORA-01422: exact fetch returns more than requested
number of rows

ORA-01422
Workaround
DELETE FROM sys.wrh$_sqltext t1
WHERE t1.dbid != (
SELECT d.dbid FROM v$database d)
AND EXISTS(
SELECT 'x'
FROM sys.wrh$_sqltext t2
WHERE t2.dbid = (
SELECT d.dbid FROM v$database d)
AND t2.sql_id = t1.sql_id)

SQL_PLAN_LINE_ID DOESN'T MATCH PLAN
DUE TO AUTOMATIC STATISTICS COLLECTION
Presence of absence of OPTIMIZER STATISTICS GATHERING operation
• Doesn't change the PLAN_HASH_VALUE
• Does change the SQL_PLAN_LINE_ID of operations after it.
This can cause ASH profile by SQL Plan line ID not to match execution plans.

NO STATISTICS COLLECTION
SQL_ID c4y8uxt5drk74, child number 0
-------------------------------------
insert into t with n as (select rownum n from dual connect by level <=
1000) select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp') from n
--------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| Cost (%CPU)| E-Time | OMem | 1Mem | Used-Mem |
--------------------------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | | | 2 (100)| | | | |
| 1 | LOAD TABLE CONVENTIONAL | T | | | | | | | |
| 2 | VIEW | | 1 | 13 | 2 (0)| 00:00:01 | | | |
| 3 | COUNT | | | | | | | | |
| 4 | CONNECT BY WITHOUT FILTERING| | | | | | 2048 | 2048 | 2048 (0)|
| 5 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 | | | |
--------------------------------------------------------------------------------------------------------------------

REAL-TIME STATISTICS BEING GATHERED
SQL_ID c4y8uxt5drk74, child number 0
-------------------------------------
insert into t with n as (select rownum n from dual connect by level <=
1000) select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp') from n
--------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows |E-Bytes| Cost (%CPU)| E-Time | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem |
--------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | 1 | | | 2 (100)| | 0 |00:00:00.01 | 57 | | | |
| 1 | LOAD TABLE CONVENTIONAL | T | 1 | | | | | 0 |00:00:00.01 | 57 | | | |
| 2 | OPTIMIZER STATISTICS GATHERING | | 1 | 1 | 13 | 2 (0)| 00:00:01 | 1000 |00:00:00.01 | 0 | | | |
| 3 | VIEW | | 1 | 1 | 13 | 2 (0)| 00:00:01 | 1000 |00:00:00.01 | 0 | | | |
| 4 | COUNT | | 1 | | | | | 1000 |00:00:00.01 | 0 | | | |
| 5 | CONNECT BY WITHOUT FILTERING| | 1 | | | | | 1000 |00:00:00.01 | 0 | 2048 | 2048 | 2048 (0)|
| 6 | FAST DUAL | | 1 | 1 | | 2 (0)| 00:00:01 | 1 |00:00:00.01 | 0 | | | |
--------------------------------------------------------------------------------------------------------------------------------------------------------------

HOW I WORK WITH ASH

As Consultant
• I am here to provide advice and guidance
• Communicate that to other team members
• And also demonstrate how to obtain information
upon which to base advice.
Every piece of analysis produces a word document
• Copy and paste queries into SQL*Plus
• Adjust the queries to fit the circumstances
• Copy and paste results back into word
• Annotate results with comments and
recommendations
HOW I WORK WITH ASH

CONCLUSION
ASH data
• Profiling DB response time is very valuable
• But there is lots more you can do
There are lots of ways to query the data
• Be imaginative!
Application instrumentation is essential
Consequences of sampling
Traps to fall into

QUESTIONS

The Practical Use of Active Session History

More Related Content

Similar to The Practical Use of Active Session History

Recently uploaded

The Practical Use of Active Session History

Editor's Notes