SlideShare a Scribd company logo
IO Waits
Kyle Hailey
Kyle_hailey@hotmail.com
http://perfvision.com
Copyright 2006 Kyle Hailey
Waits Covered in this Section
1. db file sequential read
2. db file scattered read
3. db file parallel read
4. read by other session
5. direct path read
6. direct path write
7. direct path read temp
8. direct path write temp
9. direct path write (lob)
10. Local write wait
11. data file init write
Buffer Cache I/O
PGA I/O
Special Cases
Copyright 2006 Kyle Hailey
IO Waits
Data Files
DBWR
Buffer
Cache
User1
SGA
Library
Cache
Log
Buffer
Copyright 2006 Kyle Hailey
Standard Table/Index IO
1. db file sequential read
 Single block read
2. db file scattered read
 Multi block read
3. db file parallel read
 Non-contiguous multi block read
4. read by other session
 Wait for another session to do the io
Copyright 2006 Kyle Hailey
db file sequential read
Shadow
Process
1) Search Buffer Cache for
the block by rowid
2) Fails
3) Reads of disk
• File
• Block
Buffer Cache
 The top wait
 Single Block Read
Data via Index and/or Rowid
Rollback (if expecting FTS => problem)
Select * from emp where empno = 9999;
Note: “sequential” means
A sequence like a rowid
Index on emp.empno
Copyright 2006 Kyle Hailey
db file scattered read
Shadow
Process
1) Search Buffer Cache for the
blocks
2) Fails
3) Reads off disk
• File
• Block
• Multi Block Read Count
Buffer Cache
 Multi Block Read
 Full Table Scan
 Index Fast Full Scans
SQL> select * from emp;
Note: Scattered Means
Blocks are read and
scattered throughout
buffer cache
db_file_multiblock_read_count
Copyright 2006 Kyle Hailey
db file parallel read
Shadow
Process
1) Search Buffer Cache for
the blocks
2) Fails
3) Reads that block off Disk
Buffer Cache  Process issues multiple single block reads in parallel
 Documentation says only for recovery
 But seems to happen for normal read ops
 Async Call – wait for all reads to complete
 Examples
 Not contiguous multi block read
 Index full or range scan
 Where values in Select * from emp where
Empno in (1,2,3,4,5,6);
Copyright 2006 Kyle Hailey
Read by other Session
1. Block not found in cache
2. Read block from disk
3. Found other session already
reading block from disk
4. Wait for the other session to finish
IO
S1
Buffer Cache
S2
Multiple sessions reading the
same data that requires IO
Ex) two users doing a full table
scan at the same time
Copyright 2006 Kyle Hailey
IO – Further Investigation
select parameter1, parameter2, parameter3 from v$event_name
NAME P1 P2 P3
----------------------- ----------- --------- --------
read by other session file# block# class#
db file sequential read file# block# blocks
db file scattered read file# block# blocks
db file parallel read files blocks requests
P1 and P2 are file# , block#
We can use P1, P2, P3 from ASH to get more info
Exception:
db file parallel read – p1,p2,p3 not useful
Copyright 2006 Kyle Hailey
sequential reads – blocks read
select
event,
ash.p3,
o.object_name objn,
o.object_type otype,
CURRENT_FILE# filen,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'db file sequential read'
and o.object_id (+)= ash.CURRENT_OBJ#
order by sample_time;
EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID
----------------------- -- ------------------------- ----- ----- ------ -------------
db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737
db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0
db file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9s
P3= # of
blocks
Copyright 2006 Kyle Hailey
scattered reads – multi blocks read
select
event,
ash.p3,
o.object_name objn,
o.object_type otype,
CURRENT_FILE# filen,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'db file scattered read'
and o.object_id (+)= ash.CURRENT_OBJ#
order by sample_time;
EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID
---------------------- -- ------------------------- ----- ----- ------ -------------
db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363
db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676
db file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hu
P3= # of
blocks
IO by File
select io.cnt,
round(io.cnt/(&v_minutes*60),2) aas,
io.event,
io.p1 p1,
f.tablespace_name
from ( select
count(*) cnt,
substr(event,0,25) event,
ash.p1 p1
from v$active_session_history ash
where ( event like 'db file s%' or event like 'direct%' )
and sample_time > sysdate - &v_minutes/(60*24)
group by
event ,
ash.p1
) io,
dba_data_files f
where
f.file_id = io.p1
Order by io.cnt ;
CNT AAS EVENT P1 TABLESPACE
---- --- ----------------------- --- ---------
-
1 .00 db file sequential read 1 SYSTEM
2 .00 db file sequential read 3 SYSAUX
38 .06 db file sequential read 6 SOE
179 .30 db file sequential read 7 SOEINDEX
IO by Object
select
count(*) cnt,
CURRENT_OBJ#||' '||o.object_name objn,
o.object_type otype
from v$active_session_history ash,
all_objects o
where ( event like 'db file s%' or event like 'direct%' )
and o.object_id (+)= ash.CURRENT_OBJ#
and sample_time > sysdate - &1/(60*24)
and session_state='WAITING'
group by
CURRENT_OBJ#, o.object_name ,
o.object_type
Order by count(*)
CNT AAS OBJN OTYPE
---- ------- ------------------------- ---------------
79 .00 52949 ORDER_ITEMS TABLE PARTITION
97 .00 -1
130 .00 53117 ORD_STATUS_IX INDEX
498 .01 53120 CUST_EMAIL_IX INDEX
512 .01 0
1632 .03 53112 ITEM_ORDER_IX INDEX
IO by SQL
select
round(sum(cnt) over ( partition by io.sql_id order by sql_id ) / (&v_minutes*60),2) aas,
io.sql_id,
io.cnt cnt,
100*cnt/sum(cnt) over ( partition by io.sql_id order by sql_id ) pct,
o.object_name obj,
o.subobject_name sub_obj,
o.object_type otype,
substr(io.event,8,10) event,
io.p1 file#,
f.tablespace_name tablespace_name,
tbs.contents
from
(
select
sql_id,
event,
count(*) cnt,
count(*) / (&v_minutes*60) aas,
CURRENT_OBJ# ,
ash.p1
from v$active_session_history ash
where ( event like 'db file s%' or event like 'direct%' )
and sample_time > sysdate - &v_minutes/(60*24)
group by
CURRENT_OBJ#,
event,
ash.p1,
sql_id
) io,
dba_data_files f
,all_objects o
, dba_tablespaces tbs
where
f.file_id = io.p1
and o.object_id (+)= io.CURRENT_OBJ#
and tbs.tablespace_name= f.tablespace_name
Order by tcnt, sql_id, cnt
/
AAS SQL_ID % OBJ TABLESPACE
----- ------------- --- --------------- ----------
.18 0yas01u2p9ch4 6 ITEM_PRODUCT_IX SOEINDEX
6 ORDER_ITEMS_UK SOEINDEX
88 ITEM_ORDER_IX SOEINDEX
.32 6v6gm0fd1rgrz 6 WAIT_OBJECTS SYSTEM
94 UNDO UNDOTBS1
Copyright 2006 Kyle Hailey
Missing Object IDs
 dba_extents – get object name
 Dba_extents is notoriously slow
 Options
 Create a table as select – use table
 recreate as object extents change
 Catch blocks in x$bh when waits occur
select segment_name,
segment_type
from dba_extents
where file_id = P1
and P2 between
block_id and block_id + blocks – 1;
Copy of dba_extents
Enter value for file: 3
Enter value for block: 6619
OWNER SEGMENT_NAME SEGMENT_TYPE
--------------- -------------------- ------------------
WMSYS LOG_TAB_PK INDEX
Elapsed: 00:00:41.25
create table myextents as select * from dba_extents
Table created.
Elapsed: 00:01:25.73
CNT OWN SEGMENT_NAME SEGMENT_TYPE
--- ---- ---------------- ------------
11 SYS SMON_SCN_TO_TIME CLUSTER
993 SYS _SYSSMU7$ TYPE2 UNDO
150 rows selected.
Elapsed: 00:00:01.03
Copyright 2006 Kyle Hailey
IO Solutions
1. Check average read times per file
 Should be between 5-20 ms
 Data in Statspack under “File IO Stats”
2. Check Cache buffer Hit ratio
 Check db_cache_advice 9i and higher
 Data in Statspack under “Buffer Pool Advisory”
 Want to optimize data caching
3. Tune High IO SQL
 Check reads off of UNDO
Copyright 2006 Kyle Hailey
Step 1a: Ave Read Time
File IO Stats DB/Inst:labsf03 Snaps: 1-2
Tablespace Filename
------------------------ ----------------------------------------------------
Av Mx
Av
Av Rd Rd Av Av Buffer BufWt
Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)
---------- ---- ------- ----- --- ------- ------------ -------- ---------- --
SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf
445 15 0.4 16 1.0 1,157 39 2,744 93.3
USERS /u01/app/oracle/oradata/labsf03/users01.dbf
223 7 0.5 ### 1.0 9,725 324 4 100.0
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
Copyright 2006 Kyle Hailey
Step 1b: Ave Read Time
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
select
to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time,
file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds
from
DBA_HIST_FILESTATXS f,
dba_hist_snapshot s
where f.snap_id=s.snap_id ;
SNAP_TIME FILE# AVG_READ_MS PHYRDS
---------------- ---------- ----------- ----
2008-01-05 12:00 9 36.67 39
2008-01-05 12:00 10 32.31 39
2008-01-05 13:00 1 11.63 178224
2008-01-05 13:00 2 56.37 2014
2008-01-05 13:00 3 17.38 50668
2008-01-05 13:00 4 9.39 565321
2008-01-05 13:00 5 38.78 41
2008-01-05 13:00 6 28.29 41
2008-01-05 13:00 7 27.44 39
2008-01-05 13:00 8 42.56 39
2008-01-05 13:00 9 36.67 39
2008-01-05 13:00 10 32.31 39
Copyright 2006 Kyle Hailey
Step 1c: Ave Read Time
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
select
to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time,
file_id fid,
average_read_time *10 avgrd_ms,
average_write_time *10 avgwr_ms,
physical_reads pr,
physical_writes pw
from
V$FILEMETRIC_HISTORY f
order by begin_time;
BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW
---------------- ---- -------- -------- ---------- ----------
2008-01-30 12:25 1 5.2 2.5 100 24
2008-01-30 12:25 5 80.0 20.0 2 2
2008-01-30 12:25 2 36.7 2.0 3 218
2008-01-30 12:25 4 22.8 1.8 89 2754
2008-01-30 12:25 3 22.9 2.6 14 47
Step1 & OEM
 ADDM advisory expects I/Os to be under 10ms
 Can change the behavior with
exec DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER(
'ADDM',
'DBIO_EXPECTED',
<new value>);
Copyright 2006 Kyle Hailey
Step 2 : Buffer Pool Advisory
Buffer Pool Advisory
Size for Size Buffers for Read Estimated
P Est (M) Factor Estimate Factor Physical Reads
--- -------- ------ ------------ ------ --------------
D 56 .1 6,986 2.3 58,928
D 112 .2 13,972 1.6 42,043
D 224 .4 27,944 1.0 25,772
D 336 .6 41,916 1.0 25,715
D 448 .8 55,888 1.0 25,715
D 596 1.0 74,351 1.0 25,715
D 728 1.2 90,818 1.0 25,715
D 840 1.4 104,790 1.0 25,715
D 952 1.6 118,762 1.0 25,715
D 1,064 1.8 132,734 1.0 25,715
Copyright 2006 Kyle Hailey
Step 3: Find Top I/O SQL
select count(*),sql_id, event
from v$active_session_history
where event in ('db file sequential read',
'db file scattered read',
'db file parallel read')
group by sql_id, event
order by count(*);
COUNT(*) SQL_ID EVENT
--------- ------------- ------------------------
10 8hk7xvhua40va db file sequential read
335 3hatpjzrqvfn7 db file sequential read
343 0uuqgjq7k12nf db file sequential read
549 75621g9y3xmvd db file sequential read
1311 0bzhqhhj9mpaa db file sequential read
1523 41zu158rqf4kf db file sequential read
1757 0yas01u2p9ch4 db file sequential read
Copyright 2006 Kyle Hailey
Step 3: OEM find top SQL
Copyright 2006 Kyle Hailey
Machine Memory
IO Configuration Issues
SGA
Unix File
Cache
If average IO read times
are under 7ms them
probably coming from Unix
File Cache
 Unix Buffer Cache
 Disk Spindles
 Raid 5 Not
Physical IO
Physical IO
Copyright 2006 Kyle Hailey
IO Throughput
One Disk 74G – Few IOPS
37 Disks 2G – Many IOPS
92 IO/sec
1887 IO/sec
Copyright 2006 Kyle Hailey
Disk 2Gb vs 70GB
Seagate
Barracuda 4LP
Seagate
Cheetah 73LP
Capacity 2.16GB 73.4GB
Rotation Speed 7200rpm 10000rpm
Rotational Delay(avg) 4.1ms 3ms
Time to read 32Kb 6ms 3ms
Seek Time (avg) 9.4ms 4.9
Total time for Single I/O 19.5ms 10.9ms
I/O per second
(conservative)
51 92
IOPs/sec per 100GB 2550 126
James Morle http://scaleability.com
Copyright 2006 Kyle Hailey
Raid 5
P1 P2
P1
P2
Writes require two reads
plus a write – read data,
read parity then write
Reads can conflict on
smaller stripe sizes 32K
or 64K
Readers on different
disks run without conflict
http://www.miracleas.com/BAARF/BAARF2.html
B – Battle
A – Against
A – Any
R – Raid
F - Five
Raid Levels
Copyright 2006 Kyle Hailey
http://www.miracleas.com/BAARF/1.Millsap2000.01.03-RAID5.pdf
Raid Level none 0
(stripe)
0+1
(striped then
mirrored)
5
Control file 2 1 1 3
Redo log 4 1 1 3
System tablespace 2 1 1 3
Sort segment 4 1 1 3
Rollback segment 2 1 1 5
Index read only 2 1 1 1
Sequential read only 4 1 1 3
DBWR intensive files 1 1 1 5
Direct load intensive 4 1 1 3
Data protection 4 5 1 2
Operating costs 1 1 5 3
Copyright 2006 Kyle Hailey
Direct I/O WAITS
Direct I/O :
This mechanism lets the client bypass the buffer cache for
I/O intensive operations.
The disk blocks are written into and read from process
private memory.
 direct path read :
 Parallel Query
 direct path write
 sqlldr
 loading/reading LOBs
 parallel DMLs
 create table as select
 create index
 direct path read temp , direct path write temp
 Sorting
Copyright 2006 Kyle Hailey
Direct IO
Shadow
Process
Buffer Cache
PGA
Sort Area
Direct Path
PQO Reads
X
Copyright 2006 Kyle Hailey
direct path read
_serial_direct_read = true
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
PQO
Shadow
Shadow
Process
Process
PGA
PGA
PQO
X
 Parallel Query
 See v$px_session
select parameter1, parameter2, parameter3
from v$event_name
where name='direct path read';
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
Copyright 2006 Kyle Hailey
Further Investigation ASH
----- ----- ---- ----------- ------ -- ------ ------------
-
SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID
--- ---- -- ----- ------ -- ------ -------------
149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722
144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722
60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722
54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722
149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722
select
session_id sid,
QC_SESSION_ID qsid,
ash.p3,
CURRENT_OBJ#||' '||o.object_name objn,
o.object_type otype,
CURRENT_FILE# fn,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'direct path read'
and o.object_id (+)= ash.CURRENT_OBJ#
Order by sample_time;
select
ash.SQL_ID,
QC_SESSION_ID qsid,
count(*) cnt,
count (distinct session_id) deg,
nvl(o.object_name,to_char(CURRENT_OBJ#)) obj,
o.object_type otype,
decode(session_state, 'WAITING',event,'CPU') event
from v$active_session_history ash,
all_objects o
where o.object_id (+)= ash.CURRENT_OBJ#
and qc_session_id is not null
group by qc_session_id, sql_id, o.object_name,
o.object_type, CURRENT_OBJ#, event, session_state
Order by qc_session_id, sql_id
PQO - ASH
SQL_ID QSID CNT DEG OBJ OTYPE EVENT
------------- ---- ---- --- ------------ ----- -------------------------
-
7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd
144 4 3 WAIT_OBJECTS TABLE PX qref latch
144 37 1 WAIT_OBJECTS TABLE db file sequential read
144 3 2 WAIT_OBJECTS TABLE direct path read
144 70 1 WAIT_OBJECTS TABLE CPU
144 21 4 0 PX Deq Credit: send blkd
144 12 4 0 db file sequential read
Copyright 2006 Kyle Hailey
direct path write
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
Shadow
Shadow
Process
Process
PGA
PGA
DBWR
DBWR
DBWR
DBWR
DATA
DATA
DATA
DATA
Seems to be a bug on 10g where
direct path write waits are
incorrectly recorded as CPU
select parameter1, parameter2, parameter3
from v$event_name
where name='direct path write';
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
 Occurs when:
 insert /*+ APPEND */
 sql*loader direct=y
 Create table as select
 Loading LOBS
 Parallel insert
 Create index
Copyright 2006 Kyle Hailey
Direct Path Read Temp
Direct Path Write Temp
 Sorting
 Write to Temp
 Read from Temp
Shadow
Shadow
Process
Process
Buffer Cache
Buffer Cache
PGA
PGA
TEMP
TEMP DATA
DATA
Shadow
Shadow
Process
Process
Buffer Cache
Buffer Cache
PGA
PGA
TEMP
TEMP DATA
DATA
select parameter1, parameter2, parameter3
from v$event_name
where name in ('direct path write temp‘,
‘direct path read temp’);
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
file number first dba block cnt
Copyright 2006 Kyle Hailey
Direct Path Read Temp
Direct Path Write Temp
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
TEMP
TEMP DATA
DATA
Shadow
Shadow
Process
Process
PGA
PGA
TEMP
TEMP DATA
DATA
This event happens when sorting overflows the memory buffers and has to be
written to disk. If it's a problem consider increasing parameter
pga_aggregate_target
The explain plan can give estimated sort size requirements and extended row
source stats in the explain plan can give the actual sort size usage of the SQL
statement
select s.sid, s. process,
s.sql_id,
tmp.segtype,
((tmp.blocks*8)/1024) MB,
tmp.tablespace
from
v$tempseg_usage tmp,
v$session s
where tmp.session_num=s.serial#
and segtype in ('HASH','SORT')
order by blocks desc
SID SQL_ID SEGTYPE MB TABLESPACE
--- ------------- ------- -- ----------
90 gm5s0vc0vzppc SORT 2 TEMP03
105 gm5s0vc0vzppc SORT 2 TEMP03
80 gm5s0vc0vzppc SORT 2 TEMP03
105 gm5s0vc0vzppc HASH 2 TEMP03
102 gm5s0vc0vzppc SORT 2 TEMP03
90 gm5s0vc0vzppc HASH 2 TEMP03
PGA Aggregate Target
9i, a single sessions maximum sort area using PGA_AGGREGATE_TARGET was fixed to
100M unless setting
_pga_max_size: Maximum PGA size for a single process
Defaults to 200M
_smm_max_size: Maximum workarea size for one process , defined in KILO BYTES
Defaults Min(0.05*pga_aggregate_target,0.5*_pga_max_size,100M)
In 10.2 the 100M does not apply anymore when using a PGA_AGGREGATE_TARGET > 1000M.
In 10.2 the 100M applies as long as pga_aggregate_target is smaller than 1GB.
In 10.2 pga_aggregate_target set to 5GB
_smm_max_size of 500MB (was 100MB )
_pga_max_size of 1000MB (was 200MB).
Copyright 2006 Kyle Hailey
Manual override broken 10.2.0.3 fixed in 11.1.0.7
alter session set workarea_size_policy = manual;
alter session set sort_area_size = 104857600;
Workaround, run command twice
alter session set sort_area_size = 104857600;
alter session set sort_area_size = 104857600;
But doesn’t work for parallel connections that still fall back
on default which is 64K unless otherwise specified
HASH_AREA_SIZE seems unaffected by problem
Copyright 2006 Kyle Hailey
Data file init write
 When autoextend is set and many extensions are
happening
 Can be the root cause of other waits
 Enq: TX - contention
Copyright 2006 Kyle Hailey
Local Write Wait
 Truncating a table
 Wait for Data Cache to be cleared of all blocks of
truncated table
 Wait by shadow for DBWR
 If a problem try
 GTT
 Reduce fast_start_mttr_target
Copyright 2006 Kyle Hailey
Further Investigation
NAME P1 P2 P3
----------------- ----- ------ -----
local write wait file# block#
select * from v$event_name
where name = ‘local write wait'
select
ash.p1,ash.p2, ash.SQL_ID, count(*)
from v$ash ash
where event='local write wait'
group by
ash.p1,ash.p2, ash.SQL_ID;
P1 P2 SQL_ID COUNT
--- -- ------------- -----
201 2 0q8k9xfmz0k2k 194
201 5 0q8k9xfmz0k2k 34
201 6 0q8k9xfmz0k2k 9
201 4 0q8k9xfmz0k2k 10
Copyright 2006 Kyle Hailey
Temporary File #’s
SQL> select file# from v$datafile;
FILE#
----------
1
2
3
4
5
6
7
8
SQL> select file# from v$tempfile;
FILE#
----------
2
1
SQL> show parameters db_files
NAME VALUE
---------- -------------
db_files 200
Wait Temporary File#’s =
Db_files + file#
File# 201 = v$tempfile 1
Copyright 2006 Kyle Hailey
Local Write Wait:extent allocation
CREATE TEMPORARY TABLESPACE "NEWTEMP"
TEMPFILE '/d3/temp01.dbf' SIZE 100M
AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 64K;
select a.*, b.*
from dba_objects a, dba_objects b
order by a.owner
ALTER DATABASE DEFAULT
TEMPORARY TABLESPACE newtemp;
Copyright 2006 Kyle Hailey
Local Write Waits
Copyright 2006 Kyle Hailey
Local Write Wait: Solutions
 Truncating a table
 Reduce fast_start_mttr
 Use GTT
 Temporary Table Space Allocations
 Increase extent size
Copyright 2006 Kyle Hailey
Summary I/O
 Buffer Cache IO
 db file sequential read
 db file scattered read
 db file parallel read
 Tune
 I/O’s should be 5-15ms
 Check Buffer Cache Advisory
 Tune SQL
Copyright 2006 Kyle Hailey
Summary Direct I/O and Other
 direct path read
 PQO
 direct path write
 direct path operations
 direct path read temp / direct path write temp
 sorts
 direct path write (lob)
 stored NOCACHE
 local write wait –
 Temp extensions
 Truncates
 data file init write – data file extension

More Related Content

Similar to Kyle Hailey Oracle Performance IO Waits

Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940
gopalchsamanta
 
Oracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid cloneOracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid clone
Deepti Singh
 
Oracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid cloneOracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid clone
Deepti Singh
 
Oracle forensics 101
Oracle forensics 101Oracle forensics 101
Oracle forensics 101
fangjiafu
 
FIXING BLOCK CORRUPTION (RMAN) on 11G
FIXING BLOCK CORRUPTION (RMAN) on 11GFIXING BLOCK CORRUPTION (RMAN) on 11G
FIXING BLOCK CORRUPTION (RMAN) on 11G
N/A
 
Direct SGA access without SQL
Direct SGA access without SQLDirect SGA access without SQL
Direct SGA access without SQL
Kyle Hailey
 
Oracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspackOracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspack
Kyle Hailey
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Kyle Hailey
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
Kyle Hailey
 

Similar to Kyle Hailey Oracle Performance IO Waits (20)

Oracle Database Management Basic 1
Oracle Database Management Basic 1Oracle Database Management Basic 1
Oracle Database Management Basic 1
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
 
Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940
 
Rmoug ashmaster
Rmoug ashmasterRmoug ashmaster
Rmoug ashmaster
 
Frits Hoogland - About multiblock reads
Frits Hoogland - About multiblock readsFrits Hoogland - About multiblock reads
Frits Hoogland - About multiblock reads
 
Oracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid cloneOracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid clone
 
Oracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid cloneOracle applications 11i hot backup cloning with rapid clone
Oracle applications 11i hot backup cloning with rapid clone
 
Monitoreo de MySQL y PostgreSQL con SQL
Monitoreo de MySQL y PostgreSQL con SQLMonitoreo de MySQL y PostgreSQL con SQL
Monitoreo de MySQL y PostgreSQL con SQL
 
Linux.pdf
Linux.pdfLinux.pdf
Linux.pdf
 
Oracle forensics 101
Oracle forensics 101Oracle forensics 101
Oracle forensics 101
 
FIXING BLOCK CORRUPTION (RMAN) on 11G
FIXING BLOCK CORRUPTION (RMAN) on 11GFIXING BLOCK CORRUPTION (RMAN) on 11G
FIXING BLOCK CORRUPTION (RMAN) on 11G
 
Direct SGA access without SQL
Direct SGA access without SQLDirect SGA access without SQL
Direct SGA access without SQL
 
Oracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspackOracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspack
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
Linux configer
Linux configerLinux configer
Linux configer
 
Hot and cold data storage
Hot and cold data storageHot and cold data storage
Hot and cold data storage
 
oracle dba
oracle dbaoracle dba
oracle dba
 
Javantura v2 - Replication with MongoDB - what could go wrong... - Philipp Krenn
Javantura v2 - Replication with MongoDB - what could go wrong... - Philipp KrennJavantura v2 - Replication with MongoDB - what could go wrong... - Philipp Krenn
Javantura v2 - Replication with MongoDB - what could go wrong... - Philipp Krenn
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
 

More from cookie1969

con9577-ash-deep-dive-oow2013-2031468.pdf
con9577-ash-deep-dive-oow2013-2031468.pdfcon9577-ash-deep-dive-oow2013-2031468.pdf
con9577-ash-deep-dive-oow2013-2031468.pdf
cookie1969
 
200603ash.pdf Performance Tuning Oracle DB
200603ash.pdf Performance Tuning Oracle DB200603ash.pdf Performance Tuning Oracle DB
200603ash.pdf Performance Tuning Oracle DB
cookie1969
 
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
cookie1969
 
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
Hailey_Database_Performance_Made_Easy_through_Graphics.pdfHailey_Database_Performance_Made_Easy_through_Graphics.pdf
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
cookie1969
 
Pivot query on Automatic Workload Repository _ OraDBA.pdf
Pivot query on Automatic Workload Repository _ OraDBA.pdfPivot query on Automatic Workload Repository _ OraDBA.pdf
Pivot query on Automatic Workload Repository _ OraDBA.pdf
cookie1969
 
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdfZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
cookie1969
 
02062022_ASH_Bazgutdinov_FORS.pdf
02062022_ASH_Bazgutdinov_FORS.pdf02062022_ASH_Bazgutdinov_FORS.pdf
02062022_ASH_Bazgutdinov_FORS.pdf
cookie1969
 
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdfNOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
cookie1969
 

More from cookie1969 (11)

con9577-ash-deep-dive-oow2013-2031468.pdf
con9577-ash-deep-dive-oow2013-2031468.pdfcon9577-ash-deep-dive-oow2013-2031468.pdf
con9577-ash-deep-dive-oow2013-2031468.pdf
 
200603ash.pdf Performance Tuning Oracle DB
200603ash.pdf Performance Tuning Oracle DB200603ash.pdf Performance Tuning Oracle DB
200603ash.pdf Performance Tuning Oracle DB
 
active_session_history_oracle_performance.ppt
active_session_history_oracle_performance.pptactive_session_history_oracle_performance.ppt
active_session_history_oracle_performance.ppt
 
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
dokumen.tips_oracle-10g-advanced-performance-tuning-kyle-hailey-kylelfgmailco...
 
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
Hailey_Database_Performance_Made_Easy_through_Graphics.pdfHailey_Database_Performance_Made_Easy_through_Graphics.pdf
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
 
Pivot query on Automatic Workload Repository _ OraDBA.pdf
Pivot query on Automatic Workload Repository _ OraDBA.pdfPivot query on Automatic Workload Repository _ OraDBA.pdf
Pivot query on Automatic Workload Repository _ OraDBA.pdf
 
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdfZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
ZcCsXvjIRFKcqd2Yzt4d_Shallahamer-MLPoorPerf-3d.pdf
 
R07_Senegacnik_CBO.pdf
R07_Senegacnik_CBO.pdfR07_Senegacnik_CBO.pdf
R07_Senegacnik_CBO.pdf
 
IO_Analysis_with_SAR.ppt
IO_Analysis_with_SAR.pptIO_Analysis_with_SAR.ppt
IO_Analysis_with_SAR.ppt
 
02062022_ASH_Bazgutdinov_FORS.pdf
02062022_ASH_Bazgutdinov_FORS.pdf02062022_ASH_Bazgutdinov_FORS.pdf
02062022_ASH_Bazgutdinov_FORS.pdf
 
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdfNOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
NOCOUG_201311_Fine_Tuning_Execution_Plans.pdf
 

Recently uploaded

Hall booking system project report .pdf
Hall booking system project report  .pdfHall booking system project report  .pdf
Hall booking system project report .pdf
Kamal Acharya
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
R&R Consult
 
Antenna efficency lecture course chapter 3.pdf
Antenna  efficency lecture course chapter 3.pdfAntenna  efficency lecture course chapter 3.pdf
Antenna efficency lecture course chapter 3.pdf
AbrahamGadissa
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
fxintegritypublishin
 
Automobile Management System Project Report.pdf
Automobile Management System Project Report.pdfAutomobile Management System Project Report.pdf
Automobile Management System Project Report.pdf
Kamal Acharya
 
Online blood donation management system project.pdf
Online blood donation management system project.pdfOnline blood donation management system project.pdf
Online blood donation management system project.pdf
Kamal Acharya
 
LIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.pptLIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.ppt
ssuser9bd3ba
 

Recently uploaded (20)

Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
 
Hall booking system project report .pdf
Hall booking system project report  .pdfHall booking system project report  .pdf
Hall booking system project report .pdf
 
HYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generationHYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generation
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
 
Construction method of steel structure space frame .pptx
Construction method of steel structure space frame .pptxConstruction method of steel structure space frame .pptx
Construction method of steel structure space frame .pptx
 
Antenna efficency lecture course chapter 3.pdf
Antenna  efficency lecture course chapter 3.pdfAntenna  efficency lecture course chapter 3.pdf
Antenna efficency lecture course chapter 3.pdf
 
NO1 Pandit Amil Baba In Bahawalpur, Sargodha, Sialkot, Sheikhupura, Rahim Yar...
NO1 Pandit Amil Baba In Bahawalpur, Sargodha, Sialkot, Sheikhupura, Rahim Yar...NO1 Pandit Amil Baba In Bahawalpur, Sargodha, Sialkot, Sheikhupura, Rahim Yar...
NO1 Pandit Amil Baba In Bahawalpur, Sargodha, Sialkot, Sheikhupura, Rahim Yar...
 
Top 13 Famous Civil Engineering Scientist
Top 13 Famous Civil Engineering ScientistTop 13 Famous Civil Engineering Scientist
Top 13 Famous Civil Engineering Scientist
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
 
fundamentals of drawing and isometric and orthographic projection
fundamentals of drawing and isometric and orthographic projectionfundamentals of drawing and isometric and orthographic projection
fundamentals of drawing and isometric and orthographic projection
 
weather web application report.pdf
weather web application report.pdfweather web application report.pdf
weather web application report.pdf
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
 
Automobile Management System Project Report.pdf
Automobile Management System Project Report.pdfAutomobile Management System Project Report.pdf
Automobile Management System Project Report.pdf
 
ENERGY STORAGE DEVICES INTRODUCTION UNIT-I
ENERGY STORAGE DEVICES  INTRODUCTION UNIT-IENERGY STORAGE DEVICES  INTRODUCTION UNIT-I
ENERGY STORAGE DEVICES INTRODUCTION UNIT-I
 
Online blood donation management system project.pdf
Online blood donation management system project.pdfOnline blood donation management system project.pdf
Online blood donation management system project.pdf
 
ASME IX(9) 2007 Full Version .pdf
ASME IX(9)  2007 Full Version       .pdfASME IX(9)  2007 Full Version       .pdf
ASME IX(9) 2007 Full Version .pdf
 
shape functions of 1D and 2 D rectangular elements.pptx
shape functions of 1D and 2 D rectangular elements.pptxshape functions of 1D and 2 D rectangular elements.pptx
shape functions of 1D and 2 D rectangular elements.pptx
 
Courier management system project report.pdf
Courier management system project report.pdfCourier management system project report.pdf
Courier management system project report.pdf
 
LIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.pptLIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.ppt
 
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
 

Kyle Hailey Oracle Performance IO Waits

  • 2. Copyright 2006 Kyle Hailey Waits Covered in this Section 1. db file sequential read 2. db file scattered read 3. db file parallel read 4. read by other session 5. direct path read 6. direct path write 7. direct path read temp 8. direct path write temp 9. direct path write (lob) 10. Local write wait 11. data file init write Buffer Cache I/O PGA I/O Special Cases
  • 3. Copyright 2006 Kyle Hailey IO Waits Data Files DBWR Buffer Cache User1 SGA Library Cache Log Buffer
  • 4. Copyright 2006 Kyle Hailey Standard Table/Index IO 1. db file sequential read  Single block read 2. db file scattered read  Multi block read 3. db file parallel read  Non-contiguous multi block read 4. read by other session  Wait for another session to do the io
  • 5. Copyright 2006 Kyle Hailey db file sequential read Shadow Process 1) Search Buffer Cache for the block by rowid 2) Fails 3) Reads of disk • File • Block Buffer Cache  The top wait  Single Block Read Data via Index and/or Rowid Rollback (if expecting FTS => problem) Select * from emp where empno = 9999; Note: “sequential” means A sequence like a rowid Index on emp.empno
  • 6. Copyright 2006 Kyle Hailey db file scattered read Shadow Process 1) Search Buffer Cache for the blocks 2) Fails 3) Reads off disk • File • Block • Multi Block Read Count Buffer Cache  Multi Block Read  Full Table Scan  Index Fast Full Scans SQL> select * from emp; Note: Scattered Means Blocks are read and scattered throughout buffer cache db_file_multiblock_read_count
  • 7. Copyright 2006 Kyle Hailey db file parallel read Shadow Process 1) Search Buffer Cache for the blocks 2) Fails 3) Reads that block off Disk Buffer Cache  Process issues multiple single block reads in parallel  Documentation says only for recovery  But seems to happen for normal read ops  Async Call – wait for all reads to complete  Examples  Not contiguous multi block read  Index full or range scan  Where values in Select * from emp where Empno in (1,2,3,4,5,6);
  • 8. Copyright 2006 Kyle Hailey Read by other Session 1. Block not found in cache 2. Read block from disk 3. Found other session already reading block from disk 4. Wait for the other session to finish IO S1 Buffer Cache S2 Multiple sessions reading the same data that requires IO Ex) two users doing a full table scan at the same time
  • 9. Copyright 2006 Kyle Hailey IO – Further Investigation select parameter1, parameter2, parameter3 from v$event_name NAME P1 P2 P3 ----------------------- ----------- --------- -------- read by other session file# block# class# db file sequential read file# block# blocks db file scattered read file# block# blocks db file parallel read files blocks requests P1 and P2 are file# , block# We can use P1, P2, P3 from ASH to get more info Exception: db file parallel read – p1,p2,p3 not useful
  • 10. Copyright 2006 Kyle Hailey sequential reads – blocks read select event, ash.p3, o.object_name objn, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'db file sequential read' and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time; EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID ----------------------- -- ------------------------- ----- ----- ------ ------------- db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737 db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0 db file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9s P3= # of blocks
  • 11. Copyright 2006 Kyle Hailey scattered reads – multi blocks read select event, ash.p3, o.object_name objn, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'db file scattered read' and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time; EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID ---------------------- -- ------------------------- ----- ----- ------ ------------- db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363 db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676 db file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hu P3= # of blocks
  • 12. IO by File select io.cnt, round(io.cnt/(&v_minutes*60),2) aas, io.event, io.p1 p1, f.tablespace_name from ( select count(*) cnt, substr(event,0,25) event, ash.p1 p1 from v$active_session_history ash where ( event like 'db file s%' or event like 'direct%' ) and sample_time > sysdate - &v_minutes/(60*24) group by event , ash.p1 ) io, dba_data_files f where f.file_id = io.p1 Order by io.cnt ; CNT AAS EVENT P1 TABLESPACE ---- --- ----------------------- --- --------- - 1 .00 db file sequential read 1 SYSTEM 2 .00 db file sequential read 3 SYSAUX 38 .06 db file sequential read 6 SOE 179 .30 db file sequential read 7 SOEINDEX
  • 13. IO by Object select count(*) cnt, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype from v$active_session_history ash, all_objects o where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &1/(60*24) and session_state='WAITING' group by CURRENT_OBJ#, o.object_name , o.object_type Order by count(*) CNT AAS OBJN OTYPE ---- ------- ------------------------- --------------- 79 .00 52949 ORDER_ITEMS TABLE PARTITION 97 .00 -1 130 .00 53117 ORD_STATUS_IX INDEX 498 .01 53120 CUST_EMAIL_IX INDEX 512 .01 0 1632 .03 53112 ITEM_ORDER_IX INDEX
  • 14. IO by SQL select round(sum(cnt) over ( partition by io.sql_id order by sql_id ) / (&v_minutes*60),2) aas, io.sql_id, io.cnt cnt, 100*cnt/sum(cnt) over ( partition by io.sql_id order by sql_id ) pct, o.object_name obj, o.subobject_name sub_obj, o.object_type otype, substr(io.event,8,10) event, io.p1 file#, f.tablespace_name tablespace_name, tbs.contents from ( select sql_id, event, count(*) cnt, count(*) / (&v_minutes*60) aas, CURRENT_OBJ# , ash.p1 from v$active_session_history ash where ( event like 'db file s%' or event like 'direct%' ) and sample_time > sysdate - &v_minutes/(60*24) group by CURRENT_OBJ#, event, ash.p1, sql_id ) io, dba_data_files f ,all_objects o , dba_tablespaces tbs where f.file_id = io.p1 and o.object_id (+)= io.CURRENT_OBJ# and tbs.tablespace_name= f.tablespace_name Order by tcnt, sql_id, cnt / AAS SQL_ID % OBJ TABLESPACE ----- ------------- --- --------------- ---------- .18 0yas01u2p9ch4 6 ITEM_PRODUCT_IX SOEINDEX 6 ORDER_ITEMS_UK SOEINDEX 88 ITEM_ORDER_IX SOEINDEX .32 6v6gm0fd1rgrz 6 WAIT_OBJECTS SYSTEM 94 UNDO UNDOTBS1
  • 15. Copyright 2006 Kyle Hailey Missing Object IDs  dba_extents – get object name  Dba_extents is notoriously slow  Options  Create a table as select – use table  recreate as object extents change  Catch blocks in x$bh when waits occur select segment_name, segment_type from dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1;
  • 16. Copy of dba_extents Enter value for file: 3 Enter value for block: 6619 OWNER SEGMENT_NAME SEGMENT_TYPE --------------- -------------------- ------------------ WMSYS LOG_TAB_PK INDEX Elapsed: 00:00:41.25 create table myextents as select * from dba_extents Table created. Elapsed: 00:01:25.73 CNT OWN SEGMENT_NAME SEGMENT_TYPE --- ---- ---------------- ------------ 11 SYS SMON_SCN_TO_TIME CLUSTER 993 SYS _SYSSMU7$ TYPE2 UNDO 150 rows selected. Elapsed: 00:00:01.03
  • 17. Copyright 2006 Kyle Hailey IO Solutions 1. Check average read times per file  Should be between 5-20 ms  Data in Statspack under “File IO Stats” 2. Check Cache buffer Hit ratio  Check db_cache_advice 9i and higher  Data in Statspack under “Buffer Pool Advisory”  Want to optimize data caching 3. Tune High IO SQL  Check reads off of UNDO
  • 18. Copyright 2006 Kyle Hailey Step 1a: Ave Read Time File IO Stats DB/Inst:labsf03 Snaps: 1-2 Tablespace Filename ------------------------ ---------------------------------------------------- Av Mx Av Av Rd Rd Av Av Buffer BufWt Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms) ---------- ---- ------- ----- --- ------- ------------ -------- ---------- -- SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf 445 15 0.4 16 1.0 1,157 39 2,744 93.3 USERS /u01/app/oracle/oradata/labsf03/users01.dbf 223 7 0.5 ### 1.0 9,725 324 4 100.0 IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching
  • 19. Copyright 2006 Kyle Hailey Step 1b: Ave Read Time IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching select to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time, file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds from DBA_HIST_FILESTATXS f, dba_hist_snapshot s where f.snap_id=s.snap_id ; SNAP_TIME FILE# AVG_READ_MS PHYRDS ---------------- ---------- ----------- ---- 2008-01-05 12:00 9 36.67 39 2008-01-05 12:00 10 32.31 39 2008-01-05 13:00 1 11.63 178224 2008-01-05 13:00 2 56.37 2014 2008-01-05 13:00 3 17.38 50668 2008-01-05 13:00 4 9.39 565321 2008-01-05 13:00 5 38.78 41 2008-01-05 13:00 6 28.29 41 2008-01-05 13:00 7 27.44 39 2008-01-05 13:00 8 42.56 39 2008-01-05 13:00 9 36.67 39 2008-01-05 13:00 10 32.31 39
  • 20. Copyright 2006 Kyle Hailey Step 1c: Ave Read Time IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching select to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time, file_id fid, average_read_time *10 avgrd_ms, average_write_time *10 avgwr_ms, physical_reads pr, physical_writes pw from V$FILEMETRIC_HISTORY f order by begin_time; BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW ---------------- ---- -------- -------- ---------- ---------- 2008-01-30 12:25 1 5.2 2.5 100 24 2008-01-30 12:25 5 80.0 20.0 2 2 2008-01-30 12:25 2 36.7 2.0 3 218 2008-01-30 12:25 4 22.8 1.8 89 2754 2008-01-30 12:25 3 22.9 2.6 14 47
  • 21. Step1 & OEM  ADDM advisory expects I/Os to be under 10ms  Can change the behavior with exec DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER( 'ADDM', 'DBIO_EXPECTED', <new value>);
  • 22. Copyright 2006 Kyle Hailey Step 2 : Buffer Pool Advisory Buffer Pool Advisory Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ------------ ------ -------------- D 56 .1 6,986 2.3 58,928 D 112 .2 13,972 1.6 42,043 D 224 .4 27,944 1.0 25,772 D 336 .6 41,916 1.0 25,715 D 448 .8 55,888 1.0 25,715 D 596 1.0 74,351 1.0 25,715 D 728 1.2 90,818 1.0 25,715 D 840 1.4 104,790 1.0 25,715 D 952 1.6 118,762 1.0 25,715 D 1,064 1.8 132,734 1.0 25,715
  • 23. Copyright 2006 Kyle Hailey Step 3: Find Top I/O SQL select count(*),sql_id, event from v$active_session_history where event in ('db file sequential read', 'db file scattered read', 'db file parallel read') group by sql_id, event order by count(*); COUNT(*) SQL_ID EVENT --------- ------------- ------------------------ 10 8hk7xvhua40va db file sequential read 335 3hatpjzrqvfn7 db file sequential read 343 0uuqgjq7k12nf db file sequential read 549 75621g9y3xmvd db file sequential read 1311 0bzhqhhj9mpaa db file sequential read 1523 41zu158rqf4kf db file sequential read 1757 0yas01u2p9ch4 db file sequential read
  • 24. Copyright 2006 Kyle Hailey Step 3: OEM find top SQL
  • 25. Copyright 2006 Kyle Hailey Machine Memory IO Configuration Issues SGA Unix File Cache If average IO read times are under 7ms them probably coming from Unix File Cache  Unix Buffer Cache  Disk Spindles  Raid 5 Not Physical IO Physical IO
  • 26. Copyright 2006 Kyle Hailey IO Throughput One Disk 74G – Few IOPS 37 Disks 2G – Many IOPS 92 IO/sec 1887 IO/sec
  • 27. Copyright 2006 Kyle Hailey Disk 2Gb vs 70GB Seagate Barracuda 4LP Seagate Cheetah 73LP Capacity 2.16GB 73.4GB Rotation Speed 7200rpm 10000rpm Rotational Delay(avg) 4.1ms 3ms Time to read 32Kb 6ms 3ms Seek Time (avg) 9.4ms 4.9 Total time for Single I/O 19.5ms 10.9ms I/O per second (conservative) 51 92 IOPs/sec per 100GB 2550 126 James Morle http://scaleability.com
  • 28. Copyright 2006 Kyle Hailey Raid 5 P1 P2 P1 P2 Writes require two reads plus a write – read data, read parity then write Reads can conflict on smaller stripe sizes 32K or 64K Readers on different disks run without conflict http://www.miracleas.com/BAARF/BAARF2.html B – Battle A – Against A – Any R – Raid F - Five
  • 29. Raid Levels Copyright 2006 Kyle Hailey http://www.miracleas.com/BAARF/1.Millsap2000.01.03-RAID5.pdf Raid Level none 0 (stripe) 0+1 (striped then mirrored) 5 Control file 2 1 1 3 Redo log 4 1 1 3 System tablespace 2 1 1 3 Sort segment 4 1 1 3 Rollback segment 2 1 1 5 Index read only 2 1 1 1 Sequential read only 4 1 1 3 DBWR intensive files 1 1 1 5 Direct load intensive 4 1 1 3 Data protection 4 5 1 2 Operating costs 1 1 5 3
  • 30. Copyright 2006 Kyle Hailey Direct I/O WAITS Direct I/O : This mechanism lets the client bypass the buffer cache for I/O intensive operations. The disk blocks are written into and read from process private memory.  direct path read :  Parallel Query  direct path write  sqlldr  loading/reading LOBs  parallel DMLs  create table as select  create index  direct path read temp , direct path write temp  Sorting
  • 31. Copyright 2006 Kyle Hailey Direct IO Shadow Process Buffer Cache PGA Sort Area Direct Path PQO Reads X
  • 32. Copyright 2006 Kyle Hailey direct path read _serial_direct_read = true Buffer Cache Buffer Cache Buffer Cache Buffer Cache Shadow Shadow Process Process PGA PGA PQO Shadow Shadow Process Process PGA PGA PQO X  Parallel Query  See v$px_session select parameter1, parameter2, parameter3 from v$event_name where name='direct path read'; PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt
  • 33. Copyright 2006 Kyle Hailey Further Investigation ASH ----- ----- ---- ----------- ------ -- ------ ------------ - SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID --- ---- -- ----- ------ -- ------ ------------- 149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722 144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722 54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722 149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722 select session_id sid, QC_SESSION_ID qsid, ash.p3, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'direct path read' and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time;
  • 34. select ash.SQL_ID, QC_SESSION_ID qsid, count(*) cnt, count (distinct session_id) deg, nvl(o.object_name,to_char(CURRENT_OBJ#)) obj, o.object_type otype, decode(session_state, 'WAITING',event,'CPU') event from v$active_session_history ash, all_objects o where o.object_id (+)= ash.CURRENT_OBJ# and qc_session_id is not null group by qc_session_id, sql_id, o.object_name, o.object_type, CURRENT_OBJ#, event, session_state Order by qc_session_id, sql_id PQO - ASH SQL_ID QSID CNT DEG OBJ OTYPE EVENT ------------- ---- ---- --- ------------ ----- ------------------------- - 7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd 144 4 3 WAIT_OBJECTS TABLE PX qref latch 144 37 1 WAIT_OBJECTS TABLE db file sequential read 144 3 2 WAIT_OBJECTS TABLE direct path read 144 70 1 WAIT_OBJECTS TABLE CPU 144 21 4 0 PX Deq Credit: send blkd 144 12 4 0 db file sequential read
  • 35. Copyright 2006 Kyle Hailey direct path write Buffer Cache Buffer Cache Buffer Cache Buffer Cache Shadow Shadow Process Process PGA PGA Shadow Shadow Process Process PGA PGA DBWR DBWR DBWR DBWR DATA DATA DATA DATA Seems to be a bug on 10g where direct path write waits are incorrectly recorded as CPU select parameter1, parameter2, parameter3 from v$event_name where name='direct path write'; PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt  Occurs when:  insert /*+ APPEND */  sql*loader direct=y  Create table as select  Loading LOBS  Parallel insert  Create index
  • 36. Copyright 2006 Kyle Hailey Direct Path Read Temp Direct Path Write Temp  Sorting  Write to Temp  Read from Temp Shadow Shadow Process Process Buffer Cache Buffer Cache PGA PGA TEMP TEMP DATA DATA Shadow Shadow Process Process Buffer Cache Buffer Cache PGA PGA TEMP TEMP DATA DATA select parameter1, parameter2, parameter3 from v$event_name where name in ('direct path write temp‘, ‘direct path read temp’); PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt file number first dba block cnt
  • 37. Copyright 2006 Kyle Hailey Direct Path Read Temp Direct Path Write Temp Buffer Cache Buffer Cache Buffer Cache Buffer Cache Shadow Shadow Process Process PGA PGA TEMP TEMP DATA DATA Shadow Shadow Process Process PGA PGA TEMP TEMP DATA DATA This event happens when sorting overflows the memory buffers and has to be written to disk. If it's a problem consider increasing parameter pga_aggregate_target The explain plan can give estimated sort size requirements and extended row source stats in the explain plan can give the actual sort size usage of the SQL statement select s.sid, s. process, s.sql_id, tmp.segtype, ((tmp.blocks*8)/1024) MB, tmp.tablespace from v$tempseg_usage tmp, v$session s where tmp.session_num=s.serial# and segtype in ('HASH','SORT') order by blocks desc SID SQL_ID SEGTYPE MB TABLESPACE --- ------------- ------- -- ---------- 90 gm5s0vc0vzppc SORT 2 TEMP03 105 gm5s0vc0vzppc SORT 2 TEMP03 80 gm5s0vc0vzppc SORT 2 TEMP03 105 gm5s0vc0vzppc HASH 2 TEMP03 102 gm5s0vc0vzppc SORT 2 TEMP03 90 gm5s0vc0vzppc HASH 2 TEMP03
  • 38. PGA Aggregate Target 9i, a single sessions maximum sort area using PGA_AGGREGATE_TARGET was fixed to 100M unless setting _pga_max_size: Maximum PGA size for a single process Defaults to 200M _smm_max_size: Maximum workarea size for one process , defined in KILO BYTES Defaults Min(0.05*pga_aggregate_target,0.5*_pga_max_size,100M) In 10.2 the 100M does not apply anymore when using a PGA_AGGREGATE_TARGET > 1000M. In 10.2 the 100M applies as long as pga_aggregate_target is smaller than 1GB. In 10.2 pga_aggregate_target set to 5GB _smm_max_size of 500MB (was 100MB ) _pga_max_size of 1000MB (was 200MB). Copyright 2006 Kyle Hailey Manual override broken 10.2.0.3 fixed in 11.1.0.7 alter session set workarea_size_policy = manual; alter session set sort_area_size = 104857600; Workaround, run command twice alter session set sort_area_size = 104857600; alter session set sort_area_size = 104857600; But doesn’t work for parallel connections that still fall back on default which is 64K unless otherwise specified HASH_AREA_SIZE seems unaffected by problem
  • 39. Copyright 2006 Kyle Hailey Data file init write  When autoextend is set and many extensions are happening  Can be the root cause of other waits  Enq: TX - contention
  • 40. Copyright 2006 Kyle Hailey Local Write Wait  Truncating a table  Wait for Data Cache to be cleared of all blocks of truncated table  Wait by shadow for DBWR  If a problem try  GTT  Reduce fast_start_mttr_target
  • 41. Copyright 2006 Kyle Hailey Further Investigation NAME P1 P2 P3 ----------------- ----- ------ ----- local write wait file# block# select * from v$event_name where name = ‘local write wait' select ash.p1,ash.p2, ash.SQL_ID, count(*) from v$ash ash where event='local write wait' group by ash.p1,ash.p2, ash.SQL_ID; P1 P2 SQL_ID COUNT --- -- ------------- ----- 201 2 0q8k9xfmz0k2k 194 201 5 0q8k9xfmz0k2k 34 201 6 0q8k9xfmz0k2k 9 201 4 0q8k9xfmz0k2k 10
  • 42. Copyright 2006 Kyle Hailey Temporary File #’s SQL> select file# from v$datafile; FILE# ---------- 1 2 3 4 5 6 7 8 SQL> select file# from v$tempfile; FILE# ---------- 2 1 SQL> show parameters db_files NAME VALUE ---------- ------------- db_files 200 Wait Temporary File#’s = Db_files + file# File# 201 = v$tempfile 1
  • 43. Copyright 2006 Kyle Hailey Local Write Wait:extent allocation CREATE TEMPORARY TABLESPACE "NEWTEMP" TEMPFILE '/d3/temp01.dbf' SIZE 100M AUTOEXTEND ON EXTENT MANAGEMENT LOCAL UNIFORM SIZE 64K; select a.*, b.* from dba_objects a, dba_objects b order by a.owner ALTER DATABASE DEFAULT TEMPORARY TABLESPACE newtemp;
  • 44. Copyright 2006 Kyle Hailey Local Write Waits
  • 45. Copyright 2006 Kyle Hailey Local Write Wait: Solutions  Truncating a table  Reduce fast_start_mttr  Use GTT  Temporary Table Space Allocations  Increase extent size
  • 46. Copyright 2006 Kyle Hailey Summary I/O  Buffer Cache IO  db file sequential read  db file scattered read  db file parallel read  Tune  I/O’s should be 5-15ms  Check Buffer Cache Advisory  Tune SQL
  • 47. Copyright 2006 Kyle Hailey Summary Direct I/O and Other  direct path read  PQO  direct path write  direct path operations  direct path read temp / direct path write temp  sorts  direct path write (lob)  stored NOCACHE  local write wait –  Temp extensions  Truncates  data file init write – data file extension