This document discusses various types of I/O waits in an Oracle database including sequential reads, scattered reads, parallel reads, and reads by other sessions. It provides details on identifying and troubleshooting I/O issues through metrics like average read times, buffer cache hit ratios, and top SQL statements causing I/O. Specific solutions covered include right-sizing the SGA and buffer cache, tuning high I/O SQL, and ensuring adequate disk throughput.
2. Copyright 2006 Kyle Hailey
Waits Covered in this Section
1. db file sequential read
2. db file scattered read
3. db file parallel read
4. read by other session
5. direct path read
6. direct path write
7. direct path read temp
8. direct path write temp
9. direct path write (lob)
10. Local write wait
11. data file init write
Buffer Cache I/O
PGA I/O
Special Cases
4. Copyright 2006 Kyle Hailey
Standard Table/Index IO
1. db file sequential read
Single block read
2. db file scattered read
Multi block read
3. db file parallel read
Non-contiguous multi block read
4. read by other session
Wait for another session to do the io
5. Copyright 2006 Kyle Hailey
db file sequential read
Shadow
Process
1) Search Buffer Cache for
the block by rowid
2) Fails
3) Reads of disk
• File
• Block
Buffer Cache
The top wait
Single Block Read
Data via Index and/or Rowid
Rollback (if expecting FTS => problem)
Select * from emp where empno = 9999;
Note: “sequential” means
A sequence like a rowid
Index on emp.empno
6. Copyright 2006 Kyle Hailey
db file scattered read
Shadow
Process
1) Search Buffer Cache for the
blocks
2) Fails
3) Reads off disk
• File
• Block
• Multi Block Read Count
Buffer Cache
Multi Block Read
Full Table Scan
Index Fast Full Scans
SQL> select * from emp;
Note: Scattered Means
Blocks are read and
scattered throughout
buffer cache
db_file_multiblock_read_count
7. Copyright 2006 Kyle Hailey
db file parallel read
Shadow
Process
1) Search Buffer Cache for
the blocks
2) Fails
3) Reads that block off Disk
Buffer Cache Process issues multiple single block reads in parallel
Documentation says only for recovery
But seems to happen for normal read ops
Async Call – wait for all reads to complete
Examples
Not contiguous multi block read
Index full or range scan
Where values in Select * from emp where
Empno in (1,2,3,4,5,6);
8. Copyright 2006 Kyle Hailey
Read by other Session
1. Block not found in cache
2. Read block from disk
3. Found other session already
reading block from disk
4. Wait for the other session to finish
IO
S1
Buffer Cache
S2
Multiple sessions reading the
same data that requires IO
Ex) two users doing a full table
scan at the same time
9. Copyright 2006 Kyle Hailey
IO – Further Investigation
select parameter1, parameter2, parameter3 from v$event_name
NAME P1 P2 P3
----------------------- ----------- --------- --------
read by other session file# block# class#
db file sequential read file# block# blocks
db file scattered read file# block# blocks
db file parallel read files blocks requests
P1 and P2 are file# , block#
We can use P1, P2, P3 from ASH to get more info
Exception:
db file parallel read – p1,p2,p3 not useful
10. Copyright 2006 Kyle Hailey
sequential reads – blocks read
select
event,
ash.p3,
o.object_name objn,
o.object_type otype,
CURRENT_FILE# filen,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'db file sequential read'
and o.object_id (+)= ash.CURRENT_OBJ#
order by sample_time;
EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID
----------------------- -- ------------------------- ----- ----- ------ -------------
db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737
db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0
db file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9s
P3= # of
blocks
11. Copyright 2006 Kyle Hailey
scattered reads – multi blocks read
select
event,
ash.p3,
o.object_name objn,
o.object_type otype,
CURRENT_FILE# filen,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'db file scattered read'
and o.object_id (+)= ash.CURRENT_OBJ#
order by sample_time;
EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID
---------------------- -- ------------------------- ----- ----- ------ -------------
db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363
db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676
db file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hu
P3= # of
blocks
12. IO by File
select io.cnt,
round(io.cnt/(&v_minutes*60),2) aas,
io.event,
io.p1 p1,
f.tablespace_name
from ( select
count(*) cnt,
substr(event,0,25) event,
ash.p1 p1
from v$active_session_history ash
where ( event like 'db file s%' or event like 'direct%' )
and sample_time > sysdate - &v_minutes/(60*24)
group by
event ,
ash.p1
) io,
dba_data_files f
where
f.file_id = io.p1
Order by io.cnt ;
CNT AAS EVENT P1 TABLESPACE
---- --- ----------------------- --- ---------
-
1 .00 db file sequential read 1 SYSTEM
2 .00 db file sequential read 3 SYSAUX
38 .06 db file sequential read 6 SOE
179 .30 db file sequential read 7 SOEINDEX
13. IO by Object
select
count(*) cnt,
CURRENT_OBJ#||' '||o.object_name objn,
o.object_type otype
from v$active_session_history ash,
all_objects o
where ( event like 'db file s%' or event like 'direct%' )
and o.object_id (+)= ash.CURRENT_OBJ#
and sample_time > sysdate - &1/(60*24)
and session_state='WAITING'
group by
CURRENT_OBJ#, o.object_name ,
o.object_type
Order by count(*)
CNT AAS OBJN OTYPE
---- ------- ------------------------- ---------------
79 .00 52949 ORDER_ITEMS TABLE PARTITION
97 .00 -1
130 .00 53117 ORD_STATUS_IX INDEX
498 .01 53120 CUST_EMAIL_IX INDEX
512 .01 0
1632 .03 53112 ITEM_ORDER_IX INDEX
14. IO by SQL
select
round(sum(cnt) over ( partition by io.sql_id order by sql_id ) / (&v_minutes*60),2) aas,
io.sql_id,
io.cnt cnt,
100*cnt/sum(cnt) over ( partition by io.sql_id order by sql_id ) pct,
o.object_name obj,
o.subobject_name sub_obj,
o.object_type otype,
substr(io.event,8,10) event,
io.p1 file#,
f.tablespace_name tablespace_name,
tbs.contents
from
(
select
sql_id,
event,
count(*) cnt,
count(*) / (&v_minutes*60) aas,
CURRENT_OBJ# ,
ash.p1
from v$active_session_history ash
where ( event like 'db file s%' or event like 'direct%' )
and sample_time > sysdate - &v_minutes/(60*24)
group by
CURRENT_OBJ#,
event,
ash.p1,
sql_id
) io,
dba_data_files f
,all_objects o
, dba_tablespaces tbs
where
f.file_id = io.p1
and o.object_id (+)= io.CURRENT_OBJ#
and tbs.tablespace_name= f.tablespace_name
Order by tcnt, sql_id, cnt
/
AAS SQL_ID % OBJ TABLESPACE
----- ------------- --- --------------- ----------
.18 0yas01u2p9ch4 6 ITEM_PRODUCT_IX SOEINDEX
6 ORDER_ITEMS_UK SOEINDEX
88 ITEM_ORDER_IX SOEINDEX
.32 6v6gm0fd1rgrz 6 WAIT_OBJECTS SYSTEM
94 UNDO UNDOTBS1
15. Copyright 2006 Kyle Hailey
Missing Object IDs
dba_extents – get object name
Dba_extents is notoriously slow
Options
Create a table as select – use table
recreate as object extents change
Catch blocks in x$bh when waits occur
select segment_name,
segment_type
from dba_extents
where file_id = P1
and P2 between
block_id and block_id + blocks – 1;
16. Copy of dba_extents
Enter value for file: 3
Enter value for block: 6619
OWNER SEGMENT_NAME SEGMENT_TYPE
--------------- -------------------- ------------------
WMSYS LOG_TAB_PK INDEX
Elapsed: 00:00:41.25
create table myextents as select * from dba_extents
Table created.
Elapsed: 00:01:25.73
CNT OWN SEGMENT_NAME SEGMENT_TYPE
--- ---- ---------------- ------------
11 SYS SMON_SCN_TO_TIME CLUSTER
993 SYS _SYSSMU7$ TYPE2 UNDO
150 rows selected.
Elapsed: 00:00:01.03
17. Copyright 2006 Kyle Hailey
IO Solutions
1. Check average read times per file
Should be between 5-20 ms
Data in Statspack under “File IO Stats”
2. Check Cache buffer Hit ratio
Check db_cache_advice 9i and higher
Data in Statspack under “Buffer Pool Advisory”
Want to optimize data caching
3. Tune High IO SQL
Check reads off of UNDO
18. Copyright 2006 Kyle Hailey
Step 1a: Ave Read Time
File IO Stats DB/Inst:labsf03 Snaps: 1-2
Tablespace Filename
------------------------ ----------------------------------------------------
Av Mx
Av
Av Rd Rd Av Av Buffer BufWt
Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)
---------- ---- ------- ----- --- ------- ------------ -------- ---------- --
SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf
445 15 0.4 16 1.0 1,157 39 2,744 93.3
USERS /u01/app/oracle/oradata/labsf03/users01.dbf
223 7 0.5 ### 1.0 9,725 324 4 100.0
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
19. Copyright 2006 Kyle Hailey
Step 1b: Ave Read Time
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
select
to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time,
file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds
from
DBA_HIST_FILESTATXS f,
dba_hist_snapshot s
where f.snap_id=s.snap_id ;
SNAP_TIME FILE# AVG_READ_MS PHYRDS
---------------- ---------- ----------- ----
2008-01-05 12:00 9 36.67 39
2008-01-05 12:00 10 32.31 39
2008-01-05 13:00 1 11.63 178224
2008-01-05 13:00 2 56.37 2014
2008-01-05 13:00 3 17.38 50668
2008-01-05 13:00 4 9.39 565321
2008-01-05 13:00 5 38.78 41
2008-01-05 13:00 6 28.29 41
2008-01-05 13:00 7 27.44 39
2008-01-05 13:00 8 42.56 39
2008-01-05 13:00 9 36.67 39
2008-01-05 13:00 10 32.31 39
20. Copyright 2006 Kyle Hailey
Step 1c: Ave Read Time
IO should be under 20ms
Fastest possible is around 7ms
Faster speeds are due to read caching
select
to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time,
file_id fid,
average_read_time *10 avgrd_ms,
average_write_time *10 avgwr_ms,
physical_reads pr,
physical_writes pw
from
V$FILEMETRIC_HISTORY f
order by begin_time;
BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW
---------------- ---- -------- -------- ---------- ----------
2008-01-30 12:25 1 5.2 2.5 100 24
2008-01-30 12:25 5 80.0 20.0 2 2
2008-01-30 12:25 2 36.7 2.0 3 218
2008-01-30 12:25 4 22.8 1.8 89 2754
2008-01-30 12:25 3 22.9 2.6 14 47
21. Step1 & OEM
ADDM advisory expects I/Os to be under 10ms
Can change the behavior with
exec DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER(
'ADDM',
'DBIO_EXPECTED',
<new value>);
22. Copyright 2006 Kyle Hailey
Step 2 : Buffer Pool Advisory
Buffer Pool Advisory
Size for Size Buffers for Read Estimated
P Est (M) Factor Estimate Factor Physical Reads
--- -------- ------ ------------ ------ --------------
D 56 .1 6,986 2.3 58,928
D 112 .2 13,972 1.6 42,043
D 224 .4 27,944 1.0 25,772
D 336 .6 41,916 1.0 25,715
D 448 .8 55,888 1.0 25,715
D 596 1.0 74,351 1.0 25,715
D 728 1.2 90,818 1.0 25,715
D 840 1.4 104,790 1.0 25,715
D 952 1.6 118,762 1.0 25,715
D 1,064 1.8 132,734 1.0 25,715
23. Copyright 2006 Kyle Hailey
Step 3: Find Top I/O SQL
select count(*),sql_id, event
from v$active_session_history
where event in ('db file sequential read',
'db file scattered read',
'db file parallel read')
group by sql_id, event
order by count(*);
COUNT(*) SQL_ID EVENT
--------- ------------- ------------------------
10 8hk7xvhua40va db file sequential read
335 3hatpjzrqvfn7 db file sequential read
343 0uuqgjq7k12nf db file sequential read
549 75621g9y3xmvd db file sequential read
1311 0bzhqhhj9mpaa db file sequential read
1523 41zu158rqf4kf db file sequential read
1757 0yas01u2p9ch4 db file sequential read
25. Copyright 2006 Kyle Hailey
Machine Memory
IO Configuration Issues
SGA
Unix File
Cache
If average IO read times
are under 7ms them
probably coming from Unix
File Cache
Unix Buffer Cache
Disk Spindles
Raid 5 Not
Physical IO
Physical IO
26. Copyright 2006 Kyle Hailey
IO Throughput
One Disk 74G – Few IOPS
37 Disks 2G – Many IOPS
92 IO/sec
1887 IO/sec
27. Copyright 2006 Kyle Hailey
Disk 2Gb vs 70GB
Seagate
Barracuda 4LP
Seagate
Cheetah 73LP
Capacity 2.16GB 73.4GB
Rotation Speed 7200rpm 10000rpm
Rotational Delay(avg) 4.1ms 3ms
Time to read 32Kb 6ms 3ms
Seek Time (avg) 9.4ms 4.9
Total time for Single I/O 19.5ms 10.9ms
I/O per second
(conservative)
51 92
IOPs/sec per 100GB 2550 126
James Morle http://scaleability.com
28. Copyright 2006 Kyle Hailey
Raid 5
P1 P2
P1
P2
Writes require two reads
plus a write – read data,
read parity then write
Reads can conflict on
smaller stripe sizes 32K
or 64K
Readers on different
disks run without conflict
http://www.miracleas.com/BAARF/BAARF2.html
B – Battle
A – Against
A – Any
R – Raid
F - Five
30. Copyright 2006 Kyle Hailey
Direct I/O WAITS
Direct I/O :
This mechanism lets the client bypass the buffer cache for
I/O intensive operations.
The disk blocks are written into and read from process
private memory.
direct path read :
Parallel Query
direct path write
sqlldr
loading/reading LOBs
parallel DMLs
create table as select
create index
direct path read temp , direct path write temp
Sorting
31. Copyright 2006 Kyle Hailey
Direct IO
Shadow
Process
Buffer Cache
PGA
Sort Area
Direct Path
PQO Reads
X
32. Copyright 2006 Kyle Hailey
direct path read
_serial_direct_read = true
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
PQO
Shadow
Shadow
Process
Process
PGA
PGA
PQO
X
Parallel Query
See v$px_session
select parameter1, parameter2, parameter3
from v$event_name
where name='direct path read';
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
33. Copyright 2006 Kyle Hailey
Further Investigation ASH
----- ----- ---- ----------- ------ -- ------ ------------
-
SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID
--- ---- -- ----- ------ -- ------ -------------
149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722
144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722
60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722
54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722
149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722
select
session_id sid,
QC_SESSION_ID qsid,
ash.p3,
CURRENT_OBJ#||' '||o.object_name objn,
o.object_type otype,
CURRENT_FILE# fn,
CURRENT_BLOCK# blockn,
ash.SQL_ID
from v$active_session_history ash,
all_objects o
where event like 'direct path read'
and o.object_id (+)= ash.CURRENT_OBJ#
Order by sample_time;
34. select
ash.SQL_ID,
QC_SESSION_ID qsid,
count(*) cnt,
count (distinct session_id) deg,
nvl(o.object_name,to_char(CURRENT_OBJ#)) obj,
o.object_type otype,
decode(session_state, 'WAITING',event,'CPU') event
from v$active_session_history ash,
all_objects o
where o.object_id (+)= ash.CURRENT_OBJ#
and qc_session_id is not null
group by qc_session_id, sql_id, o.object_name,
o.object_type, CURRENT_OBJ#, event, session_state
Order by qc_session_id, sql_id
PQO - ASH
SQL_ID QSID CNT DEG OBJ OTYPE EVENT
------------- ---- ---- --- ------------ ----- -------------------------
-
7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd
144 4 3 WAIT_OBJECTS TABLE PX qref latch
144 37 1 WAIT_OBJECTS TABLE db file sequential read
144 3 2 WAIT_OBJECTS TABLE direct path read
144 70 1 WAIT_OBJECTS TABLE CPU
144 21 4 0 PX Deq Credit: send blkd
144 12 4 0 db file sequential read
35. Copyright 2006 Kyle Hailey
direct path write
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
Shadow
Shadow
Process
Process
PGA
PGA
DBWR
DBWR
DBWR
DBWR
DATA
DATA
DATA
DATA
Seems to be a bug on 10g where
direct path write waits are
incorrectly recorded as CPU
select parameter1, parameter2, parameter3
from v$event_name
where name='direct path write';
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
Occurs when:
insert /*+ APPEND */
sql*loader direct=y
Create table as select
Loading LOBS
Parallel insert
Create index
36. Copyright 2006 Kyle Hailey
Direct Path Read Temp
Direct Path Write Temp
Sorting
Write to Temp
Read from Temp
Shadow
Shadow
Process
Process
Buffer Cache
Buffer Cache
PGA
PGA
TEMP
TEMP DATA
DATA
Shadow
Shadow
Process
Process
Buffer Cache
Buffer Cache
PGA
PGA
TEMP
TEMP DATA
DATA
select parameter1, parameter2, parameter3
from v$event_name
where name in ('direct path write temp‘,
‘direct path read temp’);
PARAMETER1 PARAMETER2 PARAMETER3
---------- ---------- ----------
file number first dba block cnt
file number first dba block cnt
37. Copyright 2006 Kyle Hailey
Direct Path Read Temp
Direct Path Write Temp
Buffer Cache
Buffer Cache
Buffer Cache
Buffer Cache
Shadow
Shadow
Process
Process
PGA
PGA
TEMP
TEMP DATA
DATA
Shadow
Shadow
Process
Process
PGA
PGA
TEMP
TEMP DATA
DATA
This event happens when sorting overflows the memory buffers and has to be
written to disk. If it's a problem consider increasing parameter
pga_aggregate_target
The explain plan can give estimated sort size requirements and extended row
source stats in the explain plan can give the actual sort size usage of the SQL
statement
select s.sid, s. process,
s.sql_id,
tmp.segtype,
((tmp.blocks*8)/1024) MB,
tmp.tablespace
from
v$tempseg_usage tmp,
v$session s
where tmp.session_num=s.serial#
and segtype in ('HASH','SORT')
order by blocks desc
SID SQL_ID SEGTYPE MB TABLESPACE
--- ------------- ------- -- ----------
90 gm5s0vc0vzppc SORT 2 TEMP03
105 gm5s0vc0vzppc SORT 2 TEMP03
80 gm5s0vc0vzppc SORT 2 TEMP03
105 gm5s0vc0vzppc HASH 2 TEMP03
102 gm5s0vc0vzppc SORT 2 TEMP03
90 gm5s0vc0vzppc HASH 2 TEMP03
38. PGA Aggregate Target
9i, a single sessions maximum sort area using PGA_AGGREGATE_TARGET was fixed to
100M unless setting
_pga_max_size: Maximum PGA size for a single process
Defaults to 200M
_smm_max_size: Maximum workarea size for one process , defined in KILO BYTES
Defaults Min(0.05*pga_aggregate_target,0.5*_pga_max_size,100M)
In 10.2 the 100M does not apply anymore when using a PGA_AGGREGATE_TARGET > 1000M.
In 10.2 the 100M applies as long as pga_aggregate_target is smaller than 1GB.
In 10.2 pga_aggregate_target set to 5GB
_smm_max_size of 500MB (was 100MB )
_pga_max_size of 1000MB (was 200MB).
Copyright 2006 Kyle Hailey
Manual override broken 10.2.0.3 fixed in 11.1.0.7
alter session set workarea_size_policy = manual;
alter session set sort_area_size = 104857600;
Workaround, run command twice
alter session set sort_area_size = 104857600;
alter session set sort_area_size = 104857600;
But doesn’t work for parallel connections that still fall back
on default which is 64K unless otherwise specified
HASH_AREA_SIZE seems unaffected by problem
39. Copyright 2006 Kyle Hailey
Data file init write
When autoextend is set and many extensions are
happening
Can be the root cause of other waits
Enq: TX - contention
40. Copyright 2006 Kyle Hailey
Local Write Wait
Truncating a table
Wait for Data Cache to be cleared of all blocks of
truncated table
Wait by shadow for DBWR
If a problem try
GTT
Reduce fast_start_mttr_target
41. Copyright 2006 Kyle Hailey
Further Investigation
NAME P1 P2 P3
----------------- ----- ------ -----
local write wait file# block#
select * from v$event_name
where name = ‘local write wait'
select
ash.p1,ash.p2, ash.SQL_ID, count(*)
from v$ash ash
where event='local write wait'
group by
ash.p1,ash.p2, ash.SQL_ID;
P1 P2 SQL_ID COUNT
--- -- ------------- -----
201 2 0q8k9xfmz0k2k 194
201 5 0q8k9xfmz0k2k 34
201 6 0q8k9xfmz0k2k 9
201 4 0q8k9xfmz0k2k 10
43. Copyright 2006 Kyle Hailey
Local Write Wait:extent allocation
CREATE TEMPORARY TABLESPACE "NEWTEMP"
TEMPFILE '/d3/temp01.dbf' SIZE 100M
AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 64K;
select a.*, b.*
from dba_objects a, dba_objects b
order by a.owner
ALTER DATABASE DEFAULT
TEMPORARY TABLESPACE newtemp;