DB2 Accounting Reporting

Understanding DB2 for z/OS
Accounting Information
John Campbell & Florence Dubois
IBM DB2 for z/OS Development
© 2012 IBM Corporation
campbelj@uk.ibm.com
fldubois@uk.ibm.com

Acknowledgements
• Many thanks to all people who contributed to this presentation, in particular:
– Namik Hrle
– Akira Shibamiya
– John Tobler
– Miao Zheng

Agenda
• DB2 traces: sources of key performance indicators
• Where is a DB2 transaction spending its time?
• CPU vs. suspension time
• Most common reasons for large suspension times
• SQL / Locking / BP / GBP accounting counters
• Tuning hints and tips as we go along
•

DB2 Accounting vs. Statistics Data
• DB2 statistics trace records
– Contain information about the activity of the entire DB2 subsystem (including the
DB2 system address spaces, such as db2MSTR, db2DBM1, db2IRLM, db2DIST
• DB2 accounting trace records
– Contain information about the activity performed by an individual DB2 thread or
‘transaction’
(or x threads in case of roll-up accounting)
(or x threads in case of roll-up accounting)
– The focus of this presentations is here

Synchronous/Asynchronous Agents
• Main synchronous agent
– Used to execute the SQL statement
– Usually a TCB that originates in the user’s address space
– In case of DDF, it is an (independent enclave) SRB instead of a TCB, which originates
in the DDF address space
– In case of parallelism, the TCB (SRB for DDF) is helped by synchronous SRBs
• Asynchronous agent
• Asynchronous agent
– Normally performs system or ancillary functions
– Usually an SRB, but it can also be a TCB
– The asynchronous CPU load is typically 10% of the synchronous load in most
systems
• Can be higher in DDF and/or data sharing
• More important to see if the ratio changes than looking at absolute numbers
• Can indicate a change in workload -Access path change - Maintenance causing regression

TSO attach
Call attach
Utilities attach (*)
RRS attach (incl. JCC T2)
IMS attach
CICS attach
Use TCB
Synchronous Agent
Different DB2 Attachments
• Different attachments used by DB2
CICS attach
DDF “attach”
All use XMEM services
Uses SRB
(*) Except with zIIP offload during IX build phase and RUNSTATS (V10)
Synchronous Agent

TCB
SQL
How Work Enters DB2 – Allied Thread
USER (allied) DBM1 MSTR IRLM

SRB
SQL
How Work Enters DB2 – Distributed Work
DIST
(distributed)
DBM1 MSTR IRLM

TCB
Async I/O Physical
logger
Deadlock
detection
SQL
Asynchronous ’Agents’ in DB2
USER (allied)
DBM1 MSTR IRLM
System
checkpoint Lock
resume
Open/
close
DIST
(distributed)
SRB
SQL
Auth

Synchronous/Asynchronous Agents CPU times
• The synchronous CPU load is recorded in accounting
• The asynchronous CPU load is recorded in statistics
– If a thread has to wait for the async. agent to complete its work, this elapsed time is
typically recorded in an accounting class 3 wait time counter
• Important note
• Important note
– Since DDF work doesn’t have an agent outside the DB2 address spaces to charge for
the work, the CPU time used by synchronous agents for DDF work also shows up in
DB2 statistics (as DDF enclave SRB time). So be aware of double accounting!

CPU TIMES (from Statistics) v10 TCB TIME PREEMPT SRB NONPREEMPT SRB TOTAL TIME PREEMPT IIP SRB
------------------------------- -------------- --------------- --------------- --------------- ---------------
SYSTEM SERVICES ADDRESS SPACE 8:05.013248 21:20.445149 12.782106 29:38.240503 N/A
DATABASE SERVICES ADDRESS SPACE 21:51.008289 13:35.088803 16.280806 35:42.377897 3:19:20.488131
IRLM 0.320224 0.000000 22.664271 22.984495 N/A
DDF ADDRESS SPACE 2:12.462002 2:50:01.556367 2:15.574735 2:54:29.593103 2:37:50.897412
TOTAL 32:08.803762 3:24:57.090319 3:07.301917 4:00:13.195998 5:57:11.385543
TCB
Async I/O Physical
logger
Deadlock
detection
(From Accounting)
Sync and Async Work in DB2 Stats and Accounting
USER (allied)
DBM1 MSTR IRLM
logger
System
checkpoint
detection
Lock
resume
Open/
close
DIST
SRB
SQL
SQL
Auth
AVERAGE APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 0.225604 0.099221
NONNESTED 0.220121 0.097390
STORED PROC 0.005483 0.001831
CP CPU TIME 0.002241 0.002066
AGENT 0.002241 0.002066
NONNESTED 0.001787 0.001699
STORED PRC 0.000454 0.000366
IIP CPU TIME 0.002082 0.001987
STORED PROC N/A N/A

How CPU Time is Reported (Acctg vs. Stats)
Application DBM1 MSTR IRLM
Accounting Statistics
T
SQL processing Scans
Synchronous I/O Lock requests
Open, Close
Extend
Archiving
Error checking
Allocate archive dsn
T
C
B
S
R
B
Synchronous I/O
Global lock requests*
Buffer updates
Lock requests
Logical logging
GBP reads*
The same as in the TCB
case, but only in enclave
pre-emptible SRB mode.
Reported in TCB
instrumentation.
Extend
Preformat
Asynchronous I/O
Memory management
Real-time Statistics
Castout*
P-lock negotiation*
Prefetch in CF*
SYSLGRNX updates*
GBP checkpoints*
BSDS processing
Physical logging
Checkpoints
Backouts
Deallocation
Update commit
GBP writes at commit*
Global unlock at commit*
Error checking
Management
Deadlock detection
Lock resume
Global lock conflict
resolution*
(*) data sharing specific
Allocate archive dsn

Response Time Factors
presentation network application logic network database processing
General transaction flow

Response Time Scope
TRANSACTION
PROGRAM A
STATEMENT 1
STATEMENT 2
.
.
PLAN
DBRM A (*)
STATEMENT 1
STATEMENT 2
.
.
PROGRAM B
.
STATEMENT 1
STATEMENT 2
.
.
PACKAGE B
.
STATEMENT 1
STATEMENT 2
.
.
Application space DB2 (*) No longer possible in V10

Response Time: DB2 View
thread allocation
1st SQL
2nd SQL
DB2 can collect data during the
entire thread life: activity time
It includes time spent in:
• Presentation layer (GUI)
• Application logic
• Network
nth SQL
thread deallocation
Out of DB2 In DB2
For all practical purposes, this is
the DB2 Accounting Class 1
elapsed time
• Network
• DB2 processing
activity
time

DB2 Accounting Data
• DB2 Accounting trace
– DB2 command -STA TRA(ACCTG) …. Or at DB2 start-up SMFACCT DSNZPARM
– Typically written to SMF (as SMF type 101 record)
• IFCIDs
– IFCID 3 – plan level info
– IFCID 239 – package level info
• Since V8, IFCID 239 is written as a separate record, but cut at the same time
• Since V8, IFCID 239 is written as a separate record, but cut at the same time
• Accounting Trace classes
– Class 1 – Total time (elapsed and CPU) – IFCID 3
– Class 2 – Time in DB2 (elapsed and CPU) – adds info to IFCID3
– Class 3 – Suspension time in DB2 – adds info to IFCID 3
– Class 7 – Package/DBRM info (similar to class 2)
– Class 8 – Package/DBRM info (similar to class 3)
– Class 10 – SQL/Locking/BP info at package/DBRM level

A DB2 Accounting Data is Written at:
• Thread deallocation (including abends)
• (re)Signon in case of thread reuse by RRS, IMS, CICS
– (many options in CICS)
• Commit for RRS threads using accounting-interval COMMIT on signon / auth
signon / context signon
• Commit when CMTSTAT=INACTIVE and provided connection can go inactive
– Things that prevent a connection from going inactive:
– Things that prevent a connection from going inactive:
• Touched a package that uses KEEPDYNAMIC(YES)
• Active DGTT (not explicitly or implicitly dropped)
• Open cursor with hold
• Held LOB locator
• If only reason for not going inactive is the use of KEEPDYNAMIC(YES), still cut
accounting record (and reset enclave)
– To please our SAP friends

DB2 Accounting Data is Written at: -2
• When using rollup accounting for DRDA and RRS
– Every ACCUMACC times you would have normally written an accounting record
– If gets stale (not enough occurrences of the rollup “group”)
– If rollup blocks are using too much storage
– QWACRINV indicates reason why acctg record was written
• Rollup accounting blessing and curse
– Reduces the number of accounting records
– Reduces the number of accounting records
– Can do rollup based on different “group by” criteria
– Some accounting information not available in rollup records
(some enhancements in V10)
– Makes problem shooting more complicated (see later)

When is an accounting record written?
• When the thread de-allocates – normal case
1st
SQL
Pgm
end
1st
SQL
Pgm
end
Pgm
Start
Pgm
Start
Time
Thread
create
StatsAcctg
Thread
de-allocate
Thread
create
Thread
de-allocate
StatsAcctg

When is an accounting record written? - 2
• Commit does not create an accounting record, except
– For inactive connections in DRDA (coming up)
– RRS using accounting-interval commit (and no open held cursors)
1st
SQL Commit
Pgm
end
Pgm
Start CommitCommit
Time
Thread
create
Thread
deallocate
StatsAcctg

• When a thread is reused
– Applies to IMS and CICS
– (There are additional options in CICS – see CICS Transaction Server - DB2 Guide)
1st
SQL Commit
Pgm
end
Pgm
Start
New
tran
1st SQL Commit
New
tran
1st SQL
Time
Create Thread
+ Signon
Thread
de-allocate
Signon Signon
StatsAcctg StatsAcctg StatsAcctg

Accounting pitfall
• When a thread is reused but there is a long time between transaction end and
the next transaction doing signon
1st
SQL Commit
Pgm
end
Pgm
Start
New
tran
1st SQLRegion idle
CLASS 1 time reported
Time
Create Thread
+ Signon
Thread
de-allocate
Signon
CLASS 1 time reported
StatsAcctg StatsAcctg

• When the connection goes inactive
– Distributed thread
– CMTSTAT=INACTIVE
1st
SQL Commit
Pgm
endPgm
Start
New
tran
1st
SQL Commit
New
tran
1st
SQL
Time
Acquire
DBAT
Conn.
ends
Conn.
inactive
Conn.
inactive
Acquire
DBAT
Acquire
DBAT
StatsAcctg StatsAcctg StatsAcctg

• When using rollup accounting (DDF and RRS)
– E.g. ACCUMACC=3, ACCUMUID=2 (‘group by’ transaction name)
– Values will be the totals of all 3 transactions
1st
SQL Commit
Pgm
endPgm
Start
New
tran
1st
SQL Commit
New
tran
1st
SQLCommit
Time
Acquire
DBAT
Conn.
ends
Conn.
inactive
Conn.
inactive
Acquire
DBAT
Acquire
DBAT
Conn.
inactive
Acquire
DBAT
AcctgAcctg AcctgStatsAcctg

Accounting Class 1 Data
thread allocation
1st SQL
2nd SQL
For local applications, DB2
provides an accumulative timer
that includes both CPU spent in
application and CPU spent in
DB2.
activity
Class 1
elapsed
and CPU
thread deallocation
Out of DB2 In DB2
DB2.
Activity time is very close to
Class 1 elapsed time.
For local applications, Class 1
(elapsed – CPU) time can point
to application inefficiencies.
activity
time

thread allocation
1st SQL
2nd SQL
Class 2 timers are accumulative
Time spent out of DB2
Class 1 elapsed – Class 2 elapsed
Class 1
elapsed
and CPU
Class 2
elapsed
and CPU
2nd SQL
thread deallocation
Time spent waiting in DB2
Class 2 elapsed – Class 2 CPU
Class 7 is used for package/DBRM
scope reporting.
It is equivalent to Class 2
Out of DB2 In DB2

thread allocation
1st SQL
2nd SQL
Class 8 is used for
package level reporting.
It is equivalent to Class
Class 1
elapsed
and CPU
Class 2
elapsed
and CPU
Class 3
Suspensions
2 SQL
thread deallocation
It is equivalent to Class
3.
Out of DB2 In DB2

in Appl in DB2 IRLM
Class 1 Class 2 (in DB2) 1st SQL..
SQL ..
SQL ..
...Wait for lock
(Creating Thread)
DB2 Times Terminology - Accounting Class 1,2,3
Agent
Agent, non-nested ET
(Terminating Thread)
...Wait for I/O
Agent, non-nested CPU
Class 3
(susp time)
...Commit

DB2 Times Terminology –
Class 1&2 Elapsed Time with Nested Activity
in Appl in DB2 in UDFElapsed Time
Class 1 Class 2 (in DB2)
Trigger1
1st SQL..
SQL ..
UDF
SQL ..
...UDF
(Creating Thread)
Agent
Trigger
UDF
Agent, non nested
Trigger
UDF
Trigger2 SQL ..
(Terminating Thread)

DB2 Accounting Report Class 1-2-3 Sections
TIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS
------------ ---------- ---------- ---------- -------------------- ------------ --------
ELAPSED TIME 1:42.494131:42.49413 0.111129 N/P LOCK/LATCH(DB2+IRLM) 0.000000 0
NONNESTED 0.208342 0.0571680.057168 N/A SYNCHRON. I/O 0.000000 0
STORED PROC 0.000000 0.000000 N/A DATABASE I/O 0.000000 0
UDF 1:42.238001:42.23800 0.0061650.006165 N/A LOG WRITE I/O 0.000000 0
TRIGGER 0.0477970.047797 0.0477970.047797 N/A OTHER READ I/O 0.000000 0
OTHER WRTE I/O 0.000000 0
CP CPU TIME 28.858601 0.029797 N/P SER.TASK SWTCH 0.020768 1
AGENT 28.858601 0.029797 N/A UPDATE COMMIT 0.020768 1
NONNESTED 0.016157 0.005022 N/P OPEN/CLOSE 0.000000 0
STORED PRC 0.000000 0.000000 N/A SYSLGRNG REC 0.000000 0
UDF 28.822345 0.004677 N/A EXT/DEL/DEF 0.000000 0
TRIGGER 0.020098 0.020098 N/A OTHER SERVICE 0.000000 0
PAR.TASKS 0.000000 0.000000 N/A ARC.LOG(QUIES) 0.000000 0
LOG READ 0.000000 0
SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0
SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0
CLAIM RELEASE 0.000000 0
SE CPU TIME 0.000000 0.000000 N/A PAGE LATCH 0.000000 0
NONNESTED 0.000000 0.000000 N/A NOTIFY MSGS 0.000000 0
STORED PROC 0.000000 0.000000 N/A GLOBAL CONTENTION 0.000000 0
UDF 0.000000 0.000000 N/A COMMIT PH1 WRITE I/O 0.000000 0
TRIGGER 0.000000 0.000000 N/A ASYNCH CF REQUESTS 0.000000 0
TCP/IP LOB 0.000000 0
PAR.TASKS 0.000000 0.000000 N/A TOTAL CLASS 3 0.020768 1
SUSPEND TIME 27.525561 0.020768 N/A
AGENT N/A 0.020768 N/A
PAR.TASKS N/A 0.000000 N/A
STORED PROC 0.000000 N/A N/A
UDF 27.52556127.525561 N/A N/A
NOT ACCOUNT. N/A 0.060564 N/A
DB2 ENT/EXIT N/A 104 N/A
EN/EX-STPROC N/A 0 N/A
EN/EX-UDF N/A 0 N/A
DCAPT.DESCR. N/A N/A N/P
LOG EXTRACT. N/A N/A N/P

DB2 Accounting Report Class 1-2 Sections
TIMES/EVENTS APPL(CL.1) DB2 (CL.2) TIMES/EVENTS APPL(CL.1) DB2 (CL.2)
------------ ---------- ---------- ------------ ---------- ----------
ELAPSED TIME 1:42.494131:42.49413 0.111129 ... ... ...
NONNESTED 0.208342 0.0571680.057168
STORED PROC 0.000000 0.000000 SUSPEND TIME 27.525561 0.020768
UDF 1:42.238001:42.23800 0.0061650.006165 AGENT N/A 0.020768
TRIGGER 0.0477970.047797 0.0477970.047797 PAR.TASKS N/A 0.000000
STORED PROC 0.000000 N/A
CP CPU TIME 28.858601 0.029797 UDF 27.52556127.525561 N/A
AGENT 28.858601 0.029797
AGENT 28.858601 0.029797
NONNESTED 0.016157 0.005022 NOT ACCOUNT. N/A 0.060564
STORED PRC 0.000000 0.000000
UDF 28.822345 0.004677
TRIGGER 0.020098 0.020098
PAR.TASKS 0.000000 0.000000
... ... ...

Reporting of Nested Activity
• In DB2 accounting data, you can distinguish between
– Non-nested activity
– Nested activity
(cannot distinguish between the level of nesting – that requires performance traces)
• Trigger
• Stored Procedure (SP)
• User Defined Function (UDF)
•
• For SP and UDF that run in a WLM-managed AS, DB2 can distinguish between
Class 1 (program logic - COBOL code) and class 2 (executing SQL statements)
• For triggers, SQL PL SP and non-fenced UDFs, CL1=CL2

Reporting of Nested Activity - 2
• Only actual work done by SP is reported as nested activity at the plan level
– That is work done by the CALL sp
– When using result set cursors (WITH RETURN), typically only the OPEN cursor is
done by the SP and is reported as nested activity
– The FETCHing is done by the calling application and is reported as non-nested time
at the plan level
• Package level data shows the FETCHes (as well as CPU/ET time incurred) as if
they were done by the SP package (that performed the OPEN cursor)

in Appl in DB2 In SPElapsed Time
Class 1 Class 2 (in DB2) 1st SQL..
CALL sp SP
OPEN curs
WITH RETURN
SQL ..
(Creating Thread)
Nested Activity for WITH RETURN processing
SP end
Agent
SP
Agent, non nested
SP
(Terminating
Thread)
SP end
FETCH
FETCH
FETCH

DB2 Accounting Class 1-2 (Plan Level) Sections
TIMES/EVENTS APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 0.941479 0.608886
NONNESTED 0.731353 0.4021540.402154
STORED PROC 0.2101260.210126 0.2067320.206732
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
CP CPU TIME 0.249935 0.192192
AGENT 0.249935 0.192192
NONNESTED 0.248412 0.191915
SQL DML TOTAL
-------- --------
SELECT 0
INSERT 0
ROWS 0
UPDATE 0
ROWS 0
MERGE 0
NONNESTED 0.248412 0.191915
STORED PRC 0.001523 0.000277
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
... ... ...
SUSPEND TIME 0.000323 0.000000
AGENT N/A 0.000000
PAR.TASKS N/A 0.000000
UDF 0.000000 N/A
NOT ACCOUNT. N/A 0.416693
... ... ...
MERGE 0
DELETE 0
ROWS 0
DESCRIBE 0
DESC.TBL 0
PREPARE 0
OPEN 1
FETCH 5204
ROWS 5203
CLOSE 1
DML-ALL 5206

DB2 Accounting Report Class 7-8-10 (Package Level)
• Package level data is being reported as if the FETCHes are executed by the SP
MRSBMCBM (main) TOTAL MRSBMS (SP) TOTAL
------------------ -------- ------------------ --------
SELECT 0 SELECT 0
INSERT 0 INSERT 0
UPDATE 0 UPDATE 0
DELETE 0 DELETE 0
DESCRIBE 0 DESCRIBE 0
PREPARE 0 PREPARE 0
OPEN 0 OPEN 1
FETCH 0 FETCH 5204
CLOSE 0 CLOSE 1
LOCK TABLE 0 LOCK TABLE 0
CALL 1 CALL 0

DB2 Accounting Report Class 7-8 (Package Level)
MRSBMCBM (main) VALUE MRSBMCBM TIMES
------------------ ------------------ ------------------ ------------
TYPE PACKAGE ELAPSED TIME - CL7 0.151778
LOCATION STLEC1 CP CPU TIME 0.083859
COLLECTION ID BARTCOB AGENT 0.083859
PROGRAM NAME MRSBMCBM PAR.TASKS 0.000000
CONSISTENCY TOKEN 1906C50F153EE340 SE CPU TIME 0.000000
ACTIVITY TYPE NONNESTED SUSPENSION-CL8 0.000000
ACTIVITY NAME 'BLANK' AGENT 0.000000
SCHEMA NAME 'BLANK' PAR.TASKS 0.000000
MRSBMS (SP) VALUE MRSBMS TIMES
------------------ ------------------ ------------------ ------------
------------------ ------------------ ------------------ ------------
TYPE PACKAGE ELAPSED TIME - CL7 0.4960330.496033
LOCATION STLEC1 CP CPU TIME 0.139170
COLLECTION ID BARTCOB AGENT 0.139170
PROGRAM NAME MRSBMS PAR.TASKS 0.000000
CONSISTENCY TOKEN 1906C50417D039C3 SE CPU TIME 0.000000
ACTIVITY TYPE STORED PROC SUSPENSION-CL8 0.000000
ACTIVITY NAME MRSBMS AGENT 0.000000
SCHEMA NAME SYSPROC PAR.TASKS 0.000000
SUCC AUTH CHECK NO NOT ACCOUNTED 0.356863
Greater than Nested time at
plan level - includes time for
FETCH operations

Analyzing Acctg Data: Top-Down Analysis
• Accounting report (not trace) by connection type most useful for initial analysis
• For OMPE users:
ACCOUNTING
REPORT LAYOUT(LONG) ORDER(CONNTYPE)
EXCLUDE(PACKAGE(*))
• To group by thread connection type such as TSO, CICS, DB2CALL, RRS, IMS,
DRDA, UTILITY, etc. for the period of interest
• Also combine with STATISTICS REPORT LAYOUT(LONG) for the
corresponding period

Analyzing Thread Activity Time
Thread activity time = Class 1 elapsed
Elapsed time spent out of DB2 = Class 1 elapsed – Class 2 elapsed
(in application, network, idle..)
Elapsed time spent in DB2 = Class 2 elapsed
Processing time = Class 2 CPU (GCP and zIIP)
Waiting time = Class 2 elapsed – Class 2 CPU
Suspended time = Class 3 suspension time
Not accounted time = Waiting time – Suspended time

What if Activity Time is Split as …
• Application logic inefficiency
– Often combined with
Class 2 CPU << Class 1 CPU
• Network problems
• User think (coffee?) time
• Class 2 not active all the time
In DB2
Out of DB2
• Class 2 not active all the time
• Potential pitfalls
– Thread reuse and long time between
consecutive transactions (allied threads)
– Long running DDF threads when
CMTSTAT=ACTIVE

What if Activity Time is Split as …
• What’s the actual number (not just the %)
• Is this a DB2 intensive process anyway?
•
In DB2
Out of DB2
• Need to analyze time distribution in DB2:
Where is the time really spent inside DB2?

Time in DB2 (Class 2 Time)
• Burning CPU cycles (class 2 cpu time)
– Non-nested or nested
– On a general CP or zIIP
• Suspended in DB2 for a known event
(class 3 suspension time)
– Many different types of suspensions (21 in V10)
• In DB2 but don’t know what it is we are doing
– Mostly things that happen outside of DB2’s control, such as
• Lack of CPU or paging
– Should be a small %

Accounting CPU time reporting
• CP CPU time
– CPU time used on general purpose engine
• SE CPU time
– Any CPU used on a specialty engine (zIIP or zAAP)
– Not included in GP CPU time
• Sub-capacity machines
– SE CPU time is reported as if had run on general CP
– SE CPU time is reported as if had run on general CP
– Can result in total CPU= (CP + SE CPU) > ET for CPU intensive work
• SECP CPU
– Could have run on specialty engine but didn’t
– No longer reported in V10

Accounting CPU time reporting
CONNTYPE: DRDA
AVERAGE APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 2.840103 1.913457
NONNESTED 2.839901 1.913339
STORED PROC 0.000202 0.000118
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
CP CPU TIME 0.260398 0.251655
AGENT 0.260398 0.251655
NONNESTED 0.260326 0.251584
NONNESTED 0.260326 0.251584
STORED PRC 0.000072 0.000071
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
SECP CPU 0.000726 N/A
SE CPU TIME 0.261583 0.246227
NONNESTED 0.261583 0.246227
STORED PROC 0.000000 0.000000
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000

What if In DB2 Time is Split as …
• Turn off expensive traces
• Inefficient access paths
−Run EXPLAIN
−Check transaction profile
−#SQL stmts
−#locks
Processing
Waiting
• What is the largest contributor
−Class 3 and 8 analysis
−#locks
−#getpages
Processing
Waiting

Time in DB2 (Class 2 time)
– Many different types of suspensions (21 in V10)
– Mostly things that happen outside of DB2’s control, like
– Lack of CPU or paging

• Class 3 accounting strongly recommended
• Provides information about “known” DB2 suspensions.
For suspensions, DB2 reports:
– The number of suspensions
– The total time it was suspended for those
• Always look at both #suspensions and suspension time
• Double accounting is avoided
• Double accounting is avoided
• Negligible overhead except when high internal DB2 latch contention, e.g. over
10,000/sec

Sync.EU Switch
Locking
I/O
Class3SuspensionTypes Synchronous read/write & log write
Synchronous read/write
Log write
Other agents’ read
Other agents’ write
Force-at-commit database writes (LOG NO LOBs only)
TCP/IP LOB XML
IRLM lock/latch & DB2 internal latch
IRLM lock/latch
DB2 internal latch
Page latch
Drain lock
Claim release
Synchronous Execution Unit Switch total
Open/Close
Define/Extend/Delete
V5V5 V6V6 V7,8V7,8 V9V9 V10V10
Archiving
Scheduling
Data Sharing
Class3SuspensionTypes
SYSLGRNX recording
Commit
Other services
Archive Log command
[Archive] log read
Stored procedures
UDFs
Global locks total
Parent L-locks
Child L-locks
Other L-locks
Page set/Partition P-locks
Page P-locks
Other P-locks
Sending Notify messages
Asynchronous coupling facility request completion

Lock/Latch Suspensions
• Thread can be suspended for:
– Lock contention
• includes preformatting (X’09’)
– IRLM latch contention
– DB2 internal latch contention
• Page latch contentions are reported in
a separate timer
•
w
a
UPDATE
application
address space
DB2
address space
lock suspend
lock unavailable
• DB2 reports:
– Accumulated time thread was
suspended
– Total number of suspensions
• Time/events is an average wait due to
a lock/latch conflict. The number of
true lock suspensions (from the
Locking block) is often a better divisor.
a
i
t
lock resume
buffer update

Large Lock/Latch Suspensions – Large number?
Number of IRLM lock suspensions high?
− Application problems
− Incompatible workload mix
− Page/row level locking
− Inefficient preformatting
Use PAV
Check ICF catalog performance
Number of DB2 internal latch suspensions high?
− LC06: Index tree P-lock latch contention caused most
likely by splits of GBP-dependent index pages
− LC11: Generating Identity Column
− LC14: BP LRU chain – Isolate small highly accessed
tables into separate pools
− LC19: Log – Speed up log I/O
− LC24: EDM LRU or BM latch
− LC25: EDM Pool hash chain
Waits for
lock/latch
suspensions
In DB2 Time
Number of IRLM latch suspensions high?
− If more than 10% of all IRLM requests
− IRLM Trace on
− Low IRLM dispatching priority
− Frequent IRLM Query requests (e.g. -DIS DATABASE
LOCKS, or MODIFY irlmproc, STATUS)
− Low deadlock detection cycle and many lock
suspensions
− LC25: EDM Pool hash chain
− LC32 Storage Manager Pool Header

Large Lock/Latch Suspensions – High average time?
Time per suspension large?
• Lock suspensions
– Non-committing transactions
– Concurrent DDL
• RELEASE(DEALLOCATE)
– For detail analysis, use performance trace
classes 6 and 7
Waits for
lock/latch
suspensions
In DB2 Time
classes 6 and 7
• DB2 latch suspensions
– Designed to be short duration
– Can be a symptom of another problem
– For detail analysis, use performance trace
class 11
suspensions

Notes
• Lock/Latch wait =
Lock wait +
IRLM latch wait +
Internal DB2 latch wait
– In the rare case of over 10,000 per second, disabling class 3 may significantly bring
down class 1 and 2 cpu time
– Look at IRLM lock suspension and IRLM latch suspension counter to get a better idea
which one of the 3 is likely to be the culprit
which one of the 3 is likely to be the culprit
• DB2 10
– Separate out into IRLM lock/latch and DB2 latch
– More efficient DB2 latch resume logic
• Less chance to get resumed and suspended again without ever obtaining the latch in
between

Synchronous I/O Suspensions Large?
Number of suspensions high
− Excessive Getpages
− Buffer pools contention
− Disorganized data/indexes
Synchronous
I/O Waits
In DB2 Time
Time per suspension large
− Make sure there are enough I/O resources
− Use RMF to verify
− For detailed analysis in DB2,
use performance class 4

Synchronous Database I/O Suspensions
• A common timer for:
– Synchronous database reads
– Synchronous database writes
• A separate timer for synchronous
log writes
• Use BP statistics to find out how
getpage miss
UPDATE
application
address space
DB2
address space
read begin
• Use BP statistics to find out how
many of the suspensions should
be attributed to synchronous
reads vs. writes
• Avg I/O wait time in Acctg rpt =
0.028539 sec/12.41 events =
0.002 sec / sync DB IO
w
a
i
t
read end
buffer update

Tuning the # Sync I/O suspensions
• If many database I/O waits, check
– Excessive Getpages because of a bad access path
• Use Explain to tune access path
– Buffer pools contention
• Other applications ruining our hit ratio
• BP too small
• (V)DWQT setting
•
• Look at BP statistics
– Disorganized data/indexes
• Check statistics
– Minimize index splits
– Zero freespace for data for minimum read and write I/O in heavy Insert or APPEND
YES

Tuning I/O Wait Time
• If high database I/O wait, check
– Enough I/O resources ? (Use RMF to verify)
– DASD contention
– High IOSQ time – use PAV
– Faster I/O devices (use SSD)
– Control Unit cache misses
– CPU contention
– CPU contention
– Inappropriate I/O priority
• Use I/O striping where appropriate
– Less effective with newer I/O devices
– Still beneficial for non-saturated I/O configuration with infrequent commit and large
amount of data to be read or written in one I/O, e.g. list prefetch, heavy write

Synchronous Log write I/O Suspensions
Force log buffers to disk
Commit
application
address space
DB2
address space
• “Normally” only forced write to active log
at commit time
• All log buffers that have not been written
are written at commit
• Most common cases are listed on next foil
write begin
w
a
i
t
write begin
write end
No more log buffers to write

What triggers sync log write I/Os ..
• For transactions that use 2PC, Phase 1 of 2-phase commit
• Synchronous write (to disk or GBP)
– Log write ahead protocol
• Running out of log output buffers
• Identity columns/sequences that are using NO CACHE (or low cache value) or
ORDER is used (in data sharing)
– Hidden SPRMNLGF zparm to avoid especially if identity columns are used
– Hidden SPRMNLGF zparm to avoid especially if identity columns are used
• V10 up to 40% reduction in (dual) log I/O wait because of parallel log write I/O’s

What triggers sync log write I/Os ..
• Index split in data sharing
– Reduce index page splits by
• Non-zero freespace (FREEPAGE, PCTFREE) for random insert
• PCTFREE 0 for sequential insert
• Larger index page size in V9
– Trade-off with possibly higher index page latch contention and P-lock contention
• Asymmetric index page split in V9
•
• P-lock negotiation
• IMMEDWRITE(YES) for GBP-dependent objects
• If GBP-dependent, when taking (a completely empty) index page off the IX chain
• If GBP-dependent, new (reuse after mass delete or brand-new) page used in
segmented + UTS
– PK83735/PK94122 (V9) and V8 PM17813/17817 (V8) should address this

Wait for read i/o by another
application agent or prefetch engine
Suspensions for Other Agents’ Reads
w
a
getpage miss
FETCH
application
address space
DB2
address space
prefetch begin
Number of suspensions high?
− Excessive GetPages
Use Explain to tune access path
− Buffer pools contention
Look at BP statistics
In DB2 Time
Wait for
other
agents’
reads
What if …
a
i
t
first block of pages
FETCH
getpage hit
getpage hit
Look at BP statistics
− Disorganized data/indexes
Check statistics
− Use RMF and, for detailed analysis
performance class 4 to check on:
DASD contention
Control Unit cache misses
Check SEQCACHE=YES
Consider parallelism/partitioning
Inappropriate I/O priority

Suspensions for Other Agents’ Writes
UPDATE
application
address space
DB2
address space
− Too frequent checkpoints
Check DB statistics, LOGLOAD
− Too low (vertical) deferred write
In DB2 Time
Wait for
other
agents’ writesWhat if …
buffer update
UPDATE
w
a
i
t
Page being written
page written
− Too low (vertical) deferred write
threshold for high re-reference access
pattern
Check BP statistics
− DASD contention
− CU cache misses
− Inappropriate I/O priority
buffer update deferred
write start
UPDATE

Notes
• Other write I/O wait = wait for write i/o by another application agent or write
engine, may include some time waiting for log write-ahead
• When updating a buffer being written, first agent waits for OTHER WRITE I/O. All
others wait for PAGE LATCH contention.

Synchronous EU Switch Suspensions
w
SQL
application
address space
DB2
address space
switch to a TCB/SRB
any of these tasks
• ‘Update’ commits
− Single-phase commit
− Phase 2 of two-phase commit
− Abort
− Deallocation
• Dataspace Manager tasks
A separate counter for:
w
a
i
t
task completed
− Delete/Define/Extend data set
− Reset data set
− VSAM catalog access
• OPEN/CLOSE data set
− Not include HSM RECALL of data
set
• SYSLGRNX recording
− And down-level detection
• Other services tasks
− DDF application requester
considerations

Large Suspensions for Synchronous EU Switches?
In DB2 Time
Waits for
Synchronous
• Check log efficiency
− DB statistics, RMF
• Check VSAM catalog performance
Synchronous
Execution Unit
Switches
• PCLOSET or PCLOSEN too low
• Check if DSMAX is too low

Service Task Wait - Commit Related Timers
• Main components of commit processing
– Synchronous log writes
– Cursors, locking, etc. clean-up
– Force-at-commit synchronous database writes, or GBP writes in data sharing
Class 3 Timer Read-only commit Single-phase commit Two-phase commit
1 Included in wait for synchronous EU switch for 'update' commit.
2 These are log writes that happen in phase 1 of two-phase commit.
Log writes that happen in phase 2 are included in wait for synchronous EU switch for 'update' commit.
3 Applies to LOBs with LOG(NO) only. These database writes happen in phase 1 of two-phase commit.
Wait for log writes no no1 yes2
Wait for synchronous EU
switch for update commit
no yes yes
Wait for force-at-commit
database writes
no no1 yes3

Service Task Wait - 2
• Dataset [Extend]/Define/Delete
• Aka Dataspace manager work
– Extend for preformatting, takes 0.02 to 1sec each time depending on device type and
alloc unit/size
– x'09' lock wait in async preformat since V7 so no longer shows up here
– Counting pieces of a segmented TS (at open time)
– Dataset Delete/Define when Stogroup is used
– Dataset Delete/Define when Stogroup is used
– IFCID 258 contains DBID/OBID for Dataset Extend
– IFCID 92 contains Define Cluster statement for Define/Delete

Service Task Wait - 3
• Dataset Open/Close
– Up to 20 tasks for parallel Open in V8, 40 in V9
– Typically no class3 acctg wait for dataset close as it is asynchronous
– V10 Access DB Mode(Open) – parallel data set open
– PM19528 V9/10 introduces IFCID370-371
• Others
• Others
– Time waiting for server in TCP/IP requestor accounting
– VSAM catalog update
– Parallel query cleanup
– And many others which are typically infrequent
66

Commonly Observed OTHER SERVICE TASKS
• Parallel query cleanup x’14’/x’77’ or x’84’
– Shows up in repeated execution of short-running queries in parallelism
– Use higher SPRMPTH ZPARM value to replace this (default=120(ms))
– Of course, for a long-running query, parallel query with degree ANY can result in
many times reduction in elapsed time
• VSAM catalog update x’0A’/x’94’
• VSAM catalog update x’0A’/x’94’
• DB2 for z/OS application requestor using TCP/IP waiting for server response
x’1B’/x’8F’
67

How to find which service task invoked
• Performance trace of IFCID 46 thru 50
– IFCID 46 for service task switch
• 47 and 48 for Begin and End of SRB service task
• 49 and 50 for Begin and End of TCB service task
– Shows all nested service tasks
• Or use IFCID 170 and 171 for less output
• Or use IFCID 170 and 171 for less output
– Shows one (outermost) service task for each event in class 3 accounting

Notes
• Correlation name of system agent (SYSOPR planname) in performance trace:
– eg. 021.OPNL for SYSLGRNX update in pageset open
• Chapter 9 of DB2 Diagnosis Guide and Reference contains a list of system agent
correlation identifiers as well as a list of Resource Manager id's displayed in
IFCID 46, 47, 49, and 170.

Commonly observed service tasks
• Commonly observed service tasks and RMID/FC identifiers as shown in IFCID
46, 47, 49, and 170
– Update Commit x’03’/x’49’
– Dataset Open x’0A’/x’59’
– Dataset Close x’0A’/x’4F’
– SYSLGRNX Update x’15’/x’44’
– Dataset Extend x’12’/x’65’
– Dataset Delete x’12’/x’6A’
– Dataset Define x’12’/x’68’
– Dataset Reset x’12’/x’6C’
– Dataset Count x’12’/x’88’

Notes
• FC = Function Code
• RMID = Resource Manager ID
• If a long time gap between IFCID 46 and 47 (begin SRB task) or 49 (begin TCB
task), then it is possible that we are waiting for a service task to become
available.

Log Read Suspensions
w
ROLLBACK
application
address space
DB2
address space
read active or
archive log
Wait for
other
agents’ writes
data not in log buffer
What if … Wait for
log reads
− Find out why the transactions rollback
(most rollbacks are done under EU-
switch - those don’t show up here)
− Keep to a minimum
w
a
i
t
archive log
data retrieved
backout
− Keep to a minimum
Time per suspension high?
− Increase the number and size of
active log data sets
• In V8 only archive log read wait was tracked
• Starting in V9 active log prefetch read wait time also included
• Vnext will also include sync. active log read
(currently NOT ACCOUNT time)

Archive Log Mode (Quiesce) Suspensions
UPDATE
application
address space
DB2
address space
Wait for
other
agents’ writes
What if …
Waits for
quiesce
commit
buffer update
COMMIT
ARCHIVE LOG
MODE(QUIESCE)
issued
w
a
i
t
suspend
resume
− Too low QUIESCE system parameter
or command TIME option
Time per suspension high?
− Long units of recovery
− Inappropriate timing for quiesce process
buffer update
commit
UPDATE
issued
archive
begins

Page Latch Suspensions
w
FETCH
application
address space
DB2
address space
page latch suspend
Wait for
other
agents’ writes
getpage hit,
but page latched Most often a symptom of something else
Often during high insert activity
−Space map page
What if …
Wait for
page latch
suspensions
a
i
t
page latch resume
−Space map page
Partitioning
MEMBER CLUSTER
−Data page
Randomize inserts
−Use IFCID 226, 227 for detail analysis

Page Latch Contention
• Also consider
– Higher VDWT to possibly reduce write frequency
– Faster active log and database write I/O device
– If spacemap page,
• TRACKMOD NO instead of default YES
– However, no ‘real’ incremental image copy if TRACKMOD NO
• Concurrent inserts to different partitions
• Concurrent inserts to different partitions
• Classic partitioned rather than segmented or UTS in V9
– If index page,
• V9 RANDOM instead of ASC/DESC
– Trade-off with higher data Getpage, I/O, lock requests
• V9 index page: smaller if random key insert,
bigger if sequential key insert

Global Lock Suspensions
w
a
UPDATE
application
address space
DB2
address space
lock suspend
Wait for
other
agents’ writes
lock unavailable – held by another system
• Thread can be suspended for:
− Global L-parent lock
− Global L-child (row/page) lock
− Other global L-lock
− Global pageset P-lock
− Global page P-lock
− Data
− IX leaf pages
− SM pages
a
i
t
lock resume
buffer update
− SM pages
− Other P-locks
• DB2 reports:
− Accumulated time spent suspended
− Total number of suspensions
− Sum of all IRLM, XES, and False
contentions in Data Sharing Locking
section

Wait for
other
agents’ writes
What if …
Wait for
global lock
suspensions
High Global Lock Contention Events/Time
• Keep CF utilization below 50% for availability, response time, CPU time
• If most time or most events are “OTHER” P-lock or L-lock
– -STA TRA (P) C(6) … for a short time to get detailed info

High Global lock contention events/time
• To minimize P-lock contention/negotiation
– First find out from GBP statistics what is contributing to P-lock contention
– If spacemap page P-lock and/or data page P-lock, consider using MEMBER
CLUSTER table space
– V8/9/10 Partitioned table space
–
– V10 UTS
– High spacemap page latch contention possible when the latch is held waiting for data
spacemap page P-lock contention to resolve.

• P-lock contention/negotiation - continued
– Consider setting VDWQT=0 to free page P-locks as soon as possible
– If index tree P-lock (high index splits), consider
– Freespace tuning (PCTFREE FREEPAGE)
– Minimize index key size especially for unique or semi-unique multi-column index key
– V8 non-padded index if large VARCHAR(s) in index key
– V9 bigger index page size and asymmetric page split
–
– If data page P-lock contention on small tables with LOCKSIZE ROW, consider
MAXROWS=1 and LOCKSIZE PAGE to ‘simulate’ row locking and/or reduce SM
page freespace update

• Can also affect other counters
– Update Commit service task wait P-lock unlock for each updated page
– Lock/Latch wait can increase because of
– Possibly higher IRLM latch contention
– “Mixing” local/global lock waits (next foil)

“Mixing” Global/Local Lock Suspensions
Acquire lock
Lock
TRAN A
X-Lock page 5
TRAN B
X-Lock page 5
lock suspend
TRAN C
X-Lock page 5
Release lock
TRAN A
Commit
TRAN B
X-Lock page 5
TRAN B
Commit
Lock
TRAN C
X-Lock page 5
Time Member 2
Lock
resume
Time Member 1
lock suspend
(Wait behind TRAN B)Lock suspend
Lock
resumeRelease lock
Global lock wait
Local lock wait
• Even though part of the wait time for tran C it is waiting for a global lock held by tran A,
all suspension time is reported as local because at resume time, the lock that was
released was local to this member

Time in DB2 (Class 2 time)
– Many different types of suspensions ( 21 in V10)
– Mostly things that happen outside of DB2’s control, like
– Lack of CPU or paging

What Is Left?
• Processor wait
– Wrong dispatching priorities
• NO DB2 work should run in discretionary
– Overloaded processor
• Use RMF for details
• General tuning
• Processor upgrade
•
Suspensions
In DB2 Time
Processing
• Paging
– Overcommitted real storage
• Use RMF for details
• MAXKEEPD too large
• Check Buffer pools, EDM pool, Sort,
number of threads, …
• Real storage upgrade
Not Accounted

What Is Left? - 2
• OEM monitors
• Wait time for completion of parallel tasks
• System-managed duplexing lock
requests
• HSM dataset recall
(since PQ67298)
• Waiting for requests to be returned from
Suspensions
In DB2 Time
Processing
• Waiting for requests to be returned from
SNA DB2 Server
• Portion of OPEN processing
– Accompanied with the appropriate
synchronous EU switch suspensions
Not Accounted

Notes
• Most common are:
– CPU wait under high CPU utilization, especially with lower dispatching priority
– Excessive detailed online tracing with vendor tools
• Other causes are much less frequent and widely varied
Some events not being captured by DB2, but more events are being captured in
newer DB2 versions
– See online support document:
– See online support document:
http://www.ibm.com/support/docview.wss?rs=64&context=SSEPEK&uid=swg21045823

Effect of lack of CPU resources on CL3/8 timers
w
a
getpage missUPDATE
application
address space
DB2
address space
• “Clock” on the wait time can
only be stopped after the thread
is dispatched again
• If you have to wait for the CPU
the timer keeps “ticking”
Start I/O
Start clock
a
i
t
t
I
m
e
Stop clock
buffer update
suspend
I/O interrupt
resume
I/O time
CPU wait

Accounting for DDF Work
• Processing "in DB2" (Class 2) should (almost) be the same regardless of
connection type
• Time in DB2 server is:
– Class 2 nonnested elapsed time +
Class 1 stored procedure, UDF, and trigger ET+
Nonnested (Class 1 CPU - Class 2 CPU) +
Nonnested (class 1 SE CPU - class2 SE CPU)
•
• Time outside of DB2 server is total Class 1 elapsed less previous calculation
• Active thread accounting records are created at thread deallocation (disconnect)
• Inactive thread accounting records are created at DBAT inactive (look for DBAT
inactive)

DDF Accounting at Plan Level
TIMES/EVENTS APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 3:10.24012 3.387696
NONNESTED 3:10.24012 3.387696
STORED PROC 0.000000 0.000000
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
CP CPU TIME 1.020178 0.980084
AGENT 0.979296 0.939458
NONNESTED 0.979296 0.939458
STORED PRC 0.000000 0.000000
UDF 0.000000 0.000000
• Time in DB2 = class 2 nonnested ET
+ class 1 nested activity ET
[stored procedure + UDF + Trigger]
+ nonnested (class 1 CP CPU - class 2 CP CPU)
+ nonnested (class 1 SE CPU - class2 SE CPU)
• Time in DB2 = 3.387696
+ (0.000000 + 0.000000 + 0.000000)
+ (0.979296 - 0.939458 )
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
PAR.TASKS 0.040882 0.040626
SECP CPU 0.000000 N/A
SE CPU TIME 0.008714 0.008714
NONNESTED 0.008714 0.008714
STORED PROC 0.000000 0.000000
UDF 0.000000 0.000000
TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
SUSPEND TIME 0.000000 1.440549
.. .. ..
+ (0.979296 - 0.939458 )
+ (0.008714 - 0.008714)
= 3.427534
• Time outside DB2 = class 1 ET - Time in DB2
• Time outside of DB2 =
3:10.24012 – 3.427534 = 3:06.8126
• Note: Be careful with SE CPU time when running on a
‘knee-capped’ machine – using normalized CPU times

DDF Accounting at Plan Level -2
SQL DML TOTAL
---- DISTRIBUTED ACTIVITY -----------------------------------------------------------------------------
REQUESTER : DDF1 ROLLBCK(1) RECEIVED: 0 THREADS INDOUBT : 0
PRODUCT ID : DB2 SQL RECEIVED : 601 ROWS SENT : 46515
PRODUCT VERSION : V9 R1 M5 MESSAGES SENT : 603 BLOCKS SENT : 340
METHOD : DRDA PROTOCOL MESSAGES RECEIVED : 604 CONVERSAT.INITIATED: 1
COMMITS(1) RECEIVED: 0 BYTES SENT : 2087111 NBR RLUP THREADS : N/A
BYTES RECEIVED : 809497
COMMIT(2) RECEIVED : 1 COMMIT(2) RESP.SENT: 1 PREPARE RECEIVED : 1
BACKOUT(2) RECEIVED: 0 BACKOUT(2)RESP.SENT: 0 LAST AGENT RECV. : 0
COMMIT(2) PERFORMED: 0 BACKOUT(2)PERFORMED: 0 MESSAGES IN BUFFER : 46515
TRANSACTIONS RECV. : 1 FORGET SENT : 0
SQL DML TOTAL
-------- --------
SELECT 0
INSERT 226
ROWS 282
UPDATE 30
ROWS 84
MERGE 0
DELETE 0
ROWS 0
DESCRIBE 0
DESC.TBL 0
PREPARE 144
OPEN 142
FETCH 199
ROWS 46515
CLOSE 1
DML-ALL 742
• Blocking is crucial for
performance: Rows are
put into blocks which are then
sent out in messages
• Note # rows in DML operations
for multi-row operations
(implicit or explicit)

Acctg Reporting when (CPU) Parallelism is used
• In DB2 accounting data you can distinguish between
– Agent (parent)
– Parallel task
• # acctg records depends on ZPARM PTASKROL (Opaque ZPARM)
– Yes (Default)
• One acctg record for the parent task, and one acctg record for all // activity
• When using sysplex //, coordinating member is same as above, and each assisting member
• When using sysplex //, coordinating member is same as above, and each assisting member
cuts one acctg record per // group (so can be more than one per query/tran)
– No
• Each parallel task cuts a separate acctg record (regardless of type of CPU //)
– Normally your friendly monitor takes care of ‘rolling up’ this information

OMPE Accounting Report/Trace for a Parallel Query
TIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS
------------ ---------- ---------- ---------- -------------------- ------------ --------
ELAPSED TIME 1:27.10913 1:27.10502 N/P LOCK/LATCH(DB2+IRLM) 0.067109 634
NONNESTED 1:27.10913 1:27.10502 N/A IRLM LOCK+LATCH N/A N/A
STORED PROC 0.000000 0.000000 N/A DB2 LATCH N/A N/A
UDF 0.000000 0.000000 N/A SYNCHRON. I/O 0.016820 47
TRIGGER 0.000000 0.000000 N/A DATABASE I/O 0.016820 47
LOG WRITE I/O 0.000000 0
CP CPU TIME 24.253183 24.252075 N/P OTHER READ I/O 10:38.762530 98546
AGENT 0.002148 0.001053 N/A OTHER WRTE I/O 0.000000 0
NONNESTED 0.002148 0.001053 N/P SER.TASK SWTCH 0.319902 22
Susp. time of parent
and ALL child tasks
NONNESTED 0.002148 0.001053 N/P SER.TASK SWTCH 0.319902 22
STORED PRC 0.000000 0.000000 N/A UPDATE COMMIT 0.000000 0
UDF 0.000000 0.000000 N/A OPEN/CLOSE 0.318962 10
TRIGGER 0.000000 0.000000 N/A SYSLGRNG REC 0.000823 10
PAR.TASKS 24.251035 24.251022 N/A EXT/DEL/DEF 0.000000 0
OTHER SERVICE 0.000117 2
.. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. ..
SUSPEND TIME 0.000000 10:39.1664
AGENT N/A 0.000117
PAR.TASKS N/A 10:39.1662
UDF 0.000000 N/A
NOT ACCOUNT. N/A 1:27.10385
NOT ACCT time
calculated based
on parent task only
Sum of CPU time of
all // tasks
CPU time of parent task
Note: CPU time can exceed
ET when using // for CPU
intensive queries

Understanding the Transaction ‘Profile’
• To get a better feel for type of work performed by a transaction/job, always at
least check:
– # and type of SQL statements (S/I/U/D)
– The # and type of lock request
(local and global (when data sharing)
– BP activity and GBP activity(when data sharing)

SQL Info in DB2 Accounting at Plan level
• PK62161 (V8/9) Counters added
for multi-row operations
• PM24812 (V8/V9/V10)
Counters will include only rows
that are updated, deleted or
inserted by direct user action
SQL DML AVERAGE TOTAL
------------- ---------- ----------
SELECT 5.02 1217636
INSERT 0.84 203878
ROWS 1.69 409044
UPDATE 0.87 211103
ROWS 1.01 244029
MERGE 0.00 0
DELETE 0.10 24085
ROWS 0.10 24085
• DB2 10 ‘DRDA bundled fetch’
may reduce #FETCH counter, but
#ROWS-FETCH should be
accurate
ROWS 0.10 24085
DESCRIBE 0.00 0
DESC.TBL 0.00 0
PREPARE 0.00 0
OPEN 3.14 762136
FETCH 29.10 7053106
ROWS 26.13 6333609
CLOSE 2.94 711718
DML-ALL 42.02 10183662

Locking Info in Accounting at Plan Level
• Extra info about MAX PG/ROW LOCKS HELD compared to stats
LOCKING AVERAGE TOTAL
--------------------- -------- --------
TIMEOUTS 0.00 0
DEADLOCKS 0.00 0
ESCAL.(SHARED) 0.00 0
ESCAL.(EXCLUS) 1.00 4
MAX PG/ROW LOCKS HELD 5004.00 5004
LOCK REQUEST 5042.00 20168
LOCK REQUEST 5042.00 20168
UNLOCK REQUEST 34.00 136
QUERY REQUEST 0.00 0
CHANGE REQUEST 3.00 12
OTHER REQUEST 0.00 0
TOTAL SUSPENSIONS 1.75 7
LOCK SUSPENSIONS 0.00 0
IRLM LATCH SUSPENS. 1.75 7
OTHER SUSPENS. 0.00 0

Locking Info in Accounting at Plan Level - 2
• TIMEOUTS
– The number of times lock suspension ultimately resulted in a timeout
• DEADLOCKS
– The number of times lock suspension ultimately resulted in a deadlock. This happens
when two or more application processes each hold locks on resources that the others
need and without which they cannot proceed
• ESCAL.(SHARED/EXCLUSIVE)
• ESCAL.(SHARED/EXCLUSIVE)
– Number of times the LOCKS PER TABLE(SPACE) parameter on the panel DSNTIPJ
was exceeded and the lock size was promoted from a page lock to a table space lock
for this thread. For example, an application updates most pages in a table without
issuing commits
• Further investigation
– Typically start with statistics trace class 3 info + syslog + db2MSTR log
– Zoom in with Locking traces – perf. trace class 6,7 and SQL trace – perf. trace class 3
–STA TRA(P) C(1,2,3,6,7) DEST(xxx) should do

• MAX PG/ROW LOCKS HELD from Accounting trace is a useful indicator of
commit frequency
– Page or row locks only
– AVERAGE is for average of MAX, TOTAL is for max of MAX (of Accounting records)
• So if transaction A had max. locks of 10 and transaction B had 20, then
– AVERAGE (avg. of max.) = 15
– TOTAL (max. of max.) = 20
– In general, try to issue Commit to keep max. locks held below 100
LOCKING AVERAGE TOTAL
--------------------- -------- --------
...
MAX PG/ROW LOCKS HELD 5004.00 5004

• Good Lock avoidance can significantly improve performance
– Quick check in acctg data can provide some clues whether or not lock avoidance is ok
– Lock avoidance may not be working effectively if Unlock request/commit is high
• >5/commit and > 1/3 of Lock request
• E.g. 50 Lock and 40 Unlock requests per commit
– Check table on “Lock avoidance factors” in PMTG to see under which conditions DB2
is able to use lock avoidance techniques
• Using ISO(CS) CURRENTDATA(NO) – default now – gives DB2 most opportunities to avoid
having to take locks
• Check the IRLM lock and IRLM latch suspensions in the locking section against
the L/L CL3 suspensions

Data Sharing Locking Section
DATA SHARING AVERAGE TOTAL
------------------- -------- --------
GLOBAL CONT RATE(%) 0.48 N/A
FALSE CONT RATE(%) 0.00 N/A
P/L-LOCKS XES(%) 43.83 N/A
LOCK REQ - PLOCKS 0.67 270412
UNLOCK REQ - PLOCKS 0.63 252966
CHANGE REQ - PLOCKS 0.01 2550
• Global Contention should be less than
3-5% of XES IRLM Requests
– (IRLM + XES + False SUSP) /
((IRLM + XES + False SUSP) +
(LOCK+CHANGE+UNLOCK +
CONV/ XES))
LOCK REQ - XES 5.23 2101826
UNLOCK REQ - XES 0.89 358098
CHANGE REQ - XES 0.04 16744
SUSPENDS - IRLM 0.03 11906
SUSPENDS - XES 0.00 0
CONVERSIONS- XES 0.02 7204
FALSE CONTENTIONS 0.00 0
INCOMPATIBLE LOCKS 0.00 0
NOTIFY MSGS SENT 0.01 2278
• False contention less than 1-3% of
sum of all XES IRLM Requests
• P/L LOCKS XES%
– Gives an idea how many of the locks
requests are propagated to XES
(“degree” of sharing)

Local vs. Data Sharing locks
LOCKING TOTAL DATA SHARING TOTAL
------------------- -------- ------------ --------
TIMEOUTS 0 P/L-LOCKS(%) 5
DEADLOCKS 0 P-LOCK REQ 193024
ESCAL.(SHAR) 0 P-UNLOCK REQ 191624
ESCAL.(EXCL) 0 P-CHANGE REQ 101
MAX PG/ROW LCK HELD 91 LOCK - XES 739773
LOCK REQUEST 14175179 UNLOCK-XES 193684
UNLOCK REQST 138996 CHANGE-XES 101
UNLOCK REQST 138996 CHANGE-XES 101
QUERY REQST 0 SUSP - IRLM 196
CHANGE REQST 0 SUSP - XES 57
OTHER REQST 0 CONV - XES 317
TOTAL SUSPENSIONS 12337 FALSE CONT 789
LOCK SUSPENS 3 INCOMP.LOCK 0
IRLM LATCH SUSPENS 12334 NOTIFY SENT 0
OTHER SUSPENS 0
At last – Finally – Rejoice
Requires PK85543 (DB2) PK85159(IRLM) OA12164(XES)

Local vs. Data Sharing locks - 2
• LOCK REQUESTS are L-lock requests
• P-LOCK REQ are the number of P-lock requests
– P-locks are ALWAYS sent to XES so they are included in the LOCK - XES counter
• LOCK – XES are the number of lock requests propagated to XES. This includes
– L-locks from LOCK REQUEST that need to be propagated (because of the interest)
and
– P-LOCK REQ that are always propagated to XES
– P-LOCK REQ that are always propagated to XES
– Only those that are Synchronously propagated to XES.
• With current maintenance, we can now distinguish between suspended because
of false contention vs. suspended because of XES heuristic conversions

BP Info in Accounting at Plan Level
• Not as many BP counters in DB2 accounting compared to DB2 statistics
BP0 BPOOL ACTIVITY AVERAGE TOTAL
--------------------- -------- --------
BPOOL HIT RATIO (%) 92.22 N/A
GETPAGES 711.64 12605326
BUFFER UPDATES 226.48 4011708
SYNCHRONOUS WRITE 0.00 0
SYNCHRONOUS WRITE 0.00 0
SYNCHRONOUS READ 49.71 880538
SEQ. PREFETCH REQS 0.03 597
LIST PREFETCH REQS 0.19 3313
DYN. PREFETCH REQS 1.11 19595
PAGES READ ASYNCHR. 5.67 100352

BP Info in Accounting at Plan Level - 2
• #Getpages is a good indicator of the amount of work that needs to be done by
the transaction
– Likely to change if access path changes, or object gets more disorganized
• #SYNC READs are important as you wait for those
(check against the CL3 data)
• A lot of prefetch activity can be sign of access path problem
• A lot of prefetch activity can be sign of access path problem
• BP info
– At the plan level – info for each BP
– At the package level – all BP together and only if accounting class 10 is active

GBP info
• GBP tuning is mostly done via stats
info (not acctg)
• Check GBP hit/miss ratio for
application
• Check application
page P-lock activity
– # Page P-lock requests
GROUP BP0 AVERAGE TOTAL
--------------------- -------- --------
GBP-DEPEND GETPAGES 14707.80 2191462
READ(XI)-DATA RETUR 0.32 47
READ(XI)-NO DATA RT 0.00 0
READ(NF)-DATA RETUR 101.45 15116
READ(NF)-NO DATA RT 313.67 46737
PREFETCH PAGES READ 9.48 1413
CLEAN PAGES WRITTEN 0.00 0
UNREGISTER PAGE 19.46 2899
ASYNCH GBP REQUESTS 9.31 1387
EXPLICIT X-INVALID 0.00 0
– # Page P-lock requests
– # Page P-lock suspensions
– Type of page P-lock SM, data index
leaf
• There is no GBP information at the
package level
ASYNCH SEC-GBP REQ 0.00 0
PG P-LOCK LOCK REQ 497.81 74174
SPACE MAP PAGES 17.58 2620
DATA PAGES 0.80 119
INDEX LEAF PAGES 479.43 71435
PG P-LOCK UNLOCK REQ 146.15 21776
PG P-LOCK LOCK SUSP 1.17 175
SPACE MAP PAGES 0.58 87
DATA PAGES 0.05 8
INDEX LEAF PAGES 0.54 80
WRITE AND REGISTER 176.83 26347
WRITE & REGISTER MULT 17.73 2642
CHANGED PAGES WRITTEN 304.57 45381

GBP Read Tuning
– Local BP search -> GBP search -> DASD I/O
GROUP BP14 QUANTITY /SECOND /THREAD /COMMIT
----------------------------- -------- ------- ------- -------
...
SYN.READ(XI)-DATA RETURNED 1932.00 0.09 0.01 0.00
SYN.READ(XI)-NO DATA RETURN 39281.6K 1823.66 236.31 79.22
SYN.READ(NF)-DATA RETURNED 22837.00 1.06 0.14 0.05
SYN.READ(NF)-NO DATA RETURN 6955.8K 322.93 41.85 14.03
– Local BP search -> GBP search -> DASD I/O
– SYN.READ(NF) = Local Buffer Pool miss
– SYN.READ(XI) = Local Buffer Pool hit but cross-invalidated buffer
– Most data should be found in GBP >> if not, GBP may be too small or pages have
been removed because of directory entry reclaims
Sync.Read(XI) miss ratio should be < 10%ROTROT
TOTAL SYN.READ(XI) = SYN.READ(XI)-DATA RETURNED + SYN.READ(XI)-NO DATA RETURN
Sync.Read(XI) miss ratio = SYN.READ(XI)-NO DATA RETURN / TOTAL SYN.READ(XI)

DSQGESQM VALUE DSQGESQM TIMES DSQGESQM AVERAGE TIME AVG.EV TIME/EVENT
------------------ ------------------ ------------------ ------------ ------------------ ------------ ------ ------------
TYPE PACKAGE ELAP-CL7 TIME-AVG 2:37.547703 LOCK/LATCH 0.017786 42.47 0.000419
CP CPU TIME 2:21.224699 IRLM LOCK+LATCH 0.000335 0.20 0.001641
LOCATION USTSYIPT0D AGENT 2:21.224699 DB2 LATCH 0.017451 42.27 0.000413
COLLECTION ID Q PAR.TASKS 0.000000 SYNCHRONOUS I/O 0.634727 698.49 0.000909
PROGRAM NAME DSQGESQM SE CPU TIME 0.000000 OTHER READ I/O 0.494961 404.75 0.001223
SUSPENSION-CL8 1.278264 OTHER WRITE I/O 0.026058 9.56 0.002726
ACTIVITY TYPE NONNESTED AGENT 1.278264 SERV.TASK SWITCH 0.104705 16.77 0.006242
ACTIVITY NAME 'BLANK' PAR.TASKS 0.000000 ARCH.LOG(QUIESCE) 0.000000 0.00 N/C
SCHEMA NAME 'BLANK' NOT ACCOUNTED 15.044740 ARCHIVE LOG READ 0.000000 0.00 N/C
SUCC AUTH CHECK 0 AVG.DB2 ENTRY/EXIT 112.91 DRAIN LOCK 0.000000 0.00 N/C
OCCURRENCES 93 DB2 ENTRY/EXIT 10501 CLAIM RELEASE 0.000000 0.00 N/C
NBR OF ALLOCATIONS 649 PAGE LATCH 0.000027 0.05 0.000505
SQL STMT - AVERAGE 13.94 CP CPU SU 5066356.40 NOTIFY MESSAGES 0.000000 0.00 N/C
SQL STMT - TOTAL 1296 AGENT 5066356.40 GLOBAL CONTENTION 0.000000 0.00 N/C
NBR RLUP THREADS 93 PAR.TASKS 0.00 TCP/IP LOB XML 0.000000 0.00 N/C
SE CPU SU 0.00 TOTAL CL8 SUSPENS. 1.278264 1.2K 0.001091
Package Level accounting information
SE CPU SU 0.00 TOTAL CL8 SUSPENS. 1.278264 1.2K 0.001091
DSQGESQM AVERAGE TIME AV.EVENT DSQGESQM AVERAGE TIME AV.EVENT
------------------------- ------------ -------- ------------------------- ------------ --------
GLOBAL CONTENTION L-LOCKS 0.000000 0.00 GLOBAL CONTENTION P-LOCKS 0.000000 0.00
PARENT (DB,TS,TAB,PART) 0.000000 0.00 PAGESET/PARTITION 0.000000 0.00
CHILD (PAGE,ROW) 0.000000 0.00 PAGE 0.000000 0.00
OTHER 0.000000 0.00 OTHER 0.000000 0.00 CL8 info
OCCURRENCES - # IFCID 239 entries for this package
NBR OF ALLOCATIONS - # times switched to the package
NBR RLUP THREADS (new in V10)
package level detail info with rollup (not class10) - no longer *ROLLUP*
[In OMPE averages are based on # NBR RLUP THREADS (= #OCC with no
rollup) ]
• Also SE CPU time
at package level

DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL
------------------ -------- -------- ------------------- -------- -------- --------------------- -------- --------
SELECT 0.00 0 BPOOL HIT RATIO (%) 99.25 N/A TIMEOUTS 0.00 0
INSERT 0.12 11 GETPAGES 4185.6K 389262K DEADLOCKS 0.00 0
UPDATE 0.04 4 BUFFER UPDATES 890.5K 82816786 ESCAL.(SHARED) 0.00 0
DELETE 0.06 6 SYNCHRONOUS WRITE 0.05 5 ESCAL.(EXCLUS) 0.04 4
SYNCHRONOUS READ 698.41 64952 MAX PG/ROW LOCKS HELD 477.02 44363
DESCRIBE 0.00 0 SEQ. PREFETCH REQS 2036.97 189438 LOCK REQUEST 1904.25 177095
PREPARE 0.00 0 LIST PREFETCH REQS 4.84 450 UNLOCK REQUEST 636.86 59228
OPEN 0.00 0 DYN. PREFETCH REQS 1976.46 183811 QUERY REQUEST 0.00 0
FETCH 0.00 0 PAGES READ ASYNCHR. 30603.12 2846090 CHANGE REQUEST 224.92 20918
CLOSE 0.00 0 OTHER REQUEST 0.00 0
SQL BP totals Locking
Package level Accounting - acctg CL10
CLOSE 0.00 0 OTHER REQUEST 0.00 0
TOTAL SUSPENSIONS 0.20 19
LOCK TABLE 0.00 0 LOCK SUSPENSIONS 0.00 0
CALL 0.00 0 IRLM LATCH SUSPENS. 0.20 19
OTHER SUSPENS. 0.00 0
• Only available when accounting trace class 10 is active
• Not available when rollup accounting is in effect (in plan for Vnext)
• BP info is not at BP level, but all BP together
• Note that sum SQL here is not same as SQL STMT TOTAL
(CL10 section does not include things like SET stmts, so expect <= )

DB2 V8 Enhancements
• Better control over basic accounting intervals
– KEEPDYNAMIC YES will not prevent completing the accounting interval at commit
boundary
– At the same time change of WLM enclave is enabled for the new transaction
• Accounting roll-up (for RRS and DDF work)
– Addressing the issue about the large volume of accounting data
– Aggregating accounting data for the same values of the client-side identifiers (end
– Aggregating accounting data for the same values of the client-side identifiers (end
user ID, end user transaction/application name, end user workstation name)
• zparm ACCUMUID controls which combination of IDs is used
• zparm ACCUMACC controls when the aggregated accounting records are externalized

DB2 V10 Enhancements
• Better accounting rollup (for RRS and DDF work)
– Still don’t like it
– Packages not all lumped together (up to 24 packages will be rolled up separately – if
more rest is rolled up into a single package)
– DDF info (up to 4 unique QLACs will be rolled up separately – if more the rest is rolled
up into a single one)
• Compression of SMF records (SMFCOMP=YES ZPARM)
– Should be enough to avoid a need to use ACCUMACC
– Small overhead (1%), great compression ratio (60-80%)
• Separating DB2 internal latch and IRLM lock/latch suspensions (see before)

DB2 V10 Enhancements
• Aggregate accounting statistics
– Introduced by APAR PM62797 (8/12)
– (OMPE APAR PM72949 to support this in stats reports)
– High level info accounting information per connection type
– New statistics class 9 will write IFCID369 (if acctg trace class 1 is also active)
– Written at 1 min (stats) interval
– Basically class 1,2,3 time info per connection type
– Basically class 1,2,3 time info per connection type
• Fields from DSNDQWAC and DSNDQWAX
– ‘Draft’ sample OMPE report
CONNTYPE CL1 ELAPSED CL1 CPU CL1 SE CPU CL2 ELAPSED CL2 CPU CL2 SE CPU CL3 SUSP CL2 NOT ACC THREADS
-------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ --------
BATCH 1:14.326062 8.882753 0.000000 1:14.234073 8.879979 0.000000 1:05.997069 0.000000 3.00
CICS N/P N/P N/P N/P N/P N/P N/P N/P 0.00
DDF 3:20.927303 9.873839 8.240478 55.917303 8.697064 6.742272 39.852088 7.368151 15895.00
IMS N/P N/P N/P N/P N/P N/P N/P N/P 0.00
RRSAF 1:04.091521 0.143129 0.000000 0.050749 0.044200 0.000000 0.000062 0.006487 10.00
UTILITY 21.317271 0.712988 0.031434 1.224435 0.189245 0.031434 0.987360 0.047829 72.00

© Copyright IBM Corporation 2012. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE
EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS
PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION,
Disclaimer
© 2012 IBM Corporation110
PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION,
THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO
CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED
IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR
REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF
ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE
IBM, the IBM logo, ibm.com, and DB2 are trademarks or registered trademarks of International Business Machines Corporation in the
United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information
with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this
information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

DB2 Accounting Reporting

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DB2 Accounting Reporting

Similar to DB2 Accounting Reporting (20)

Recently uploaded

Recently uploaded (20)

DB2 Accounting Reporting