Click to edit Master title style
Monitoring IMS Performance
for
Faster Problem Solving
Lih M. Wang
IBM System z Client Tec...
Topics
Introduction
– IMS Overview
– Performance Basics

IMS Transaction Flow
– Key Performance Indicators

Case Study
Sum...
Click to edit Master title style

Introduction

3

3

© 2013 IBM Corporation
IMS TM and DB Overview
D
L
I
S
A
S

C
O
N
T
R
O
L

BMP
D
B
R
C

I
R
L
M

BMP
MP
MP

ACBLIB

MFS

MSG
Queue

OLDS

WADS

RD...
IMS Java Transaction Support

5

5

© 2013 IBM Corporation
Emergence of online – All roads lead to IMS

IMS
IMS
Connect
IMS
Connect
IMS
Connect

IMS

Connect
Web Browser

WebSphere
...
Click to edit Master title style

What to Monitor?

7

7

© 2013 IBM Corporation
Performance Basics
CPU
Memory
I/O
Network
Locking
Storage
Applications

8

8

© 2013 IBM Corporation
IMS System Health At-a-Glance

Proactively monitor IMS address spaces and health indicators
Out-of-the Box graphics showin...
Anatomy of an I/O Request
Front-end Time

IOSQ
T1

Queue
Device

EXCP
Issued

10

+ PEND
T2

Queue
Path

Back-end Time

+ ...
Monitoring I/O Response Time

I/O Response Time = IOSQ + PEND + CONN + DISCONNECT
11

11

© 2013 IBM Corporation
Monitoring TCP/IP Connections

12

12

© 2013 IBM Corporation
Global Locking – Top Blocker

New area
containing list of
top blockers

13

13

© 2013 IBM Corporation
Click to edit Master title style

IMS Transaction Flow and
Key Performance Indicators

14

14

© 2013 IBM Corporation
IMS Transaction Flow
Communications

Control

MP
Scheduler
Common
Services

Transaction

Appl.
Pgm.

Log

15

DL/I

Data
B...
IMS Communication Overview

Communications

Control

MP
Scheduler
Common
Services

Transaction

Appl.
Pgm.
DL/I
Log
Data
B...
Communication Pools
ECSA
VTAM
NODE

IOBUFs

SAPs

IMS CONTROL REGION

VTAM

CIOP
HIOP

RECANY

OUTBUF

SPAP
MFS/FRE

Forma...
Selective Dispatching
------------------------------------------------------------ 10/27/12 9:09:45
IMS Status
I91A
+-----...
Storage Pool Information

19

19

© 2013 IBM Corporation
IMS Scheduler Overview
Communications

Control

MP
Scheduler
Common
Services

Transaction

Appl.
Pgm.
DL/I
Log
Data
Bases
...
Scheduling Facility
Priority

Class

SCHEDULER

Load
Balancing

Message
Queue
Pool

21

Processing
Limit Count

Scheduler
...
Scheduling Waits
CONTROL REGION
BLDL
LIST

PSB WORK
AREA POOL

PSB POOL
PSB
A

PSB
B

PSB
C
RESIDENT
PSB’s

ACBLIB

INTENT...
PSB Status - Clean Up The Unusable

23

23

© 2013 IBM Corporation
Schedule End to First DL/I Call
Time before the first DL/I call to retrieve a message
– Loading the application program
– ...
Monitor Non-DL/I Wait
PDEX
------SHORT TERM %---------LONG TERM %----+ (ELAPSED TIME= 1:41 MN) % 0_______ 50_______100
% 0...
IMS Message Queuing

Communications

Control

MP
Scheduler

Transaction

Common
Services
Appl.
Pgm.

Log

DL/I
Data
Bases
...
Input/Output Messages
CONTROL REGION

QPOOL
MESSAGE

SMB

QUEUE
BLOCK

MESSAGE

MESSAGE

MESSAGE

CNT

SHORT
MESSAGE

LONG...
Message Queue Considerations
Allocate sufficient QBUFs
Optimize LRECL sizes to make efficient use of pool
space and balanc...
Message Queue I/O
------------------------------------------------------------ 08/28/11 14:39:10
Message Queue Buffers and...
If MSG-Q BUF# Became Low...
Enough active MP regions?
Scheduling Delayed because of short on PSB, PSBW, or
EPCB pools?
Rem...
DL/I Interface
Communications

Control

MP
Scheduler
Common
Services

Transaction

Appl.
Pgm.
Log

DL/I
Data
Bases

31

Me...
Online Database Operations
Interregion
Communication

32

ENQ/DEQ Pool

DL/I Buffer
Handler Pool

Program
Isolation

OSAM
...
VSAM Hiperspace
DLISAS
B
U
F
F
E
R

16 MEG

33

P
R
E
F
I
X
E
S

Buffers

VSAM Hiperspace

Buffers

Hiperspace
Buffers
Buf...
VSAM Buffer Pool
------------------------------------------------------------ 08/28/12 14:48:08
VSAM Buffer Pools Status
I...
OSAM Buffer Pool
4 Gig limit
Page fix only the buffer
Sequential Buffering
DFSVSMxx member in IMSVS.PROCLIB
– IOBF=(4096,5...
IMS Checkpoint Processing
Communications

Control

MP
Scheduler
Common
Services

Transaction

Appl.
Pgm.

Log

36

DL/I

D...
Common Logging Facility
z/OS
IMS/TM

DBRC

DLISAS

Control
Region

DB Recovery
Control

DL/I Subordinate
Address Space

WA...
Logging I/O
------------------------------------------------------------ 08/28/12
Logging Buffers and Statistics
I91A

7:1...
Monitor IMS System Datasets

39

39

© 2013 IBM Corporation
Cross Reference – Part 1
Reference: IMS Performance Guide Redbook, SG24-4637-00
EVENT

Message Input

ACTIVITY

POOLs/LIST...
Cross Reference – Part 2
EVENT

ACTIVITY

POOLs/LISTs

Program Elapsed

DC Calls

QBUF

MSGQs

DL/I Elapsed

DB Calls

OSA...
Click to edit Master title style

Monitoring IMS Connect

42

42

© 2013 IBM Corporation
IMS Connect
IMS Connect is the TCP/IP interface for IMS

– IMS Connect is delivered with IMS
– IMS Connect runs in a separ...
Submitting IMS transaction messages
over TCP/IP
IMS
TCP/IP Application

IMS
Connect

OTMA
XCF

IMS Connect and IMS communi...
One IMS Connect to Many IMSs
IMS
IMS
OTMA

TCP/IP
Application

IMS
Connect

OTMA
XCF

IMS
OTMA

IMS Connect can communicat...
Several IMS Connects to one IMS

TCP/IP
Application

IMS
IMS
Connect
IMS
Connect
IMS
Connect
Connect

IMS

Several IMS Con...
Using IMS Connect with Sysplex
Distributor
IMS
IMS
Connect
IMS
Connect
IMS
Connect
Connect
Web Browser

WebSphere
Applicat...
Monitoring TCP/IP Connections

48

48

© 2013 IBM Corporation
TCP/IP.PROFILE Definitions
PORT statement: IMS Connect PORT set to NODELAYACKS
– Allows ACKS to be sent immediately
– Defa...
Customizing Unix System Services
SYS1.PARMLIB (BPXPRMxx)
– MAXSOCKETS sets total number of active
sockets per stack
– MAXF...
Network Key Indicators to Watch
TCP/IP Active Connection
FTP long running jobs
VTAM Buffer Pools

51

Backlog Connection R...
Click to edit Master title style

Case Study: Database I/O Wait

52

52

© 2013 IBM Corporation
Database I/O Wait
PDEX
------SHORT TERM %---------LONG TERM %----+ (ELAPSED TIME= 1:41 MN) % 0_______ 50_______100
% 0____...
Database Status
------------------------------------------------------------ 08/28/12 14:44:25
Details for Database : PAY2...
Database Buffer Pool Status
------------------------------------------------------------ 08/28/12 14:47:27
OSAM Buffer Poo...
Volume Response Time
------------------------------------------------------------ 08/28/12 14:47:32
Device Statistics
I91A...
Dataset Response Time
Actions
Options
Index View
Help
---------------------------------------------- SMF ID: SYSA - 08/28/...
Who Has I/O?
>

Profile of the DASD Device

DEV 4E8 Volser=PPSMPN Online Alloc
===========================================...
Click to edit Master title style

Summary

59

59

© 2013 IBM Corporation
Steps to Managing Performance
Set up objectives
Monitor transaction response time
– Snap shot and trending

Pinpoint the b...
Correlate Metrics with z/OS, TCP, DB2, MQ, SMS
CPU – top CPU Consumers, looping Jobs, excessive CPU tasks,
GCP, zIIP
Memor...
Summary
Monitor and measure system routinely
Understand workloads and priorities
Obtain a global view of your entire syste...
References
IMS product documentation
IMS Webcasts by Deepak Kohli
– IMS regions simplified, clarified and demystified: PAR...
Thank You

64

64

© 2013 IBM Corporation
Upcoming SlideShare
Loading in …5
×

Monitoring IMS Performance for Faster Problem Solving v8 - IMS UG Philly November 2013

899 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
899
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Monitoring IMS Performance for Faster Problem Solving v8 - IMS UG Philly November 2013

  1. 1. Click to edit Master title style Monitoring IMS Performance for Faster Problem Solving Lih M. Wang IBM System z Client Technical Specialist Email: LihWang@us.ibm.com Philadelphia IMS User Group Meeting November 20, 2013 1 1 © 2013 IBM Corporation
  2. 2. Topics Introduction – IMS Overview – Performance Basics IMS Transaction Flow – Key Performance Indicators Case Study Summary 2 2 © 2013 IBM Corporation
  3. 3. Click to edit Master title style Introduction 3 3 © 2013 IBM Corporation
  4. 4. IMS TM and DB Overview D L I S A S C O N T R O L BMP D B R C I R L M BMP MP MP ACBLIB MFS MSG Queue OLDS WADS RDS RECON PGMLIB User DB KPI: CPU, Memory, I/O, Network, Locking 4 4 © 2013 IBM Corporation
  5. 5. IMS Java Transaction Support 5 5 © 2013 IBM Corporation
  6. 6. Emergence of online – All roads lead to IMS IMS IMS Connect IMS Connect IMS Connect IMS Connect Web Browser WebSphere Application Server Sysplex Distributor WebSphere MQ ATM Network 6 Distributed Systems 6 © 2013 IBM Corporation
  7. 7. Click to edit Master title style What to Monitor? 7 7 © 2013 IBM Corporation
  8. 8. Performance Basics CPU Memory I/O Network Locking Storage Applications 8 8 © 2013 IBM Corporation
  9. 9. IMS System Health At-a-Glance Proactively monitor IMS address spaces and health indicators Out-of-the Box graphics showing Paging/EXCP and CPU usage Alert notification and intelligent drill-down for root cause analysis 9 9 © 2013 IBM Corporation
  10. 10. Anatomy of an I/O Request Front-end Time IOSQ T1 Queue Device EXCP Issued 10 + PEND T2 Queue Path Back-end Time + CONN + DISC T3 Seek T4 Set Sector CCW Fetch + DISC T5 Latency (Set Sector) + DISC T6 RPS Reconnect + CONN + CONN T7 T8 Search ID Equal SSCH CC=0 Data Transfer I/O Complete 10 © 2013 IBM Corporation
  11. 11. Monitoring I/O Response Time I/O Response Time = IOSQ + PEND + CONN + DISCONNECT 11 11 © 2013 IBM Corporation
  12. 12. Monitoring TCP/IP Connections 12 12 © 2013 IBM Corporation
  13. 13. Global Locking – Top Blocker New area containing list of top blockers 13 13 © 2013 IBM Corporation
  14. 14. Click to edit Master title style IMS Transaction Flow and Key Performance Indicators 14 14 © 2013 IBM Corporation
  15. 15. IMS Transaction Flow Communications Control MP Scheduler Common Services Transaction Appl. Pgm. Log 15 DL/I Data Bases Message Queue Checkpoint/Restart Transaction arrival rate, Transaction stopped Database stopped, ACB stopped VSAM OSAM Buffer Pool, DB I/O IMS Logging datasets, Message Queue datasets, DBRC Locking conflicts 15 © 2013 IBM Corporation
  16. 16. IMS Communication Overview Communications Control MP Scheduler Common Services Transaction Appl. Pgm. DL/I Log Data Bases 16 Message Queue Checkpoint/Restart 16 © 2013 IBM Corporation
  17. 17. Communication Pools ECSA VTAM NODE IOBUFs SAPs IMS CONTROL REGION VTAM CIOP HIOP RECANY OUTBUF SPAP MFS/FRE Format MSG QUEUE POOL Short MSG Q Long MSG Q SMBs SMBs 17 SAP – Save Area Prefix FRE – Fetch Request Element CIOP – Communication I/O Pool HIOP – High I/O Pool (outbound messages) SPAP – Scratch Pad Area Pool (for conversational messages) 17 © 2013 IBM Corporation
  18. 18. Selective Dispatching ------------------------------------------------------------ 10/27/12 9:09:45 IMS Status I91A +-----------------------------------------------------------------------------+ ¦ ¦ ¦ IMS restart date . : 05.297 IMS restart time . . . .: 08:48:11 ¦ ¦ ¦ ¦ Checkpoints taken . : 3 Current checkpoint id . : 05297/143735 ¦ ¦ ¦ ¦ MPPs active . . . . : 1 BMPs active . . . . . . : 0 ¦ ¦ ¦ ¦ Msg enqueue rate . : .00/sec Msg dequeue rate . . . .: .00/sec ¦ ¦ ¦ ¦ Transactions queued : 778 Selective dispatch . . .: Inactive ¦ ¦ ¦ ¦ VTAM authorize Path : Active VTAM ACB . . . . . . . .: Open ¦ ¦ ¦ ¦ START DC . . . . . .: Performed Logtape write-ahead . . : Active ¦ ¦ ¦ +-----------------------------------------------------------------------------+ SAP CIOP MFS/FRE MSG Queue ***** ITASK must wait when the resource shortage is detected 18 18 SPAP © 2013 IBM Corporation
  19. 19. Storage Pool Information 19 19 © 2013 IBM Corporation
  20. 20. IMS Scheduler Overview Communications Control MP Scheduler Common Services Transaction Appl. Pgm. DL/I Log Data Bases 20 Message Queue Checkpoint/Restart 20 © 2013 IBM Corporation
  21. 21. Scheduling Facility Priority Class SCHEDULER Load Balancing Message Queue Pool 21 Processing Limit Count Scheduler Related Pools Wait for Input Scheduler Related Control Blocks 21 © 2013 IBM Corporation
  22. 22. Scheduling Waits CONTROL REGION BLDL LIST PSB WORK AREA POOL PSB POOL PSB A PSB B PSB C RESIDENT PSB’s ACBLIB INTENT LISTS DMB X DMB Y RESIDENT DMB’s DMB POOL 22 ***** If DMB space is unavailable, the least recently referenced DMB in the pool is flushed, all of its data sets have to be CLOSED - OPEN / CLOSE overhead, performance degradation. ***** If PSB pool space is unavailable, the least recently referenced PSB is removed, next time that PSB must be reloaded from ACBLIB when requested. © 2013 IBM Corporation 22
  23. 23. PSB Status - Clean Up The Unusable 23 23 © 2013 IBM Corporation
  24. 24. Schedule End to First DL/I Call Time before the first DL/I call to retrieve a message – Loading the application program – Initializing the program – Opening any non-DL/I files – Issuing any database calls 24 24 © 2013 IBM Corporation
  25. 25. Monitor Non-DL/I Wait PDEX ------SHORT TERM %---------LONG TERM %----+ (ELAPSED TIME= 1:41 MN) % 0_______ 50_______100 % 0_______ 50_______100 + USING CPU: 21.0|--->. . . .| 20.5|--->. . . . + USING CPU IN APPL (11.2)|--> . . . .| (11.0)|->- . . . . + USING CPU IN IMS (9.8)|--> . . . .| (9.5)|-> . . . . + SCHEDULING WAITS: 2.8|> . . . .| 2.0|> . . . . + WAIT FOR MPP (0.2)|> . . . .| (0)|> . . . . + WAIT FOR GU (2.6)|> . . . .| (2.0)|> . . . . + IMS ACTIVITY: 16.5|--> . . . .| 15.5|--> . . . . + SPA I/O (1.2)|> . . . .| (1.2)|> . . . . + SYNC POINT WAIT (5.6)|> . . . .| (5.6)|> . . . . + ISWITCHED TO CTL (9.7)|-> . . . .| (8.7)|-> . . . . + DATABASE I/O WAITS 26.7|-----> . . .| 21.7|----> . . . + ADA221 (15.4)|--> . . . .| (11.4)|--> . . . . + ADA021S (11.3)|-> . . . .| (10.3)|-> . . . . + MVS WAITS: 35.8|-------> . . .| 30.8|------> . . . + CPU WAIT (MPP/BMP) (13.7)|--> . . . .| (10.7)|--> . . . . + PROGRAM FETCH I/O (22.1)|---> . . .| (20.1)|---> . . . + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Avg. Trans Executing: 5.3 4.0 25 25 © 2013 IBM Corporation
  26. 26. IMS Message Queuing Communications Control MP Scheduler Transaction Common Services Appl. Pgm. Log DL/I Data Bases 26 Message Queue Checkpoint/Restart 26 © 2013 IBM Corporation
  27. 27. Input/Output Messages CONTROL REGION QPOOL MESSAGE SMB QUEUE BLOCK MESSAGE MESSAGE MESSAGE CNT SHORT MESSAGE LONG MESSAGE MESSAGE MESSAGE SMB – Scheduler Message Block built at SYSGEN, a list of all valid transaction codes – TRANSACT Macro CNT – Communication Name Table, built at SYSGEN, a list of all valid logical terminal names – NAME macro CVB – Command and Verb Block, a list of all valid IMS/VS 27 27 © 2013 IBM Corporation
  28. 28. Message Queue Considerations Allocate sufficient QBUFs Optimize LRECL sizes to make efficient use of pool space and balance I/O activity between the SHMSG and LGMSG datasets If large SPAs are defined, allocate the size of the LGMSG LRECL to include the size of the SPA plus the length of the input message Use fixed length SPAs 28 28 © 2013 IBM Corporation
  29. 29. Message Queue I/O ------------------------------------------------------------ 08/28/11 14:39:10 Message Queue Buffers and Datasets I91A +-----------------------------------------------------------------------------+ ¦ Message Queue ¦ Total ¦ Short ¦ Long ¦ QBLKS ¦ Free ¦ ¦---------------------+----------+----------+----------+----------+-----------¦ ¦ No. of Buffers ¦ 5 ¦ 2 ¦ 2 ¦ 1 ¦ 0 ¦ ¦ Percentage ¦ ¦ 40.0% ¦ 40.0% ¦ 20.0% ¦ .0 ¦ +-----------------------------------------------------------------------------+ +-----------------------------------------------------------------------------+ ¦ Dataset ¦ % Used ¦ Shutdown ¦ I/O Rate ¦ ¦------------+---------------------------+-------------+----------------------¦ ¦ Short ¦ 14.36%, 968 of 6740 ¦ 98.53% ¦ .00 per second ¦ ¦ Long ¦ .96%, 6 of 620 ¦ 86.11% ¦ .00 per second ¦ ¦ QBLKS ¦ .24%, 32 of 13100 ¦ 99.24% ¦ .00 per second ¦ +-----------------------------------------------------------------------------+ 29 29 © 2013 IBM Corporation
  30. 30. If MSG-Q BUF# Became Low... Enough active MP regions? Scheduling Delayed because of short on PSB, PSBW, or EPCB pools? Remote printers are active? Increase QBUF= in control region JCL If region is stopped but messages are still coming, stop transactions. Otherwise the % utilization of short message dataset will increase. If the % used of long message dataset increases, an application may be looping while inserting output messages, stop the region (especially if it’s BMP). 30 30 © 2013 IBM Corporation
  31. 31. DL/I Interface Communications Control MP Scheduler Common Services Transaction Appl. Pgm. Log DL/I Data Bases 31 Message Queue Checkpoint/Restart 31 © 2013 IBM Corporation
  32. 32. Online Database Operations Interregion Communication 32 ENQ/DEQ Pool DL/I Buffer Handler Pool Program Isolation OSAM Buffer Pool OSAM/VSAM Buffer Handlers Data Bases DL/I Call Analyzer DL/I Action Modules WADS Database Work Area Pool VSAM Shared Resource Pool 32 © 2013 IBM Corporation
  33. 33. VSAM Hiperspace DLISAS B U F F E R 16 MEG 33 P R E F I X E S Buffers VSAM Hiperspace Buffers Hiperspace Buffers Buffers Hiperspace Buffers 33 © 2013 IBM Corporation
  34. 34. VSAM Buffer Pool ------------------------------------------------------------ 08/28/12 14:48:08 VSAM Buffer Pools Status I91A Lines 1 to 11 of 11 +-----------------------------------------------------------------------------+ ¦ Subpool ¦ Number of ¦ Size of ¦ Hit ¦ Buffers ¦ Blocks ¦ ¦ ¦ Buffers ¦ Buffers ¦ Ratio ¦ Page-Fixed? ¦ Page-Fixed? ¦ ¦------------+------------+-----------+-----------+-------------+-------------¦ ¦ 1 ¦ 5 ¦ 512 ¦ .0% ¦ YES ¦ YES ¦ ¦ 2 ¦ 15 ¦ 1024 ¦ 82.0% ¦ YES ¦ YES ¦ ¦ 3 ¦ 20 ¦ 2048 ¦ 90.0% ¦ YES ¦ YES ¦ ¦ 4 ¦ 5 ¦ 4096 ¦ .0% ¦ YES ¦ YES ¦ ¦ 5 ¦ 55 ¦ 4096 ¦ 70.0% ¦ YES ¦ NO ¦ ¦ 6 ¦ 5 ¦ 4096 ¦ .0% ¦ YES ¦ NO ¦ ¦ 7 ¦ 3 ¦ 32768 ¦ .0% ¦ YES ¦ YES ¦ ¦ 8 ¦ 5 ¦ 2048 ¦ .0% ¦ YES ¦ YES ¦ ¦ 9 ¦ 5 ¦ 512 ¦ .0% ¦ YES ¦ YES ¦ ¦ 10 ¦ 5 ¦ 1024 ¦ .0% ¦ YES ¦ YES ¦ ¦ 11 ¦ 5 ¦ 1024 ¦ .0% ¦ YES ¦ YES ¦ +-----------------------------------------------------------------------------+ Optimal Index Hit Ratio > = 90% Data Hit Ratio > = 70% 34 34 © 2013 IBM Corporation
  35. 35. OSAM Buffer Pool 4 Gig limit Page fix only the buffer Sequential Buffering DFSVSMxx member in IMSVS.PROCLIB – IOBF=(4096,50,N,Y) – IOBF=(6144,8) IOBF=(length, number, fix1, fix2) Fix1 specifies the buffers long-term-page-fixing option Y – all buffers and buffer prefixes associated with this subpool will be longterm-page-fixed at initializaiton of the subpool. N – no buffer associated with this subpool will be long-term-page-fixed at init. Of the subpool. Fix2 specifies the buffer-prefix long-term-pagepfixing option. 35 35 © 2013 IBM Corporation
  36. 36. IMS Checkpoint Processing Communications Control MP Scheduler Common Services Transaction Appl. Pgm. Log 36 DL/I Data Bases Message Queue Checkpoint/Restart 36 © 2013 IBM Corporation
  37. 37. Common Logging Facility z/OS IMS/TM DBRC DLISAS Control Region DB Recovery Control DL/I Subordinate Address Space WADS Buffers Buffers RECON 1 OLDS1 OLDS2 WRITE AHEAD DATA SETS RECON 2 OLDS3 OLDS4 RECON 3 OLDS5 LOG ARCHIVE UTILITY SYSTEM LOG DATA SETS SLDS1 SLDS2 37 RLDS1 RLDS2 RECOVERY LOG DATA SETS 37 © 2013 IBM Corporation
  38. 38. Logging I/O ------------------------------------------------------------ 08/28/12 Logging Buffers and Statistics I91A 7:15:47 +-----------------------------------------------------------------------------+ ¦ Size : 22 K Number of Buffers : 25 Number Available : 4 ¦ +-----------------------------------------------------------------------------+ +----------------------------------------------------------+ ¦ I/O Type ¦ Rate/sec ¦0..................¦ ¦---------------------------+----------+-------------------¦ ¦ OLDS writes. . : 225 ¦ .00 ¦ ¦ ¦ OLDS reads . . : 0 ¦ .00 ¦ ¦ ¦ WADS 2K writes : 1058 ¦ .00 ¦ ¦ ¦ WADS EXCPVRS . : 234 ¦ .00 ¦ ¦ +----------------------------------------------------------+ BMP Abending? ***** If WADS I/O rate is too high, monitor the dataset response time, and re-evaluate OLDs buffer size. 38 38 © 2013 IBM Corporation
  39. 39. Monitor IMS System Datasets 39 39 © 2013 IBM Corporation
  40. 40. Cross Reference – Part 1 Reference: IMS Performance Guide Redbook, SG24-4637-00 EVENT Message Input ACTIVITY POOLs/LISTs TP CIOP RECANY MFP SPAP/CWAP QBUF LOG MFS SPA GET QUEUING DATASETs LINEs FORMAT MSGQs IMSLOG Scheduling QBUF PSBP DMBP LOG MSGQs ACBLIB ACBLIB IMSLOG Schedule End to First DL/I Call 40 Scheduling PSB Load DMB Load Program Load Initialization BLDL VLF/LLA PRELOAD STEPLIBs 40 © 2013 IBM Corporation
  41. 41. Cross Reference – Part 2 EVENT ACTIVITY POOLs/LISTs Program Elapsed DC Calls QBUF MSGQs DL/I Elapsed DB Calls OSAM VSAM LOG Databases Databases IMSLOG SPA Insert SPAP/CWAP LOG OSAM VSAM QBUF LOG IMSLOG QBUF MFP CIOP/HIOP LOG MSGQs FORMAT LINEs IMSLOG IWAIT Elapsed Sync Point Termination Message Out 41 Dequeuing MFS TP TP DATASETs Databases Databases MSGQs IMSLOG 41 © 2013 IBM Corporation
  42. 42. Click to edit Master title style Monitoring IMS Connect 42 42 © 2013 IBM Corporation
  43. 43. IMS Connect IMS Connect is the TCP/IP interface for IMS – IMS Connect is delivered with IMS – IMS Connect runs in a separate address space – IMS Connect is a TCP/IP Server – IMS Connect is used for: • Submitting IMS transaction messages over TCP/IP • Accessing IMS data (IMS Open Database) over TCP/IP • MSC, ISC over TCP/IP • Application Remote Messaging 43 43 © 2013 IBM Corporation
  44. 44. Submitting IMS transaction messages over TCP/IP IMS TCP/IP Application IMS Connect OTMA XCF IMS Connect and IMS communicate using XCF (cross-system coupling facility) i.e IMS Connect and IMS do not have to be on the same LPAR IMS Connect and IMS interface thru OTMA (Open Transaction Manager Access IMS Connect provides exit routines for transaction message formatting 44 44 © 2013 IBM Corporation
  45. 45. One IMS Connect to Many IMSs IMS IMS OTMA TCP/IP Application IMS Connect OTMA XCF IMS OTMA IMS Connect can communicate with multiple IMSs IMS Connect could do load balancing across the IMSs IMS Connect can know the status of the IMSs So if an IMS is unavailable, IMS connect could re-direct the transaction to another IMS 45 45 © 2013 IBM Corporation
  46. 46. Several IMS Connects to one IMS TCP/IP Application IMS IMS Connect IMS Connect IMS Connect Connect IMS Several IMS Connects could communicate with one IMS Might want to do that for availability 46 46 © 2013 IBM Corporation
  47. 47. Using IMS Connect with Sysplex Distributor IMS IMS Connect IMS Connect IMS Connect Connect Web Browser WebSphere Application Server IMS Sysplex Distributor Can front end IMS connect with Sysplex Distributor (A software function in z/OS that increases availability through a combination of dynamic VIPA and the z/OS Workload Manager). 47 47 © 2013 IBM Corporation
  48. 48. Monitoring TCP/IP Connections 48 48 © 2013 IBM Corporation
  49. 49. TCP/IP.PROFILE Definitions PORT statement: IMS Connect PORT set to NODELAYACKS – Allows ACKS to be sent immediately – Default is to wait for up to 200ms before ACKing each transmission SHAREPORT – Allows IMS Connect PORTS to be shared by multiple IMS Connect instances on the same TCP/IP stack TCPCONFIG INTERVAL or KEEPALIVEOPTIONS INTERVAL – Allows TCP/IP to maintain a connection that can be inactive for long periods of time SOMAXCONN – Specifies the max. num of connection requests queued for any listening socket. Default is 10. – Must be defined large enough for maximum concurrent IMS Connect requests 49 49 © 2013 IBM Corporation
  50. 50. Customizing Unix System Services SYS1.PARMLIB (BPXPRMxx) – MAXSOCKETS sets total number of active sockets per stack – MAXFILEPROC sets total number of file (socket) descriptors for process IMS Connect – HWSCFGxx Option – MAXSOC sets total number of concurrent sessions for this IMS Connect. – One connection per port is reserved as a listener connection. Default is 50 50 50 © 2013 IBM Corporation
  51. 51. Network Key Indicators to Watch TCP/IP Active Connection FTP long running jobs VTAM Buffer Pools 51 Backlog Connection Rejected Packet Retransmission Rate OSA Adapter Performance 51 © 2013 IBM Corporation
  52. 52. Click to edit Master title style Case Study: Database I/O Wait 52 52 © 2013 IBM Corporation
  53. 53. Database I/O Wait PDEX ------SHORT TERM %---------LONG TERM %----+ (ELAPSED TIME= 1:41 MN) % 0_______ 50_______100 % 0_______ 50_______100 + USING CPU: 21.0|--->. . . .| 20.5|--->. . . . + USING CPU IN APPL (11.2)|--> . . . .| (11.0)|--> . . . . + USING CPU IN IMS (9.8)|--> . . . .| (9.5)|--> . . . . + SCHEDULING WAITS: 2.8|> . . . .| 2.5|> . . . . + WAIT FOR MPP (0)|> . . . .| (0.2)|> . . . . + WAIT FOR GU (2.8)|> . . . .| (2.3)|> . . . . + IMS ACTIVITY: 16.5|--> . . . .| 12.5|--> . . . . + SPA I/O (1.2)|> . . . .| (1.0)|> . . . . + SYNC POINT WAIT (5.6)|> . . . .| (5.2)|> . . . . + ISWITCHED TO CTL (9.7)|-> . . . .| (6.3)|--->. . . . + DATABASE I/O WAITS 35.7|-------> . . .| 26.7|-----> . . . + PAY221 (25.4)|-----> . . .| (15.4)|--> . . . . + ADA021S (10.3)|-> . . . .| (11.3)|-> . . . . + MVS WAITS: 25.8|------> . . .| 22.7|---> . . . . + CPU WAIT (MPP/BMP) (13.7)|--> . . . .| (12.6)|--> . . . . . + PROGRAM FETCH I/O (12.1)|-> . . . .| (10.1)|-> . . . . + ---+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + Avg. Trans Executing: 5.3 3.5 53 53 © 2013 IBM Corporation
  54. 54. Database Status ------------------------------------------------------------ 08/28/12 14:44:25 Details for Database : PAY221 I91A +-----------------------------------------------------------------------------+ ¦ ¦ ¦ ¦ Status . . . . . : Available ¦ Block size . . . . : 1024 ¦ ¦ ¦ ¦ ¦ Residency Status : Resident ¦ Buffer Subpool . . : Default ¦ ¦ ¦ ¦ ¦ Recoverability . : Recoverable ¦ Type . . . . . . . : OSAM ¦ ¦ ¦ ¦ +-----------------------------------------------------------------------------+ Lines 1 to 1 of 1 +-------------------------------------+ ¦ Volser ¦ Unit ¦ ¦------------------+------------------¦ ¦ PPSMPN ¦ 4E8 ¦ +-------------------------------------+ 54 54 © 2013 IBM Corporation
  55. 55. Database Buffer Pool Status ------------------------------------------------------------ 08/28/12 14:47:27 OSAM Buffer Pools Status I91A Lines 1 to 3 of 3 +-----------------------------------------------------------------------------+ ¦ Subpool ¦ Subpool ¦ Number of ¦ Hit ¦ Size of ¦ Buffers ¦ Prefix ¦ ¦ ¦ Id ¦ Buffers ¦ Ratio ¦ Buffers ¦ Page-Fixed? ¦ Page-Fixed? ¦ ¦---------+---------+-----------+-------+---------+-------------+-------------¦ ¦ 1 ¦ none ¦ 64 ¦ 62.5%¦ 1024 ¦ NO ¦ NO ¦ ¦ 2 ¦ none ¦ 32 ¦ 62.0%¦ 2048 ¦ NO ¦ NO ¦ ¦ 3 ¦ none ¦ 16 ¦ 41.3%¦ 4096 ¦ NO ¦ NO ¦ +-----------------------------------------------------------------------------+ 55 55 © 2013 IBM Corporation
  56. 56. Volume Response Time ------------------------------------------------------------ 08/28/12 14:47:32 Device Statistics I91A Lines 1 to 7 of 7 +-----------------------------------------------------------------------------+ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦I/Os per sec¦ ¦ ¦ ¦ Volser¦ Unit ¦Response¦ IOSQ ¦ Pend ¦ Conn ¦ Disc ¦------------+Queue ¦% ¦ ¦ ¦ ¦ Time ¦ Time ¦ Time ¦ Time ¦ Time ¦Total¦ IMS ¦Length¦Busy ¦ ¦-------+------+--------+------+------+------+------+-----+------+------+-----¦ ¦ PPSMPE¦ 14E ¦ 7.5 ¦ .0¦ 3.2 ¦ 2.6¦ 1.6¦ .7¦ .0 ¦ .00 ¦ .7¦ ¦ CAN001¦ 2A4 ¦ 7.2 ¦ .0¦ 1.5 ¦ 2.1¦ 3.5¦ .5¦ .0 ¦ .00 ¦ 2.5¦ ¦ CAN009¦ 2B9 ¦ 6.8 ¦ .0¦ 1.0 ¦ 2.7¦ 2.9¦ .4¦ .0 ¦ .00 ¦ .4¦ ¦ PPSMPF¦ 4E1 ¦ 4.8 ¦ .0¦ 2.6 ¦ 1.5¦ .6¦ 30.9¦ .0 ¦ .00 ¦ 3.4¦ ¦ PPSMPH¦ 4E5 ¦ 15.0 ¦ .0¦ 3.5 ¦ 1.1¦ 10.2¦ .0¦ .0 ¦ .00 ¦ .0¦ ¦ PPSMPI¦ 4E7 ¦ 7.6 ¦ .0¦ .6 ¦ 3.0¦ 4.0¦ .6¦ .0 ¦ .00 ¦ .2¦ ¦ PPSMPN¦ 4E8 ¦ 54.7 ¦ 34.3¦ 1.3 ¦ 14.5¦ 4.6¦ 7.8¦ .0 ¦ .02 ¦ 11.5¦ +-----------------------------------------------------------------------------+ 56 56 © 2013 IBM Corporation
  57. 57. Dataset Response Time Actions Options Index View Help ---------------------------------------------- SMF ID: SYSA - 08/28/12 14:47:36 KDFS0450 Dataset Details Status: Dataset . . . . PRODIMS.PAYROLL.DATA S=Show details +------------------------------------------------------------------------------+ ¦Volser . . . : PPSMPN Device type . . : 3390 Volume MSR . . : 54.7 ¦ ¦Data class . . Creation date . : 05/09/09 ¦ ¦Mgmt class . . NOACTION Last reference : 08/28/12 ¦ ¦Storage class . DATABASE Expiration date : None ¦ ¦------------------------------------------------------------------------------¦ ¦DSORG : VS RECFM . : U ¦ Response Time Data ¦ Cache Statistics ¦ ¦LRECL : 4089 BLKSIZE : 4096 ¦----------------------+---------------------¦ ¦Key length . : 0 ¦I/O Count . : 3 ¦Eligible % . : 100.0 ¦ ¦Key position : 0 ¦I/O per sec : 0.0 ¦Inhibit % . : 0.0 ¦ ¦ ¦Avg. IOSQ . : 33.3 ¦Bypass % . . : 0.0 ¦ ¦Alloc . : 13 Used : 13 ¦Pending . . : 0.3 ¦Total hit % : 50.0 ¦ ¦Volumes : 1 % Used : 100.0 ¦Disconnect . : 12.5 ¦Read hit % . : 50.0 ¦ ¦Secondary value : 13 ¦Connect . . : 1.7 ¦Write hit % : 0.0 ¦ ¦Secondary unit : Track ¦I/O service : 14.6 ¦Read % . . . : 100.0 ¦ ¦No. of extents . 1 ¦Total MSR . : 47.9 ¦Seq'l I/O % : 0.0 ¦ +------------------------------------------------------------------------------+ 57 57 © 2013 IBM Corporation
  58. 58. Who Has I/O? > Profile of the DASD Device DEV 4E8 Volser=PPSMPN Online Alloc ============================================================================== dplt01 ________ OMEGAMON Peek at Unit=4E8 Volser=PPSMPN _______ + |DBsy DDDDD DDDDDDDDDDDDDDDDD DDDDDDDDDDDDDDDD DDDDDDDD| Samples: 50 + |IOQ | Interval: 2 + |I/O# >27 >28 >29 | I/O's: 3 + |CPU ..............................................| + |User >D81ADBM1 | + |Cyl >217 | + |Nrdy | + |Resv | ============================================================================== What if DB2 data was placed on the same pack with an IMS database? 58 58 © 2013 IBM Corporation
  59. 59. Click to edit Master title style Summary 59 59 © 2013 IBM Corporation
  60. 60. Steps to Managing Performance Set up objectives Monitor transaction response time – Snap shot and trending Pinpoint the bottleneck – Message flooding – Locking contention – Database I/O – Logging and DBRC – Network congestion Correct the problem 60 60 © 2013 IBM Corporation
  61. 61. Correlate Metrics with z/OS, TCP, DB2, MQ, SMS CPU – top CPU Consumers, looping Jobs, excessive CPU tasks, GCP, zIIP Memory / Storage – ECSA/CSA storage leaks I/O – high volume response time, IOSQ, pending, disconnect, connect Times WLM - Performance Index >1 Sysplex - Coupling Facility Unix System Services – processes, files I/O, threads connection pooling for distributed systems FTP Jobs – long holding a dataset Batch Jobs - delay, long waits, back-out issues DB2 - lockouts TCP/IP - timeouts, backlog limit, backlog connections rejected …and monitor third party gateway / connectors 61 61 © 2013 IBM Corporation
  62. 62. Summary Monitor and measure system routinely Understand workloads and priorities Obtain a global view of your entire system 62 62 © 2013 IBM Corporation
  63. 63. References IMS product documentation IMS Webcasts by Deepak Kohli – IMS regions simplified, clarified and demystified: PART 1 & PART 2 (Aug 27 & Oct 15, 2013) – Registration URL: http://ibm.co/16WNtBE IMS Performance Monitoring and Tuning Update Redbook, SG24-6404-00 IMS Performance Guide Redbook, SG24-4637-00 IMS Slideshare and SHARE conference presentations IBM OMEGAMON XE for IMS User’s Guide www.ibm.com/ims 63 63 © 2013 IBM Corporation
  64. 64. Thank You 64 64 © 2013 IBM Corporation

×