IBM Impact session Ed addison nuts and bolts ws

© 2014 IBM Corporation
ACI-1787 - Debugging CICS
Web Services in a Mobile Age
Ed Addison – STSM L2 Support

Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
2

CICS TS Feature Pack for Mobile Extensions V1.0
A fully supported addition to base CICS installed using SMP/E
Separately orderable from Shopz (PID 5655-Y48)
Common to CICS TS 4.2 and 5.1
Requires enabling APARs PM82511 (4.2) and PM82519 (5.1)
Most of the functionality is actually provided by the APARs to
allow code reuse
3

CICS TS Feature Pack for Mobile Extensions V1.0 Capabilities
Expose existing CICS applications as JSON web services
(bottom-up approach)
Create new RESTful interfaces to CICS applications (top-down
approach)
Call existing JSON web services from your CICS applications
Create Java applications in Liberty applications using JAX-RS
4

Feature Pack Contents
USS - <pathprefix>/usr/lpp/cicsts/mobilefp
• Shell scripts for DFHJS2LS and DFHLS2JS
• json_schema
– JSON schema for IVP
• Pipelines
– Sample pipeline conifg file
• Wsbind
– WSBIND file for IVP
Datasets
• SDFHMOBI - DFHJS2LS and DFHLS2JS JCL procedures
• SDFHMOBL – IVP load module
• SDFHMOBS – Source, language structure and CSD defintions
for IVP
5

What is JSON
Javascript Object Notation
A lightweight, text-based data interchange format
• Typically half the size of an XML payload
Natively supported in Javascript
Commonly used in mobile applications
6

Producing JSON in CICS
Support for JSON web services is based closely on CICS support for
SOAP web services so the development and deployment steps are
very similar.
Decide whether to start with a language structure (bottom-up
approach) or from a JSON schema (top-down approach)
Run the JSON assistant batch job, supplying the appropriate artifacts.
This generates a WSBIND file.
Define and install a PIPELINE resource in CICS with an appropriate
pipeline configuration file for JSON.
Copy the WSBIND file to the PIPELINE's web services directory, and
perform a PIPELINE SCAN to generate the WEBSERVICE and
URIMAP resources.
Your JSON web service is ready to process requests.
7

Differences Between JSON and SOAP
The content of a SOAP message is XML data, whereas a JSON message contains JSON data.
JSON and XML are different encoding mechanisms for describing structured data. JSON tends to be
a more efficient encoding mechanism, so a typical JSON message will be smaller than the
equivalent XML message.
JSON is easy to integrate in JavaScript applications, but XML isn't. This makes JSON a preferred
data format with many mobile application developers.
SOAP provides a mechanism to add Headers to a message, and a family of specifications for
qualities of service (such as security configuration, and distributed transactions). JSON does not
provide this mechanism, it instead relies on the services of the underlying HTTP network protocol.
This results in fewer options for securing and configuring a workload.
SOAP web services have an explicit error format involving SOAP Fault messages. There's no
equivalent for JSON.
8

Similarities Between JSON and SOAP
The CICS implementation of JSON is derived from the SOAP architecture, and shares many of the
concepts and artifacts.
Both involve offline utility programs that assist with mapping application data to and from the external
data representation. For SOAP there is DFHLS2WS and DFHWS2LS, for JSON there is DFHLS2JS
and DFHJS2LS.
The deployment mechanism for both technologies involve a PIPELINE resource, a WEBSERVICE
resource, and a URIMAP resource.
9

10
Problem One: Disabled Pipeline

11
Explorer View of Disabled Pipeline DFH0MOBI

Messages Received
ICH408I USER(USASSC9 ) GROUP(TSOUSER ) NAME(ADDISON, EDWARD A. ()
/u/ephoppe/IYNX43/PIPELINE/DFH0MOBI/jsonjavaprovider.xml
CL(DIRSRCH ) FID(00000001000000010000000000000000)
INSUFFICIENT AUTHORITY
DFHPI0705 E IYNX43 CICSUSER PIPELINE DFH0MOBI encountered an error writing the
configuration to the derived shelf /u/ephoppe/IYNX43/PIPELINE/DFH0MOBI/.
The response code from the HFS write was X'0000006F' and the reason code was
X'EF786015'.
DFHPI0709 E IYNX43 CICSUSER PIPELINE DFH0MOBI resolution failed because the XML configuration file cannot be
copied to the derived shelf.
12

Explorer view of Pipeline Definition
13
Note: Shelf Directory called out in message DFHPI0705 is /u/ephoppe

Change Pipeline Shelf Directory
16

Problem Two: DFHPI0735 Web Service Not Found
18

CEDA Install of DFH0MOBI Pipeline and URIMAP
ENTER COMMANDS
NAME TYPE GROUP LAST CHANGE
DFH0MOBI PROGRAM DFH0MOBI 16/08/13 21:02:54
DFH0MOBI URIMAP DFH0MOBI *n INSTALL SUCCESSFUL
DFH0MOBI PIPELINE DFH0MOBI *n INSTALL SUCCESSFUL
DFH0MOBI WEBSERVICE DFH0MOBI 16/08/13 21:57:54
SYSID=NX43 APPLID=IYNX43
RESULTS: 1 TO 4 OF 4 TIME: 16.24.41 DATE: 10/04/14
PF 1 HELP 2 SIG 3 END 4 TOP 5 BOT 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL
19

Messages During dfh0mobi Web Service Creation
DFHRD0124 IYNX43 IYCNTC49 CICSUSER CEDA INSTALL PIPELINE(DFH0MOBI)
DFHPI0703 IYNX43 CICSUSER PIPELINE DFH0MOBI is about to scan the WSDIR directory.
DFHPI0901 IYNX43 CICSUSER New WEBSERVICE dfh0mobi is being created during a scan against PIPELINE
DFH0MOBI.
DFHPI0910 IYNX43 CICSUSER WEBSERVICE dfh0mobi within PIPELINE DFH0MOBI has been created.
DFHPI0915 IYNX43 CICSUSER WEBSERVICE dfh0mobi is now INSERVICE and is ready for use.
DFHPI0704 IYNX43 CICSUSER PIPELINE DFH0MOBI Implicit scan has completed. Number of wsbind files found in the
WSDIR directory: 000001. Number of successful WEBSERVICE creates: 000001.
Number of failed WEBSERVICE creates: 000000.
20

Explorer Display of Web Service
21

Attempt to run Mobile Feature Pack IVP
22

Message from MSGUSR
DFHPI0735 E IYNX43 Error 'com.ibm.cics.domains.DomainException:
Class: Dfhpiwrj, function: INQUIRE_WEBSERVICE,
response: EXCEPTION, reason: WEBSERVICE_NOT_FOUND'
occurred whilst configuring WEBSERVICE DFH0MOBI
within a JVMSERVER.
23

DFHPI0735 Web Service not found – What we know so far
Client downloaded Mobile Feature Pack
Client CEDA installed PIPELINE and URIMAP DFH0MOBI
Pipeline scan built Web Service dfh0mobi
CICS installed Web Service dfh0mobi and put it in-service
Attempt to run the Mobile IVP resulted in DFHPI0735 WEBSERVICE _NOT_FOUND
24

CEMT Inquire of Web Service dfh0mobi
I WEBSERVICE
STATUS: RESULTS - OVERTYPE TO MODIFY
Webs(dfh0mobi ) Pip(DFH0MOBI)
Ins Pro(DFH0MOBI)
RESPONSE: NORMAL TIME: 22.16.58 DATE: 02/04/14
PF 1 HELP 3 END 5 VAR 7 SBH 8 SFH 9 MSG 10 SB 11 SF
25

CEDA view PIPELINE for DFH0MOBI
26
OBJECT CHARACTERISTICS CICS RELEASE = 0680
CEDA View PIpeline( DFH0MOBI )
PIpeline : DFH0MOBI
Group : DFH0MOBI
DEScription : IVP PIPELINE FOR MOBILE FEATURE PACK
STatus : Enabled Enabled | Disabled
Respwait : Deft Default | 0-9999
COnfigfile : /u/usassc9/ephoppe/jsonjavaprovider.xml
(Mixed Case) :
:
:
:
SHelf : /u/usassc9
(Mixed Case) :
:
:
:
Wsdir : /u/usassc9/mobilewsbind
(Mixed Case) :

WSBind file in OMVS
NOTE: This will create a lowercase dfh0mobi Web Service when the Pipeline
Scan is completed. This is due to the name of the bind file in z/FS.
27

CEDA View URIMAP for DFH0MOBI
OBJECT CHARACTERISTICS CICS RELEASE = 0680
CEDA View Urimap( DFH0MOBI )
Urimap : DFH0MOBI
Group : DFH0MOBI
DEScription : IVP URL FOR MOBILE FEATURE PACK
STatus : Enabled Enabled | Disabled
USAge : Pipeline Server | Client | Pipeline | Atom
| Jvmserver
UNIVERSAL RESOURCE IDENTIFIER
SCheme : HTTP HTTP | HTTPS
POrt : No No | 1-65535
HOST : *
(Mixed Case) :
PAth : /DFH0MOBI
(Mixed Case) :
:
ASSOCIATED CICS RESOURCES
TCpipservice :
ANalyzer : No No | Yes
COnverter :
TRansaction : CPIH
PRogram :
PIpeline : DFH0MOBI
Webservice : DFH0MOBI (Mixed Case)
ATomservice :
28

DFHPI0735 Web Service not found – What we know so far
Reviewing the URIMAP for DFHM0BI it has the Web Service coded as DFH0MOBI (Upper Case)
Reason for lower case dfh0mobi was due to DFH0MOBI PIPELINE SCAN finding dfh0mobi.wsbind
in wsdir /u/usassc9/mobilewsbind
29

CEDA Install of DFH0MOBI WEBSERVICE
ENTER COMMANDS
NAME TYPE GROUP LAST CHANGE
DFH0MOBI PROGRAM DFH0MOBI 16/08/13 21:02:54
DFH0MOBI URIMAP DFH0MOBI 16/08/13 21:02:54
DFH0MOBI PIPELINE DFH0MOBI 31/03/14 20:21:30
DFH0MOBI WEBSERVICE DFH0MOBI *n INSTALL SUCCESSFUL
RESULTS: 1 TO 4 OF 4 TIME: 16.24.41 DATE: 10/04/14
PF 1 HELP 2 SIG 3 END 4 TOP 5 BOT 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL
30

Explorer Display of Web Services
31

Successful Execution of DFH0MOBI
32

Mobile Feature Pack run from Android Phone
33

DFHPI0735 Web Service not found – What we know
Reviewing the URIMAP for DFHM0BI it has the Web Service coded as DFH0MOBI (Upper Case)
Reason for lower case dfh0mobi was due to DFH0MOBI PIPELINE SCAN finding dfh0mobi.wsbind
in wsdir /u/usassc9/mobilewsbind
Directions from Infocenter on installing Mobile Feature Pack:
• Install the sample application: Edit the sample PIPELINE DFH0MOBI in group DFH0MOBI. Ensure that
the path to the configuration file,
/pathprefix/usr/lpp/cicsts/mobilefp/samples/pipelines/jsonjavaprovider.xml, is correct for your installation.
• Edit the sample WEBSERVICE DFH0MOBI in group DFH0MOBI. Ensure that the path to the wsbind
file, /pathprefix/usr/lpp/cicsts/mobilefp/samples/wsbind/dfh0mobi.wsbind, is correct for your installation.
• Install group DFH0MOBI (PROGRAM, PIPELINE, WEBSERVICE, and URIMAP).
34

35
Problem Three: JVM Server Hang

JVM Server Hang
Customer called Support Center indicating access to their JVM
Server for Mobile Web Service stalled
36

IPCS Primary Menu
------------------- IPCS PRIMARY OPTION MENU ---------------------------------
OPTION ===>
0 DEFAULTS - Specify default dump and options
1 BROWSE - Browse dump data set
2 ANALYSIS - Analyze dump contents
3 UTILITY - Perform utility functions
4 INVENTORY - Inventory of problem data
5 SUBMIT - Submit problem analysis job to batch
6 COMMAND - Enter subcommand, CLIST or REXX exec
T TUTORIAL - Learn how to use the IPCS dialog
X EXIT - Terminate using log and list defaults
Enter END command to terminate IPCS dialog
F1=HELP F2=SPLIT F3=END F4=RETURN F5=RFIND F6=MORE F7=UP
F8=DOWN F9=SWAP F10=LEFT F11=RIGHT F12=CURSOR
Ed Addison 37

IPCS Default Menu
Ed Addison 38
------------------------- IPCS Default Values ---------------------------------
Command ===>
You may change any of the defaults listed below. The defaults shown before
any changes are LOCAL. Change scope to GLOBAL to display global defaults.
Scope ==> BOTH (LOCAL, GLOBAL, or BOTH)
If you change the Source default, IPCS will display the current default
Address Space for the new source and will ignore any data entered in
the Address Space field.
Source ==> DSNAME('USASSC1.JVMHANG.DUMP')
Address Space ==> ASID(X'0044')
Message Routing ==> NOPRINT TERMINAL
Message Control ==> FLAG(WARNING) NOCONFIRM VERIFY
Display Content ==> MACHINE REMARK REQUEST NOSTORAGE SYMBOL
Press ENTER to update defaults.
Use the END command to exit without an update.

IPCS Primary Menu
OPTION ===>
0 DEFAULTS - Specify default dump and options
1 BROWSE - Browse dump data set
2 ANALYSIS - Analyze dump contents
3 UTILITY - Perform utility functions
4 INVENTORY - Inventory of problem data
5 SUBMIT - Submit problem analysis job to batch
6 COMMAND - Enter subcommand, CLIST or REXX exec
T TUTORIAL - Learn how to use the IPCS dialog
Ed Addison 39

Problem Three - ST SYS
SYSTEM STATUS:
Nucleus member name: IEANUC01
Sysplex name: SYSPLEX1
TIME OF DAY CLOCK: CCFC281A C867F408 04/10/2014 15:57:46.700415 local
TIME OF DAY CLOCK: CCFC1AB1 8E27F408 04/10/2014 14:57:46.700415 GMT
Program Producing Dump: SVCDUMP
Program Requesting Dump: DFHKETCB
Incident token: SYSPLEX1 MV23 04/10/2014 14:57:45.943332 GMT
Ed Addison 41
41
Note: You want to know the exact time the dump was taken. You also want
to know the difference between CICS and GMT time.
Some CICS Verbx Outputs are in GMT time and some are in Local.

42
CICS Verbexit optionsKeyword Functional area
AI = 0|2 Autoinstall Model Manager
AP = 0|1|2|3 Application Domain
APS=<TASKID=nnnnn>
AU = 0|2 CICS affinities utility
BA = 0|1|2|3 Business application manager
BR = 0|1|2|3 3270 bridge
CC = 0|2 CICS catalog domain
CP = 0|2 Common Programming Interface
CQ = 0|1|2 Auto install model manager
CSA=0|2 CICS Common System Area
DB2=0|1|2|3 The CICS DB2 interface
DD = 0|1|2|3 Directory Domain
DH = 0|1|2|3 Document handling domain
DLI = 0|2 CICS DL/I Interface
DM = 0|1|2|3 Domain Manager
DP = 0|1|2|3 Debug Profiles manager
DS = 0|1|2|3 Dispatcher Domain
DU = 0|2 Dump Domain
EC = 0|1|2|3 Event Capture domain
EJ = 0|1 Enterprise JAVA
EM = 0|1|2|3 Event manager domain for BTS
EP = 0|1|2|3 Event Processing domain
FCP=0|2 File Control Program
FT = 0|1|2|3 CICS WEB Interface
ICP = 0|2 Interval Control Program
IE = 0|1|2|3 IP ECI Domain
II = 0|1|2|3 IIOP
IND = 0|1|2|3 Page number indexes for output
IS = 0|1|2|3 IP Interconnectivity domain
JCP=0|2 Journal Control Program
KE = 0|1|2|3 CICS Kernel
LD = 0|1|2|3 Loader Domain
LG = 0|1|2|3 Logger Domain
LM = 0|1|2|3 Lock Manager domain
ME = 0|2 Message domain
ML = 0|1|2|3 Markup Language domain
MN = 0|1|2|3 Monitoring domain
MQ = 0|1|2|3 CICS-MQ interface
Keyword Functional area
MRO=0|2 CICS Multi-Region Operation
NQ = 0|1|2|3 Enqueue Manager
OT = 0|1|2|3 Object Transaction Domain
PA = 0|2 Parameter manager domain
PCP=0|2 Program Control Program
PCT=0|2 Program Control Table
PG = 0|1|2|3 Program Manager Domain
PI = 0|1|2|3 Pipeline Domain
PR = 0|2 Partner Resource management
PT = 0|1|2|3 Partner Domain
RD = 0|2 Resource definition manager
RL = 0|1|2|3 Resource Lifecycle domain
RM = 0|2 Recovery Management
RS = 0|1|2|3 Region Status domain
RX = 0|1|2|3 Recoverable EXCI domain
RZ = 0|1|2|3 Request Streams
SH = 0|1 Scheduler services domain for BTS
SJ = 0|1|2|3 JVM Domain
SM = 0|1|2|3 Storage Manager domain
SO = 0|1|2|3 Sockets domain
SSA=0|2 Static Storage Areas
ST = 0|1|2|3 Statistics domain
SZ = 0|1 Front End Programming Interface
TCP= 0|1|2|3Terminal Control Program
TDP= 0|1|2|3Transient Data Program
TI = 0|1|2|3 Timer domain
TMP=0|2 Table Manager Program
TR = 0|1|2|3 Trace domain
TRS=<trace selection parameters>
TS = 0|1|2|3 Temporary Storage Program
UEH=0|2 User Exit Handler
US = 0|1|2|3 User Domain
WB = 0|1|2 The web interface
W2 = 0|1|2|3 Web 2.0 domain
XM = 0|1|2|3 The transaction manager
XRF=0|2 The extended recovery facility
XS = 0|1 Security Domain
Ed Addison 42

Problem Three - VERBX DFHPD680 ‘DS=3’
DS_TOKEN KE_TASK T S F P TT RESOURCE RESOURCE_NAME W TIME OF TIMEOUT DTA AD ATTACHER MD SUSPAREA XM_TXN_TOKEN
TYPE SUSPEND DUE (DSTSK) TOKEN
0D020005 14F65800 N S N N - USERWAIT WorkWait M 18:02:24.930 - 1A8E7200 XM 15028700 QR 14F70728 150287000000044C
0D060003 14F74800 N S N N - USERWAIT EMSTATUS M 14:53:34.060 - 1A8E7500 XM 15028B00 QR 14F7F728 15028B000000046C
0B080199 14F47100 N S P N - ICWAIT S 14:53:19.944 - 1A8E3680 XM 166AC300 QR 1A8E3680 166AC3000006902C
0D0A02C9 14F65100 N S P N - JVMTHRED DFH$AXIS S 14:53:34.992 - 1A8E7800 XM 166AC700 QR 1A8E7800 166AC7000006908C
0600A279 0A394080 N S P N - ENQUEUE FCDSRECD C 19:53:35.267 - 0E747080 XM 0B371340 QR 0A4FEC85 0B3713400011613C
0D080375 14F38800 N S P N - ICWAIT S 14:53:34.865 - 1A8E7680 XM 166AC900 QR 1A8E7680 166AC9000006907C
0604C013 0A359780 N S P N - ENQUEUE FCDSRECD C 19:51:10.696 - 0E747380 XM 0970A960 QR 0A4FEC85 0970A9600011204C
0606B575 0A3B1400 N S P N - ENQUEUE FCDSRECD C 19:44:57.451 - 0E747500 XM 0970A1B8 QR 0B348CEC 0970A1B80010048C
0B120167 14F56100 N S P N - ICWAIT S 14:53:34.559 - 1A8E3E00 XM 166AC500 QR 1A8E3E00 166AC5000006904C
06080CC5 0A3B1080 N S P N - ENQUEUE FCDSRECD S 19:44:53.971 - 0E747680 XM 0970A340 QR 0E747680 0970A3400010033C
==DS: TASKS USING OPEN TCBS SUMMARY
DS_TOKEN KE_TASK XM_TXN_TOKEN TCB_ID DS_TCB MVS_TCB
0B080199 14F47100 166AC3000006902C T8000 1395B500 00000000
0B120167 14F56100 166AC5000006904C T800I 1395B000 00000000
0D080375 14F38800 166AC9000006907C T800J 13890E00 00000000
Note: ST SYS showed dump was taken at 14:57:45.943332
Task waiting in JVMTHRED suspend for over four minutes – Needs a thread to JVM server DFH$AXIS
Majority of user tasks are in ICWAIT
All owners of T8 TCBs are in ICWAIT
JVMTHRED: The thread limit for the JVM server has been reached, as specified in the THREADLIMIT attribute of the JVMSERVER resource.
The JVM server must wait until a thread becomes available before starting another task. To reduce the frequency of waits for a
particular JVM server, increase the value of the THREADLIMIT attribute for that JVMSERVER resource.
Ed Addison 43
43

T8 TCB Thread Management
CICS uses the open transaction environment (OTE) to run JVM server work.
Each task runs as a thread in the JVM server and is attached using a T8 TCB.
When CICS enables a JVM server, the JVM server runs on a Language
Environment process thread. This thread is a child of the TP TCB. Every CICS
task is attached to a thread in the JVM using a T8 TCB.
• You can control how many T8 TCBs are available to the JVM server by setting
the THREADLIMIT attribute on the JVMSERVER resource.
The T8 TCBs that are created for the JVM server exist in a virtual pool and
cannot be reused by another JVM server running in the same CICS region.
• The maximum number of T8 TCBs that can exist in a CICS region across all
JVM servers is 2000
• The maximum for a specific JVM server is 256.
44

CEMT Display of JVM Server
45
I JVMSERVER(DFH$AXIS)
STATUS: RESULTS - OVERTYPE TO MODIFY
Jvm(DFH$AXIS) Ena Prf(DFHJVMAX) Ler(DFHAXRO )
Threadc(003) Threadl( 003 ) Cur(7758784)
RESPONSE: NORMAL TIME: 13.58.41 DATE: 11/04/14
PF 1 HELP 3 END 5 VAR 7 SBH 8 SFH 9 MSG 10 SB 11 SF

JVM Server Hang – What We Know So Far
Client called support indicating access to the JVM Server for Mobile Web Services stalled
Dump was taken at 14:57:45.943332 GMT
Task 06908 was in a wait for DFH$AXIS JVM Server Thread for four minutes
CEMT Display of JVM Server showed three threads defined and all threads used
Which tasks own the JVM Server Threads?
46

Problem Three - DFHPD680 ‘SJ=3’
47
SJ JSR fields
JVMSERVER Name : DFH$AXIS JVM Process ID : 050303E8
LE Runtime Options : DFHAXRO JVM Profile : DFHJVMAX
State : enabled resolved
GC Policy : -Xgcpolicy:gencon
Initial Heap Size : 00000000 01000000 Max Heap Size : 00000000 10000000
Current Heap Size : 00000000 0098B808 Peak Heap Size : 00000000 00AC8980
IPT TCB Token : 13890C00 00000001
Thread Limit : 00000003 Current Threads : 00000003
Thread HWM : 00000003 Thread Wait Count : 00000001
Thread Number : 00000001 TCB Token : 1395B500 00000001
Thread Number : 00000002 TCB Token : 1395B000 0000000B
Thread Number : 00000003 TCB Token : 13890E00 00000009
Task Token : 0B080199
Task Token : 0B120167
Task Token : 0D080375
NOTE: Take Task Token back to DS=3 Output to find transactions owning the threads

Problem Three - VERBX DFHPD680 ‘DS=3’
DS_TOKEN KE_TASK T S F P TT RESOURCE RESOURCE_NAME W TIME OF TIMEOUT DTA AD ATTACHER MD SUSPAREA XM_TXN_TOKEN
TYPE SUSPEND DUE (DSTSK) TOKEN
0D020005 14F65800 N S N N - USERWAIT WorkWait M 18:02:24.930 - 1A8E7200 XM 15028700 QR 14F70728 150287000000044C
0D060003 14F74800 N S N N - USERWAIT EMSTATUS M 14:53:34.060 - 1A8E7500 XM 15028B00 QR 14F7F728 15028B000000046C
0B080199 14F47100 N S P N - ICWAIT S 14:53:19.944 - 1A8E3680 XM 166AC300 QR 1A8E3680 166AC3000006902C
0D0A02C9 14F65100 N S P N - JVMTHRED DFH$AXIS S 14:53:34.992 - 1A8E7800 XM 166AC700 QR 1A8E7800 166AC7000006908C
0600A279 0A394080 N S P N - ENQUEUE FCDSRECD C 19:53:35.267 - 0E747080 XM 0B371340 QR 0A4FEC85 0B3713400011613C
0D080375 14F38800 N S P N - ICWAIT S 14:53:34.865 - 1A8E7680 XM 166AC900 QR 1A8E7680 166AC9000006907C
0604C013 0A359780 N S P N - ENQUEUE FCDSRECD C 19:51:10.696 - 0E747380 XM 0970A960 QR 0A4FEC85 0970A9600011204C
0606B575 0A3B1400 N S P N - ENQUEUE FCDSRECD C 19:44:57.451 - 0E747500 XM 0970A1B8 QR 0B348CEC 0970A1B80010048C
0B120167 14F56100 N S P N - ICWAIT S 14:53:34.559 - 1A8E3E00 XM 166AC500 QR 1A8E3E00 166AC5000006904C
06080CC5 0A3B1080 N S P N - ENQUEUE FCDSRECD S 19:44:53.971 - 0E747680 XM 0970A340 QR 0E747680 0970A3400010033C
==DS: TASKS USING OPEN TCBS SUMMARY
DS_TOKEN KE_TASK XM_TXN_TOKEN TCB_ID DS_TCB MVS_TCB
0B080199 14F47100 166AC3000006902C T8000 1395B500 00000000
0B120167 14F56100 166AC5000006904C T800I 1395B000 00000000
0D080375 14F38800 166AC9000006907C T800J 13890E00 00000000
Task Token 0B080199 is associated with transaction number 6902
Task Token 0B120167 is associated with transaction number 6904
Task Token 0D080375 is associated with transaction number 6907
Note: ST SYS showed dump was taken at 14:57:45.943332
Task waiting in JVMTHRED suspend for over four minutes – Needs a thread to JVM server DFH$AXIS
Majority of user tasks are in ICWAIT and FCDSRECD Enqueue
All owners of T8 TCBs are in ICWAIT
JVMTHRED: The thread limit for the JVM server has been reached, as specified in the THREADLIMIT attribute of the JVMSERVER resource.
The JVM server must wait until a thread becomes available before starting another task. To reduce the frequency of waits for a
particular JVM server, increase the value of the THREADLIMIT attribute for that JVMSERVER resource.
Ed Addison 48
48
FIND 0B080199

Problem Three – VERBX DFHPD680 ‘WB’
49
==WB: WEB DOMAIN WEBREQUEST BLOCK SUMMARY
Task No Service Program Userid Clientaddress HTTP 32K+ Recv Complete API Send Analyzer
_______________________________________________________________________________________________
06904 EXMPPORT DFH0MOBI CICSUSER 9.27.47.174 Y N Y N DFHWBAAX
06907 EXMPPORT DFH0MOBI 9.27.46.236 Y N Y N DFHWBAAX

Problem Three - VERBX DFHPD680 ‘KE=3’
50
===KE: Kernel Domain KE_TASK Summary
KE_NUM KE_TASK STATUS TCA_ADDR TRAN_# TRANSID DS_TASK KE_KTCB ERROR
0001 137F5000 KTCB Step 00000000 00000000 13839038
0002 137F5680 KTCB QR 00000000 1383DE00 1383C200
0003 13804000 KTCB RO 00000000 1383DF00 1383B168
0004 13804680 KTCB FO 00000000 13884F00 1383A0D0
00EF 14F38800 Not Running 13A9F700 06907 CPIH 1A8E7680 1383C200
00F0 14F47100 Not Running 13A9E700 06902 CPIH 1A8E3680 1383C200
00F1 14F47800 Not Running 13A9D100 06883 CEDA 19E03080 1383C200
00F2 14F56100 Not Running 13A9E100 06904 CPIH 1A8E3E00 1383C200
00F3 14F56800 ***Running** 13A9D700 06910 CEMT 1A8E6800 1383C200
00F4 14F65100 Not Running 13A9F100 06908 CPIH 1A8E7800 1383C200
00F5 14F65800 Not Running 13A9C100 00044 COI0 1A8E7200 1383C200
Transaction 06902: TRANSID CPIH: TCA Address: 13A9E700 – Owns JVM Thread
Transaction 06904: TRANSID CPIH: TCA Address: 13A9E100 – Owns JVM Thread
Transaction 06907: TRANSID CPIH: TCA Address: 13A9F700 – Owns JVM Thread
Transaction 06908: TRANSID CPIH: TCA Address: 13A9F100 – Waiting for JVM Thread
FIND 6902

JVM Server Hang – What We Know So Far
Transactions 06902, 06904 and 06907 own the JVM Server Threads and are in ICWAIT
Who put the tasks in an ICWAIT?
51

Finding Information About the Request
Ed Addison 52
TCA.06902 13A9E700 Task Control Area (User Area)
0000 13A9E800 00000000 00000000 00041048 00000000 14DB5C90 00000000 00000000 *.zY...................*.......*
0020 00000000 0006902C 00000000 00000000 00000000 00000000 008B2000 15120048 *..............................*
0040 95241692 0008ACBC 00000000 95241528 14E9CDE0 00000000 20000000 00000000 *n..k........n....Z...........*
0060 C4C6F0F0 F2F6F1C1 0000500C 00000000 00000000 00000000 00000000 00000000 *DF00261A..&...................*
SYSEIB.06902 13A9EA80 System EXEC Interface Block
-0008 5CE2E8E2 C5C9C240 * *SYSEIB *
0000 0155319C 0114100F C3D7C9C8 0006902C 00000000 00000000 00000010 04000000 *........CPIH..................*
0020 00000000 00000000 00000000 00000000 00000040 40404040 40404000 00000000 *................. .....*
0040 00000000 00000000 00000000 00000000 00000000 00 *................... *
EIUS.06902 15120008 EXEC Interface User Structure
0000 00E86EC4 C6C8C5C9 E4E24040 40404040 15120668 00000000 151268C0 00000000 *.Y>DFHEIUS ...........{..*
0020 00000000 00000000 00000000 00000000 00000000 15120100 00000000 1512BB28 *..............................*
0040 00000000 00000000 151265E0 9409DCC6 00540F20 1409DDF8 14DFB840 14DFBA14 *...........m..F.......8... ..*
0060 140A1388 14DFB040 00041048 14F47100 13A9E988 166AC300 140A1409 14DFB898 *...h... .....4...zZh..C......q*
0080 153EECA8 13A9E700 00000000 00000000 00000000 00000000 00000000 00000000 *...y.zX.......................*
00A0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 *..............................*
00C0 00000000 00000000 00000000 00000000 00000000 00000000 1512003C 15120040 *............................. *
00E0 00000000 00000000 *........ *
EIB.06902 15120100 EXEC Interface Block
-0010 00656EC4 C6C8C1D7 6DC4C6C8 C5C9C25C * ..>DFHAP_DFHEIB**
0000 0155319C 0114100F C3D7C9C8 0006902C 00000000 00000000 00000034 16000000 *........CPIH..................*
0020 00000000 00000000 00000000 00000000 00000040 40404040 40404000 00000000 *................... .....*
0040 00000000 00000000 00000000 00000000 00000000 00 *................... *
*
DFHPD680 ‘AP=3’
Application Register Savearea
EIB Function DELAY
If the EIB function code in the (User) EIB is different than the function code in the System EIB, then typically the current
EXEC CICS request has not yet completed.
The function code in the System EIB is the most current request.
The current EXEC CICS request above is an EXEC CICS DELAY. The previous request was a PUT CONTAINER
EIB Function PUT CONTAINER

IPCS Primary Menu
OPTION ===>
********************
0 DEFAULTS - Specify default dump and options * USERID - USASSC1
1 BROWSE - Browse dump data set * DATE - 06/29/13
2 ANALYSIS - Analyze dump contents * JULIAN -
3 UTILITY - Perform utility functions * TIME - 16:46
4 INVENTORY - Inventory of problem data * PREFIX - USASSC1
5 SUBMIT - Submit problem analysis job to batch * TERMINAL- 3278
6 COMMAND - Enter subcommand, CLIST or REXX exec * PF KEYS - 24
T TUTORIAL - Learn how to use the IPCS dialog ********************
Ed Addison 53

Browse Mode – Register Savearea
Ed Addison 54
ASID(X'0043')
Command ===> L 1512BB28
1512BB28 00104001 1512B980 | .. ..... |
1512BB30 1512BCD0 96718612 00000000 1512CE54 | ...}o.f......... |
1512BB40 1512BC28 1512CEC0 1512D010 16718060 | .......{..}....- |
1512BB50 14DFB840 14DFB7F8 00000000 1512CF40 | ... ...8....... |
1512BB60 1512CD18 16718154 16718314 1512AC50 | ......a...c....& |
1512BB70 00000000 1512BCD0 00000000 00000000 | .......}........ |
Note: Registers 14 through 12 are stored in the Application Savearea at offset X’C’
when a CICS call is issued.
Register 14 will point to where the CICS call was issued from.
Register 1 will contain the parameters when the CICS call was issued.

Browse Mode – Register 14
Ed Addison 55
ASID(X'0043') ADDRESS(166100BE.) STORAGE --------------------------------------------
Command ===> L 16718612
16718612 58C0 90E840F0 2008D210 D108A071 | .{.Y 0..K.J... |
16718620 4130D108 5030D100 9680D100 4110D100 | ..J.&.J.o.J...J. |
16718630 58F0A000 4100913C 58C09080 0DEF58C0 | .0....j..{.....{ |
16718640 90E840F0 200847F0 B34C9120 D08447E0 | .Y 0...0.<j.}d. |
16718650 B34C5830 905C58F0 30F44110 A1920DEF | .<...*.0.4..~k.. |
16718660 5830915C 5B30C000 5030915C 91289054 | ..j*$.{.&.j*j... |
16718670 4770B372 48F02008 58D0D004 58E0D00C | .....0...}}..}. |
Note: Register 14 will point to where the CICS call was issued from. Backing up from Register
14 will tell you the module that made the call.
16718000 C4C6C8E8 C9F6F8F0 E3F0021C 001758F0 | DFHYI680T0.....0 |
16718010 F0D058F0 F01458F0 F00C58FF 000C07FF | 0}.00..00....... |
16718020 5CC6C9D3 D3C9D55C 47F0F028 00C3C5C5 | *FILLIN*.00..CEE |
16718030 000001A8 00000014 47F0F001 98CEAC00 | ...y.....00.q... |
16718040 167180DE 00000000 00000000 00000000 | ................ |
16718050 00104001 00000008 C4C6C8F0 D4D6C2C9 | .. .....DFH0MOBI |
Backup from R14 to start of module

Browse Mode – Register 1
Ed Addison 56
ASID(X'0043') ADDRESS(1406B0.) STORAGE ---------------------------------------------------
Command ===> L 1512BC28
1512BC28 1512BC60 1512D010 | ...-.... |
1512BC30 1512BC80 1512BC88 1512BC90 1512BC98 | .......h.......q |
Note: Register 1 will point to the parameters when CICS was called.
ASID(X'0043') ADDRESS(1406B0.) STORAGE ---------------------------------------------------
Command ===> L 1512BC60
1512BC60 10048010 07000420 80F0F0F0 F4F24040 | .........00042 |
ASID(X'0043') ADDRESS(16610148.) STORAGE ---------------------------------------------------
Command ===> L 1512D010
1512D010 00000005 00000000 00000000 00000000 | ................ |
Parameters
First Parameter (EIB Function Code 1004 DELAY)
Second Parameter (Five Minutes)

JVM Server Hang – What We Know
Transactions 06902, 06904 and 06907 own the JVM Server Threads and are in ICWAIT
Program DFH0MOB1 issued EXEC CICS DELAY for 5 minutes
57

Summary
Pipeline Disabled
Web Service Not Found
JVM Server Hang
58

Useful Debugging Tools
Postman REST client plugin for Chrome
• Send HTTP requests and headers
• Formats JSON responses
• Allows saving requests
• https://chrome.google.com/webstore/detail/postman-rest-
client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en
Online JSON validator
• If JSON fails to validate in CICS, this tool will highlight the area of error
• http://jsonlint.com/
Online schema validator
• Validate schema syntax – if schema is not accepted or produces strange results in
DFHJ2LS
• Validate input message against schema – check request structure if transformation errors
occur
• http://json-schema-validator.herokuapp.com/
59

Social media: Facebook, YouTube, Twitter, Aggregator
Visit the WebSphere and CICS Support Facebook page
• https://www.facebook.com/WebSphereSupport
Check out our CICS videos on YouTube
• http://www.youtube.com/user/CICSfluff
Follow CICS Support on Twitter
• http://www.ibm.com/support/docview.wss?uid=swg21384915 or
https://twitter.com/IBM_CICS
See multiple channels by using the Social Media Aggregator
• https://www.ibm.com/social/aggregator/cics
Subscribe to the Quarterly Support Newsletter for CICS Transaction Server and CICS Tools
• http://www.ibm.com/support/docview.wss?uid=swg27006900 (on-line version)
See the Blog Post for CICS
• https://www.ibm.com/developerworks/community/blogs/aimsupport/tags/cics
60

Software Services and Support Zone
61
Visit us in the Solution EXPO
ibm.com/software/expertise
Site features:
 Client successes
 Specialized practices
 Services catalog
 Solution brochures
 Practitioner stories
 Video library
 Social connections
 Consultant profiles
 and more
Technology
to propel you
Expertise
to help you
Resources
to educate you
Successes
to assure you
People
to guide you

Legal Disclaimer
• © IBM Corporation 2014. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you
will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and
other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration
purposes only.
64

IBM Impact session Ed addison nuts and bolts ws

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (14)

Similar to IBM Impact session Ed addison nuts and bolts ws

Similar to IBM Impact session Ed addison nuts and bolts ws (20)

More from nick_garrod

More from nick_garrod (20)

Recently uploaded

Recently uploaded (20)

IBM Impact session Ed addison nuts and bolts ws

Editor's Notes