Ibm session tac 2104 - ctg presentation for impact 2013 final

Session #TAC-2104 - Achieving highest availability and best performance with CICS TG on z/OS HA Architecture - User Experience
Achieving highest availability and best performance
with CICS TG on z/OS HA Architecture – User Experience
Elena Nanos
Lead Systems Architect
Health Care Service Corporation
IBM Certified Infrastructure Systems Architect
TOGAF Certified Architect - The Open Group Architecture Framework V8.1.1
IBM Certified Solution Expert - Dynamic Infrastructure Technical Leader
IBM Certified System Administrator - WebSphere Application Server ND V7.0
IBM Certified Solution Expert - CICS Web Enablement
IBM Certified System Specialist - WebSphere MQSeries
Email – ELENA_NANOS@BCBSIL.COM
Session TAC-2104
1

2
CICS TG Product Positioning
Presented by Catherine Moxey, IBM STSM, Email: moxeyc@uk.ibm.com

 Health Care Service Corporation (HCSC) is the fourth largest health
insurance company in the nation.
 Largest customer-owned health insurer in the U.S., founded in 1936,
now with more than 13 million members, HCSC operates Blue Cross
and Blue Shield Plans in Illinois, New Mexico, Oklahoma and Texas.
 We're greater than 18,000 employees strong with 60 local offices and
state-of-the-art technology, including two Tier IV data centers — the
industry's highest reliability level – that provide the speed and data
security to meet our customers' current and future business needs.
Who Are We? About Health Care Service Corporation
3

Common Architectures and Standard Used with CICS
Connectivity
option
Required
middleware
Main capabilities Usage notes
Web services CICS-only solution • Inbound and outbound
• Low coupling
• Synchronous (HTTP)
• Asynchronous (WMQ)
• QoS based on transport type
• Support for some WS-* standards
Should be first consideration for service
enabling CICS applications, particularly
when you need to support multiple service
requester types or need bi-directional
support
JCA with
CICS TG
CICS TG
(normally WebSphere
Application Server)
• Inbound to CICS
• Medium coupling
• Synchronous
• High qualities of service (QoS)
Most appropriate solution when service
requester is JEE component and when high
QoS required
JCA with
WOLA
WebSphere
Application Server
for z/OS
• Inbound and outbound
• Tight coupling
• Synchronous
• High QoS
Particularly useful for JCA access to and
from CICS, and for very high throughput
and performance requirements.
CICS web
support
CICS-only solution • Inbound and outbound
• Medium coupling
• Synchronous
• Medium QoS
Use with web services, RESTful services,
and Atom feeds, or when remote
client/server only supports HTTP
WebSphere
MQ for z/OS
CICS-only solution • Inbound and outbound
• Medium coupling
• Asynchronous, with almost-
synchronous capabilities
• Assured delivery
Exploit WMQ for basic messaging and
flowing web services
4

HCSC CICS TG Topology
• Topology 1 was our main WASd/CICS TG configuration.
• Topology 2 was proposed configuration moving CICS TG off distributed platform into
z/OS, using CICS TG on z/OS stand-alone (Remote mode). This is our current
topology.
• Topology 3 is also used with WAS on z/OS and CICS TG configuration, using CICS TG
in Local mode. Note: Local mode on z/OS can use either EXCI (which is MRO cross memory), or
IPIC (which is still sockets based, but will be optimized in TCPIP to Fast Local Sockets - effectively cross
memory - when co-located within the same LPAR as CICS).
5

Current Configuration
6
Previous HCSC Distributed Configuration

7
HCSC CICS TG HA Architecture

CICS TG V7.0
 WLM Health reporting for storm drain avoidance that required use of the
SHAREPORTWLM keyword (a variant of SHAREPORT)
 WLM Feedback support – Storm drain avoidance
 Feedback of CICS region status to IP load balancing mechanism via WLM
 Allows more intelligent autonomic decisions to be performed by IP load balancers
 Pushes CICS health out to the IP load balancer
CICS TG V7.2
 Sysplex-wide support for XA with HA which used Sysplex Distributor.
 Port-sharing across Sysplex but this could only be combined with DSS for 1-phase commit
transactions and synconreturn.
 Support for XA transactions in High Availability topologies using TCPIP load balancing in a z/OS
Sysplex
All CICS TG versions have ability to use SHAREPORT on the PORT statement in the TCP/IP profile
 Enables a group of cloned servers to listen on the same port
 New connections are distributed across available servers using a weighted round-robin
8
High Availability and XA Transactions
History of Enhancements

CICS TG V8.0
 Provides additional High Availability capabilities, allowing dynamic selection of CICS server
within the Gateway daemon
 Provides a working, configurable CICS request exit (including source) for performing
failover or round robin
 Support for cross-sysplex port sharing XA/2PC requests in combination with DSS
CICS TG V8.1
 Introduced the "out-of-the-box" configurable DSSPOLICY as an alternative to CREXIT, for
those wanting a basic roundrobin or failover implementation. CREXIT still provides maximum
flexibility when it comes to custom algorithms.
CICS TG V9.0
 Routing or filtering by client IP address
 Full request details for XA transactions
 New Request Monitoring exit point “RequestDetails”
 Invoked just before calling CICS, after Dynamic
Server Selection
9
High Availability and XA Transactions
History of Enhancements

 Health reporting is effective only in TCP/IP load balancing topologies, with the CICS Transaction
Gateway running in remote mode.
 At intervals specified by the Health interval, the Gateway daemon monitors error codes like
ECI_ERR_NO_CICS, ECI_ERR_RESOURCE_SHORTAGE, ECI_ERR_SYSTEM_ERROR
to determine the health of communications with the CICS subsystem. The load balancer can then
use this health to set priorities when creating new incoming IP socket connections to cloned
Gateway daemons.
 MVS console command can be used to display Gateway health status –
F <JOB>,APPL=STATS,GS=GD_CHEALTH
10
WLM Feedback

 TCP/IP load balancing
• TCP/IP load balancing provides high availability of the Gateway daemon by distributing
connections from clients across multiple Gateway daemons. The efficiency of load balancing
can also be increased by taking into account CICS server status provided by dynamic feedback to
the IBM z/OS Workload Manager (WLM).
 Sysplex Distributor
• Sysplex Distributor provides TCP/IP load balancing across multiple LPARs.
 Port sharing
• TCP/IP port sharing enables requests for work to be shared between several
Gateway daemons through a single TCP/IP port. Port sharing provides TCP/IP
load balancing in a single LPAR. When connections are established between a Client application
and a Gateway daemon, the TCP/IP port sharing component of the z/OS TCP/IP. Communications
subsystem distributes requests across multiple Gateway daemons. When the connection is
established, subsequent requests from the Client application continue to use the same connection.
 Health monitoring
• A TCP/IP load balancer that is allocating a connection to the Gateway daemon detects whether or
not a CICS server is available. The Gateway daemon reports the health of its CICS server
connections to the TCP/IP load balancer.
 Support Pack CH51 – CICS TG Simple Request Monitor Utility
 Support Pack CA1T – CICS TG High Availability User Exits
 MICS report using SMF 111 records
 CICS Explorer
11
Technology Utilized in HCSC CICS TG HA Architecture

Two different workload balancing polices (round robin or fail over) are supplied – both of
which use server rules from the HA configuration file to remap and retry failed requests.
For example, the following server rule defines the server alias SERVER1 as mapping
to 4 actual CICS servers CICSA, CICSB, CICSC & CICSD:
SERVER1=CICSA,CICSB,CICSC,CICSD
 The round robin policy will distribute requests in a round robin fashion, so that
requests are evenly distributed across servers CICSA, CICSB, CICSC & CICSD.
 The fail over policy will distribute requests in a hierarchical order so that all requests
are sent to CICSA, if this is unavailable requests will be sent to CICSB, if CICSB is
unavailable requests will be sent to CICSC, and if CICSC is unavailable requests will
be sent to CICSD.
http://www-01.ibm.com/support/docview.wss?uid=swg24021484
12
SupportPack CA1T – CICS TG High Availability User Exits

Three different CICS request exit implementations are provided as follows:
1. FailOver – uses a list of CICS servers to define how server names are remapped.
Requests will be retried using a static list of servers, each server being used in turn from
the list until the list is exhausted or the request succeeds.
2. RoundRobin – distributes ECI requests around a list of CICS servers in round-robin
fashion. Requests will be distributed evenly across the servers, and retried until the
request succeeds or all serves have been used.
3. RequestValidation – validates the transaction ID, program and user ID parameters in
the ECI request against a defined list. No server name remapping is performed.
Latest enhancements
CA1T V3.0 support pac has been recently updated and gives ability to set new
environment variable CTG_HAVALIDATION=EXCLUSIVE, then any request
which matches one of the USERID, TRANSID and PROGRAM validation rules will
be rejected with ECI_ERR_INVALID_CALL_TYPE (-14).
Note: Configuration is based on a flexible set of rules defined in a configuration file. On z/OS, the configuration file can reside either on HFS or
in an MVS dataset. Rules for workload balancing can be generic per Gateway, based on server aliases, or on the payload type (for example,
channel or commarea). Rules for request validation can be specified based on user IDs, transaction IDs, or program names, and operate as
either a white list (INCLUSIVE) or black list (EXCLUSIVE). All rules can be dynamically updated while the Gateway daemon is running. A
configurable retry interval is provided to improve the efficiency of request distribution when using workload balancing.
http://www-01.ibm.com/support/docview.wss?uid=swg24021484
ftp://public.dhe.ibm.com/software/htp/cics/support/supportpacs/individual/ca1t_v110.pdf 13
(cont.)
SupportPack CA1T V3.0 – CICS TG High Availability User Exits

Sample output of CICS Request Exit –
03/11/13 08:26:38:943 CH51 Flow(XaPrepare) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)
RespTime(8)ms CICScall(8)ms
03/11/13 08:26:38:956 CH51 Flow(XaCommit) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)
RespTime(6)ms CICScall(4)ms
03/11/13 08:26:39:196 CH51 Flow(XaStart) Srv(NEWCICSR) Trnid(null) Userid(WRDBC01) ClientIP(10.99.129.223)
RespTime(0)ms
03/11/13 08:26:39:209 CH51 Flow(XaEci) Srv(NEWCICSR) Pgm(PRDFSLCT) MirrorTrnid(1234) Userid(WRDBC01)
ClientIP(10.99.129.223) RespTime(11)ms CICScall(10)ms
….
3/11/13 11:11:04:827 CH51 Flow(EciSynconreturn) Srv(OLDCICSR/NEWCICSR) Pgm(PRDPREPR) MirrorTrnid(ABCD)
Userid(PROD016) ClientIP(10.99.10.5) RespTime(43)ms CICScall(37)ms wait(2)ms
03/11/13 11:11:05:063 CH51 Flow(EciSynconreturn) Srv(NEWCICSRC/NEWCICSR) Pgm(PRDD026) MirrorTrnid(ABC3)
Userid(PRODSY4) ClientIP(10.99.11.32) commarea(28462)bytes RespTime(5)ms CICScall(5)ms
03/11/13 11:11:05:287 CH51 Flow(EciSynconreturn) Srv(OLDCICSR/NEWCICSR) Pgm(PRDPREPR) MirrorTrnid(ABCD)
Userid(PROD016) ClientIP(10.99.10.5) RespTime(17)ms CICScall(10)ms CICS TG_rc(-7) Abend(AEI0)
14
Support Pack CH51: CICS TG
Simple Request Monitor Utility

Set at CTGSTART_OPTS=
-j-Dcom.ibm.CTG.samples.requestexit.out=/usr/cicstg/CH51LOGS/CTGname01.log
-j-Dcom.ibm.CTG.samples.requestexit.log.number=5 -j-Dcom.
ibm.CTG.samples.haexit.out=//DD:HALOG -j-Dcom.ibm.CTG.samples.haexit.ca1t=1
-statsport 3101 -applid=CTGname -j-Xms256M -j-Xmx512M - j-
Xdump:system:dsn=yourprefix.CTGname.JAVA.DUMP
Logging levels available – set at -j-Dcom.ibm.CTG.samples.haexit.ca1t
15
Support Pack CH51: CICS TG
Simple Request Monitor Utility Setup
Can also issue console command. For example to enable level 3 logging issue the following
command - /F <JOB>,APPL=CREXIT,CMD=CA1T_3

 This SupportPac allows integration test tools to intercept ECI calls to CICS
TG from Java™ applications before they are processed by CICS TG.
 Allows integration test tools ( for example Rational Test Workbench / Rational
Integration Tester V8.0.1.1) to intercept External Call Interface (ECI) calls to
IBM CICS TG, originating from Java Standard Edition (Java SE) and Java
Enterprise Edition (JEE) applications. This, therefore, allows such applications
to be tested without requiring a running CICS TG or CICS Transaction
Server.
This SupportPac provides updated versions of the CICS TG V8.0 Java base classes and non-extended
architecture (non-XA) ECI resource adapter. These updated files support the use of a Gateway intercept plug-
in that can intercept application calls made through the Java Application Programming Interfaces (APIs).
Documentation for the APIs is also provided by this SupportPac.
When enabled, a Gateway intercept plug-in is notified before a connection to a Gateway daemon is opened or
closed, and before a request is sent to the Gateway daemon. Before a connection is opened, the plug-in can
change properties of the connection object, and can prevent the connection from opening. Before a request is sent
over a connection, the plug-in can change properties of the request object, and can cause the request to be
returned to the application without being sent. Before a connection is closed, the plug-in can prevent the
connection from closing. This allows a variety of scenarios to be simulated for application testing.
Can be downloaded from – http://www.ibm.com/support/docview.wss?uid=swg24033968
16
Support Pack CA5F: IBM CICS TG
'gateway' Intercept Plug-in Support

( set in Runtime env CICSCLI)
SECTION IPICSERVER = SYS1CIC1
description = IPIC connection to CICS region #1
hostname = your.host.net
port = 3101
tcpkeepalive = Y
srvidletimeout = 60
srvretryinterval = 0
connecttimeout = 10
sendsessions = 200
ENDSECTION
port = 3102
tcpkeepalive = Y
srvidletimeout = 60
connecttimeout = 10
sendsessions = 200
ENDSECTION
port = 3103
tcpkeepalive = Y
srvidletimeout = 60
connecttimeout = 10
sendsessions = 200
ENDSECTION
port = 3104
tcpkeepalive = Y
srvidletimeout = 60
connecttimeout = 10
sendsessions = 200
ENDSECTION
On the Gateway you can set - CONNECTTIMEOUT, which is a numeric value between 0 and 3600, specifying the maximum
time in seconds that establishing a connection is allowed to take. The default value of 0 means that no limit is set. For
more details see - http://publib.boulder.ibm.com/infocenter/txformp/v7r1/index.jsp?topic=/com.ibm.cics.tx.doc/tasks/t_config_ipic.html
17
Sample CICS TG CONFIG File

CICS TG V8.1 and WebSphere V6, V7 & V8 - http://www-01.ibm.com/support/docview.wss?uid=swg21577454
Runtime Environments for Java and JEE Applications
18
CICS TG Compatibility with WebSphere

 CICS TG V8.0 or prior has two CICS Resource Adapters – XA and non-XA.
 CICS TG V8.1 has support for Java Enterprise Edition 6
 The CICS connectors included with CICS TG V8.1 exploit JEE 6 / JCA 1.6
 JEE 6 certified application servers (e.g. WAS 8.0) benefit from:
 A single ECI resource adapter (rather than two) including support for both XA (two-phase
commit) and non-XA transactions
 A more flexible approach to setting the transactional capabilities of the CICS ECI resource
adapter
 Lower administrative and maintenance overheads
Resource adapters compatible with JEE 5 (or earlier) application servers are available
use with CICS TG V8.1 in remote mode only from Support Pac CC03
19
CICS JCA Resource Adapters

 The CICS JCA resource adapters are provided, in order to support
connectivity from JEE application servers to CICS using the facilities
of the CICS Transaction Gateway.
 The latest service levels of the CICS Transaction Gateway Java
Enterprise Edition Connector Architecture (JCA) resource adapters for
CICS TG are available for download for deployment into supported
Java Enterprise Edition (JEE) application servers. These resource adapters can be
used to provide JCA connectivity into CICS using the facilities of a licensed CICS TG installation, and are
designed around the JCA specification as supported by JEE application servers such as WebSphere Application
Server and Oracle Weblogic Server.
 This Support Pack includes the CICS JCA resource adapters from the
following CICS Transaction Gateway for Multiplatforms releases:
V7.0.1.1, V7.1.0.4, V7.2.0.4, V8.0.0.4, V8.1.0.1, V9.0.0.0
http://www-1.ibm.com/support/docview.wss?uid=swg24008817&myns=swgother&mynp=OCSSGMJ2&mync=E
20
Support Pack CC03: CICS JCA Resource
Adapters for Use With JEE Application Servers

J2C connection factories > CICS Connection Factory > Connection pools
Maximum connections
 Specifies the maximum number of physical connections that you can create in this pool. These are
the physical connections to the backend resource. Once this number is reached, no new physical
connections are created and the requester waits until a physical connection that is currently in use
returns to the pool, or a onnectionWaitTimeoutException is thrown. Ensure that there are no waits
for connections and that CICS can handle concurrent volume set here.
Minimum connections
 Specifies the minimum number of physical connections to maintain. If the size of the connection pool
is at or below the minimum connection pool size, the Unused Timeout thread does not discard
physical connections. Best to set to zero, to allow stale connections to be discarded.
Reap time
 Specifies the interval, in seconds, between runs of the pool maintenance thread. Do not set too
low, as it will increase the overhead.
Unused timeout
 Specifies the interval in seconds after which an unused or idle connection is discarded. Set the
Unused Timeout value higher than the Reap Timeout value for optimal performance.
Aged timeout
 Specifies the interval in seconds before a physical connection is discarded. Setting Aged Timeout
to 0 supports active physical connections remaining in the pool indefinitely. Do not set it at
zero to avoid connection retries on stale connections.
21
Connection Pool Tuning in WAS

The graph shows the difference in processor usage cost between EXCI and IPIC.
 The CICS address space costs were less when using EXCI, because of the efficiency of using
cross-memory MRO for communication between the gateway address space and the CICS
address space.
 Athough the overall cost to use IPIC was greater, the amount of processing offloaded to a
zAAP was also greater, which made the use of general-purpose CPs in the gateway address
space less critical for IPIC.
22
EXCI vs. IPIC CPU Usage

Data from redpaper IBM CICS Performance Series: A Processor Usage Study of Ways into CICS
at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4906.pdf
The graph shows the total microseconds of processor usage per transaction to get
32 KB of data into CICS and back to the client using the access methods tested
Environment
Hardware
The following hardware was used:
• IBM zEnterprise® 196 (Z196) 2817-779 model M80
• LPAR with three dedicated central processors (CPs)
and one dedicated IBM System z®
• (zAAP) on zIIP)
• DASD DS8800
• Internal coupling facility peer (ICP)
• 1 GB OSA-Express3
• Crypto Express3 coprocessor
• CP Assist for Cryptographic Function feature (enabled)
Software
The following software was used:
• IBM z/OS® Version 1, Release 13
• CICS Transaction Server Version 4, Release 2
• IBM WebSphere MQ Version 7, Release 1
• Communication Server IP Sockets for CICS
• CICS Transaction Gateway Version 8, Release 1
23
CPU Cost Getting CICS Data

24
POC Testing – High zAAP Offload Rate

25
CA Wily Introscope CICS TG Metrics

26
Wily Introscope ECI Pool Size Metrics

Compare to week prior
27
Wily Introscope Response Time Metrics

28
Wily Introscope CICS TG/JCA Metrics

CICS Transaction Gateway - Activity Summary
_____________________________________________
0STTG0001 Printed at 11:29:15 3/01/2013 Data from 08:00:00 2/21/2013 to 08:00:00 2/21/2013 Page 1
CICS TG GATEWAY 02/21/13 REPORT
Gateway ID Start time Up time Number of Number of Peak Peak TPS time Connect Worker
Intervals requests TPS Timeouts Timeouts
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 7days 17.34.09 95 1643676 13 2013-02-21-17.15.00 0 0
Gateway ID Start time Collection time Region Java heap Clients Workers EXCI pipes IPIC sessions
Size/Max Size/Max Peak/Max Peak/Max Num/Max Num/Avail
ZOSCGRP0.CICS TG2P003 2013-02-14 06.10 2013-02-21-16.00.00 875M/1000M 71M/512M 19/500 3/800 0/0 0/400
CICS Performance Analyzer V5.1 now includes two CICS TG batch reports, "Activity Summary", and
"Usage and Capacity" which can be run against SMF 111 records from CICS TG for z/OS.
29
CICS Performance Analyzer for z/OS - Sample Reports

CICS Transaction Gateway plug-in for the CICS Explorer V2.0.0.0 is now available for CICS Explorer V5.1, easily
installed through the common V5.1 update site which is pre-configured in the CICS Explorer "Install new software"
facility.
30
CICS Transaction Gateway Plug-in For The CICS Explorer

CSSYS1CIC1_SIPADDR=yourSYS1.network.net (CICS server
TCP/IP address)
CSSYS1CIC1_SIPPORT=3168 (CICS server TCP/IP port)
CSSYS1CIC1_LALLREQ=4549989 (Number of requests
processed)
CSSYS1CIC1_CWAITING=-2 (Number of requests waiting on a
response)
CSSYS1CIC1_SSESSMAX=200 (Number of requested IPIC
sessions)
CSSYS1CIC1_CSESSMAX=200 (Number of negotiated IPIC
sessions)
CSSYS1CIC1_CSESSCURR=0 (Number of IPIC sessions in use)
CSSYS1CIC1_LCOMMSFAIL=49 (Number of CICS
communication failures)
CSSYS1CIC1_LCONNFAIL=4484 (Number of connection failures)
CSSYS1CIC1_LLOSTCONN=32 (Number of lost connections)
CSSYS1CIC1_LSESSFAIL=0 (Number of IPIC session failures)
CSSYS1CIC1_LIDLETIMEOUT=0 (Number of timed out
connections)
CSSYS1CIC1_LREQDATA=28649890161 (Amount of CICS
request data)
CSSYS1CIC1_LRESPDATA=29043330152 (Amount of CICS
response data)
CSSYS1CIC1_LAVRESP=64 (Average CICS response time)
CSSYS1CIC1_CAPPLIDQ=ILA (APPLID qualifier of connected
CICS)
CSSYS1CIC1_CAPPLID=YourCICSApplid (APPLID of connected
CICS)
CSSYS1CIC1_ICOMMSFAIL=0 (Number of CICS communication
failures)
CSSYS1CIC1_ICONNFAIL=0 (Number of connection failures)
GD_IAVRESP=42 (Average Gateway daemon response time)
GD_LAVRESPIO=109 (Average Gateway daemon response time with
I/O)
GD_IAVRESPIO=98 (Average Gateway daemon response time with
I/O)
GD_SNAME=CICS TGPRD1 (Gateway daemon name)
GD_CHEALTH=100 (Gateway daemon health)
GD_LHAEXIT=4687171 (Number of CICS request exit calls)
GD_IHAEXIT=642 (Number of CICS request exit calls)
GD_SDFLTSRV= (Default server name)
GD_LRUNTIME=1758839 (Gateway daemon running time)
GD_SSTATINT=001500 (Length of the statistics interval HHMMSS)
GD_SSTATEOD=000000 (Logical End Of Day time HHMMSS)
GD_CNEXTRESET=150000 (End of interval time HHMMSS)
GD_IRUNTIME=199 (Interval running time)
WT - Worker thread
WT_SMAX=800 (Maximum number of worker threads)
WT_SINIT=200 (Initial number of worker threads)
WT_CCURR=200 (Current number of worker threads)
WT_CALLOC=0 (Currently allocated worker threads)
WT_LTIMEOUTS=0 (Number of times worker time out reached)
WT_ITIMEOUTS=0 (Number of times worker time out reached)
With CICS TG on z/OS stand alone we can use MVS console commands to display statistical
information about the CICS TG, or can obtain statistics using the Statistics API interface.
/F CICS TGname,APPL=STATS,GS
31
MVS Console Commands

If you would like to analyze CICS TG SMF 111 records, 1st you would need to extract these
records in a separate dataset by the IFASMFDP utility. Here is a sample JCL -
//* COPY SMF RECORDS FROM DAILY TAPES
//* THIS PARTICULAR JOB COPIES SMF 111 RECORDS FOR CTG
//* BEFORE RUNNING, SET DATE AND LPAR
// SET SMFHLQ=TSMF <<== HLQ OF SMF
// SET SMFTYPE=CTG
// SET LPAR=A <<== LPAR <- set LPAR name
// SET DATE=D032013 <- set date
//STEP2 EXEC PGM=IFASMFDP
//SYSPRINT DD SYSOUT=*
//SMFIN DD DISP=SHR,DSN=&SMFHLQ..SMF.&LPAR.NOSSDAY.&DATE
//DUMPOUT DD DSN=&SMFHLQ..SMF.&LPAR..&SMFTYPE..&DATE,
// DISP=(NEW,CATLG),UNIT=3490E,RETPD=9
INDD(SMFIN,OPTIONS(DUMP))
OUTDD(DUMPOUT,TYPE(111))
You can collect and report on the following statistics -
 CICS Server Instance (CSx) SMF statistics for EXCI
 CICS Server Instance (CSx) SMF statistics for IPIC
 Gateway daemon (GD) statistics SMF data
 Protocol handler (PH) statistics SMF data
 System environment (SE) statistics SMF data
 Worker thread (WT) statistics SMF data
For more details please see 'CICS Transaction Gateway Version 8 or 9 z/OS Administration' – section 'Displaying statistics‘.
32
CICS TG SMF 111 Records ReportingCICS TG SMF 111 Records Reporting

Inquiry : CICLDG CA MICS - CICS Transaction Gateway Daily Thr
Run Date: 07MAR13 Data Presented by HOUR for each CT
REPORT PAGE: Data for Date=06MAR SYSID : 1
Gateway ID: A001 APPLID: CICS TG1P001 Release Level:8.0.0
+------------- Gateway Daemon Activity ------------+
Day Client Client Client Client Client CICS
of Requests Request Resp Request Avg Resp Requests
Week HOUR Processed MB Data MB Data per/sec Time Processed
---- --------- ------- ------- ------- -------- -------- --------
Wed 1 16272 158.01 188.92 4.520 0.1250 1627
….
6 28525 9845.18 9877.02 7.924 0.0890 2852
7 57660 17330.8 17389.7 16.017 0.0820 5766
8 92410 17048.2 17124.3 25.669 0.0920 9241
9 204752 16063.8 16036.2 56.876 0.0690 20475
10 245463 8153.38 8069.41 68.184 0.0730 24547
11 270587 7347.09 7206.97 75.163 0.0680 27057
12 224123 12615.7 12494.0 62.256 0.0630 22412
13 197175 9026.42 8954.67 54.771 0.0670 19717
14 224994 7117.15 6964.56 62.498 0.0710 22499
15 201345 2994.01 2853.23 55.929 0.0560 20134
16 183724 3153.67 3025.13 51.034 0.0500 18372
17 144724 2194.84 2086.04 40.201 0.0540 14472
33
Sample MICS CICS TG Report

34
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0

35
 Performance Tuning
Helps to prevent CICS becoming overloaded or underutilized
Allows for optimization of workflow through CICS TG and onto CICS
Alerts users to changes in behavior when abnormal workloads hit CICS TG
Provides users with an account of transactions per minute and CPU
consumption
 Availability
Alerts users to communication problems between CICS TG and CICS
 Managablity
Helps to determine where to control resources in CICS TG or CICS
 Monitoring
Transaction Monitoring in Gateway daemon and WebShere Application
Server on z/OS
Transaction Response Time Analysis, Payload and Wiresize per
transaction
 End To End Response Time Analysis
 Problem Determination
Identifies the bottlenecks within CICS TG daemon
Identifies Abended transactions
Correlates all application Flows within a Transaction
Provides Transaction Flow Trace
 Capacity Planning
Helps to plan for workload growth, perform trend analysis, detect system
anomalies and to identify performance bottlenecks using the historical data
via the Tivoli Data Warehouse repository
 Total Cost of Ownership
Opens up the “black-box” of CICS TG thread management and helps to
save precious resources in both CICS TG and CICS
Other Monitoring Tools Available
IBM Tivoli OMEGAMON XE for CICS on z/OS V5.1.0

Issue
We implemented new Application using CICS TG on z/OS, which had invalid Userid set on
Connection Factory. This caused ID to get suspended and Application was redriving error routine
to restablish CICS TG connection, after each failure. This caused connections not getting
released, causing all connections to be in use across all 4 CICS TGs, resulting in CICS TG -16
return (ECI_ERR_RESOURCE_SHORTAGE).
Solution
Setup automation to issue CICS TG console command –
/F CICS TGname,APPL=STATS,GS every 5 minutes and send alert if field CS_CSESSCURR
(total # of IPIC sessions in use with all CICS servers) is at 80% or higher from CS_CSESSMAX (#
of IPIC sessions negotiated with CICS servers). Can also have automation (OPS/MVS for
example) check the values of CM_CALLOC and WT_ALLOC and alert if the threshold set
has been passed.
CICS TG on z/OS V8.0
PM38908: ECI_ERR_RESOURCE_SHORTAGE reported when sending extended mode ECI
requests over an IPIC connection to a CICS server
Error description
When an extended mode ECI request does not complete successfully (possibly due to
security reasons) before the CICS mirror transaction is attached, the IPIC session is not
freed.
http://www-01.ibm.com/support/docview.wss?uid=swg1PM38908
36
Issues To Watch Out For

GD_IAVRESP - The average time taken in milliseconds for the Gateway daemon to respond to API (ECI, ESI, EPI)
and XA requests from remote clients. Successful and failed requests are included. This value is inclusive of the CICS
response time, as provided by the corresponding CS_IAVRESP statistic.
GD_IAVRESPIO - The average time in milliseconds for the Gateway daemon to respond to API (ECI) and XA
requests from remote clients including network I/O time. Successful and failed requests are included. This value is
inclusive of the Gateway response time, as provided by the corresponding GD_IAVRESP statistic.
The latest versions of the CICS TG plug-in for CICS Explorer V1.1 for CICS Explorer V1.1 and V2.0 for CICS Explorer
V5.1 now shows Network Latency data. You can get lifetime average value for network latency, under the column
"GwyRespTimeIO", and is included in the pre-defined Column layout "Requests", of the Gateway daemons view.
37
CICS TG V8.1 New SMF Network Latency Data

 Limitations of 31-bit Gateway daemon
 31-bit Gateway daemon, 375MB heap, REGION 0M (1632MB)
 1350 Connection Mangers
 999 Worker threads
 1 IPIC connection with 999 sessions
 32KB payload
 64-bit Gateway daemon scalability
Note: Data above is a rough guide, your ‘milage’ may vary …
38
CICS TG Scalability

 For best performance with 64-bit HWM, use compressed references
(-j-Xcompressedrefs)
 Migration to 64-bit can reduce number of address spaces
 Fully exploit IPIC and Dynamic Server Selection after migrating from the
constraint of 250 EXCI pipes
 Rule of thumb for Gateway MEMLIMT
MEMLIMIT should be at least
128MB (core) + Heap size + (#CM+WT threads) * 3MB
39
Tuning Suggestions

GA December 14, 2012
Please do not miss IMPACT session TAC-3022 ‘Connecting to CICS with CICS
Transaction Gateway V9’ scheduled for 3:45pm on May 1st, presented by Andrew Smithson.
40
CICS Transaction Gateway V9.0

Routing or filtering by client IP address
 CICS Request Exit now provides the TCPIP address of the application
originating the request
 InetAddress map object “ClientLocation”
 Allows routing decisions based upon client location (e.g. subnet)
 Allows rejection of requests from unauthorised locations
Full request details for XA transactions
 CICS Request Exit allows custom Dynamic Server Selection
 CICS TG V8 has limited request meta-data for XA transactions
 CICS TG V9.0 provides all available request meta-data for both XA and non-
XA ECI requests
 Allows customized routing logic for High Availability solutions with full
distributed two-phase commit
New Request Monitoring exit point “RequestDetails”
 Invoked just before calling CICS, after Dynamic Server Selection
 Allows clear visibility of target CICS server for HA solutions
 Very useful for time outs, or hung request scenarios
41
CICS TG V9 HA Enhancements

Cross component trace (XCT) contexts
 Available on requests originating from WebSphere
Application Server V8.5, with XCT support enabled
 Enabled/disabled
• Dynamically through the WAS admin console run time options
• Statically through the WAS admin console configuration
 Uses High Performance Extensible Logging (HPEL)
XCT contexts
 XCT context Begin and End demarcate component boundaries
• Contexts are hierarchical
 A thread of execution can have upto 3 XCT contexts at any one time:
• Root – initial context (Request ID) of the component at the point of entry
• Parent – context of the calling component
• Current – context of the current component
42
CICS TG V9 Cross Component Trace

43
CICS TG V9 Enhancements To IPIC Connectivity

RFE ID Headline
17173 Asynchronous call support for ECI Version 2 applications
17258 CICS Gateway Support for OEL 5u4
19319 CICS TG Explorer Plugin - Logical grouping of Gateway daemons
21081 Transaction tracking from WAS into CICS through CICS TG
21082 Support for Gateway daemon running in a 64-bit JVM
21085 Additional exit point for Request Monitoring
21633 64-bit compile and run time support for ECI Version 2
21642 Remote Mode SSL for IPIC Connections
21828 Bi-directional language support
21871 Make program and transaction available to the CICS Request Exit for XA transactions
21872 ECI_TIMEOUT default value for IPIC connections
21875 64-bit C module compilation support for ECIv2 on AIX (IBM POWER) and Red Hat (x86)
22036 CICS Request Exit to provide client IP address
22049 Assembler DSECTs for mapping the SMF 111 records
22081 Sample SMF formatter CICS TGSMFRD support for mixed-release installations
22088 Information about channels in request monitoring exits
22108 Asserted identity support for Multiplatforms C applications making ECI requests
See details at http://www-01.ibm.com/support/docview.wss?uid=swg27037911
44
Customer Requirements Answered in CICS TG V9.0

45
Questions

We love your Feedback!
Don’t forget to submit your Impact session and speaker
feedback!
• Your feedback is very important to us – we use it to improve
next year’s conference
• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):
– Use the session ID number to locate the session
– Click the “Take Survey” link
– Submit your feedback
46

Legal Disclaimer
• © IBM Corporation 2013. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by
you will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and
other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration
purposes only.
47

Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
48

Ibm session tac 2104 - ctg presentation for impact 2013 final

More Related Content

What's hot

Similar to Ibm session tac 2104 - ctg presentation for impact 2013 final

More from Elena Nanos

Recently uploaded

Ibm session tac 2104 - ctg presentation for impact 2013 final