Making It Hard for Those IMS Problems to Hide - IMS UG March 2013 DC_JAX_PA

1,093 views
995 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,093
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Making It Hard for Those IMS Problems to Hide - IMS UG March 2013 DC_JAX_PA

  1. 1. IBM Software Group Making It Hard for Those IMS Problems to Hide Session Number 3262A Jeff Maddix IBM Silicon Valley Lab maddix@us.ibm.com1
  2. 2. Making it Hard for those IMS Problems to Hide Please note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.c IBM Corporation, 2012 2
  3. 3. Making it Hard for those IMS Problems to Hide Making it Hard for those IMS problems to Hide - Agenda Section 1: MVS Setup Recommendations Section 2: IMS Setup Recommendations Section 3: Preservation of Standard Documentation Section 4: Manual Dump Intervention Section 5: Situation Specific Diagnosticsc IBM Corporation, 2012 3
  4. 4. Making it Hard for those IMS Problems to Hide Section 1. MVS Setup Recommendationsc IBM Corporation, 2012 4
  5. 5. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Maximize System Trace Table Size Maximize System Trace Table Size. Advantages: Extremely valuable for a large variety of problem types such at ABENDs, loops, and performance issues. Setup Set MVS System Trace table size to 999K or 3M in z/OS 1.9 or higher –Default size is only 1M –z/OS 1.9 max size is 9G The MVS command "TRACE ST,999K" or “TRACE ST,3M” can be specified in the MVS COMMNDxx SYS1.PARMLIB member. See the z/OS MVS Systems Commands Manual for further detailc IBM Corporation, 2012 5
  6. 6. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Maximize System Trace Table Size Maximize System Trace Table Size. Use the MVS Display command to ensure the system trace table is active: –DISPLAY TRACE IEE843I 13.17.07 TRACE DISPLAY 479 SYSTEM STATUS INFORMATION ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) Considerations: System Trace Table is page fixed storage. Supply a reasonable value to the nnnM or nG parameter after considering the available central storage and the actual storage required for system trace. Supplying a large value for nnnM or nG might cause a shortage of pageable storage in the system.c IBM Corporation, 2012 6
  7. 7. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Increase Master Trace Table Size Increase master trace table size. Advantages: Master trace maintains a table of the most recently issued operator messages. Allows view of external events at the time of failure. Setup Set MVS Master Trace Table size to 1000K –Default size is only 24K "TRACE MT,1000K" can be specified in the SCHEDxx member of SYS1.PARMLIB. See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning Guide/ Ref, and z/OS MVS System Commands manuals for complete detailsc IBM Corporation, 2012 7
  8. 8. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Increase Master Trace Table Size Increase master trace table size Use the MVS Display command to ensure the master trace table is active: –DISPLAY TRACE IEE843I 13.17.07 TRACE DISPLAY 479 SYSTEM STATUS INFORMATION ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) Considerations: Ensure Master Trace Table is large enough to span most error time frames Uses Subpool 229 Key 0 High Private Pageable Storage of the master scheduler address spacec IBM Corporation, 2012 8
  9. 9. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Use the Common Storage Tracker Use the Common Storage Tracker Advantages: SVC dumps (or RMF reports) will provide CSA/ECSA ownership information with jobname, time, and requesting module information. Great for common storage outage (storage leaks) problems and fragmentation analysis. Setup Use the DIAGxx SYS1.PARMLIB member to contain the request. Specify DIAG=xx in the IPL system parameters or use the SET DIAG=xx operator command. –Example: DIAGxx member: VSM TRACK CSA(ON) SQA(ON) See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning Guide/ Reference, and z/OS MVS System Commands manuals for complete details.c IBM Corporation, 2012 9
  10. 10. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Use the Common Storage Tracker Use the Common Storage Tracker Display the Common Storage Tracker status by using the following MVS command: DISPLAY DIAG IGV007I 11.44.05 DIAG DISPLAY 713 VSM TRACK CSA(ON) SQA(ON) VSM TRACE GET(OFF) FREE(OFF) VSM ALLOWUSERKEYCSA(YES) VSM BESTFITCSA(NO) VSM USEZOSV1R9RULES(YES) TRAPS NAME() CBLOC VIRTUAL24() VIRTUAL31() REUSASID(NO) AUTOIPL SADMP(NONE) MVS(NONE) Considerations: Negligible Performance impact. Table stored in ESQAc IBM Corporation, 2012 10
  11. 11. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Tune CHNGDUMP MAXSPACE Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump. Advantages: Higher likelihood that SVC dumps will be captured in their entirety without worry of "Partial Dump". Setup Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump. Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate CHNGDUMP command at IPL time. –Example: CD SET,SDUMP,MAXSPACE=3000M - Default size is 500M 3000M is standard for large multi-address space SVC Dumps. –See z/OS MVS Diagnosis: Tools and Service Aids, z/OS MVS System Commands, and z/OS Initialization and Tuning Guide for additional details. Continued on next pagec IBM Corporation, 2012 11
  12. 12. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Tune CHNGDUMP MAXSPACE Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump. Display the MAXSPACE setting using the following MVS Command: D DUMP,OPTIONS SDUMP- ADD OPTIONS (ALLPSA,NUC,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP, Q=YES,GRSQ,XESDATA),BUFFERS=00000000K, MAXSPACE=00003000M,MSGTIME=99999 MINUTES, MAXSNDSP=015 SECONDS,AUXMGMT=ON ,DEFERTND=NO •Continued on next pagec IBM Corporation, 2012 12
  13. 13. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Tune CHNGDUMP MAXSPACE Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump. Considerations: MAXSPACE is the total virtual storage used by SDUMP SDUMP data capture is serialized so only one dump can be written at a time. However, MVS releases the serialization after data capture, so another dump can be started before the first one gets written out (especially if something causes a delay in writing the first dump out to DASD). So MAXSPACE virtual storage might have to be temporarily "shared" by more than one dump. Ensure central storage and/or local page datasets are large enough to contain their normal peak load plus additional SVC dumps! See MVS Initialization and Tuning Guide for additional details.c IBM Corporation, 2012 13
  14. 14. Making it Hard for those IMS Problems to Hide MVS Setup Recommendations Automatic Dump Dataset Allocation Ensure automatic dump dataset allocation is in place. Advantages: SVC dumps will be allocated to the correct size without worry of "Partial Dump". Setup Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate DUMPDS commands to set this up. –Example: DUMPDS NAME=, DUMPDS ADD, and DUMPDS ALLOC=ACTIVE –See z/OS MVS Diagnosis: Tools and Service Aids and z/OS MVS Systems Commands manuals for complete details. Considerations: Ensure assigned storage class has enough space to ensure adequate SVC dump storage requirementsc IBM Corporation, 2012 14
  15. 15. Making it Hard for those IMS Problems to Hide Section 2. IMS Setup Recommendationsc IBM Corporation, 2012 15
  16. 16. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations Use FMTO Option D Specify IMS control region EXEC parm value: FMTO=D Produces an SDUMP for terminating and non-terminating errors Non-terminating errors include: – IMS dynamic allocation failures. – Some ESAF (External Subsystem Attach Facility) failures. Allows use of SYSMDUMP as a backup dump in case of SDUMP failures.c IBM Corporation, 2012 16
  17. 17. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations SYSMDUMP DD for IMS Control Regions The SYSMDUMP specification will be used as a backup dump by IMS in the event that SDUMP processing should fail. Specify SYSMDUMP DD in JCL of: IMS CTL Region. IMS DLI/SAS Region. IMS DBRC Region. The following dump options should be specified in the SYS1.PARMLIB(IEADMR00) member to ensure that adequate areas of MVS storage are dumped to diagnose the problem under most circumstances: SDATA=(CSA,RGN,SQA,SUM,TRT,NUC,LPA) The SYSMDUMP SDATA options can be added via the CHNGDUMP command CD SET,SYSMDUMP=(CSA,RGN,SQA,SUM,TRT,NUC,LPA)c IBM Corporation, 2012 17
  18. 18. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations SYSMDUMP DD, Continued SYSMDUMP DD Specification DISP=MOD should be specified because it is possible for multiple tasks to generate dumps during the processing of an error. We probably will need to see the first dump produced. GDG Example //SYSMDUMP DD DISP=(MOD,CATLG),DSN=IMSSP05.IBA1IMS.SYSMDUMP(+1), // DCB=(LRECL=4160,RECFM=FB,BLKSIZE=29120),UNIT=3390, // SPACE=(CYL,(750,100),RLSE),DATACLAS=dtcls,VOL=SER=volser Testing the SYSMDUMP function can be done on a test system by temporarily setting the SDUMP to NODUMP via the CHNGDUMP command. This forces a SYSMDUMP to be created. CD SET,SDUMP,NODUMPc IBM Corporation, 2012 18
  19. 19. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations SYSUDUMP DD The SYSUDUMP specification will be used by IMS dependent regions for failure events. Many application dump formatter tools rely on the SYSUDUMP DD statement to be present in the JCL. Specify SYSUDUMP DD in JCL of: IMS Dependent Regions. The following dump options should be specified in the MVS SYS1.PARMLIB(IEADMP00) member to ensure that adequate areas of MVS storage are dumped. SDATA=(CB,ERR,SUM) PDATA=(JPA,LPA,PSW,REGS,SA,SPLS)c IBM Corporation, 2012 19
  20. 20. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations IMS Table Traces Activated Turn on the IMS Dispatcher, Scheduler, DLI, and Lock traces to capture general system diagnostics and database access history Specify the following options in the IMS PROCLIB member: DFSVSMxx DISP=ON, SCHD=ON, DL/I=ON, LOCK=ON Or, use IMS command: TRA SET ON TABLE nnnn Where nnnn is alternately = DISP, SCHD, DLI, and LOCK These traces have relatively low CPU overhead SCHD being the lowest and DLI being the highest. For non-production environments, when recreating failures, run with traces for the functional areas being testedc IBM Corporation, 2012 20
  21. 21. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations External Trace Environment IMS external tracing allows for IMS trace table output to be placed on IMS trace datasets rather than on the IMS OLDS when: DFSVSMxx PROCLIB member with nnnn=OUT is specified rather than nnnn=ON DISP=ON, SCHD=ON, DL/I=OUT, LOCK=OUT The IMS TRACE command specifies OPTION LOG /TRA SET ON TABLE nnnn OPTION LOG Increases IMS system throughput should external tracing be desired versus sending trace data to the IMS OLDS. External trace dataset logging adds less than 1% to the system assuming IO subsystem isn’t constrained External trace datasets are allocated in the following order: 1) DASD JCL - DFSTRA01 and DFSTRA02 DDNAMES 2) DASD MDA- DFSTRA01 and DFSTRA02 Dynamic Allocation Members 3) TAPE MDA - DFSTRA0T Dynamic Allocation Member 4) IMS OLDS - If none of the above are foundc IBM Corporation, 2012 21
  22. 22. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations External Trace Environment IMS external tracing specifications • DFSTRA01 and DFSTRA02 DSORG Sequential RECFM VB LRECL 4004 BLKSIZE 20024 (LRECLx n +4) -Set up unique GDG with - minimum 24 hour limit for archive - ideal to last IMS restart or longer hlq.imsid.TRACE.ARCHIVE(0) -Use automation to archive filled trace dataset off to trace archive GDG datasets on external trace dataset switch DFS2864I EXTERNAL TRACE DATA SET xxxxxxxx FULL - SWITCHING TO yyyyyyyy. imsid - and for IMS ABEND Scenario – DFS3896I ATTEMPTING TO PURGE OLDS BUFFERSc IBM Corporation, 2012 22
  23. 23. Making it Hard for those IMS Problems to Hide IMS Setup Recommendations External Trace Environment Display IMS external tracing specification Command: /DIS TRACE XTRC. IMS ACTIVE TRACES IMS EXTERNAL TRACE IS TRACING TO DISK XTRC DDNAME ALLOC STATUS STATUS DFSTRA01 DYNA ALLOCATED CLOSED DFSTRA02 DYNA ALLOCATED ACTIVE DFSTRA0T DYNA UNALLOCATED CLOSEDc IBM Corporation, 2012 23
  24. 24. Making it Hard for those IMS Problems to Hide Section 3. Preservation of Standard Documentationc IBM Corporation, 2012 24
  25. 25. Making it Hard for those IMS Problems to Hide Preservation of Standard Documentation Consider implementing normal operating procedures to preserve the following documentation near the time of error. MVS Console (SYSLOG) SYS1.LOGREC IMS OLDS/SLDS JES JOBLOG of jobs related to failure Any Dumps Produced For elusive problems, these items might be requested a month or more after the problem has been reported.c IBM Corporation, 2012 25
  26. 26. Making it Hard for those IMS Problems to Hide Ideal Time Periods for Syslog, Logrec, SLDS, and Joblog Preservation MVS Console (SYSLOG), Logrec, SLDS and Joblogs –The ideal time frame: Back to the last IMS restart. MVS Console from the prior clean execution (For comparison) The moderate time frame: 24 hours of MVS Console messages. The minimum time frame: Two IMS system checkpoint intervals.c IBM Corporation, 2012 26
  27. 27. Making it Hard for those IMS Problems to Hide Dump Preservation All dumps created near the time of IMS failure should be retained. SYS1.DUMP Datasets should be examined. Multiple dumps may be created. Keep all dumps at time of failure regardless of the subsystem. SYSMDUMP for the IMS Control, DLI/SAS, and DBRC regions need to be examined in case of primary SYS1.DUMP failures. Save these datasets if a dump was produced. If GDG is used for copies, examine in ISPF for tracks used SYSUDUMP should be saved for affected IMS dependent regions.c IBM Corporation, 2012 27
  28. 28. Making it Hard for those IMS Problems to Hide Section 4. Manual Dump Interventionc IBM Corporation, 2012 28
  29. 29. Making it Hard for those IMS Problems to Hide Additional Manual Dump Intervention While IMS produces SDUMPs for most internal errors without human intervention, IMS Wait/Loop or partial loss of function conditions will require intervention to produce an SVC dump IMS hangs can be caused by interaction with many address spaces including: CTL DLI/SAS DBRC IRLM CQS MQ IMS (DBDC, Connect Series * DBCTL, DCCTL) Subsys * ODBA IBM OM SCI RM ODBM * DB2 Tools * * Subsys * ap thr Non- MPP JMP JBP cctl IBM IFP BMP * * * * * * Tools ap thr ap thr ap thr ap thr ap thr ap thr IXGLOGR TCPIP Other? RRS APPC VTAM WLMc IBM Corporation, 2012 29
  30. 30. Making it Hard for those IMS Problems to Hide How Much Data to Dump? Due to the complex interactions between the many address spaces, it is sometimes difficult to determine exactly where the source of the problem lies without a dump of the associated address spaces and their associated data areas. Omission of any interrelated address space or data area adds to the possibility that the dump may not be sufficient to solve the problem. MVS dumps have a maximum job total of 15 per dumpc IBM Corporation, 2012 30
  31. 31. Making it Hard for those IMS Problems to Hide IMS Dump Techniques IMS SVC Dumps can be requested using various techniques MVS SYS1.PARMLIB IEADMCxx DUMP Command Parmlib MVS SYS1.PARMLIB IEASLPxx SLIP Command Parmlib Manual MVS DUMP command Customized JCL can be built and submitted Others?....c IBM Corporation, 2012 31
  32. 32. Making it Hard for those IMS Problems to Hide IEADMCxx, MVS SYS1.PARMLIB DUMP Command Parmlib Member Create SYS1.PARMLIB members called IEADMCxx for each customized dump command. Advantages – Can be used to customize IMS dumps prior to error event. – Simple operator interface. Disadvantages – All Jobs must be active or no dump is produced on the system on which the dump command is entered. Remote systems without a jobname match will dump the DUMPSRV address space. – Avoid issues: –Use wildcards * (multiple chars) and ? (single char) and similar naming conventions can be used to specify the jobnames. –Edit the Parmlib member jobnames prior to issuing the command See z/OS MVS Initialization and Tuning Guide and z/OS System Commandsc IBM Corporation, 2012 32
  33. 33. Making it Hard for those IMS Problems to Hide IEADMCxx Example for IMS Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters: JOBNAME=(j1,j2,j3,j4,j5,j6,j7), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ) Where: j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname. j3 = IMS DLI Region Jobname. Jobname. j4 = IRLM Region Jobname. j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2 j7 = IMS CCTL Region 3. Note: It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down. Example: Using the following naming conventions, the JOBNAME statement could be reduced: IMSPCTLA, IMSPDLIA, IMSPDRCA IMSPRLMA CICSP1A CICSP2A CICSP3A JOBNAME=(IMSP*A,CICSP?A)c IBM Corporation, 2012 33
  34. 34. Making it Hard for those IMS Problems to Hide IEADMCxx DUMP Activation To request a dump from the IEADMCI1 parmlib member, enter the following MVS commands: DUMP TITLE=(DUMP OF IMS and CCTL Regions ),PARMLIB=(I1) A dump dataset will be created on the MVS image from which the dump command is enteredc IBM Corporation, 2012 34
  35. 35. Making it Hard for those IMS Problems to Hide IMS SYSPLEX Dump Considerations IMS SYSPLEX implementations need to consider the possibility that a hang or problem on one IMSplex member may be due to a problem originating from another member, particularly in the area of IRLM locking Problems such as IMS Wait/Loops or partial loss of function conditions which require intervention to produce an SVC dump, should include SVC dumps from other members of the IMSplex Challenge: Ensure that a dump is taken for all necessary address spaces on each system.c IBM Corporation, 2012 35
  36. 36. Making it Hard for those IMS Problems to Hide Sysplex IEADMCxx Example Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters: JOBNAME=(j1,j2,j3,j4,j5,j6,j7), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE), DSPNAME=(j4.*,XCF*.*), REMOTE=(SYSLIST=(*(j1,j2,j3,j4’,’j5’,’j6’,’j7’), SDATA,DSPNAME)) Where: j1 = IMS Control Region Jobname j2 = DBRC Region Jobname j3 = IMS DLI Region Jobname j4 = IRLM Region Jobname j5 = CCTL Region 1 J6 = CCTL Region 2 j7 = CCTL Region 3 It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down. Note: The XESDATA, COUPLE and REMOTE parms are for use in SYSPLEX environmentsc IBM Corporation, 2012 36
  37. 37. Making it Hard for those IMS Problems to Hide Sysplex IEADMCxx DUMP Activation To request a dump from the IEADMCI1 parmlib member, enter the following MVS command: DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=(I1) A dump dataset will be created on each MVS image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs.c IBM Corporation, 2012 37
  38. 38. Making it Hard for those IMS Problems to Hide Storage as a Moving Target Storage can be a moving target while being dumped, especially if multiple address spaces interact. Here are some considerations: If the MVS sdump CHNGDUMP SET, SDUMP=(Q=Y) command , is specified, all address spaces that are not exempt from system-wide nondispatchability are set nondispatchable while global storage (CSA, SQA) is dumped. –By quiescing most system activity during SVC dump global storage capture, the dump will contain a more consistent image however, your system activity will nearly cease during global storage capture. – This option is generally not recommended for this reason.c IBM Corporation, 2012 38
  39. 39. Making it Hard for those IMS Problems to Hide Storage as a Moving Target, Continued Storage can be a moving target, continued… •In all cases (Q=Y or Q=N), dumped address spaces are made nondispatchable while global storage and the private region storage is being dumped for those address spaces There is no intentional dispatching coordination between requested address spaces in a dump. All private storage is dumped through scheduling of SRBs to the requested address spaces. This SRB activity is asynchronous. Multiple SRBs to multiple address spaces will run independently and loosely in parallelc IBM Corporation, 2012 39
  40. 40. Making it Hard for those IMS Problems to Hide Section 5. IMS Situation Specific Diagnosticsc IBM Corporation, 2012 40
  41. 41. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Control Region Wait/Hang Control Region Wait/Hang Take a SVC dump using one of the methods discussed earlier in the presentation. Consider dumping related regions IMS Control Region Troublesome IMS Dep Regs TCPIP IMS DLI/SAS Region CCTL Regions Websphere DBRC Region ODBA ESAF - DB2, MQSeries, others IRLM Region IXGLOGRC Other Regions ??? CQS RRS Other IMSplex members with Operations Mgr APPC all their related regions Resource Mgr WLM ODBM Structure Call Interface VTAM = Cant do without If a partial wait occurs, consider taking two of the above mentioned SVC Dumps to identify normal activity versus abnormal activity. Note: Modify dumps of IMS are no longer the preferred method to obtain wait documentation.c IBM Corporation, 2012 41
  42. 42. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Control Region Wait/Hang, Continued IEADMCxx Parmlib (IEADMCA1) example for a slow moving performance problem or hang in a DBCTL data sharing environment with IRLM JOBNAME=(isa?ims,isa?dls,isa?irlm,ci1?sa??), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE), DSPNAME=(ISA?IRLM.*,XCF*.*), REMOTE=(SYSLIST=( SP1A(isa?ims,isa?d*,isa?irlm,ci1?sa??), SP1C(isa?ims,isa?d*,isa?irlm,ci1?sa??), SP1E(isa?ims,isa?d*,isa?irlm,ci1?sa??), SP1F(isa?ims,isa?d*,isa?irlm,ci1?sa??), SDATA,DSPNAME)) Dump Command to activate the above dump: DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=(A1) One dump will be created on each of the specified sysplex membersc IBM Corporation, 2012 42
  43. 43. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Application System Timeout Application lockouts are detected by IRLM and MSG DXR162I is issued. DXR162I irlmx CYCLE NUMBER nnnnnnnn PROCESSED FOR TIMEOUT Explanation: A TIMEOUT candidate has been detected on at least one IRLM. If an incompatible blocker was found, the IMS TIMEOUT EXIT has been driven to present WAITER/BLOCKER information Set the following SLIP to capture a dump for this event SLIP SET,A=(SVCD),MSGID=DXR162I, SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE), JL=(ISA*,CI1?SA??),ID=M162,DSPNAME=(ISA?IRLM.*,XCF*.*), REMOTE=(SYSLIST=(SP1A,SP1C,SP1E,SP1F),JOBLIST,DSPNAME,SDATA), ENDc IBM Corporation, 2012 43
  44. 44. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: IMS Loop If the IMS Control, DLI, DBRC or Dependent Region enters a loop state: Turn on the MVS System Trace Table with BR=ON TRACE ST,3M,BR=ON - (size greater than 1m in z/OS 1.9 and above) If possible, and not already set, use IMS command: /TRA SET ON TABLE nnnn Where nnnn alternately = DISP, SCHD, DLI, LOCK, and LATCH. Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions. Note: IMSplex partner dumps are probably not required for loop problems, unless they are also looping. Reset the MVS system trace table and IMS trace tables to their original settings.c IBM Corporation, 2012 44
  45. 45. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DBRC Related Problems DBRC Related Problems Obtain a listing of the DBRC RECONs prior to repair attempts. Use Recover Control Utility (DSPURX00) - LIST.RECON command. If the RECON dataset listing cannot be obtained due to size, then alternatively obtain a subsystem listing (LIST.RECON is preferred). Use Recover Control Utility (DSPURX00) - LIST.SUBSYS ALL commandc IBM Corporation, 2012 45
  46. 46. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DBRC Related Problems, Continued DBRC Related Problems, Continued Use D GRS,CONTENTION command on each system sharing the RECON to determine if the RECON is held at the exclusion of other waiters D GRS,CONTENTION S=SYSTEMS DSPURI01 RECON1.IMSBP1 SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS BT21 IBA2FDBR 0124 009AF7F8 EXCLUSIVE OWN BT23 IBA1FDBR 0090 009AF1B0 EXCLUSIVE WAIT BT21 IBA4FDBR 05E0 009AF7F8 EXCLUSIVE WAIT BT26 IBA4DBRC 05DE 009F15B8 EXCLUSIVE WAIT BT21 IBA1DBRC 00D7 009FF118 EXCLUSIVE WAIT BT23 IBA2DBRC 0600 009F15C8 EXCLUSIVE WAIT If so, dump the owning address space DUMP COMM=(dump title) R nn,JOBNAME=(j1),SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ),ENDc IBM Corporation, 2012 46
  47. 47. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DBRC Related Problems, Continued Obtain before and after copies of the RECONs JCL Example for Recon dump to file //JJJJJJJJ JOB ( ) //* DUMP RECON TO FILE //STEP1 EXEC PGM=ADRDSSU,REGION=0M //SYSPRINT DD SYSOUT=* //DASD DD DSN=hhhhhhhh.RECON.BACKUP,DISP=(NEW,CATLG), // UNIT=3390,SPACE=(CYL,(5,5),RLSE) //SYSIN DD * DUMP DS(INCLUDE(IMS.RECON*)) - OUTDD(DASD) /*c IBM Corporation, 2012 47
  48. 48. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DBRC Related Problems, Continued JCL Example for Recon restore from file //JJJJJJJJ JOB ( ) //* TO DOWNLOAD TAPE LOADED WITH DFSDSS //* NEED TO USE RENUNC TO RENAME TO YOUR HLQ //* TAPE MAP WILL NOT SHOW LRECL OR BLKSIZE.... //S01RSTR EXEC PGM=ADRDSSU //SYSPRINT DD SYSOUT=* //IN DD DISP=SHR,DSN=hhhhhhhh.RECON.BACKUP //DASD DD UNIT=3390,DISP=OLD,VOL=SER=SYS285 //SYSIN DD * RESTORE DATASET(INCLUDE( ** )) - INDDNAME(IN) - OUTDDNAME(DASD) - CANCELERROR - CATALOG - REPLACE - IMPORT - TGTALLOC(SOURCE) - RENAMEUNCONDITIONAL(hhhhhhhh) - WAIT(2,2) /*c IBM Corporation, 2012 48
  49. 49. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DBCTL Related Problems DBCTL Related Problems Issue IMS /DISPLAY ACTIVE and /DISPLAY CCTL commands, multiple times /DIS ACTIVE /DIS CCTL ALL /DIS CCTL cctlname Set the AP portion of the CICS trace to level 1-2 Set the FC (FILE CONTROL) portion of the CICS trace to level 1-2 Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions, especially the CCTL regionsc IBM Corporation, 2012 49
  50. 50. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Database Problems For problems in the database arena, obtain Damaged database dataset Database Image copy of broken database in state prior to damage Image Copy of logically related databases IMS SLDS from all data sharing IMS subsystems Save to last good database image copy of damaged database If possible, and not already set, use IMS command and save output: /TRA SET ON TABLE nnnn OPTION LOG – Where nnnn alternately = DLI and LOCK SYSOUT from the Pointer Checker broken database SYSOUT from batch jobs that accessed the damaged databasec IBM Corporation, 2012 50
  51. 51. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Database Problems, Continued For problems in the database arena, obtain (Cont...) DBD for broken databases PSBs that access the broken DB List.RECON and LIST.HISTORY DBD(broken database) SMF 60 (VSAM Volume Data Set Updated), 62 (VSAM Component or Cluster Opened), and 64 (VSAM Component or Cluster Status) records from all data sharing systems Back to last good image copy of broken databasec IBM Corporation, 2012 51
  52. 52. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: Database Problems, Continued For problems in the database arena, obtain (continued) For VSAM datasets, close the broken KSDS and run IDCAMS commands: VERIFY EXAMINE ITEST NODTEST EXAMINE NOITEST DTEST LISTCAT ALL If the dataset cannot be closed, run IDCAMS commands: EXAMINE ITEST NODTEST EXAMINE NOITEST DTEST EXAMINE ITEST NODTEST EXAMINE NOITEST DTEST LISTCAT ALL If the data set is open and being updated then an EXAMINE may reflect what appears to be a problem because the data set is in a state of flux. Thus if you run the EXAMINEs twice and the errors are the same, there is probably a problem. If the errors are different, then there is no problem.c IBM Corporation, 2012 52
  53. 53. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: ESAF Interface Problems Problems involving the ESAF (External Subsystem Attach Facility) Turn on the IMS SUBS TRACE to the external trace dataset ƒ IMS command: /TRA SET ON TABLE SUBS OLDS Take a SVC dump of the related regions before and after the recreate attempt. Ensure that pertinent regions for the affected subsystem (db2mstr, db2dbm1, mqseries) are included with the other related IMS regions Note: IMSplex partner dumps are probably not required for ESAF problems, unless they are also experiencing the problem.c IBM Corporation, 2012 53
  54. 54. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: DC Related Problems Problems involving Data Communications Issue IMS DISPLAY NODE x command. /DIS NODE nodename Turn on IMS NODE trace. Data will be captured in the IMS TYPE6701 log record. Print records using IMS utility programs DFSERA10/DFSERA30. /TRA SET ON NODE nodename Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS NODE trace. Note: GTF must be active with the USR option. F NET,TRACE,TYPE=BUF,ID=nodename F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG) Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM.c IBM Corporation, 2012 54
  55. 55. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: APPC Related Problems Problems involving APPC Turn on the IMS LUMI Trace to the external trace dataset. /TRACE SET ON TABLE LUMI OPTION LOG /TRACE SET ON LUNAME XXXXXXX INPUT /TRACE SET ON LUNAME XXXXXXX OUTPUT – (XXXXXXX is the partner LU) Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS LUMI trace. Note: GTF must be active with the USR option. F NET,TRACE,TYPE=BUF,ID=nodename F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG) Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM and the APPC regions.c IBM Corporation, 2012 55
  56. 56. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: APPC Related Problems, Continued Problems involving APPC (cont...) Turn on the program trace for TPPCB DL/1 calls. /TRACE SET ON PROGRAM pppppppp – pppppppp is the program name of the application. Turn on the MVS APPC component trace. TRACE CT,ON,200M,COMP=SYSAPPC – nn,OPTIONS=(GLOBAL),END When the problem has been recreated, stop the CTRACE. TRACE CT,OFF,COMP=SYSAPPC APPC component trace sends its trace buffers to a SYS1.DUMP dataset. The following IPCS commands can be used to format the trace: CTRACE COMP(SYSAPPC) SHORT - One line summary entries CTRACE COMP(SYSAPPC) FULL - Detailed trace entryformattingc IBM Corporation, 2012 56
  57. 57. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS Additional Manual Dump Intervention IMS/CQS environment additional dump considerations include: Structure dumps – Primary, Overflow, Lock entries CQS Region Other CQS Clients with their related CQS regions Above regions from the other IMSplex members MVS Loggerc IBM Corporation, 2012 57
  58. 58. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - IEADMCxx Example with Structures Create two SYS1.PARMLIB members: SYS1.PARMLIB IEADMCIA – IMS Partners JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), DSPNAME=(‘ DSPNAME=(‘j4.*,XCF*.*),REMOTE= (SYSLIST=(*(j1,j2,j3,j4,j5‘ j6’ j7’ j8’ j9’ (SYSLIST=(*(j1,j2,j3,j4,j5‘,’j6’,’j7’,’j8’,’j9’), SDATA,DSPNAME)) Where: j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j5 = IMS CQS Region j6 = APPC Region j7 = APPC Scheduler j8 = VTAM j9 = Other CQS Client Region It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down.c IBM Corporation, 2012 58
  59. 59. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - IEADMCxx Example with Structures SYS1.PARMLIB member for structures and MVS logger SYS1.PARMLIB IEADMCIB JOBNAME=(IXGLOGR),DSPNAME=(IXGLOGR.*), SDATA=(ALLNUC,LPA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA,COUPLE), STRLIST= imsmsgq01,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), (STRNAME=imsmsgq01 (STRNAME=imsmsgq01,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), STRNAME=imsmsgq01oflw imsmsgq01oflw,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER) STRNAME=imsmsgq01oflw,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER) , STRNAME=mvslogqmsg01 mvslogqmsg01,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER, STRNAME=mvslogqmsg01,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER, ADJ=CAPTURE)) Where: imsmsgq01.................is the main structure name imsmsgq01oflw.............is the overflow structure name mvslogqmsg01..............is the associated logger structure structurec IBM Corporation, 2012 59
  60. 60. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - IEADMCxx DUMP Activation To request a dump from the IEADMCIA and IEADDMCIB parmlib members, enter the following MVS command: DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB) Partners/Structures),PARMLIB Structures),PARMLIB=(IA,IB) One combined dump dataset will be created on the MVS image from which the command is entered. One dump dataset will be created on each image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs for PARMLIB member IA. NOTE: MVS logger support only needs the MVS logger address space from the system which experiences the problem.c IBM Corporation, 2012 60
  61. 61. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - MVS Log Stream The merged MVS log stream can be used to examine CQS log records. IEBGENER can be used along with the default log stream subsystem exit routine, IXGSEXIT, to copy the log records at time of failure for later analysisc IBM Corporation, 2012 61
  62. 62. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - MVS Log Stream, JCL Example //CQSCPYLG JOB USERID,USERID,MSGLEVEL=1,CLASS=K //********************************************************************* //* This job copies a CQS log stream to a dataset (max 32K / record) * //* * //* - Replace the DSN on the SYSUT1 card with your CQS logstream * //* name. * //* * //* - Replace the DSN on the SYSUT2 card with your desired output * //* dataset name. You may also need to adjust the space * //* allocations, depending on the size of your logstream. * //* * //* Changes: * //* 02/24/97 ben Creation * //********************************************************************* //STEP1 EXEC PGM=IEBGENER,REGION=1024K //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSIN DD DUMMY //SYSUT1 DD DSN=SYSLOG.MSGQ01.LOG, // SUBSYS=(LOGR,IXGSEXIT), // DCB=(BLKSIZE=32760) //SYSUT2 DD DSN=CQS.LOG.COPY, // DISP=(NEW,KEEP,DELETE), // VOL=SER=USER05, // SPACE=(CYL,(2,10)), ƒ // UNIT=SYSDAc IBM Corporation, 2012 62
  63. 63. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CQS - SRDS Save the most recent CQS SRDS (Structure Recovery Data Set) for each structure dumped Use IDCAMS repro if LRECL is acceptable (less than 32761) If LRECL greater that 32761, use DFSMS ADRDSSU utilityc IBM Corporation, 2012 63
  64. 64. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: MSC Related Problems If you use IMS MSC (Multiple Systems Coupling) and experience a related problem Take a SVC dump of the coupled IMS regions (minimally the CTL regions, but the problem may reside in any IMS related region as discussed earlier) In addition, ensure that the VTAM and/or TCPIP address space is included. Do this as close to the problem time as possible, prior to attempting to fix the problem Save the IMS OLDS for both coupled systems from the time of the message creation Issue DISPLAY LINK ALL and DISPLAY LINK ALL MODE commands to show the status and queue counts for the logical link and physical link information for partnerc IBM Corporation, 2012 64
  65. 65. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: MSC Related Problems, Cont. MSC related problems, continued If the problem is recreatable, If using VTAM, Turn on the VTAM Internal Trace for both coupled systems – F NET,TRACE,TYPE=VTAM,OPT=(API,PIU,MSG),DSPSIZE=5,SIZE=999 Using these options, the VIT will be created in a VTAM dataspace. When the problem has been recreated, the dump parameters should also include the VTAM dataspace: – DSPNAME=(NET.ISTITDS1) The MSC LINK trace should also be turned on for both coupled systems (link# or ALL is acceptable for the link specification) – TRACE SET ON LINK link# LEVEL 4 MODULE ALL IMS TYPE6701 records containing TM control blocks for each message at key pointsc IBM Corporation, 2012 65
  66. 66. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CTL/DLI Region “Out of Storage” IMS control region “out of storage” conditions (ABEND80A ABEND878) can result in ABEND40D (Recursive errors in MVS RTM) • Problem: Unable to diagnose problem because the dump dataset doesn’t contain private storage for the failing region. • Solution: Set the following four SLIP traps to capture the initial “out of storage” prior to RTM gaining controlc IBM Corporation, 2012 66
  67. 67. Making it Hard for those IMS Problems to Hide Situation Specific Diagnostics: CTL/DLI Region “Out of Storage”, Continued Enter the following two SLIP traps for both the IMS Control and IMS DLI/SAS regions (4 slips total) SET,C=80A,J=jobname jobname, SLIP SET,C=80A,J=jobname, SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA), ID=xxxx,END ID=xxxx,END SET,C=878,J=jobname jobname, SLIP SET,C=878,J=jobname, SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA), ID=xxxx,END ID=xxxx,END Where jobname = Alternatively, Jobname of the IMS CTL region, = Jobname of the IMS DLI/SAS regionc IBM Corporation, 2012 67
  68. 68. Making it Hard for those IMS Problems to Hide Making Hard for those IMS Problems to Hide - Summary MVS Setup Recommendations IMS Setup Recommendations Preservation of Standard Documentation Manual Dump Intervention Situation Specific Diagnosticsc IBM Corporation, 2012 68
  69. 69. Making it Hard for those IMS Problems to Hide Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. © Copyright IBM Corporation 2012. All rights reserved. – U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.com, IMS, MVS, z/OS, DB2, CICS, and WebSphere are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml. Other company, product, or service names may be trademarks or service marks of others.c IBM Corporation, 2012
  70. 70. Making it Hard for those IMS Problems to Hide Communities • On-line communities, User Groups, Technical Forums, Blogs, Social networks, and more – Find the community that interests you … • Information Management bit.ly/InfoMgmtCommunity • Business Analytics bit.ly/AnalyticsCommunity • Enterprise Content Management bit.ly/ECMCommunity • IBM Champions – Recognizing individuals who have made the most outstanding contributions to Information Management, Business Analytics, and Enterprise Content Management communities • ibm.com/championc IBM Corporation, 2012 70
  71. 71. Making it Hard for those IMS Problems to Hide Thank You! Your Feedback is Important! • Access SmartSite to complete your session surveys – Any web or mobile browser at iodsmartsite.com – SmartSite applications for iPhone, iPad, Droid and Blackberry – Any SmartSite kiosk onsitec IBM Corporation, 2012 71

×