• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
DB2 Connect to DB2 z/OS - Problem Determination
 

DB2 Connect to DB2 z/OS - Problem Determination

on

  • 2,031 views

 

Statistics

Views

Total Views
2,031
Views on SlideShare
2,031
Embed Views
0

Actions

Likes
0
Downloads
57
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    DB2 Connect to DB2 z/OS - Problem Determination DB2 Connect to DB2 z/OS - Problem Determination Document Transcript

    • Session: B02 DB2 Connect to DB2 z/OS – Problem Determination & Performance Tips & Tricks Melanie Stopfer IBM North America Lab Services mstopfer@us.ibm.com 13 October 2008 • 16:15 – 17:15 Platform: DB2 z/OS Title: DB2 Connect to DB2 z/OS - Problem Determination & Performance Tips & Tricks Abstract: Learn how to resolve problems that arise in your DB2 Connect to DB2 z/OS environment. By using the right tools, develop a methodology to isolate and resolve problems and improve your ability to meet your On Demand business requirements. Attend this session and learn how to prepare yourself to successfully perform problem determination in your DB2 Connect to DB2 z/OS environment. Melanie will include tips and tricks that she has learned from customer experiences. 1
    • Objectives • Describe data flow within a DB2 Connect environment and develop a problem determination methodology • Perform problem determination analysis using tools available on DB2 Client • Capture, parse and examine CLI and JCC Traces, configure DB2 Connect to improve performance, and analyze Snapshot DCS monitor output • Run and interpret network commands and use DB2 commands to perform problem determination • Analyze accounting, statistics and performance trace output to identify and resolve connection and performance problems 2 Outline: I. Preparing for Problem Determination Distributed DB Environment - Data Flow DBM CFG Settings DB2 Registry Variable Settings ZPARM Settings II. Problem Determination - Where's the Problem? Methodology for Isolating Problem's Source Failure to Connect Checklist III. Monitoring and Flow Details Which Tool to Use? IV. Analyzing with Client Tools Diagnostic Log Configurations Analyzing CLI Trace Output Analyzing JDBC/Java Trace Information V. Analyzing with DB2 Connect Tools List DCS Applications Extended Snapshot DCS Applications DRDA Trace 2
    • Problem Determination – Where’s the Problem? DDF? VTAM? Application? z/OS System? Network? DB2 Connect? DB2 Subsystem? Client System? UNIX System Services? LUW System? DB2 Run-Time? RACF? DRDA? TCP/IP? 3 A typical problem description in a DB2 Connect environment is “It just doesn't work”. Data flows through a large number of components, all of which are required to support an application. You should be aware of the various components that make up a DB2 Connect environment. Problems can occur anywhere along the data flow path. There is no problem determination tool that can look at or analyze the global picture. Each component has its own logs and monitoring tools. As the system administrator, you need to know what components are in the data flow path, what logs to go to and what tools are available to monitor or trace a particular component. So how do we approach problems in this environment? 3
    • Distributed Database Environment – TCP/IP Client DB2 Connect z/OS Instance DDF DB2 CLI/ODBC DB2 Run-Time Application UNIX System db2agent TCP/IP TCP/IP TCP/IP TCP/IP Services DBAT Driver DB2 DBM Database db2cli. DBM Database BSDS ZPARMS CFG Directory ini CFG Directory Node Node Directory Directory DCS Directory 4 DB2 Connect operates in a distributed environment consisting of multiple components. Each component provides some required function in moving information between clients and servers. Each component is critical to the successful flow of data. When performing problem determination or performance studies, understanding what components data must flow through is critical for a successful resolution. Starting with the client system, SQL requests flow from an application through some networking protocol stack and onto the physical network. The data flows to a DB2 Connect system using DRDA protocol format and is handled by an agent task running under a DB2 Connect instance. The data flows into a z/OS system and is received by the DDF component of DB2 for z/OS, where any necessary codepage conversion is done. As of DB2 for z/OS V8, all SQL is parsed in UNICODE (CCSID 1208). Then the connection is assigned to a thread in the DBM1 address space. The thread is responsible for executing your SQL request and returning a response. 4
    • Preparing for Problem Determination DBM CFG – DIAGLEVEL/NOTIFYLEVEL = 4 – DFT_MON_STMT = ON – DFT_MON_UOW = ON DB2 Registry Variable – DB2CONNECT_IN_APP_PROCESS = NO Allows Sysplex, Monitoring, Connection Concentrator ZPARMS – EXTSEC = YES – SMFACCT = 1,2,3,7,8 – SMFSTAT = YES (Default) 5 On average, each additional EDU agent takes 3MB memory in a process model environment. ExtSEC is set to YES by default and allows detailed security reason codes to be returned to DRDA and passwords to be changed. Only two monitor switches are used by DRDA. DB2 starts accounting classes 1, 2, and 3 automatically. The statistics trace is a system wide trace. Because many (or most) problems that you will encounter are unintentional, unplanned, unlikely, unanticipated, and so forth, they always seem to come at the most inopportune times, when you least expect it and can least afford it. So the more you can be prepared for problem situations, the sooner you can identify the problems, bring appropriate problem tools to bear to isolate the problem, and resolve the problem. Many problem determination and analysis tools, like monitors and traces, are very effective in locating and resolving problems. However, by their very nature, they may be very resource intensive. Starting them automatically and running them all the time may put an additional load on the system, which could be counterproductive, and mask problems or lead to additional problems. There are some things you can do to customize or alter the configuration of your system, however, which will probably not use a great deal of additional resources, and at the same time improve your ability to perform problem analysis more quickly and effectively. On your DB2 Connect system, you can make sure the following settings are in place: • In the DBM CFG, make the following updates: - NOTIFYLEVEL=4 5
    • Failure to Connect Missing Link F t DD nec Con DB2 Application cannot connect to database db2 connect to ... 6 Missing link between DDF and DB2 Connect is often the reason for a connection failure. If a DB2 Connect server configuration is being used, the missing link could also be between the remote client system and the DB2 Connect server. If one thinks in terms of a DB2 Connect server perspective, the inbound link is between the remote client and DB2 Connect server. The outbound link is between DB2 Connect server and DDF. 6
    • Failure to Connect Checklist Is DRDA AS active? • DDF started • DDF LU active to VTAM Is TCP/IP correctly configured? • IP addresses and host names • Routers operational? • Service name map to correct port numbers? Is LOGON info correct? • User name • Password • DRDA AS password expired • User name been disabled on DRDA AS Check z/OS console/master logs for DSN* messages Check diagnostic/error logs for messages Run DRDA trace on DB2 Connect Server Is DB2 Connect info correct? • Node, System Database, and DCS directories 7 FYI: Even if this is a TCP/IP only environment, DDF must have a LU defined under VTAM and that LU must be active. Problem Isolation Checklist: - Make sure DDF is started on the DB2 for z/OS DRDA AS. - Make sure the DDF APPL LU is active under VTAM - even if this is a TCP/IP only environment, DDF must have an LU defined under VTAM and the LU must be active. - If using TCP/IP: • Is TCP/IP properly configured on the host and at the DB2 Connect system? • IP addresses and host names are correct? • Routers are operational and a path exists between the host and DB2 Connect system? • Service names used are correct and map to correct port numbers? - Verify Database, Node, and DCS directory entries on DB2 Connect Server. - Verify database and node directory entries on client. - Check the z/OS console log or DB2 for z/OS master log for DSN messages. - Check error logs for messages. - Run a DRDA trace on the DB2 Connect Server. - Try it locally. - Deliberately crash each component. • DB2 Connect message identifiers consist of a three-character message prefix 7
    • DB2 Connect Correlations Worksheet Using TCP/IP Node Directory Linux/UNIX/Windows z/OS Services file TCPIP.DATA Node Service name: DOMAINORIGIN db2aserv etchosts 446 Hostname Port number: : HOSTNAME Service name Hosts file or DNS tcpip.ETC.SERVICES Hostname: Hostname: drda IP /tcp Protocol TCPIP IP address: address: tcpip.HOSTS.LOCAL or DNS System DB Directory HOST : : TCPPARMS(PROFILE) Alias PORT TCP DB2ADIST Database PORT TCP DB2ADIST HOME TR1 Node VTAM DCS Directory APPL LUNAME= Local Database Target Database DB2 for z/OS BSDS DB2 CONNECT TO LOCATION LUNAME PORT RPORT 8 This graphic provides a correlation chart of z/OS, DB2 for z/OS, DB2 Connect, and client parameters. These parameters are required to establish connectivity between DB2 Connect and a DB2 for z/OS subsystem. It also provides students with a shell to document their own parameter values that need to match. 8
    • Problem Determination - Methodology Client Narrow Scope of Problem! DB2 Connect z/OS 9 The first step is to start isolating the problem to as few components as possible. A good starting point is to see if the problem is with only one application on one workstation or if the problem is widespread. If the problem is only on one workstation or only with one application, then you've narrowed the scope of the problem. If the problem is widespread, then you need to start looking at facilities being used by all users, such as the network, the DB2 Connect server, or the z/OS system. If there is a connection problem, make sure: 1. You can ping/aping from the DB2 client to the DB2 Connect system and the DB2 Connect system to the z/OS system. 2. You can connect from the DB2 Connect system. 3. The DB2 Connect system is up and the instance started. 4. The DDF address space is started. 5. The DB2 for z/OS subsystem is started. If the problem is not a connection problem: 1. Check whether any maintenance was applied or changes made to the DB2 for z/OS subsystem. 2. Try to determine what assumptions were made by the application developers and retest the assumptions. 9 3. If the problem cannot be isolated, gather your results and contact the IBM Support
    • Data Flow - Client Client System DB2 Instance Network Protocol DB2 Run-Time Client Adapter LAN Application DB2 Driver DBM db2cli.ini CFG Database Node Directory Directory 10 So lets start isolating the problem – starting with the data flow through the client. 10
    • Monitoring and Flow Summary Client DB2 Connect z/OS A O D D p D B B T T T 2 C T p B 2 C C l C P C R P P U i C U / P c L / Agent / S DDF DB2 a N I / I P I I S t / T I i I P P thread J P o C M n C E db2 ping ping ping Class 1 Class 2 snapshot Nonnested jcctrace CPU - Non- nested clitrace network analyzer network analyzer Class 2 Elapsed Nonnested (In db2drdat CPU Appli- jcctrace cation) snapshot snapshot clitrace (In DB2) Class 1 Elapsed - Class 2 Nonnested Elapsed Class 1 Elapsed 11 The key to solving the problem is knowing where time is spent. Is it spent in the application? In the network? On the client? In DB2 Connect? In DDF? In DB2 z/OS? Well let’s look at what traces are available to help us isolate the problem and know who to point the finger at! Whose to blame? This is a great chart to show you all your options. This graphic summarizes all the performance monitoring tools discussed previously and graphically shows what each tool is measuring. Normally, the best place to start a performance analysis is on the client system where the application is executing. Using the CLI and JCC Traces and trace parser facilities, elapsed times can be broken down between In Application time and In DB2 time. This information can direct you towards looking at performance issues in the application or outside the application (in the DB2 Client code, network, DB2 Connect, or DB2 for z/OS. The CLI and JCC traces can also be used to measure elapsed time to complete every SQL statement. Statements with a higher than expected response time could then be analyzed further, using a tool like Visual Explain. Next, by looking at DB2 for z/OS accounting trace Class 1 and Class 2 nonnested elapsed times, time can be separated by DB2 for z/OS processing time (Class 2 elapsed time) and distributed processing time (Class 1 elapsed time less Class 2 elapsed time). If Class 2 nonnested elapsed time seems excessive, take a percentage of Class 2 nonnested CPU time to Class 2 nonnested elapsed time. This will show the percentage of elapsed time executing SQL. If the percentage is small, then analyze wait times. Calculate what percentage of Class 2 nonnested elapsed time was spent on 11 locks I/O and other wait elements If there is a performance problem within the DB2
    • Client tools – Traces – CLI trace [COMMON] trace = 1 tracecomm = 1 traceerrimmediate=1 tracefilename = d:tempcli.trc tracepathname = d:temp traceflush = 0 db2cli.ini traceflushonerror=1 tracelocks = 0 tracepidlist = tracepidtid = 1 tracerefreshinterval = n Traces CLI/ODBC tracestmtonly = 0 tracetime = 1 Programs tracetimestamp = 1 12 Trace off is the default. Specify either the trace filename or trace pathname, but use path name for multi-threaded applications only. Only set trace flush to 1 if applications do not exit normally. Use TRACECOMM=1 to get network request information. Set tracetimestamp to 1. Trace pid ids by setting tracepidtid to 1. Only the keywords that apply to all connections to DB2 through the DB2 CLI/ODBC driver are included in the COMMON section. The CLI/ODBC trace facility is an essential tool for problem determination and general understanding of an application. All function calls executed are recorded in a text file for later analysis. In addition to functional information, the trace file contains time information which can be extremely useful for application and database tuning. To obtain a CLI trace, the db2cli.ini file must be updated to include a [COMMON] section. There are several ways to update the db2cli.ini file. 1. Use the Configuration Assistant: Select a database (it must be one registered for ODBC), then Properties, CLI/ODBC Settings, Advanced, and Service. 2. DB2 Command Window: • db2 update cli cfg for section common using trace 1 • db2 update cli cfg for section common using tracefilename filename • db2 update cli cfg for section common using tracecomm 1 Confirm the db2cli.ini configuration: db2 GET CLI CFG FOR SECTION COMMON 3. Edit the db2cli.ini file and add the following lines. Only the keywords that apply to all connections to DB2 through the DB2 CLI/ODBC driver are included in the COMMON section: [COMMON] • TRACE=1 (0 is the default) • TRACECOMM=1 (Use a 1 to include information about each network request in the trace file - 0 is the default) • TRACEERRIMMEDIATE=1 (helps in determining when errors occur during application execution by writing diagnostic records to the CLI/ODBC trace file at the time the records are generated) • TRACEFILENAME=<fully_qualified_filename> OR • TRACEPATHNAME=<fully_qualified_pathname> (Multi-threaded apps only) • TRACEFLUSH=1 (only use if applications do not exit normally - 0 is the default) 12 • TRACEFLUSHONERROR=1 (forces the DB2 CLI driver to close and re-open the trace file each time an
    • Client Tools – Traces – CLI Trace Output SQLPrepare( hStmt=1:6, pszSqlStr="SELECT A.BRANCH_ID, B.BRANCH_NAME, COUNT(*), SUM(A.BALANCE) FROM ACCT A, BRANCH B WHERE A.BRANCH_ID = B.BRANCH_ID GROUP BY A.BRANCH_ID, B.BRANCH_NAME", cbSqlStr=-3 ) ---> Time elapsed - +0.000000E+000 seconds ( StmtOut="SELECT A.BRANCH_ID, B.BRANCH_NAME, COUNT(*), SUM(A.BALANCE) FROM ACCT A, BRANCH B WHERE A.BRANCH_ID = B.BRANCH_ID GROUP BY A.BRANCH_ID, B.BRANCH_NAME FOR FETCH ONLY" ) SQLPrepare ( ) <--- SQL_SUCCESS Time elapsed - +0.000000E+000 seconds SQLExecute (hStmt=1:6 ) ---> Time elapsed - +0.000000E+000 seconds sqlccsend( ulBytes - 436 ) sqlccsend( Handle - 12744784 ) sqlccsend( ) - rc - 0, time elapsed - +0.000000E+000 seconds sqlccrecv( ) sqlccrecv( ulBytes - 163 ) - rc - 0, time elapsed - +4.000000E-002 SQLExecute ( ) <--- SQL_ERROR Time elapsed - +4.000000E-002 seconds SQLFreeStmt ( hStmt=1:6, fOption=SQL_CLOSE ) ---> Time elapsed - +0.000000E+000 seconds ( Unretrieved error message="SQL0206N "A.BRANCH_ID" is not a colum in an inserted table, updated table, or any table identified in a FROM clause or is not a valid transition variable for the subject table of a trigger. SQLSTATE=42703 " ) ( COMMIT=0 sqlccsend( ulBytes - 196 ) sqlccsend( Handle - 12744784 ) sqlccsend( ) - rc - 0, time elapsed - +0.000000E+000 sqlccrecv( ) sqlccrecv( ulBytes - 27 ) - rc - 0, time elapsed - +3.000000E-002 13 The .002 negative means move decimal place two to the left so 4.0 -002 seconds is .04 seconds or 4 milliseconds. There was deferred prepare turned on by default. The execute had an error so didn’t take long to run the statement. The DB2 Run-Time Client CLI driver writes trace records when it is entered and when it exits, to reflect the activity just completed. In the above graphic, the SQLPrepare entered into the CLI driver. You can tell entry records by the ---> symbol on the Time elapsed line. The second entry for SQLPrepare was the exit from the CLI driver. The <--- symbol represents an exit from the CLI driver. The Time elapsed is the time spent in DB2 to process the SQLPrepare call. The in DB2 time includes the time in the CLI Driver, the DB2 Run-Time Client, the entire communication infrastructure between client and the database server, time spent on the DB2 Connect Server, and the time spent in the DB2 for z/OS subsystem. The second SQLExecute record shows an SQL error occurred during execution of the SQL statement. The line, SQLFreeStmt, shows the SQL error, SQL0206N, and a description of the error. The COMMIT statement flows because autocommit is on by default. The trace example was captured using TRACECOMM=1. What was added to the trace were the lines in the entry record for SQLExecute following the “Time elapsed” line. By specifying TRACECOMM=1, you can: 1. Find out when a client-to-server communication occurs, either locally or over the network. Many CLI functions are processed completely on the client and therefore do not represent a performance problem. 2. Find out the number of bytes sent and received in each communication. 3. Break down CLI call elapsed times into their CLI and their communications components. The CLI trace information provides you with several timings: • The time between SQL requests • The time to process the send and get a confirmation back • The time from the end of the send to the end of the receive • The total time spent "in DB2" Two CLI keywords allow you to timestamp each record and to also include process and thread IDs. • TraceTimeStamp adds a timestamp to the beginning of each line. There are three formats available, with the default value being 0 (off). 1. =1 [<number of seconds since 01/01/1970>.<microseconds>-<formatted timestamp>] 2. =2 [<number of seconds since 01/01/1970>] 3. =3 [<formatted timestamp>] 13 • TracePidTid causes each line to begin with the process ID and thread ID of the application thread issuing the CLI
    • Sample CLITraceParser Output Overall Trace statistics ======================================================== 10023 statements in trace. Download CLITraceParser code: 12.400 seconds total trace time. 0.172 seconds spent for application processing. 12.228 seconds spent for CLI processing. ftp://ftp.software.ibm.com/ps/ Network Specific CLI processing time statistics ======================================================== products/db2/tools/ 10002 network flows sent to transmit 1230482 bytes, requiring a total of 0.476 seconds. 10002 network flows received, transmitting 860386 bytes, requiring a total of 9.214 seconds. End of overall trace statistics report *************************************************************************************************** Function specific statistics ======================================================== Timing Network Send Network Receive Function Name Total Application CLI Flows Bytes Time Flows Bytes Time --------------------------------------------------------------------------------------------------- SQLGetInfo 3 0.001 0.000 0 0 0.000 0 0 0.000 SQLSetConnectAttr 1 0.000 0.000 0 0 0.000 0 0 0.000 SQLExecDirect 1 0.000 0.008 1 10 0.000 1 54 0.008 SQLGetFunctions 1 0.000 0.000 0 0 0.000 0 0 0.000 SQLGetDiagRec 1 0.000 0.000 0 0 0.000 0 0 0.000 SQLPrepare 1 0.000 0.001 0 0 0.000 0 0 0.000 SQLFreeHandle 3 0.000 0.001 0 0 0.000 0 0 0.000 SQLAllocHandle 3 0.000 0.002 0 0 0.000 0 0 0.000 SQLConnect 1 0.000 0.076 1 342 0.000 1 208 0.002 SQLDisconnect 1 0.000 0.000 0 0 0.000 0 0 0.000 SQLExecute 10000 0.170 12.139 10000 1230130 0.476 10000 860124 9.204 SQLBindParameter 2 0.000 0.000 0 0 0.000 0 0 0.000 SQLGetStmtAttr 4 0.000 0.000 0 0 0.000 0 0 0.000 SQLSetEnvAttr 1 0.000 0.000 0 0 0.000 0 0 0.000 End of function specific statistics report Network time = (9.214 + 0.476) = 9.69 Application time = 0.172 CLI Driver time 12.228 - 9.69 = 2.538 DB2 Time 14 Here’s the CLI Trace output which has been ran thru the CLI TRACE PARSER which is available for download at the site listed. This summarizes into a much cleaner report. Note the network time is 9.69 seconds. The time in application is .172. The time that you left application until returned to application is 12.228. CLITraceParser parses a DB2 Client CLI trace and produces a summary of the trace. The first section, Overall Trace statistics, shows the total trace time and the breakdown of time in the application and in CLI processing (everything outside of the time in the application). The next section, Network Specific CLI processing time statistics, summarizes the number of sends and receives and the total bytes transmitted. The network CLI processing times include DB2 Connect and DB2 server processing times. The last section, Function specific statistics, shows the number of calls by function, each function's elapsed time in application and CLI, and the related network activity. You can use the CLITraceParser output to determine very quickly if the elapsed time is being spent in your application or outside the application. It also provides a good summary of which CLI calls are being done the most and which ones are taking the most time to complete. Another factor to examine is the amount of data being sent and received by the application. In a lot of cases, network delay times can be significant, 14 so by reducing the amount of data being transmitted you might improve on
    • I1 How to Take a DB2 Universal JDBC Driver Trace • Trace as standalone JCC application • DataSource interface for connection to JCC • DB2DataSource > setTraceLevel > default TRACE_ALL • -javax.sql.DataSource.setLogWriter > TRACE_ALL only available • DriverManager interface for connections to JCC • DriverManager.getConnection • Set the traceLevel property in the info parameter or URL parameter • DriverManager.setLogWriter • Specify trace destination and turn on the trace • Within WebSphere, embed the JCC trace points • Set JDBC trace properties in WebSphere Application Server • Go to Resources > JDBC Provider > Data Sources > Additional Properties > Custom Properties. • Set the property: traceLevel(-1 means full trace TRACE_ALL) • Turn on the trace • Go to Troubleshooting > Logs and Trace > Pick the server > Diagnostic Trace > Trace Specification: RRA=all=enabled:WAS.database=all=enabled • Specify two trace strings separated by ':', one for WAS resource adaptor and one for database (JDBC driver) 15 For more information, refer to: •Understand the DB2 UDB JDBC Universal Driver on http://www- 128.ibm.com/developerworks/db2/library/techarticle/dm-0512kokkat/ •DB2 application development: Tracing with the DB2 Universal JDBC Driver on http://www- 128.ibm.com/developerworks/db2/library/techarticle/dm-0506fechner/ •Enabling DB2 Universal JDBC Driver (JCC) tracing on WebSphere Application Server V5 on http://www- 1.ibm.com/support/docview.wss?rs=71&context=SSEPGG&q1=DRDA&uid=sw g21181878&loc=en_US&cs=utf-8&lang=en •Application Development Guide: Programming Client Applications for additional information concerning JDBC tracing with the DB2 Universal JDBC Driver (especially Chapter 20. Diagnosing JDBC and SQLJ problems) Tracing data at the interface between JDBC application and DB2 z/OS or DB2 iseries or DB2 Linux, Unix, or Windows database provides the developer with information to identify program errors and to optimize database access. JDBC tracing is a method for providing Java application developers with valuable information to aid in their database application development efforts. • Searching for program logic or initialization errors -- A database connection may fail because of a wrong URL, a query that is expected to be executed once is called again and again, database inconsistencies may occur because of a transaction that is not correctly defined, and so on. In all these cases, it is likely that trace data can be used to uncover the cause of the problem. 15 • Performance tuning -- Performance problems in multi-tier environments are
    • Slide 15 I1 IBM_USER, 4/11/2006
    • Taking JCC Trace Without Modifying the Application (Option 1) 1. Create plain text file on client a. named DB2JccConfiguration.properties b. with one line of text 2. To enable JCC tracing automatically: c. Add file to the CLASSPATH DB2JccConfiguration.properties db2.jcc.override.traceFile=c:jcc.trc 16 A Java Call Control (JCC) trace can be taken without having to modify the application. If you create a plain text file on the client named DB2JccConfiguration.properties with only one line of text and add it to the CLASSPATH, it will enable JCC tracing automatically: db2.jcc.override.traceFile=c:jcc.trc This is very useful in cases where you cannot change any of the source code or JCC driver properties (for example, when using a third-party product that internally uses the JCC driver). You can change the properties file to specify that these properties should override any data source and connection tracing properties. The global properties that are specified in the provided properties file are default values only. If any data source or connection tracing properties are specified in an application, those values will be used instead. To override the data source and connection tracing properties in the application, you can use the following code: db2.jcc.override.traceFile=jccTrace.txt db2.jcc.override.traceFileAppend=true db2.jdd.override.traceDirectory=c:test 16
    • Taking JCC Trace Without Modifying the Application (Option 2) 1. Create configuration file with trace properties: jcc.properties db2.jcc.traceDirectory=c:temp db2.jcc.traceFile=trace db2.jcc.traceFileAppend=false db2.jcc.traceLevel=-1 2. When Java program is executed, specify the filename via the Option -D java -Ddb2.jcc.propertiesFile=jcc.properties JccTraceExample2 a. The configuration file is placed in same directory as Java class file or complete path for configuration file can be specified. b. Trace level cannot be specified as constant 17 To control tracing without changes to the source code, create a separate configuration file containing the trace properties. DB2 JDBC trace properties db2.jcc.traceDirectory=c:temp db2.jcc.traceFile=trace db2.jcc.traceFileAppend=false db2.jcc.traceLevel=-1 There are no naming conventions for this configuration file. The filename is specified via the option -D when the Java program is executed. For example, if the configuration file is named jcc.properties, the program call looks like this. DB2 JDBC trace properties file java -Ddb2.jcc.propertiesFile=jcc.properties JccTraceExample2 In this case, the configuration file is placed in the same directory as the Java class file. Otherwise a complete path for the configuration file can be specified too. If a configuration file is used, the trace level cannot be specified as constant, instead the corresponding integer values have to be used, for example -1 for TRACE_ALL or 17 6 for TRACE STATEMENT CALLS | TRACE RESULT SET CALLS (in this
    • JCC Trace Parser Features Trace time analysis – Application processing time – JCC processing time – Network processing time – Time analysis for overall trace – Time analysis for overall trace on per connection basis – Time analysis per JCC function call Network flow analysis – Bytes sent or received during flush/fill – Time taken during flush/fill – Total number of flows – Network flow analysis on a per JCC function call basis Error report identifying – Prints the entire exception – The line number of the error Automatic extraction of multiple traces from a single file – For example, if there are many connections in a single trace file then that many numbers of trace outputs will be generated. – To maintain the relationship between connection and respective method calls unique thread name has taken into consideration. Input = Valid JCC trace file Output = One report file which may contain multiple trace reports depending upon different connections information found in a given trace file 18 For some applications, the majority of the elapsed time is spent in the application, not in DB2. In this case, tuning your DB2 server is a waste of time. Application and in JCC times can be summarized using a Java tool called JCCTraceParser. You want to get the file JCCTraceParser.zip. The README.TXT file has information on installing and using the tool. Files included in JCCTraceParser.zip ==================================== JCCTraceParser.sh - a script to run JCCTraceParser on Unix. JCCTraceParser.bat - a script to run JCCTraceParser on MS-DOS. JCCTraceParser.jar - an archive containing all the classes used in JCC Trace Parser. Readme file for JCC Trace Parser 18
    • JCCTrace Report Generated by JCCTraceParser Overall Trace Statistics for Connection: #1 ThreadName: ORB.thread.pool : 3 ======================================================== 208 methods called in trace. 15031.000 mili seconds total trace time. 2177.000 mili seconds total trace time per connection. 2093.000 mili seconds spent for application processing. 84.000 mili seconds spent for JCC processing. Network Specific JCC processing time statistics ======================================================== 18 network flows sent 4256 bytes, and received 4384 bytes, and consumed a total network time of 2093 time in Application 72.000000 mili seconds. 84 JCC – 72 Network = 12 in DB2 End of overall trace statistics report Function specific statistics ======================================================== Timing Network Information Function Name Total Application JCC Flows SentBytes ReceiveBytesNetworkTime ------------------------------------------------------------------------------------------------------------- setMaxRows 14 N/A N/A N/A N/A N/A N/A setLong 27 N/A N/A N/A N/A N/A N/A close 14 N/A N/A N/A N/A N/A N/A getString 38 2.000 0.000 0 0 0 0.000 rollback 4 2086.000 12.000 4 0 128 10.000 setAutoCommit 2 N/A N/A N/A N/A N/A N/A setString 9 N/A N/A N/A N/A N/A N/A prepareStatement 1 0.000 0.000 0 0 0 0.000 next 23 2.000 0.000 0 0 0 0.000 setObject 36 N/A N/A N/A N/A N/A N/A wasNull 6 0.000 0.000 0 0 0 0.000 executeQuery 14 3.000 72.000 14 4256 4256 62.000 getTimestamp 2 0.000 0.000 0 0 0 0.000 getLong 18 0.000 0.000 0 0 0 0.000 End of function specific statistics report 19 JCCTraceParser parses a DB2 Client JCC trace and produces a summary of the trace. The first section, Overall Trace statistics, shows the total trace time and the breakdown of time in the application and in JCC processing (everything outside of the time is in the application). The next section, Network Specific JCC processing time statistics, summarizes the number of sends and receives and the total bytes transmitted. The network JCC processing times include DB2 Connect (DB2 Connect processing time is included only for a Type 2 JCC trace - not for a Type 4 JCC trace since Type 4 does not need to connect through DB2 Connect) and DB2 server processing times. The last section, Function specific statistics, shows the number of calls by function, each function's elapsed time in application and JCC, and the related network activity. You can use the JCCTraceParser output to determine very quickly if the elapsed time is being spent in your application or outside the application. It also provides a good summary of which JCC calls are being done the most and which ones are taking the most time to complete. Another factor to examine is the amount of data being sent and received by the application. In a lot of cases, network delay times can be significant, so by reducing the amount of data being transmitted, you might improve on performance. 19
    • Client Tools – Java Application Monitoring Java API for application monitoring See – DB2SystemMonitor monitor= z/OS manual: ((DB2Connection)conn).getDB2SystemMonitor(); SC18-7414 – monitor.enable(true); Redbook: – monitor.start(com.ibm.db2.jcc.DB2SystemMonitor.RESET_TIMES); – monitor.stop(); SG24-6435 – monitor.getServerTime() – monitor.getNetworkIOTime() – monitor.getCoreDriverTime() – monitor.getApplicationTime() Java App. Univ. Driver SQLJ/JDBC DB2 Server prepareStatement/ executeUpdate executeUpdate executeUpdate 20 There is a Java API for application monitoring built into the IBM DB2 Driver for JDBC and SQLJ as of V8 Fix pack 4 of DB2 for Linux, UNIX, and Windows. The monitor will tell you how much time was spent in: • Application time - The time between the start() and stop() calls • Core driver time - The time spent in the JAVA driver; this includes network I/O time and server time • Network I/O time - The time used to flow the DRDA protocol stream across the network • Server time - The time spent in the DB2 server itself This information can be obtained by turning on the corresponding Java API at the Java client. The monitor itself can be dynamically turned on or off with the right API. Example: import com.ibm.db2.jcc.DB2Connection; import com.ibm.db2.jcc.DB2SystemMonitor; ... DB2SystemMonitor monitor = conn.getDB2SystemMonitor(); monitor.enable(true); monitor.start(DB2SystemMonitor.RESET_TIMES); 20
    • Client Tools – Visual Explain Output 21 The Visual Explain output consists of a graphic representation of the SQL access plan on the right side of your display. Different geometric shapes are used to indicate different types of objects. A rectangle represents a table, while a triangle represents an index. The hexagon shape denotes an operation. The flow of the access plan is from bottom to top. In the above graphic, a join operation is being performed. The table feeding the join from the left will be the outer table and the table feeding the join from the right will be the inner table. One of the objects will be highlighted. Information about the highlighted object will be on the left side of the display. In this case the object is the last object in the access plan and represents a summary of the SQL statement. The statistical information shows the estimated CPU cost to be 27 milliseconds and a service unit cost of 308. Note that when you select one of the statistic lines, an explanation of the statistic is displayed in the Selection details window. The statistics vary by type of object, with tables and index objects showing runstats statistics collected on the table or index. You want to identify which SQL statements have the highest service unit costs. These statements can then be analyzed to see if you can improve on their performance. If the application is using static SQL, the package can be explained and the most expensive SQL statements can easily be identified. If the application is using dynamic SQL and is a CLI/ODBC or Java application, the CLI and JCC traces can identify which SQL statements are taking the longest to run. The SQL text could then be extracted from the CLI trace and submitted directly to Visual Explain. 21
    • Data Flow - DB2 Connect DB2 Connect System DB2 Instance Network Protocol Network Protocol Adapter Adapter DB2 DB2 LAN Agent Agent LAN Listener DB2 DB2 Inbound from client Agent Agent Outbound to host DBM CFG Database Node DCS Directory Directory Directory 22 Returning to our Data Flow of DB2 Connect diagram and our requirement to isolating the problem, lets see what we can do to isolate the data flow through the DB2 Connect machine. 22
    • db2diag Tool Provides the ability to filter and format db2diag.log messages Dozens of options available - run the following for details: db2diag -h [ <option> | brief | examples | tutorial | notes | all ] General help Help for all options, More advanced All help, usage, but no examples examples and examples Help for a specific option Displays various examples Usage notes and restrictions Examples: db2diag -filter instance=DB2 -time 2005-03-08-17.42.35.164191 db2diag -gi "level=severe" -H 1d 23 Examples: • To display all severe error messages produced by the process with the process ID (PID) 52356 and on node 1, 2 or 3, enter: db2diag -g level=Severe,pid=952356 -n 1,2,3 • To display all messages containing database SAMPLE and instance aabrashk, enter: db2diag -g db=SAMPLE,instance=aabrashk • To display all severe error messages containing the database field, enter: db2diag -g db:= -gi level=severe • To display all error messages containing the DB2 ZRC return code 0x8 040055, and the application ID G916625D.NA8C.068149162 29, enter: db2diag -g msg:=0x8 040055 -l Error | db2diag -gi appid^=G916625D.NA • To display all messages not containing the LOADID data, enter: db2diag -gv data:=LOADID • To display only logged records not containing the LOCAL pattern in the application ID field, enter: db2diag -gi appid!:=local or db2diag -g appid!:=LOCAL 23
    • db2pd Command Examples Examples: – List agents under the current instance, repeating every 10 seconds: db2pd -agents –repeat 10 – Shows memory sets and operating system details db2pd -inst -memsets -osinfo – Shows subsystems currently being used for workload balancing in sysplex environment db2pd -sysplex db=DSND - Shows active and delayed database manager config values db2pd -dbmcfg – Developer Works article: http://www-128.ibm.com/developerworks/ db2/library/techarticle/dm-0504poon2 24 The db2pd utility retrieves information from the DB2 for Linux, UNIX, and Windows memory sets. Authorization: One of the following: • On Windows operating systems, the sysadm authority level. • On UNIX operating systems, the sysadm authority level. You must also be the instance owner. Required connection: There is no minimum connection requirement. However, if a database scope option is specified, that database must be active before the command can return the requested information. The db2pd utility retrieves various statistics, internal metadata, and snapshot information from a running DB2 instance. Run "db2pd -help" for options. It is completely non-intrusive and since it doesn't acquire latches, it is a Very fast retrieval and doesn't impact the engine in any way. It can be run even if the system is hung. Please refer to the Developer Works article, http://www- 128.ibm.com/developerworks/db2/library/techarticle/dm-0504poon2, for more information. 24
    • DB2 Connect – db2 LIST DCS APPLICATIONS EXTENDED Shows detail information for each application 25 The Status Change Time is only recorded if UOW switch on. The LIST DCS APPLICATIONS EXTENDED command provides more detailed information about each of the application connections. The output shows: •Client Application ID - The name is in three parts, separated by periods. The first part of the name identifies the client host. If the connection from client to DB2 Connect is TCP/IP, the first part of the name is the IP address of the client, in hex. Every pair of hex numbers represents the decimal value in the IP address. For example, the first part of the Client Application ID in our example is C0A801F8. If we break this into hex pairs, we get C0.A8.01.F8. If each pair of numbers is converted to decimal, we get the IP address of the client: 192.168.1.248. The second position is the client's port number used for the connection, in hex. If the first or second parts of the Client Application ID start with a value of G- P (like the PB11 value above in the second part of the ID), the G-P value is converted to 0-9 respectively to obtain the real hex value. So in the above example, the P is converted to a 9, and then the hex value 9B11 is converted to a decimal value of 39697 for the port number. The third position is a unique identifier generated for this client connection. It is based on the timestamp of when the client connection was established, but 25 because there is no guarantee it will be unique for example if you have
    • DB2 Connect – Snapshot DCS Connections Use Snapshot Monitor to display detail information Monitor Switches for DB2 Connect – STATEMENT – UOW Snapshot Options GET SNAPSHOT FOR DBM GET SNAPSHOT FOR ALL DCS DATABASES GET SNAPSHOT FOR ALL DCS APPLICATIONS 26 You can use the snapshot monitor with DB2 Connect Enterprise Edition to monitor remote client connections. To monitor local clients, the DB2CONNECT_IN_APP_PROCESS registry variable must be set to NO. The snapshot monitor is always active, but if you want more detailed information you need to turn on the STATEMENT and UOW switches. This can be done for a session by issuing the DB2 UPDATE MONITOR SWITCHES USING STATEMENT ON UOW ON command or permanently by updating the DBM CFG file. You must have SYSADM, SYSCTRL, SYSMAINT, or SYSMON authority to use the SNAPSHOT command. There are five command formats that are applicable to DB2 Connect: • GET SNAPSHOT FOR ALL DCS DATABASES • GET SNAPSHOT FOR ALL DCS APPLICATIONS • GET SNAPSHOT FOR DCS APPLICATION APPLID appl_id • GET SNAPSHOT FOR DCS APPLICATION AGENTID appl_handle • GET SNAPSHOT FOR DCS DATABASE ON db-alias • GET SNAPSHOT FOR DCS APPLICATIONS ON db-alias 26 In addition instance information can be retrieved by executing the command:
    • DB2 Connect – db2 Get Snapshot DBM 27 A snapshot at the database manager level provides statistics for the active database manager instance. If there is no instance attachment, a default instance attachment is created. Some of the key elements to look for are: • Instance name - The name of the database manager instance for which the snapshot was taken. • Database manager status - The current status of the instance of the database manager. • Product name - Details of the version of the DB2 instance that is running. • Service level - This is the current corrective service level of the DB2 instance. • Start Database Manager timestamp - The date and time when the database manager was started using the db2start command. • Snapshot timestamp - The date and time when the database system monitor information was collected. • Remote connections to db manager - The current number of connections initiated from remote clients to the instance of the database manager that is being monitored. • Remote connections executing in db manager - The number of remote applications that are currently connected to a database and are currently processing a unit of work within the database manager instance being monitored. 27
    • DB2 Connect – Snapshot DCS Database db2 get snapshot for dcs database on tdsnl 28 A snapshot at the database level provides information on database connections, performance, errors and throughput of SQL requests. Some of the key elements to look for are: • Most recent elapsed time to connect - The elapsed time between the start of connection processing and actual establishment of a connection, for the most recent DCS application that connected to this database. Use this element as an indicator for the time it takes applications to connect to a particular host database. • Most recent elapsed connection duration - The elapsed time that the DCS application which was most recently disconnected from this host database was connected. • Host response time - The sum of the elapsed times for all the statements that were executed for a particular database. Use this element with Outbound Number of Bytes Sent and Outbound Number of Bytes received to calculate the outbound response time (transfer rate): ((outbound bytes sent + outbound bytes received) / host response time). • Number of SQL statements attempted - The number of SQL statements that have been attempted since the database activated or the last reset monitor switches command was issued. Use this to measure the rate of SQL activity against a host database. By taking multiple snapshots and calculating the difference between number of SQL statements attempted and dividing by the elapsed time between the snapshots, you can get a rate at which SQL statements are being processed at the DB2 Connect Server. • Failed statement operations - The number of SQL statements that were 28 d b f il d f b ff db h i h dl f il d
    • DB2 Connect – Snapshot DCS Applications (1 of 2) db2 get snapshot for dcs applications 29 Host response time is sum of elapsed times for all statements that were executed for a particular application. Time spent on gateway processing is time in seconds/microseconds at DB2 Connect server to process an application request. Idle time is number of seconds since an application has issued any requests to the server. An application snapshot provides information on activity on a specific client application-DB2 Connect Agent-DDF thread connection. Some the of key elements to monitor are: • Application status - The status of a DCS application at the DB2 Connect Server. The values can be: - connect pending - outbound - The application has initiated a database connection from the DB2 Connect server to the host database, but the request has not completed yet. - waiting for request - The connection with the host database has been established, and the DB2 Connect agent is waiting for an SQL statement from the client application. - waiting for reply - An SQL statement has been sent to the host 29 d t b d th ti iti f l
    • DB2 Connect – Snapshot DCS Applications (2 of 2) 30 © Copyright IBM Corporation 2006 f the UOW monitor switch is turned on, an application snapshot will return additional information about the most recent unit of work (transaction). Key fields to monitor are: • UOW start timestamp - The date and time that the unit of work first required database resources. The time is when the first SQL statement of a unit of work is sent to the host for the most recent transaction. This time is used to calculate how long transactions are taking and to see if any applications have not issued a commit or rollback for a long time. • UOW stop timestamp - The data and time that the most recent unit of work completed, which occurs when a commit or rollback is issued. By subtracting UOW start timestamp from UOW stop timestamp, you can tell how long transactions are taking to complete, or to see if an application has failed to issue a commit or rollback for an extensive period of time. • Elapsed time of last completed UOW - The elapsed time of the most recently completed unit of work in seconds and microseconds. The timing is tracked by DB2 Connect solely based on its own processing. This elapsed time starts counting when DB2 Connect starts the new unit of work. The counting stops when DB2 Connect finishes the unit of work, which includes processing the commit/rollback reply from the DB2 for z/OS server. If the STATEMENT monitor switch is turned on, an application snapshot will return additional information about the most recent SQL statement. Key fields to monitor are: 30
    • Analyzing DCS applications snapshot DB2 Connect Network DB2 z/OS 1. 2. 3. 4. Time in seconds Element name in output Element name/identifier in manual 1. Elapsed time of last completed stmt Most recent statement elapsed time/stmt_elapsed_time 2. Host response time Host response time/host_response_time 3. Host execution elapsed time Statement execution elapsed time/elapsed_exec_time 4. Time spent on gateway processing Elapsed time spent on DB2 Connect gateway/gw_exec_time 31 Elapsed time of last completed stmt contains the highest value of all three elements as it represents all time spent processing a statement inside DB2 Connect. This includes all time spent on the network communications between it and the host, that is, Host response time. The next highest value is Host response time which represents only all time spent on the network communications between it and the host. As such, this time includes time spent processing the statement on the host, that is, Host execution elapsed time. The lowest value is Host execution elapsed time as it represents only the time spent for processing a statement on the host as reported by the host DB2 back to DB2 Connect. At the DCS statement level, host_response_time or Host Response Time, monitor element is the elapsed time between the time that the statement was sent from the DB2 Connect gateway to the host for processing and the time when the result was received from the host. At DCS database and DCS application levels, it is the sum of the elapsed times for all the statements that were executed for a particular application or database. Time spent on gateway processing is the time in seconds and microseconds at the DB2 Connect server to process an application request (since the connection was established). At the application level, this is the cumulative time spent processing requests. At the statement level, this is the time to process the last SQL statement. This element can be used to determine how much time is spent within the DB2 Connect server. An average time per SQL statement can be calculated using the formula: (Time spent on gateway processing / number of SQL statements attempted). 31
    • DB2 Connect – db2drdat Command db2drdat - DRDA Trace on db2drdat -r -s -l=length path off -f -t=tracefile -p=pid 1. -r - TRACE RECEIVE BUFFERS 2. -s - TRACE SEND BUFFERS TYPE OF DRDA TYPE OF DRDA REPLY/OBJECT REQUEST 32 The db2drdat trace utility provides a record of the data interchanged between the DB2 Connect server (on behalf of an application client) and a DB2 for z/OS subsystem. This is an I/O trace of the DRDA protocol flow between the agent on the DB2 Connect server and the thread on the DB2 for z/OS system, and also between the agent and the client system if the client is at V8 (thus it would also use DRDA to communicate with the DB2 Connect server). No communication protocol information is captured in this trace. The trace can be used for problem determination, by displaying exactly what DRDA data is being sent and received, and performance tuning. For performance tuning, each entry is time stamped which allows us to determine the elapsed time between data sent and the response received. The db2drdat DRDA trace allows the user to capture the DRDA data stream exchanged between a DRDA Application Requestor (AR) and the DRDA Application Server (AS). Although this tool is most often used for problem determination, by determining how many sends and receives are required to execute an application, it can also be used for performance tuning in a client/server environment. 32
    • Network - DB2 PING Command Syntax ping db_alias request packet_size response packet_size [ n times ] Example: db2 connect to tdb2390 user <id> using <pswd> db2 ping tdb2390 request 100 response 200 2 times Elapsed time: 26754 microseconds Elapsed time: 27352 microseconds Client DB2 Connect DB2 z/OS Host T C U D P db2 ping host Response Time / S D DB2 I S F P Response Time 33 DB2 for Linux, UNIX, and Windows Version 7 Fix pack 2 introduced the db2 ping command. This command tests the network response time of the underlying connectivity between a client and a database server where DB2 Connect is used to establish the connection. No special authority is required to issue the command. As of V8, the db2 ping command supports the specification of a request and response packet size. The command syntax is: db2 ping dbalias REQUEST packet_size RESPONSE packet_size number_of_times where: dbalias - Specifies the database alias for a DRDA server that the ping is being sent to. REQUEST packet_size - Specifies the size, in bytes, of the packet to be sent to the server. The size must be between 0 and 32767 inclusive. The default is 10 bytes. This option is only valid on servers running DB2 for Linux, UNIX, and Windows Version 8 or later, or DB2 for z/OS Version 8 or later. RESPONSE packet_size - Specifies the size, in bytes, of the packet to be 33 returned back to client The size must be between 0 and 32767 inclusive The
    • Network – TCP/IP NETSTAT - Statistics > netstat -e -n -s -p tcp Interface Statistics Received Sent Bytes 68180872 3067496 Unicast packets 50847 40949 Non-unicast packets 80022 384 Discards 0 0 Errors 0 17 Unknown protocols 20302 TCP Statistics Active Opens = 1111 Passive Opens = 0 Failed Connection Attempts = 27 Rest Connection = 789 Current Connections = 7 Segments Received = 38814 Segments sent = 29613 Segments Retransmitted = 31 34 For Windows platforms: Network statistical information can be displayed by using the NETSTAT command with the ESP options. In this example, the command netstat -ensp tcp is issued to display Ethernet interface statistics and per protocol statistics for tcp in numerical form. We are displaying network interface statistics and TCP statistics. DB2 Connect users, both inbound and outbound, will only use TCP packets. The Interface Statistics information is for all protocols and can be used to develop transmission rates. By taking successive displays, a data rate for received and sent rates can be computed. This could be used to track usage and trends. There is also an Error value shown. If the percentage of errors is high (over 1 to 2 percent), then this needs to be investigated. Under TCP Statistics, you can get a rate at which TCP connections are occurring. If Segments Retransmitted is high (1 to 2 percent) to Segments Sent, then this could indicate a problem within the network. For UNIX systems: The netstat -v command shows network statistics for each network adapter. You can 34 thi i f ti t d t i t k dd l t d t t h
    • Data Flow - z/OS z/OS System UNIX System Services Network Protocol Comm MSTR DBM1 Cntlr IRLM DDF LAN O S A RACF BSDS CDB ZPARMS 35 Returning to our Data Flow of DB2 Connect diagram and our requirement to isolating the problem, lets see what we can do to isolate the data flow through the z/OS machine. 35
    • Display DDF Detail Number of # of DBATs Current # Connections CMTSTAT Pooled of DBATs Inactive (I or A) 36 The DISPLAY DDF command is a DB2 for z/OS command that displays information regarding the status and configuration of DDF, as well as statistical information regarding connections or threads controlled by DDF. This command can be issued from a z/OS console, a DSN session under TSO, or a DB2I panel (DB2 COMMANDS). DB2 commands issued from a z/OS console are not associated with any secondary authorization IDs. To execute this command, the privilege set of the process must include one of the following: • DISPLAY privilege • SYSOPR, SYSCTRL, or SYSADM authority The DETAIL option displays additional configuration and statistical information. The DISPLAY DDF command displays the following output: • STATUS - The operational status of DDF. • LOCATION - The location name of DDF. • LUNAME - The fully qualified logical unit name (LUNAME) of DDF. • GENERICLU - The fully qualified generic LUNAME of DDF. • IPADDR - The IP address of the z/OS system that DDF is running on. • TCPPORT - The SQL listener port used by DDF. • RESPORT - The resync listener port used by DDF, when involved in two-phase commit processing. 36
    • DB2 for z/OS – DISPLAY THREAD Command Syntax >>DISPLAY THREAD_______________________________________________> | <_,_____________________________| | |_(______connection-name_________|____) | | |_partial-connection*___| | |_(_*_)__________________________________| >______________________________________________________________> | ACTIVE____ | |_TYPE(_|_INDOUBT_____|_)__| | |_*___________| | |_INACTIVE____| | |_POSTPONED___| >______________________________________________________________> | <_,________________________| |_DETAIL_| |____LOCATION(____location-name__|__)__| | | |partial-location*_| | | |_*_____________________| | <__,_______________________| |____LUWID___(_____luwid__________|__)_| | |_partial-luwid*__| | |_token___________| 37 The DB2 for z/OS command DISPLAY THREAD displays current status information about DB2 for z/OS threads. A DB2 for z/OS thread is either an allied thread, a database access thread, or a parallel task thread. Threads can be active, inactive, indoubt, or postponed. Distributed threads are those threads that have a connection with a remote location (active or inactive) or that had a connection with a remote location (indoubt). An allied thread and a parallel thread can be distributed or nondistributed; a database access thread is always distributed. For most applications using DB2 Connect, the thread will be a distributed thread (DBAT). If an application is using a stored procedure however, an allied thread is used for the stored procedure. The privilege needed to display thread information is DISPLAY privilege or SYSOPR, SYSCTRL, or SYSADM authority. You can limit the threads you display by specifying the TYPE of thread, or the LOCATION, or for a specific LUWID. The LOCATION name is the IP address of the DB2 Connect system. The LUWID is the Host application ID from a DB2 LIST DCS APPLICATIONS command. The DETAIL option displays additional information about each thread. A thread is associated with a DB2 Connect agent, which is associated with a client application. When an application is processing a transaction (unit of work), the thread is ACTIVE. 37 Immediately following a COMMIT or ROLLBACK the thread goes INACTIVE This
    • Display Thread Detail 38 The above graphic represents the output from a DISPLAY THREAD (*) TYPE(INACTIVE) DETAIL command. The following are key items to look for on the display: • NAME value of SERVER - A 1 to 8-character variable representing the connection name used to establish the thread. For distributed database access threads using application-directed access from a non-DB2 for z/OS requester, this variable displays the constant SERVER. • ST value of R2 - A distributed thread is performing a remote access on behalf of a request from another location. The thread is currently a Type 2 inactive thread and is waiting for an agent to become available to process. • A - The A column represents the active indicator status of the thread. A value of asterisk is displayed if the thread is active within DB2. The value is blank otherwise. • REQ - A wraparound counter showing the number of requests received by this thread. If this counter is increasing, then you know that this thread is performing work. • ID - The application name passed to the thread at connection time. • AUTHID - The primary authid of the end user making the connection. • DISTSERV value of PLAN - A 1 to 8-character variable representing the plan name associated with the thread. For distributed database access threads using application-directed access from a non-DB2 for z/OS requester, this variable displays the constant DISTSERV. • V445 - The LUWID (Logical Unit 38 f W k ID tifi ) i d t thi th d Thi th H t
    • DB2 for z/OS: Display Database Locks Who is holding the lock? Select dbname , tsname from sysibm.systables where name=‘ACCT2' and creator = 'ADMIN' DBNAME TSNAME CF63T23 CF63T000 39 The DISPLAY DATABASE command displays information about specific DB2 for z/OS databases, table spaces, tables in segmented table spaces, LOB table spaces, index spaces within a database, indexes on auxiliary tables, and partitions of partitioned table spaces and index spaces. This command can be used to see what resources are being used by an application and if any locks are being held and by which application. The DISPLAY DATABASE LOCKS output shows locking information for table spaces, index spaces, and partitions. The key fields are: • NAME - The table space name, index space name, or table object ID. • TYPE - A TS for table space, IX for index, or TB for table. • PART - The partition number. It is blank for a simple table space or index space. • STATUS - Status of object. Will normally show RO - the table space, table space partition, index space or index space partition is started for read-only activity, or RW - the table space, table space partition, index space or index space partition is started for read and write activity. • CONNID - A connection identifier for the thread. For DBAT threads using application-directed access from a non-DB2 for z/OS requester, this field contains the constant SERVER. • CORRID - The name of the application holding the connection. 39
    • Using Accounting Trace Information SQL DML TOTAL AVERAGE APPL(CL.1) DB2 (CL.2) -------- -------- ------------ ---------- ---------- SELECT 50974 ELAPSED TIME 24:59.7823 3:50.24315 INSERT 30949 NONNESTED 24:59.7823 3:50.24315 UPDATE 13029 STORED PROC 0.000000 0.000000 DELETE 1292 UDF 0.000000 0.000000 TRIGGER 0.000000 0.000000 DESCRIBE 0 DESC.TBL 0 CPU TIME 1:05.93530 40.218570 PREPARE 0 AGENT 1:05.93530 40.218570 OPEN 5670 NONNESTED 1:05.93530 40.218570 FETCH 15100 STORED PRC 0.000000 0.000000 CLOSE 5670 UDF 0.000000 0.000000 TRIGGER 0.000000 0.000000 DML-ALL 122684 PAR.TASKS 0.000000 0.000000 --- DISTRIBUTED ACTIVITY ------- SUSPEND TIME N/A 1:18.03509 REQUESTER : 10.10.10.10 AGENT N/A 1:18.03509 COMMITS(1) RECEIVED: 4738 PAR.TASKS N/A 0.000000 SQL RECEIVED : 101914 MESSAGES SENT : 106653 NOT ACCOUNT. N/A 1:51.98949 MESSAGES RECEIVED : 106653 DB2 ENT/EXIT N/A 213306 BYTES SENT : 12943505 EN/EX-STPROC N/A 0.00 BYTES RECEIVED : 21614937 EN/EX-UDF N/A 0.00 MESSAGES IN BUFFER : 9430 ROWS SENT : 13326 BLOCKS SENT : 5670 Time in DB2 = 3:50.24315+(1:05.93530-40.218570) = 4:15.95988 Time outside of DB2 = 24:59.7823-4:15.95988 = 20:43.82242 40 This is an accounting trace of a recent performance problem. Even though the time in the DB2 server is about 14%, there is an indication in this information that the entire platform has a problem. Can you tell what it is? Look at the NOT ACCOUNT. time in DB2. It is almost half the total time in DB2. This indicates a serious delay waiting for processing resources which can equate to either there are other more important tasks in the same LPAR or, with floating processor resources, there are other more heavily weighted LPARs taking precedence on the overall system. The latter was the case here. The number of messages sent in this case is the total of the SELECT (singleton), INSERT, UPDATE, DELETE, and OPEN DMLs processed plus the number of commits. This is an active thread accounting trace report. Finally, if the application was dynamic, then PREPAREs would be part of the inbound message usually chained with a DESCRIBE INPUT (JDBC), and INSERT/UPDATE/DELETE/OPEN. If a DESCRIBE was chained with a PREPARE, then the other kinds of DML would be separate messages. Distributed threads always fall under the NONNESTED category or element in the trace output. So in the above output the Class 2 NONNESTED ELAPASED TIME is the amount of time spent processing in DB2 for this thread. 40 Th Cl 1 NONNESTED CPU TIME i th CLASS 2 NONNESTED CPU
    • DB2 for z/OS Timings z/OS Host T C U P S DDF DB2 / S I P Class 1 Nonnested CPU - Class 2 Class 2 Nonnested Elapsed Time Nonnested CPU - Class 1 Elapsed - Class 2 Nonnested Elapsed Class 1 Elapsed 41 Class 2 nonnested elapsed time measures the time spent processing in the DB2 for z/OS subsystem, including wait time. Class 1 elapsed time measures the time spent processing in the DB2 for z/OS subsystem and outside DB2. This time reflects the total elapsed time since the application connected to the database. The time just in DDF can be computed by taking the Class 1 nonnested CPU time and subtracting the Class 2 nonnested CPU time. DDF processing is all CPU time. The time spent outside the DB2 for z/OS subsystem can be computed by taking Class 1 elapsed time and subtracting Class 2 nonnested elapsed time. Then by subtracting the time in DDF, all of DB2 for z/OS can be eliminated. You are not looking for any specific values. Calculate percentages of Class 1 time to see where most of the time is being spent. Then investigate that component. 41
    • z/OS - TCP/IP KEEPALIVE Interval Client DB2 Connect DB2 for z/OS Application Agent Thread SQL X ? SQL X Agent ? Set TCP/IP KEEPALIVE Interval – Network Setting – TCPKPALV ZPARM 42 TCP/IP has a configurable option to periodically send a keepalive packet over a connection if there has been no activity on that connection for some specified period of time. Since TCP/IP keepalive support is optional, not all applications will respond to the keepalive packet. DB2 for z/OS, however, does recognize the TCP/IP keepalive packet. It is recommended that TCP/IP keepalive be configured and the keep alive interval be set to 5 minutes or less. This will allow lost connections to be detected either because of a network outage or a system crash. Resources being held by an application can then be released. To enable TCP/IP keepalive and set a keep alive interval, the client, the DB2 Connect, and the DB2 for z/OS systems must all be configured (assuming TCP/IP is being used between all systems). On the z/OS system, the keep alive enablement and interval setting can be set within TCP/IP to cover all applications. In the TCP/IP PROFILE file, there is a parameter called KEEPALIVEOPTIONS in the TCPPARMS(PROFILE) on z/OS. One of the keywords in this option is INTERVAL, which sets the keep alive interval in terms of seconds. Check with your z/OS TCP/IP person to find out what was used on your system. To override the default KEEPALIVE setting for DB2 for z/OS, you can set a TCP/IP KEEPALIVE value during installation on the DSNTIP5 installation panel. You can also modify the TCPKPALV ZPARM value. Possible settings are: • ENABLE - Do not override the TCP/IP KEEPALIVE configuration value. 42
    • LUW - TCP/IP KEEPALIVE Settings Operating Parameter wait time Parameter interval Parameter maximum retry Unit of System before probing the between retry probes probes measure connection AIX tcp_keepidle tcp_keepintvl n/a half-seconds HP-UX 11i tcp_timewait_interval tcp_keepalive_interval tcp_keepalives_kill (1) milliseconds Linux tcp_keepalive_time tcp_keepalive_intvl tcp_keepalive_probes seconds Solaris tcp_time_wait_interval tcp_keepalive_interval n/a milliseconds Windows KeepAliveTime KeepAliveInterval TepMaxDataRetransmissions milliseconds Note: (1): tcp_keepalives_kill cannot be modified on HP. It is set to 1. 43 TCP/IP uses operating system keepalive parameters to detect when the client or server side of an idle connection is no longer responding. DB2 sets the TCP/IP keepalive setting on both the client and server by default. You may wish to decrease the keepalive parameters on the server side machine to improve detection of client failures, or decrease the keepalive parameters on the client side machine to improve detection of server failures. Alternatively, you may wish to increase the keepalive parameters on the server side machine to prevent client disconnects on idle connections, or increase the keepalive parameters on the client side machine to prevent server disconnects on idle connections. Each keepalive parameter comes with a default setting; many parameters are configurable. In general, the parameters: • Determine how long to wait before probing the idle connection. On most platforms, the default is 2 hours. • Determine how long to wait before retrying the probe after initial failure to respond. • Determine the maximum number of times to retry the probe. Modifying any keepalive parameter may involve trade-offs and affects applications on your entire machine. For example, changing these parameters affects rlogin, ssh, and telnet. 43
    • Registry Variables – Detect if Client or Server Alive DB2CHECKCLIENTINTERVAL - Server Registry DB2TCP_CLIENT_CONTIMEOUT - Client Variables DB2TCP_CLIENT_RCVTIMEOUT - Client • See the Communications variables topic in the product Information Center at: • http://publib.boulder.ibm.com/infocenter/db2luw/v8/ index.jsp?topic=/com.ibm.db2.udb.doc/admin/r0005660.htm 44 DB2 registry variables to help detect when a client or server is not responding. If you do not wish to adjust the keepalive operating system settings, DB2 has additional registry variable settings that you can use to help detect some situations where a client or server is not responding. DB2CHECKCLIENTINTERVAL - Adjusting this variable on the server determines how quickly the DB2 server can detect a client-server connection has been terminated (for example, a client kills a DB2 application). Setting this variable does not help with situations where the client is not responding because of an abnormal machine termination where TCP cannot respond (operating system keepalive values must be adjusted to handle this condition). DB2TCP_CLIENT_CONTIMEOUT - Adjust this variable on a client to guarantee that a connection will be established or will fail within a specified amount of time. This is useful when the server is not responding because the machine is down or overloaded. DB2TCP_CLIENT_RCVTIMEOUT - Adjust this variable on the client to terminate the connection if data is not received from the server within a specified amount of time. This is useful in situations where a connection has already been established with the server but the server is no longer responding because the machine is down or overloaded. 44
    • DB2 Connect Enhancements • BINARY, VARBINARY, and DECFLOAT data type support in .NET and CLI client applications • Client support for trusted connections to DB2 for z/OS databases • Command line processor (CLP) 64 KB limit for SQL statements has been changed to 2 MB • Developer Workbench replaces the Development Center in DB2 9 • Data Studio replaces Developer Workbench in DB2 9.5 • XML and XQuery support • JDBC and SQLJ driver enhancements • Connection timeout support for database applications • Support for IPV6 • Statement level isolation for nicknames • Two-phase commit for multivendor data sources • User mapping retrieval from an external repository is supported • NetBIOS and SNA communication protocols are no longer supported 45 The above list includes enhancements to DB2 Connect 9 and DB2 9.5. The only enhancement that is unique to DB2 9.5 is that Data Studio replaces Developer Workbench which replaced the Development Center in DB2 9. 45
    • Summary • Describe data flow within a DB2 Connect environment and develop a problem determination methodology • Perform problem determination analysis using tools available on DB2 Client • Capture, parse and examine CLI and JCC Traces, configure DB2 Connect to improve performance, and analyze Snapshot DCS monitor output • Run and interpret network commands and use DB2 commands to perform problem determination • Analyze accounting, statistics and performance trace output to identify and resolve connection and performance problems 46 In summary, we have covered the above objectives. 46
    • Session B03 DB2 Connect to DB2 z/OS – Problem Determination & Performance Tips & Tricks Melanie Stopfer IBM North America Lab Services mstopfer@us.ibm.com For additional information, see www.ibm.com/training • CF632 DB2 Connect to DB2 for z/OS Problem Determination & Performance • CF602 DB2 Connect to DB2 for z/OS DRDA Implementation 47 For additional information, see www.ibm.com/training for courses CF632 and CF602. 47