• Save
Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative
Upcoming SlideShare
Loading in...5
×
 

Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative

on

  • 7,828 views

 

Statistics

Views

Total Views
7,828
Views on SlideShare
7,825
Embed Views
3

Actions

Likes
1
Downloads
0
Comments
0

1 Embed 3

http://www.linkedin.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative Presentation Transcript

  • Lessons Learned from Isbank –A Story of a DB2 for z/OS initiativeCüneyt GöksuIBM DB2 Gold ConsultantVBTSession Code: F13Wednesday, 10 November 2010 | Platform: z/OS 1
  • Due to Isbank’s confidentiality regulations some ofthe performance numbers, naming conventions andfigures are removed, rearranged or renamed. 2
  • Agenda• How did we start?• Installation, Parallel Sysplex and More• Brand New Corebank Application and Java• IMS & DB2 Coexistence• Unicode, SQL and Application Tuning Experiences• Moving Windows ODS Application to z/OS• V9 Migration• Thinking V10 3 View slide
  • How did we start?• In 2007• 2 x System z9 EC• z/OS 1.7• DB2 V8• Omegamon for DB2 V4.1.0• DB2 Admin Tool from IBM• No Application! 4 View slide
  • Installations, Parallel Sysplex and More • Sandbox LPAR, V8 Installation and Naming Conventions. • Install DB2 V8 NFM with Codepage 1026 (Turkish) • One-Way Data Sharing Enabled • Two-Way Data Sharing... • CF Sizing and DataSharing Failure Test Scenarios • CF350 - DB2 for z/OS and Data Sharing Implementation Workshop • CF83 - DB2 for z/OS Database Admin Workshop • CV721 - DB2 9 for z/OS Application Programming Workshop • CE130 – Advanced SQL Workshop 5
  • Test Environment• 4 LPARs• 5 DataSharing Groups• 4 Members / Group WHY?!?!?! DEV3 DEV4 DEV1 DEV2 TEST3 TEST4 TEST1 TEST2 PAT3 PAT4 PAT1 PAT2 UAT3 UAT4 UAT1 UAT2 PERF3 PERF4 PERF1 PERF2 LPAR1 LPAR1 LPAR1 LPAR1 6
  • Production Environment• 4 LPARs• 1 DataSharing Group• 4 Members / Group DEV1 Prod 2 Prod 3 Prod 4 Prod 1 TEST1 PAT1 UAT1 PERF1 LPAR1 LPAR1 LPAR1 LPAR1 7
  • Explore the DB2• IpPlex / VIPA• RTS and Utility Automation• Dynamic SortWork Allocation for DB2 Utilities PK45916: REMOVE THE NEED TO SPECIFY SORTNUM ON UTILITY STATEMENTS FOR DYNAMICALLY ALLOCATED SORT WORK DATA SETS• UTSORTAL = YES IGNSORTN = YES MAX_UTIL_PARTS  PK51853 8• SORTWKnn, SW01WKnn, DATAWKnn, STATWKnn, ST01WKnn
  • Explore the DB2Monitor and Action against critical DB2 messages such as• DSNJ111E (OUT OF SPACE IN ACTIVE LOG DATA SETS)• DSNI014I(Broken Page)• DSNP007I(EXTEND FAILED)SMFACCT=(1,2,3,7,8,10)SMFSTAT=(1,3,4,5,6,8)ACCUMACC = NO  Development-PerformanceACCUMACC = 10  ProdSYSIBM.SYSPACKAGE WHERE OPERATIVE <> Y ORVALID <> YPARMLIB(CSVLLAxx)  ....SDSNEXIT/F LLA,UPDATE=xx 9
  • Explore the DB2• “Automated Space Management” or “sliding secondary extent” MGEXTSZ=YES, PriQTY = -1, SecQTY = -1• CLI Packages (SYSxynzz) x: L or S : package size y: H or N : hold or nohold n: 1,2,3,4 : 1=UR, 2=CS, 3=RS, 4=RR SYSLH3xx, SYSLH4xx, SYSSH3xx, SYSSH4xx  DISABLED odbc.ini (WithHold=1)  1: With HOLD, 0: No HOLD 10
  • Corebank Application & JAVA• Insert ONLY Banking Application. DELETEs and UPDATEs are logical.• Member Cluster• Trackmod NO• FREEPAGE 0 & PCTFREE 10• DB2 for z/OS Optimizing INSERT Performance John Campbell Insert Performance in DB2 UDB for z/OS V7/V8 Akira Shibamiya 11
  • Corebank Application & JAVA• Optimistic Locking 1. Read row 2. Modify data 3. Try to write back data 4. Before updating the row check whether it has been changed in the meantime; if so return with error• Lock Avoidance Dirty Read on package level or statement level 12
  • Corebank Application & JAVA• Date, Time, TimeStamp, Type Conversion Costs• Timestamp  STRING Datatype in Java• SELECT ..., CHAR (AAPCR_START_DATE), CHAR (DIP_EFCTV_DATE), ..., CHAR (DIL_HOST_TS), CHAR (DIL_LOCAL_TS), DIL_WS_ID, DIL_USER_ID, RULE_SYSTEM_ID, LDBID FROM DRVD_INTRST_LINE D• 10,000 rows in table 13
  • Corebank Application & JAVA• Avg CPU Cost of retrieving a row %1,3 with SQL Function is 0,000107026 without SQL Function is 0,000105629107,500000000 107,025927708107,000000000106,500000000 CPU Usage with SQL Func106,000000000 CPU Usage w/o SQL Func 105,628945059105,500000000105,000000000104,500000000 1000000 14
  • Corebank Application & JAVA 15
  • Corebank Application & JAVA• Beware!  PreTest Connection Properties 16
  • Corebank Application & JAVA• SELECT ....FROM ...WHERE ? BETWEEN C1 AND C2 Non-Indexable• SELECT ....FROM ...WHERE C1 <= ? AND C2 >=? Indexable 17
  • Corebank Application and Package Versioning 2800+ Program 30,000 Package with all versions! Impacts on SPT01! +-------------------------------------------------------------------------------------------------------------+ | COLLECTION | PACKAGE | VERSION | CREATETIME | +-------------------------------------------------------------------------------------------------------------+ 1_| CRBFID0 | AA01MP | CJ45_8210_TDV | 2008-06-06-09.03.57.265416 | 2_| CRBFID0 | AA01MP | isb45_8210_1 | 2008-10-30-10.44.13.749868 | 3_| CRBFID0 | AA01MP | isb45_8210fp3_Hotfix001.081021 | 2009-01-14-09.31.34.361108 | 4_| CRBFID0 | AA01MP | isb45_8210fp5_Hotfix001_090216 | 2009-02-27-11.27.59.529836 | 5_| CRBFID0 | AA01MP | isb45_8210fp7_Hotfix001_090310 | 2009-04-30-15.53.47.888161 | 6_| CRBFID0 | AA01MP | isb45_8210fp8_Hotfix001_090609 | 2009-06-16-14.27.03.542562 | 7_| CRBFID0 | AA01MP | isb45_8210fp8_Hotfix001_090615_WAS6.1 | 2009-06-17-18.24.05.582171 | 8_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090722 | 2009-07-27-15.53.44.351321 | 9_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090730_WAS6.1 | 2009-08-04-09.48.02.599017 | 10_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090819 | 2009-08-20-17.37.08.907330 | 11_| CRBFID0 | AA01MP | isb45_8210fp9_Hotfix001_090821_WAS6.1 | 2009-08-21-16.21.56.794659 | +-------------------------------------------------------------------------------------------------------------+ 18
  • Corebank Application and Package Versioning• Keep the last 5 versionsSELECT DISTINCT SUBSTR(SYSPACK.VERSION,1,46) AS VERSION,DATE(TIMESTAMP) AS TARIH FROM SYSIBM.SYSPACKAGE SYSPACK WHERESYSPACK.COLLID LIKE CRB%‘ ORDER BY TARIH DESC; +-------------------------------------------------------------+ ! VERSION ! TARIH ! +-------------------------------------------------------------+1_! isb50_9440fp5_DEV_100701 ! 2010-07-12 !2_! isb50_9440fp4_DEV_100416 ! 2010-05-06 !3_! isb50_9440fp3_DEV_100402 ! 2010-04-09 !4_! isb50_9440fp3_DEV_100305 ! 2010-03-18 !5_! isb50_9440fp3_DEV_100305 ! 2010-03-17 !6_! isb45_8210fp10_Hotfix001_091109_zOS ! 2009-11-12 !7_! isb45_8210fp10_Hotfix001_091023_WAS61 ! 2009-10-23 ! +-------------------------------------------------------------+ 19
  • Corebank Application and Package Versioning• DISABLE 4 versions except with the highest CREATETIMEREBIND PACKAGE (CRBMIG0.*.(isb50_9440fp4_DEV_100416)) FLAG(E) -DISABLE(BATCH,CICS,DB2CALL,DLIBATCH,IMSBMP,IMSMPP,REMOTE,RRSAF) )DSNT408I SQLCODE = -807, ERROR: ACCESS DENIED: PACKAGE PGM01IS NOT ENABLED FOR ACCESS FROM BATCH 20
  • Corebank Application and Package Versioning• Free the rest of the packages FREE PACKAGE (CRBDEV0.*.(isb45_8210fp10_Hotfix001_091109_zOS)) ) 21
  • IMS & DB2 IMS DB2 Co-existence• All DB2s in Data Sharing z-OS LPAR LPAR COBOL LPAR LPAR Application IMS DB DB2 DB2 SQLs Data Sharing AIX API• Proven Architecture JVM• High Availability, Scalability, Flexibility with DB2Data Sharing and IPPlex• Additional SQL Workload for IMS LPAR 22
  • Cobol & Unicode• CREATE TABLE .... CCSID UNICODE• Unicode Support in Enterprise COBOL• Compiler Option : NSYMBOL(NATIONAL) BIND Option : ENCODING(UNICODE)• DATE Datatype 77 W-TARIH-YYAG PIC N(10). DISPLAY FUNCTION DISPLAY-OF (W-TARIH-YYAG, 1026). 23
  • Cobol & Unicode• IDTYP_VALUE GRAPHIC(4)• 77 V-IDTYP-MUSNO PIC N(4). ... MOVE Ncsid TO V-IDTYP-MUSNO. SELECT COUNT(*) FROM CRBFID0.IP_ID WHERE IDTYP_VALUE = :V-IDTYP-MUSNO 24
  • Unicode & Unload• UNLOAD TABLESPACE DMIGCUST.SIP PUNCHDDN TEMP_2 UNLDDN TEMP_1 FROM TABLE CRBMIG0.IP WHEN (IP_USER_ID<>DMUSER)• IP_USER_ID GRAPHIC(6)• IP_USER_ID<>x444D55534552‘• MR1030081910  Not working for V8, V9, V10 25
  • Unicode & Load• All tables have GRAPHIC • Lot’s of external data in datatype such as; Delimeted Format such as;IFS_IDFR DECIMAL(15, 0) 11.,”9310”,2010-01-01CARDT_VALUE GRAPHIC(4) 22.,”9343”,2010-01-02CARD_EXPR_DATELOAD DATA INDDN CXZ837AQ LOG NO REPLACE EBCDIC CCSID(01026,01208,01200) GRAPHIC (4)  CHAR(4) FORMAT DELIMITED INTO TABLE ÜCRBMIG0Ü. ÜFNRATLÜ ( ÜIFS_IDFRÜ POSITION(*) DECIMAL , ÜCARDT_VALUEÜ POSITION(*) CHAR(004) , ÜCARD_EXPR_DATEÜ POSITION(*) DATE EXTERNAL ) 26
  • IDTHTOIN• IDTHTOIN : Idle Thread Timeout in seconds IDTHTOIN = 0 ; Disables Timeout!• IDTHTOIN = 900 or 3600 or else... • MR1122081733 : Different IDTHTOIN settings for different cases!• DB2 V10 : Profile Tables  27
  • IDTHTOIN• DB2 V10 GA • For monitoring threads and connections• SYSIBM.DSN_PROFILE_TABLE PROFILEID : unique identifier for the profile AUTHID, PLANNAME, COLLID, PKGNAME, IPADDR : Monitored Items• SYSIBM.DSN_PROFILE_ATTRIBUTES PROFILEID : unique identifier for the profile KEYWORDS, ATTRIBUTE1, ATTRIBUTE2 = MONITOR IDLE THREADS, WARNING or EXCEPTION, threshold value 28
  • INCORROUT• SELECT * FROM CRBFID0.ISB_PROVISION_INPUT T1 WHERE T1.PRV_IDFR = 134000001420080 AND T1.PRVIN_HOST_TS = (SELECT MAX (PRVIN_HOST_TS) FROM CRBFID0.ISB_PROVISION_INPUT WHERE PRV_IDFR = T1.PRV_IDFR AND PRVIT_TYPE = T1.PRVIT_TYPE)• SELECT * FROM CRBFID0.ISB_PROVISION_INPUT T1, ( SELECT MAX (PRVIN_HOST_TS) PRVIN_HOST_TS, PRVIT_TYPE FROM CRBFID0.ISB_PROVISION_INPUT WHERE PRV_IDFR = 134000001420080 GROUP BY PRVIT_TYPE ) AS T2 WHERE T1.PRV_IDFR = 134000001420080 AND T1.PRVIN_HOST_TS = T2.PRVIN_HOST_TS• Tested with DB2 for z V8, V9, DB2 for LUW & Oracle• PMR - 20549,001,862 = Apar Fix: AM05402 29
  • Tuning ExperiencesCLASS 2 TIME DISTRIBUTION-------------------------------------CPU !==> 4%SECPU !NOTACC !========================> 48%SUSP !========================> 48%  Accounting ReportsAVERAGE APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 2:06:13.02 1:46:04.64 SQL and Index Design Buffer Pools Physical Design, Partitioning... 30
  • Tuning ExperiencesDASD AVG RESP TIME : .927CPU Activity:LPAR Busy: 40.04  RMF ReportsMVS BUsy: 100.0Distribution of address spaces in INREADY QUEUE <=N: 3.2 % Physical CPU was “saturated” Little Paging as well. 31
  • Tuning ExperiencesAVERAGE APPL(CL.1) DB2 (CL.2)------------ELAPSED TIME ---------- 20:13.1970 ---------- 17:41.2783  Accounting Reports Almost %500 Better Performance ! 32
  • Tuning ExperiencesBatch CobolWill Update 100M Records with nice Commit freq. 1-2Million Updates in 15-20 secondsThen slows down significiantly+ Waits Count Total+ --------------------------------- ---------- ------------+ Synchronous I/O Wait 19834 00:00:21.688+ Asynchronous Read I/O Wait 34 00:00:00.074+ Log Write I/O Wait 360252 00:07:48.482+ Sync EX Unit Sw-com/abort/dealloc 22 00:00:00.032+ ------------------------------------------------------------+ Total Class 3 Wait Time 00:08:27.970 Check RMF Reports! CPU Cap after 10-20 sec. With enough CPU  40,000 Update/sec 33
  • ODS Application Migration ODS ApplicationNear 7/24 ApplicationBusiness Logic : Cobol Stored ProcedureData Migration  DRDA & XLoader & Parallelism 34
  • ODS Application MigrationDSNL030I -PR4B DSNLILNR DDF PROCESSING FAILURE FORLUWID=GA521E67.E4ED.000000000000AUTHID=-N/A-, REASON=00D31034The MAX REMOTE CONNECTED limit was reachedETL JobsFeeding ODS from OLTP, DW and more!...Idle Thread  3600 !CondBat (max remote connected) is increased! 35
  • V9 Migration Update RSU0908 Applied to V8 Migration/fallback : II14401, II14464, II14441 Development and Prod DB2 in Sandbox Lpar V9 Order Arrived V9 CM Install to Sandbox Lpar Fallback, Remigrate, ENFM, NFM Test CM, Prod CM, Test NFM, Prod NFM36 36
  • V9 Migration Gotchas! Before Plan Stability usage: SPT01 is enlarged 3 times more GRAPHIC was seen as VARGRAPHIC and vice versa in IBM Admin Tool 7.2.0 APAR PM17236  PMR 23637,001,862 DSNJ111E -AT3B OUT OF SPACE IN ACTIVE LOG DATA SETS DSNJ115I -AT3B OFFLOAD FAILED, COULD NOT ALLOCATE AN ARCHIVE DATA SET AT3BMSTR(REGION=32K  REGION=0M) DB2 Crash during OMPE Start  Addresed in PK6837737 37
  • Thinking V10GA Since Oct 22ndDB2 10 is optimized on zEnterpriseCPU savings … right out of the box zEnterprise Upgrade to Hardware DB2 10 100 19% less CPU usage 80 60 40 20 0 DB2 9.1 DB2 10 38
  • Thinking V10• Performance improvement Categories1. Improvements that you receive by simply migrating to Version 102. Improvements that are implemented by DBA-level changes, without requiring application changes3. Improvements that require application changes4. Improvements from scalability enhancements 39
  • Thinking V101. Improvements that you receive by simply migrating to V10• Out-of-the box savings in NFM• Improved index matching for OR and IN predicates• Parallelism enhancements such as for multi-row fetch• Collection of autonomic statistics. Stored procedures will determine if statistics need to be collected then DB2 performs the collection. 40
  • Thinking V102. Improvements that are implemented by DBA-level changes, without requiring application changes• New Access Path: Hash Access WHO can get benefit ? 1-Queries uses equal predicates to access a single row 2-Tables of a predictable and reasonably static size. 41
  • Thinking V103. Improvements that require application changes• Access to currently committed data it minimizes transaction suspension HOW ? 1- a new bind option CONCURRENTACCESSRESOLUTION 2- two new PREPARE statement clauses, USE CURRENTLY COMMITTED and WAIT FOR OUTCOME, which you can specify in the attribute-string. 42
  • Thinking V104. Improvements from scalability enhancements• The scalability enhancements contributes to improved performance Reductions in log latch contention Most cases, DB2 holds the log latch for less time when creating log records, because of the increased use of compare and swap logic. Enhancement does NOT require configuration or application changes. Provides reductions in transaction and CPU time when logging rates are high. 43
  • Thinking V10 MORE...• 10x more concurrent users, up to 20,000 / subsystem (MAXDBAT zparm)• Temporal Data• Thread Management thru Profile Tables 44
  • Cuneyt GoksuVBTIBM DB2 Gold Consultantcuneyt.goksu@vbt.com.trF13Lessons Learned from Isbank – A Story of a DB2 forz/OS initiative 45