IMS CT Test Envir - IMS UG November 2012 Phoenix


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

IMS CT Test Envir - IMS UG November 2012 Phoenix

  1. 1. Data Management ToolsCloning Test Systems –Cost, CPU, and Time Savings Rosemary Shay Email: Ron Bisceglia Email: © 2012 IBM Corporation M
  2. 2. Data Management Tools – IMS ToolsTopics Why Do We Test Copying Data, the Fast Way Cloning Automation – Cloning Terminology – Challenges of Cloning – Clone IMS systems – Refresh IMS databases Additional Test Environments2 © 2012 IBM Corporation
  3. 3. Data Management Tools – IMS ToolsWhat Causes Outages Based on extensive feedback from clients, we estimate that, on average, unplanned application downtime is caused: 20 percent of the time by hardware (e.g., server and network), OSs, environmental factors (e.g., heating, cooling and power failures) and disasters; 40 percent of the time by application failures including "bugs," performance issues or changes to applications that cause problems (including the application code itself or layered software on which the application is dependent); and 40 percent of the time by operator errors, including not performing a required operations task or performing a task incorrectly (e.g., changes made to infrastructure components that result in problems and incur unexpected downtime). Thus, approximately 80 percent of unplanned downtime is caused by people and process issues, while the remainder is caused by technology failures and disasters. Improving availability requires a different strategy and set of investment choices for each of the three unplanned downtime categories.” -- Gartner Group3 © 2012 IBM Corporation
  4. 4. Data Management Tools – IMS ToolsDatabase Downtime Drives Up Costs Most organizations spend an extra $1.5M USD per year because of unplanned database downtime Very few organizations have perfect or near-perfect datacenter uptime Only 3% of organizations have uptimes of 100% Only 4% of organizations have uptimes of 99.9% Source: Aberdeen Group, 20104 © 2012 IBM Corporation
  5. 5. Data Management Tools – IMS ToolsWhy is Test Data Management Important? Improving business critical applications – Improve application reliability – Increase functionality – Shorten time to market – Reduce development and testing costs Sharing resources and skills for development and testing – Setting up a test environment usually involves: • Systems programmer • DBA • Application developer • Tester – Test databases are typically shared Meeting technical requirements – Application testing done in a production replica – Technical changes tested before production implementation Meeting regulatory requirements – Protection of personal or sensitivity data5 © 2012 IBM Corporation
  6. 6. Data Management Tools – IMS Tools Copying Data, the Fast Way6 © 2012 IBM Corporation
  7. 7. Data Management Tools – IMS ToolsHost Based Data Copy Options Data copy processes use host based CPU and I/O facilities More costly and slower than storage-based fast replication Host-based Copy Process Volume copy options – DFSMSdss (IBM) – FDR (Innovation Data Processing) – TDMF (IBM) – FDRPAS (Innovation Data Processing) Data set copy options – DFSMSdss (IBM) – FDR (Innovation Data Processing)7 © 2012 IBM Corporation
  8. 8. Data Management Tools – IMS ToolsStorage Processor = Storage + Processor Storage – Non-volatile Host – Computer components and recording media that based APIs retain digital data Processor – Computing system that executes software programs Storage Processors – Enable data movement through host-based API – Allows CPU reduction from host-based systems8 © 2012 IBM Corporation
  9. 9. Data Management Tools – IMS ToolsWhat is Storage-based Fast Replication? For volume replication a relationship is established between a source and a target (Pair) – Geometrically similar devices A near instant copy of a volume/data set at a Storage specific point in time Processor-based Copy Process – Builds a bitmap to describe the source volume – After the bitmap has been created, the source and target volume data can be used immediately Data movement (CPU and I/O) offloaded to storage processor – Frees up resources on host processor – No host CPU or I/O costs Consistency Groups – Group of volumes copied at exactly the same point in time while maintaining the order of dependent writes –9 © 2012 IBM Corporation
  10. 10. Data Management Tools – IMS ToolsFast Replication Data Copy Options Fast copy processes offloaded to the storage processor No host CPU or I/O resources Volume based fast replication options Fast Replication – FlashCopy (IBM,EMC,HDS) Commands from z/OS – SnapShot (IBM,STK) – TimeFinder/Clone Volume Snap (EMC) – TimeFinder/Snap (EMC) – Mirror processes •PPRC (IBM,EMC,HDS) •TimeFinder/Mirror, SRDF (EMC) •ShadowImage HUR (HDS) Data set based fast replication – Data set FlashCopy (IBM,EMC,HDS) – Data set SnapShot (IBM,STK) – TimeFinder/Clone data set Snap (EMC)10 © 2012 IBM Corporation
  11. 11. Data Management Tools – IMS ToolsDatabase and Storage Integration Repurpose resources that you already own: Storage systems – You already have a significant investment in storage systems – Use storage systems to do more than you thought they could • Simplify database administration tasks • Speed up database administration tasks • Reduce the costs of database administration tasks Database knowledgeable and storage aware – Storage aware tools understand the language and the environment of the database administrator • Assists the DBA by doing research that the DBA would normally have to do. • Translates database administration clone, copy and refresh functions into commands used by the storage subsystem • Provides simulation and undo options • Provides a safe, user friendly interface between the DBA and powerful storage utilities – Allows the Storage Administrator to breath easier while the DBA uses storage tools11 © 2012 IBM Corporation
  12. 12. Data Management Tools – IMS Tools Database and Storage Integration Mainframe Application and Database Database Management Systems Domain • Organizational Integration • New Backup Methods Storage Aware • New Recovery Strategies Database Tools • Business Recovery Monitoring • Cloning Automation • Disaster Restart Solutions Storage Administration Source and Database Business Continuity Backup, Domain Clone, DR12 © 2012 IBM Corporation
  13. 13. Data Management Tools – IMS Tools Cloning Automation13 © 2012 IBM Corporation
  14. 14. Data Management Tools – IMS ToolsCloning Terminology A clone is an exact replica – Clone an IMS system by volume – Clone an IMS Database by data set IMS system cloning and database refresh – The act of replicating the data, making the replica accessible, and then using the replica in lieu of the original data IMS system cloning automation – Clones a complete IMS system including all its databases IMS database refresh – Refreshes specific IMS databases14 © 2012 IBM Corporation
  15. 15. Data Management Tools – IMS Tools IMS System ‘Cloning’ Today Production LPAR Test LPAR IMS001 IMS001 User Catalogs User Catalogs IMS002 IMS002 IMS System Data IMS System Data Sets Sets IMS003 IMS003 Application Application Libraries Libraries IMS004 IMS004 RECON Data Sets RECON Data Sets IMS005 IMS005 Data Base Data Data Base Data Sets Sets15 © 2012 IBM Corporation
  16. 16. Data Management Tools – IMS Tools IMS Database ‘Cloning’ Today z/OS Production IMS Test IMS IMS System Data IMS System Data Sets Sets Application ??? Application Libraries Libraries RECON Data Sets Unload Load RECON Data Sets Data Base Data Data Base Data Sets Sets Image Copy Recovery Index Rebuild16 © 2012 IBM Corporation
  17. 17. Data Management Tools – IMS ToolsChallenges to Using Volume Fast Replication IMS system cloning is best done using volume replication – Large amounts of data – Volume replication much faster than data set replication • No z/OS dataset allocation Volume data is replicated fast and easy but… – How do you access the cloned data? Inherent Problems: – VOLSERs may have the same volume names as the source – Volume VTOC, VTOCIX and VVDS would reference the source VOLSERs – Data set names are the same on source and target volumes17 © 2012 IBM Corporation
  18. 18. Data Management Tools – IMS ToolsIMS Cloning Tool - Provides Data Access On the Same or Shared LPAR Rapid Volume Reconditioning – IMS Cloning Tool performs rapid volume reconditioning and data set renaming on cloned system volumes to solve the data access challenges – Provides access to data sets on target volumes • Renames the VTOC, VTOCIX, and VVDS to match the target volume • Renames and catalogs all data sets to a new HLQ – Solution to enable the use of the data sets on the target volumes on the same image • No more complicated cloning • No more requirement for a separate image18 © 2012 IBM Corporation
  19. 19. Data Management Tools – IMS Tools IMS Cloning Tool - Provides Data Access On the Same or Shared LPAR Source Target A1.CAT PIMS01 TIMS01 ICF User VTOC VVDS VTOC VVDS Catalog VTOCIX VTOCIX A.DSN3 B.DSN3 A.DSN2 B.DSN2 A.DSN1 B.DSN11. Target Volume is renamed.2. Volume VTOC, VTOCIX and VVDS now reference the target VOLSERs.3. Data sets are renamed and cataloged.24 © 2012 IBM Corporation
  20. 20. Data Management Tools – IMS ToolsAdjusts Target IMSTo Accommodate and Accept the Cloned Data RECONs data sets – Data set names, IMS subsystem IDs, and VOLSERs are updated in the following RECON records: header record, database data set records, online log records, and back-out records IMS PROCLIB and JOBS and user JCL libraries – New values for IMSID, VOLSERS, and data set names in the JCL members within these libraries MDA (MVS Dynamic Allocation) members for databases and system data sets, are updated to reflect the new data set names. Updates include: – The RECON data sets – Online Log data sets (OLDS) – Write-ahead data sets (WADS) If IMS data sharing is involved, each additional IMS data sharing member is updated20 © 2012 IBM Corporation
  21. 21. Data Management Tools – IMS ToolsIMS Cloning Tool Delivers Solutions Leverages Storage-based fast replication – Very fast -13TB in .4 seconds - no outage of source system – Large systems cloned in less then 30 minutes – Saves hours over traditional cloning – Saves CPU and I/O processing Automation – We condition the copied volumes for usability – We condition the cloned IMS’s RECON, PROCLIB,JOBS, MDA members – everything you need to bring up your cloned IMS – Databases are copied, underlying data sets renamed, DBRC updated – Repeatable process With IMS Cloning Tool it is now easy and inexpensive to create cloned environments for testing, offloading reporting, moving data or aiding in problem determinations.21 © 2012 IBM Corporation
  22. 22. Data Management Tools – IMS ToolsDo I have to Use Storage-based Fast Replication? IMS Cloning Tool works with Host- Host-based Copy Process based copy options - Consider the availability requirements of the source system - Consider the amount of data being copied All the same test environments can be created Rapid volume reconditioning will be performed to make the copied volumes accessible The target IMS datasets will be conditioned22 © 2012 IBM Corporation
  23. 23. Data Management Tools – IMS Tools Walking Through the Cloning Process23 © 2012 IBM Corporation
  24. 24. Data Management Tools – IMS ToolsPre-System Cloning Steps Review and Define Target System PROCLIB members – Determine what components and configuration options you want • IRLM, CSL, XRF, IMS Connect, etc. By default, because the clone is an exact replica of the source, the clone would have the same components as the source – Review IMS.PROCLIB members (DFSPBxxx, etc.) – IMS Configuration Manager can review and create new members24 © 2012 IBM Corporation
  25. 25. Data Management Tools – IMS ToolsIMS System Cloning Automation Performs IMS system cloning automation – Takes an existing IMS system (complete installation and Production system generation process completed) and creates a new, Database or cloned, IMS system from it without having to repeat the entire installation and system generation processes – Simplifies IMS cloning processes IMS – Reduces cloning time and administration costs IMS Clone Leverages fast replication facilities to clone data IMS Cloning Tool – Host-based copy can also be used Cloning Automation Performs rapid volume reconditioning and data set renaming on cloned volumes to solve the data access challenges – Target volumes retain their target volume label – Renames the VTOC, VTOCIX, and VVDS to match the target volume Source Clone Database Database – Renames and catalogs all data sets to a new HLQ Volumes Volumes Adjusts target IMS to accommodate and accept the cloned data – IMS RECONs, PROCLIB, JOBS, JCL, MDA members25 © 2012 IBM Corporation
  26. 26. Data Management Tools – IMS ToolsIMS System Cloning Steps Production IMS Target IMS ‘Source’ IMS IMS Clone 1 IMS volume 5 Volume Recondition selection 2 A. Log Suspend 6 Data set rename B. Consistency Group 7 Update cloned IMS meta data 3 Volume copy 8 Start cloned IMS 4 Log Resume if 2A Source Clone Database Database Volumes Volumes26 © 2012 IBM Corporation
  27. 27. Data Management Tools – IMS Tools Status of In-Flight Transactions An online cloning solution often results in transactions in flight These in-flight transactions, cloned to the target system result in the same target system action that would happen on the source system if it were to have died at that same time and then restarted The target restart is essentially an emergency restart of a failed system27 © 2012 IBM Corporation
  28. 28. Data Management Tools – IMS ToolsRefresh IMS Databases Using IMS Cloning Tool Database Refresh Automation Performs automated IMS database refresh Production operations Database – Fast refresh of IMS databases – IMS DB support (FF, HALDB, DEDB) IMS IMS Verifies source and target database compatibility Source Target IMS Cloning Tool IMS data copied using storage-based data set fast- Database Refresh replication Automation – Target takes up the same amount of space as the source – Host-based copy can also be used Performs target system meta-data management – Updates DBRC information for target databases – Supports HALDB special requirements • Primary and secondary indexes and ILDS can be copied to eliminate index and ILDS rebuild time • Partition and reorg numbers will reflect the source RECON IMS IMS System Clone Volumes Volumes 28 © 2012 IBM Corporation
  29. 29. Data Management Tools – IMS ToolsRefresh IMS Databases Using IMS Cloning Tool Production IMS Target IMS ‘Source’ 1 Create target IMS databases IMS IMS if they do not exist Target 2 7 Update DBRC Database Selection for Target Databases 3 Compatibility Check 8 Start Target Databases Stop Source and 4 Target IMS databases 5 Data Set Copy Process Start Source 6 IMS databases29 © 2012 IBM Corporation
  30. 30. Data Management Tools – IMS Tools Additional Test Environments30 © 2012 IBM Corporation
  31. 31. Data Management Tools – IMS ToolsIMS System Skeleton Clone IMS System Skeleton clone – Clones an IMS system without cloning any data – The replicated system is accessible and is usable in lieu of the original system without requiring a system generation – Contains all of the database and application definitions – Creates an IMS system to test the apply process31 © 2012 IBM Corporation
  32. 32. Data Management Tools – IMS Tools Create an IMS Cloned System Skeleton Production IMS Target IMS ‘Source’ Test new releases of IMS 3 Create Cloned IMS System Without IMS IMS Cloned Database Clone Data IMS System Volumes ONLY IMS System IMS IMS 4 Install new 1 IMS release Volume Selection System Skeleton or upgrades Data System Sets 2 Perform IMS 5 Start Your System IMS Cloning Skeleton Data- Automation base Using System Data Sets Volumes Only32 32 © 2012 IBM Corporation
  33. 33. Data Management Tools – IMS ToolsWhat you can test with an IMS System Skeleton Clone IMS version migration – Test your end to end apply process • Test with full production RECON • Re-clone for different environment configuration new/existing • Can you bring your IMS up? – Activate new functionality • Migration to Striping and Buffers in 64-bit Storage • RECON – changing MINVERS • Dynamic Resource Definitions (DRD) – Convert to DRD – IMS Repository Test with full production RECON Manually clean up a RECON – This may take more then one try; simply re-clone between tests33 © 2012 IBM Corporation
  34. 34. Data Management Tools – IMS ToolsIMS Cloned System Skeleton Copy Databases for Application Testing Production IMS Target IMS ‘Source’ 3 Create Cloned IMS System Without IMS IMS Cloned Database Clone Data IMS Database Refresh Environment IMS System and Database Volumes Refresh IMS Databases IMS System IMS IMS 4 Refresh IMS 1 Database A Volume System Selection Data Skeleton Refresh IMS Sets System 5 2 Perform IMS System Database B Cloning Automation Using System Data 6 Test Set Volumes Only A A B B34 34 © 2012 IBM Corporation
  35. 35. Data Management Tools – IMS ToolsIMS Cloned System Skeleton Copy Databases for Application Testing IMS version migration – Focused Application and Database testing • IVP validation steps • IMS Exit Routines • Systems with a history of having issues during past migrations – Test new functionality • FF- Dynamic Database Buffering • DEDB – Secondary Index processing • DEDB- Buffer Manager 64 bit Enhancements Determine if an issues is environmental or application related. Specific application full volume testing35 © 2012 IBM Corporation
  36. 36. Data Management Tools – IMS ToolsFull System Clone Exact copy of production environment for the highest quality testing • Test interfaces with other systems • Application acceptance testing • Load test high volumes • Reset the environment by re-running the cloning process IMS version migration - performance • zAAP/zIIP Accounting Log Records – Distinguish between CP (Central Processor) and zAAP/zIIP times • Logging Enhancements – Striping – improved logging throughput – Moved above 2 GB boundaries – ECSA saves Repeatable performance testing • QCF (Queue Control Facility) • IMS Performance Analyzer36 © 2012 IBM Corporation
  37. 37. Data Management Tools – IMS ToolsIMS Gold Copy Cloned to Multiple Test Environments IMS Targets IMS IMS Source Target A IMS Target IMS Source IMS Cloning Cloning Target B Tool Tool IMS Target C IMS Target D37 © 2012 IBM Corporation
  38. 38. Data Management Tools – IMS ToolsFederated Gold Copy Cloned to Multiple Test Environments Optim reduces and scrubs data from IMS, DB2, VSAM, and sequential files Cloning Tool creates target environments, refreshes databases and tablespaces IMS/DB2 Target A Reduced IMS data data IMS/DB2 Cloning Target B Optim Tool IMS/DB2 DB2 data Target C IMS/DB2 Target D VSAM and sequential38 © 2012 IBM Corporation
  39. 39. Data Management Tools – IMS ToolsIMS Cloning Tool V1.2 Preview ISPF user friendly interface IMS Database Refresh – Data Masking – Apply Logs after refresh process for databases consistency – Support Database Quiesce System Cloning – Enhance data set RENAME masking – Allow storage groups to be used – Support Preserve Mirror Flash Copy – Support Incremental Flash Copy – Support Space Efficient Flash Copy Target Volumes – Support EMC Timefinder/Clone Volume Snap39 © 2012 IBM Corporation
  40. 40. Data Management Tools – IMS ToolsIMS Cloning Tool Benefits Significantly reduced cost by providing quality test environments – Creates many types of testing environments to meet your needs – Creates exact copies of your production environments – Easily leveraged in many test environment schemas Significantly lower CPU and time needed when cloning multiple systems and databases verses traditional methods – Leverage Fast Replication – Perform Rapid Volume Reconditioning – Condition the IMS meta data – It takes 30 minutes vs. 4 days on average (customer experience) Significantly reduced administrator time – IBM IMS Cloning Tool does the administrative work for you – Freeing up people and systems for other requirements40 – It) © 2012 IBM Corporation
  41. 41. Data Management Tools – IMS ToolsQuestions © 2012 IBM Corporation
  42. 42. Data Management Tools – IMS Tools © 2012 IBM Corporation