Replicating IMS Data with InfoSphere Classic - IMS UG Phoenix 12-2013
Upcoming SlideShare
Loading in...5
×
 

Replicating IMS Data with InfoSphere Classic - IMS UG Phoenix 12-2013

on

  • 413 views

 

Statistics

Views

Total Views
413
Views on SlideShare
413
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Replicating IMS Data with InfoSphere Classic - IMS UG Phoenix 12-2013 Replicating IMS Data with InfoSphere Classic - IMS UG Phoenix 12-2013 Presentation Transcript

  • Replicating IMS Data with InfoSphere Classic Replication Gregory Meimers NA IIDR Specialist
  • IMS Replication InfoSphere IMS Replication for z/OS delivers a native IMS-to-IMS software replication solution that supports highavailability IMS data environments. This solution synchronizes the contents of IMS databases on a single site, or across geographically dispersed locations, in near real time with full recovery.
  • The Role of Data Replication Homogeneous and Heterogeneous Data Synchronization Heterogeneous: Synchronize data across multiple systems. Data distribution and synchronization for Billing, Inventory, Financials, Customer data, etc. es ng a Ch Changes Ch ang es Homogeneous: Synchronize data between two or more data centers for Business Continuity & Disaster Recovery & Query Offload Heterogeneous: Synchronize Operational Data Stores with Reporting data bases, Information Server (ETL/Quality), MDM, Data Warehouses, and Datamarts for Business Intelligence, Analytics, Data Consolidation & Migrations 3
  • Consider the New Ways to Use Copied Data Mobile Applications, Business Analytics and Big Data will all drive dramatic increases in query workloads A dedicated query data environment maintained by data replication is a quick and easy way to handle the growth! − Offload the query workload to a real time copy of the data on a secondary system No new programming No visible impact on users No impact on the existing transaction environment − “Update” workload continues on the primary system Optimizes update environment − Using dual sites also introduces continuous availability benefits ! Secondary environment ready and able to carry the load during both planned and unplanned outages
  • The Business Value Reducing Cost (Extracting, Moving & Loading Data) legacy • • • • CPU usage/cost/time for extracting batch data? Is more hardware required to support growing batch extraction process? What are the development costs for supporting custom batch extraction? What is the impact of intraday queries on performance / processing costs? apps dbs xls, xml, flat Extending Application Availability (Batch Windows) • • • Are business applications unavailable during extract or load processes? How many hours are these applications unavailable? What is their data volume growth rate? If batch is not a problem today, will it be tomorrow? warehouse z/OS + Revenue / - Risk (Decisions Based on Current Data) • custom • 5 How is revenue generation impacted by lack of timely access to data (crosssell, up-sell, service)? What are the risks of making important decisions based on outdated or unavailable data?
  • The IBM Data Replication Model Efficient Capture & Delivery of Changes to Enterprise Data Log Based Captures independent of source applications − Minimizes impact on source platform resources − Easy to restart and recover via spill or archive logs Native Apply APIs to optimize throughput − Efficient I/O to targets Broadest Source, Target, OS and Platform support − No other vendor comes close to IBM’s coverage DBMS Message Queue Log ETL Source Data Flat files Capture 6 Push Apply SOA
  • B. InfoSphere Data Replication Expansive support that is continuously expanding SOURCES TARGETS O/S PLATFORM DB2 (z/OS, i, LUW) All Sources § z/OS System z Informix Pure Data for Analytics Red Hat / SuSE System z Oracle Information Server AIX System p MS SQL Server Cognos Now! IBM i OS System i Sybase Solid DB Red Hat / SuSE Intel / AMD IMS Teradata MS Windows Intel / AMD MQ Series / JMS HP-UX HP- Itanium WebMethods / BEA / TIBCO HP-UX HP PA-RISC MY SQL / Greenplum ÷ Solaris Sun Sparc §IMS Target ONLY for an IMS Source VSAM Target ONLY for a VSAM Source ÷ Customized solution, some restrictions
  • IMS Replication
  • InfoSphere Data Replication for IMS for z/OS Two models in one product A. High speed, low latency IMS to IMS data replication spanning unlimited distances IMS databases InfoSphere IMS Replication IMS databases B. Heterogeneous IMS to non-IMS replication*** (when used with InfoSphere Data Replication’s CDC Target Engines) IMS logs InfoSphere IMS Replication IBM InfoSphere Data Replication (CDC) databases IMS databases ETL e.g. DataStage Message queues Applications *** Also offered stand-alone, without the IMS to IMS capability, as IBM InfoSphere Classic Change Data Capture for z/OS
  • Subscriptions Subscription A programming object consisting of multiple replication mappings that identify the source tables that you want to replicate as a consistent group to the target tables. Subscription methods: Continuous Net change Refresh Persistent subscriptions start replicating automatically: When you start the data server After transient failures, such as communication outages The data server disables persistency: When you explicitly stop replication After a non-recoverable error
  • InfoSphere Data Replication for IMS for z/OS Two models in one product A. High speed, low latency unidirectional IMS to IMS data replication spanning unlimited distances Support for updates made via DB/TM, DBCTL and Batch DL/I programs Requires DBRC to “chase” logs for scheduled & recovery modes Replication monitoring is built in Independent initial load of target DB is required Integration with Tivoli for GDPS Active-Active Continuous Availability IMS databases InfoSphere IMS Replication IMS databases
  • IMS CCA – DBD Example DBD NAME=DI21PART,ACCESS=(HISAM,VSAM) DATASET DD1=DI21PART,DEVICE=3380,OVFLW=DI21PARO, SIZE=(2048,2048),RECORD=(678,678) SEGM NAME=PARTROOT,PARENT=0,BYTES=50,FREQ=250, x EXIT=(*,NOKEY,DATA,NOPATH,(NOCASCADE),LOG) FIELD NAME=(PARTKEY,SEQ),TYPE=C,BYTES=17,START=1 SEGM NAME=STANINFO,PARENT=PARTROOT,BYTES=85,FREQ=1 FIELD NAME=(STANKEY,SEQ),TYPE=C,BYTES=2,START=1 SEGM NAME=STOKSTAT,PARENT=PARTROOT,BYTES=160,FREQ=2, EXIT=(*,KEY,DATA,NOPATH,(CASCADE,KEY),LOG) FIELD NAME=(STOCKEY,SEQ),TYPE=C,BYTES=16,START=1 SEGM NAME=CYCCOUNT,PARENT=STOKSTAT,BYTES=25,FREQ=1 FIELD NAME=(CYCLKEY,SEQ),TYPE=C,BYTES=2,START=1 SEGM NAME=BACKORDR,PARENT=STOKSTAT,BYTES=75,FREQ=0 FIELD NAME=(BACKKEY,SEQ),TYPE=C,BYTES=10,START=1 DBDGEN FINISH END x
  • A. IMS to IMS Data Replication Read source IMS logs - Send committed changes - Apply changes in parallel Source IMS DBs Replication Metadata ACBLIB IMS Classic Data Architect Replication Metadata ACBLIB Target IMS DBs Bookmark DB IMS Admin. Services IMS Logs C BR D RECON I AP Log Read/ Merge Unit of Recovery Capture SOURCE SERVER Admin. Services TCP/IP Unit of Recovery Analysis IMS Apply TARGET SERVER IMS DRA Interface
  • A. IMS to IMS Data Replication Capture – Capture Engine reads source IMS log records – Records from distinct logs are merged and properly sequenced – Committed changes are pushed based on unit-of-recovery criteria – All changes for a unit-of-work are sent together Source IMS DBs Replication Metadata ACBLIB IMS Classic Data Architect Bookmark DB IMS Admin. Services IMS Logs C BR D RECON 14 Replication Metadata ACBLIB Target IMS DBs I AP Log Read/ Merge Unit of Recovery Capture SOURCE SERVER Admin. Services TCP/IP Unit of Recovery Analysis IMS Apply TARGET SERVER IMS DRA Interface
  • A. IMS to IMS Data Replication Details of IMS Source Capture Classic routine associated with IMS’s Partner Program Exit. ACTION: Notify Classic Server when an IMS system starts. WHAT: IMS TM / DB* IMS Databases Start Notification Exit Routine TCP/IP Notification IMS Logs BATCH DL/I Batch Start-Stop Exit Routine Replication Metadata Log Reader Service TCP/IP Notification DBRC API Classic routine associated with IMS’s Partner Program Log Info RECON Exit. ACTION: Notify Classic Server when a Batch DL/I job starts or stops. Change Stream Ordering Capture Services WHAT: SOURCE SERVER * includes BMP and DBCTL
  • A. IMS to IMS Data Replication Push – TCP/IP Conversations with the Target Engine • Two threads per conversation • One “control” conversation for source engine • One “data” conversation per subscription Source IMS DBs Replication Metadata ACBLIB IMS Classic Data Architect Bookmark DB IMS Admin. Services IMS Logs C BR D RECON 16 Replication Metadata ACBLIB Target IMS DBs I AP Log Read/ Merge Unit of Recovery Capture SOURCE SERVER Admin. Services TCP/IP Unit of Recovery Analysis IMS Apply TARGET SERVER IMS DRA Interface
  • A. IMS to IMS Data Replication Apply – Receive committed transactions • Serialization based on resources updated by unit of recovery – Apply change to the target in parallel when possible – Replication Bookmark Database required to manage recovery & restarts Source IMS DBs Replication Metadata ACBLIB IMS Classic Data Architect Bookmark DB IMS Admin. Services IMS Logs C BR D RECON 17 Replication Metadata ACBLIB Target IMS DBs I AP Log Read/ Merge Unit of Recovery Capture SOURCE SERVER Admin. Services TCP/IP Unit of Recovery Analysis IMS Apply TARGET SERVER IMS DRA Interface
  • A. IMS to IMS Data Replication Target Engine Details DRA thread Staged Unit-of-Recovery Data E G es N A ag H C ess M CHANGE Messages Writer Services Apply Service IMS Writer Services Dependency Analysis TARGET SERVER WHAT: Patented Dependency Analysis ACTION: Parallelizes writes to the target when possible to increase end to end throughput.
  • A. IMS to IMS Data Replication Sample Replication Performance Source System − 8-way IMSPLEX − Peak OLTP workload: − Peak Batch workload: 100,000 – 120,000 updates/second 2,200 – 2,600 transactions/second Target System − 2-way IMSPLEX − OLTP Latency: Sub-second end-to-end − Batch Latency: Maximum 10 minute delay during peak processing only
  • InfoSphere Data Replication for IMS for z/OS Two models in one product B. Heterogeneous IMS to non-IMS replication Continuous mirroring: Apply changes at target as made at source Refresh: Apply a snapshot version of source system − Shares IMS to IMS Capture Engine but Requires “Classic” IMS meta data to map to a relational target model − Requires InfoSphere Data Replication’s CDC Target engine for: 1. Broad targeting and target engine transformations 2. Map source to target replication subscriptions 3. View and report on data flow characteristics IMS logs InfoSphere IMS Replication IBM InfoSphere Data Replication (CDC) databases IMS databases ETL e.g. DataStage Message queues Applications
  • B. IMS to Non-IMS Data Replication Classic Data Architect Source IMS DBs Management Console Access Server ACBLIB Replication Metadata IMS IMS Admin Agent Classic Server IMS Logs C BR D RECON I AP Log Read/ Merge Admin. Services Unit of Recovery Capture MetaData Admin API TCP/IP Comm Layer Target Engine SOURCE SERVER IBM InfoSphere Data Replication for IMS (Classic CDC) Apply Agent TARGET IBM InfoSphere Data Replication (CDC)
  • B. IMS to Non-IMS Data Replication Metadata Management, Configuration and Monitoring Classic Data Architect Management Console Access Server Replication Metadata Classic ‒ Map IMS segments to logical tables using imported copybooks & DBDs • Automates translation of legacy data types • Handles legacy constructs like recurring data, redefines, variable lengths, etc. • Metadata-driven WHERE filtering clauses – Configure/Manage Classic address space e.g. Tracelevel, heartbeat interval, logging, IIDR – Define and Manage Subscriptions: • Main configuration object for replication • Link one or more Classic “tables” or “views” to targets • Define source to target communication connection • Start and stop data movement
  • B. IMS to Non-IMS Data Replication Source IMS Capture with “Classic” Data Reformatting Classic Data Architect Source IMS DBs Management Console Access Server ACBLIB Replication Metadata IMS IMS Classic Server IMS Logs RC DB RECON I AP Log Read/ Merge Admin. Services Unit of Recovery Capture SOURCE SERVER IBM InfoSphere Data Replication for IMS (Classic CDC) Capture – IMS Replication Source Server with additional components for: • Transforming captured IMS data into the relational table and view model based on the metadata mapping
  • B. IMS to Non-IMS Data Replication Leverage InfoSphere Data Replication’s Targeting Classic Data Architect Source IMS DBs Access Server ACBLIB Apply Receive committed transactions • Apply any target transformations • Apply changes to the target Replication Metadata IMS IMS Classic Server IMS Logs RC DB RECON 24 Management Console I AP Log Read/ Merge Admin Agent Admin. Services Unit of Recovery Capture TCP/IP Comm Layer Target Engine SOURCE SERVER IBM InfoSphere Data Replication for IMS (Classic CDC) MetaData Admin API Apply Agent TARGET IBM InfoSphere Data Replication (CDC)
  • The Value of Data Replication Reduced down time, MIPS savings, Faster solution delivery Optimize resource utilization by eliminating massive batch movements − Lower costs by saving CPU and network resources Shorten batch windows by streaming changes as they happen − Extend application availability Reduce data latency with “right time” updating − Improve the bottom line with accurate information when and where needed Leading petroleum refiner Major Chinese bank Insurance provider Large U.S. Telco Cuts batch window by 60% Shortens weekly planned outage by 90+% Saves $100K / month $500K savings by eliminating home-grown