• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Data Integration Solutions - Michał Kuratczyk
 

Data Integration Solutions - Michał Kuratczyk

on

  • 1,224 views

EXAbreakfast, 25.08.2011 r.

EXAbreakfast, 25.08.2011 r.

Statistics

Views

Total Views
1,224
Views on SlideShare
1,224
Embed Views
0

Actions

Likes
1
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Data Integration Solutions - Michał Kuratczyk Data Integration Solutions - Michał Kuratczyk Presentation Transcript

    • <Insert Picture Here>Data Integration SolutionsMichał KuratczykData Integration Solutions, EMEA
    • The Data Integration Business Problem Application Screen Scrape Download Application Message File Application Queue Screen Application Scrape Sockets Screen Transaction Transaction Scrape File File Application Sockets Download CICS Gateway RPC File ORB APPC Application Message ORB Application Application Message Transaction Queue File Application Message Queue CICS Gateway Transaction Screen File Scrape APPC Application Message Download RPC File
    • The Data Integration Business Problem Application Screen Scrape Fragmented, Inaccurate, Stale Data Download Application Message File Application Queue Screen Application Scrape Sockets High Cost of Integration, Operational Impact Screen Transaction Transaction Scrape File File Application Sockets CICS Gateway Download File Data Explosion , Increased batchORB RPC windows, APPC Poor performing ETL Application Message ORB Application Application Message Transaction Queue File Uptime of Distributed, Replicated Systems Application Message Queue CICS Gateway Transaction Screen File Scrape APPC Application Message Download RPC File
    • Oracle Data Integration SolutionsEnterprise-Wide Solutions OGG ADG Zero Downtime Migrations & Upgrades New DB/HW/OS/APP Disaster Recovery Heterogeneous OGG ADG Source Systems Active/Active High Availability Fully Active Distributed DB OGG ADG ADG Query Off-Loading OGG Reporting Database ODI OGG ADG ODS OGG ODI Data Warehouse Real-Time & Batch BI OGG ODI Data Distribution Global Data Centers BPM OGG ODI BPM SOA/EDA CEP 4
    • Oracle Data Integration SolutionsPortfolio ODI OGG EDQ ADG• Oracle Data Integrator (ODI) o Next generation ELT technology with broad heterogeneous connectivity (including real-time via Oracle GoldenGate) offering industry leading Data Quality and Profiling options. Pre-built knowledge modules and optimized to work with all Oracle DIS products• Oracle GoldenGate (OGG) o Real-time, low-impact, transactional data capture, distribution and delivery (replication) across heterogeneous systems and platforms.• Oracle Data Quality (EDQ) o The best of breed option for managing your data quality implementations as part of the integration process. Pre-built knowledge modules and quality rules as well as optimized for Oracle Data Integrator (ODI).• Oracle Active Data Guard (ADG) o The best performing maximum availability solution for Oracle 11g databases, offering access to standby databases and supporting production offloading with enhanced recovery options 5
    • Oracle Data Integration Solutions ODI OGG EDQ ADG Compete with Oracle Data Integrator • Fastest E-LT bulk/batch performance Faster • Leverage underlying system parallelism • Faster project setup, design and delivery • Simpler to setup, configure, manage, & monitor Simpler • Map using declarative SQL interfaces • Deploy with fewer hardware devices • Extend with knowledge module & code templates • Less HW & Energy Costs with E-LT Architecture Saves Money • Less Time Wasted on Unnecessary ETL(Lower TCO, Higher ROI) Mappings, Scripting, and Complex Training • Less Integration Overhead Integrating with Applications, SOA, and Management Software 6
    • ODI OGG EDQ ADG Declarative Design Developer ProductivityBenefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-IT pros 7
    • ODI OGG EDQ ADG Knowledge Modules Hot-Pluggable: Modular, Flexible, ExtensibleBenefits Tailor to existing best practices Ease administration work Reduce cost of ownership 8
    • ODI-EE Knowledge Modules ODI OGG EDQ ADG Pluggable Architecture for Improved Flexibility Pluggable Knowledge Modules Architecture Reverse Load from Journalize Check Integrate, Data Engineer Source to (CDC) Constraints Transform Service Metadata Staging Sample out-of-the-box Knowledge Modules Oracle Oracle External TPump/ Oracle Web SAP ERP SAP BW JMS Queues Oracle Merge Datapump DBLink Tables Multiload Services eBusiness Oracle Siebel EIM DB2 Web Siebel DB2 Exp/Imp J.D Edwards Type II SCD Suite SQL*Loader Schema Services• Key Architecture Benefits: • Ease of maintenance, • Flexibility, tailored to existing best practices, • Reduces cost of ownership 9
    • Report to Source Data Lineage ODI OGG EDQ ADGFrom Target Columns to Source Columns 10
    • Oracle Data Integrator Enterprise Edition Best ETL Platform for Exadata• ODI’s E-LT architecture is best for Exadata • Exadata powers all transformations • No additional hardware is required • Every join benefits from Smart Scan • ODI jobs can take advantage of Smart Flash Cache • ODI performance scales linearly with Exadata infrastructure • ODI transforms the data without it ever leaving the Exadata Machine• ODI implements Exadata best practices • Fastest Loading Mechanisms • External Tables, SQL*Loader, DBLINK, Data Pump • Out of the box Index Management • Support for bitmap indexes, unique/non-unique indexes • Advanced Oracle Features Support • Optimizer hints (Direct Path loading, Parallel, etc) • Merge & Multi-Table Insert statement • Any Oracle function can be leveraged (incl. analytic functions) • Oracle OLAP and Spatial support 11
    • ODI on Exadata Benchmark ResultsUp to 7TB per hour of real world data loading and complex transformations ODI ELT w/ Exadata  ODI scales with Exadata  Loads increase linearly as Exadata scales  ODI runs on Exadata – no ETL hardware required  No new hardware required as data sets grow  ODI processes used only during integration runs  Exadata continually available for OLTP, BI, DW, etc  Common administration, monitoring and management  All the benefits of rapid tools-based ETL development Conventional ETL  As data sets grow additional hardware ($$) needed to scale  ETL parallel optimization and design ($$$) is heavily dependent on resources available to the ETL environment  Sources, integrations, targets must be designed to match processing power of ETL environment  Source flat files split to match # of ETL engine CPU’s  Integration grid setup appropriately to match # of ETL engine CPU’s  Target partitions, table spaces to match # of ETL engine CPU’s  ETL engine hardware resources only used for ETL  Cannot be utilized for OLTP, BI, DW, etc.  Hardware not co located, multiple vendors  Different management, monitoring and administration from database and BI infrastructure ($$) 12
    • Oracle Data Integration Solutions ODI OGG EDQ ADGCompete with Oracle GoldenGate • Sub-second latency for data movement Performance • Low-impact capture, routing, transformation, and delivery of transactional data • Open, modular architecture Flexible & • Supports heterogeneous sources and targets Extendable • Agile and adaptable to modern data integration architectures • Maintains transactional integrity Reliable • Resilient against interruptions and failures • Fast Recovery after process interruptions. 13
    • ODI OGG EDQ ADG Expanded HeterogeneityDatabases O/S and PlatformsOracle GoldenGate Capture:  Oracle • Linux  DB2 • Sun Solaris  Microsoft SQL Server  Sybase ASE • Windows 2000, 2003, XP  Teradata • HP NonStop  Enscribe • HP-UX  SQL/MP  SQL/MX • IBM AIX  MySQL • IBM z Series  JMS message queues • zLinuxOracle GoldenGate Delivery:  All listed above, plus:  TimesTen, IBM System i  Netezza, Greenplum, HP Neoview and any ODBC compatible databases  ETL products 14
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Capture Source TargetOracle & Non-Oracle Oracle & Non-Oracle Database(s) Database(s) 15
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Trail: stages and queues data for routing. Trail Capture Source TargetOracle & Non-Oracle Oracle & Non-Oracle Database(s) Database(s) 16
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Trail Capture Pump Source TargetOracle & Non-Oracle Oracle & Non-Oracle Database(s) Database(s) 17
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Trail Trail Capture Pump Source TargetOracle & Non-Oracle Oracle & Non-Oracle Database(s) Database(s) 18
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Delivery: applies data with transaction integrity, transforming the data as required. Trail Trail Capture Pump Delivery Source TargetOracle & Non-Oracle Oracle & Non-Oracle Database(s) Database(s) 19
    • ODI OGG EDQ ADG How Oracle GoldenGate Works Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Delivery: applies data with transaction integrity, transforming the data as required. Trail Trail Capture Pump Delivery Source TargetOracle & Non-Oracle Bi-directional Oracle & Non-Oracle Database(s) Database(s) 20
    • Oracle Real Time Business Intelligence Complete, Flexible, Integrated OBI EE Suite Plus Real-time Analytics Historic Analytics Batch Feed Oracle DIM DIM Data Integrator &EMP DEPT Data Quality Log-based, Real-time Data Feeds FACT 22 Oracle GoldenGate EMP DEPT DIM DIMSource OLTP System ODS Schema DW Schema Oracle Exadata 22
    • ODI OGG EDQ ADG GoldenGate for Database Upgrades Eliminate Downtime During Upgrades to Oracle Database 11g R2• Zero database Switchover downtime for upgrades from 8i, 9i, Application Real-time changed data 10g to 11g• Leverage new features of Oracle Database 11g Oracle Oracle without impacting Database Database 8i/9i/10g Failback Data Flow 11g Release 2 business operations• Minimize risks with failback option Compare & Verify 23
    • Exadata and Oracle Data Integration Solutions ODI OGG EDQ ADG Customers Running DIS Products on Exadata Swiss Media US: Marketing Turkey: TelcoGoldenGate to migrate US: Grocery and bring up an Logistics Solutions ODI used to load and entirely new GoldenGate for a move large volumes infrastructure ODI with Exadata to phased migration to of data between including Exadata. allow for future Exadata with more than 20 source systems,Availability, scalability business growth - 12 terabytes of data 12 datamarts, and and security are key replacement of from heterogeneous their overall DW on requirements. manual processes. legacy environments Exadata.US: Retail, Logistics UK: Tools Retailer US: Wireless Turkey: Telco ODI and EDQ to Chose GoldenGate Manufacturer move and cleanse and ODI with ODI and GoldenGate OGG for offloading to offload reporting data in an automated Exadata for new operational reporting from OLTP then fashion into Exadata, custom CDI project. to Exadata for populate DW on replacing a improved OLTP Exadata in real-time. previously intensive performance. manual process. 24