Tech oracle data integrator


Published on

Tech oracle data integrator

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Tech oracle data integrator

  1. 1. <Insert Picture Here> Oracle Data Integrator – Technical Deck Mark Pare Sr. Sales Consultant Oracle Higher Education
  2. 2. 3 <Insert Picture Here> • 4 Key Differentiators • Architecture • 6 Steps to Production • ODI or ESB? • Popular Usage Scenarios Agenda
  3. 3. 4 Oracle Data Integrator 4 Key Differentiators
  4. 4. 5 Data Integration Data Warehousing Master Data Management Real Time Messaging FederationMigration Data in Disparate Sources ERP --- ------ --- --- --- CRM - - - Legacy --- ------ --- --- --- --- ------ --- --- --- Best-of-breed Applications Information How and Where you Want It Business Intelligence Corporate Performance Management Business Activity Monitoring Business Process Management HAVE… NEED… - - - - - - - - - - - - Data Synchronization Why Data Integration?
  5. 5. 6 Challenges & Emerging Solutions In Data Integration CHALLENGE EMERGING SOLUTION 1. Increasing data volumes; decreasing batch windows 2. Non-integrated integration 3. Complexity, manual effort of conventional ETL design 4. Lack of knowledge capture Shift from E-T-L to E-LT Convergence of integration solutions Shift from custom coding to declarative design Shift to pattern-driven development
  6. 6. 7 Oracle Data Integrator • Data Movement and Transformation from Multiple Sources to Heterogeneous Targets 1. Performance: Heterogeneous “E-LT” 2. Flexibility: Active Integration Platform 3. Productivity: Declarative Design 4. Hot-Pluggable: Knowledge Modules BENEFITS KEY DIFFERENTIATED FEATURES
  7. 7. 8 Differentiator: E-LT Architecture High Performance Conventional ETL Architecture Extract LoadTransform Next Generation Architecture “E-LT” LoadExtract Transform Transform Transform in Separate ETL Server • Proprietary Engine • Poor Performance • High Costs • IBM & Informatica’s approach Transform in Existing RDBMS • Leverage Resources • Efficient • High Performance Benefits  Optimal Performance & Scalability  Easier to Manage & Lower Cost 1
  8. 8. 9  Enables real-time data warehousing and operational data hubs  Services plug into Oracle SOA Suite for comprehensive integration Oracle Data Integrator Data-oriented Integration Event Conductor Event-oriented Integration Service Conductor Service-oriented Integration Declarative Design Metadata Data Conductor Differentiator: Active Integration Batch, Event-based, and Service-oriented Integration • Evolve from Batch to Near Real-time Warehousing on Common Platform • Unify the Silos of Data Integration • Data Integrity on the Fly • Services Plug into Oracle SOA Suite • Benefits 2
  9. 9. 10 Differentiator: Declarative Design Developer Productivity Conventional ETL Design Specify ETL Data Flow Graph • Developer must define every step of Complex ETL Flow Logic • Traditional approach requires specialized ETL skills • And significant development and maintenance efforts Declarative Set-based Design • Simplifies the number of steps • Automatically generates the Data Flow whatever the sources and target DB • Example: [SALAH] Benefits  Significantly reduce the learning curve  Shorter implementation times  Streamline access to non-IT pros ODI Declarative Design Define How: Built-in Templates Define What You Want Automatically Generate Dataflow 1 2 3
  10. 10. 11 Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse Engineer Metadata  Tailor to existing best practices  Ease administration work  Reduce cost of ownership Reverse Journalize Load Check Integrate Services Pluggable Knowledge Modules Architecture CDC Sources Staging Tables Error Tables Target Tables W S W S W S SAP/R3 Siebel Log Miner DB2 Journals SQL Server Triggers Oracle DBLink DB2 Exp/Imp JMS Queues Check MS Excel Check Sybase Oracle SQL*Loader TPump/ Multiload Type II SCD Oracle Merge Siebel EIM Schema Oracle Web Services DB2 Web Services Sample out-of-the-box Knowledge Modules Benefits Differentiator: Knowledge Modules Hot-Pluggable: Modular, Flexible, Extensible 4
  11. 11. 12 Oracle Data Integrator Architecture
  12. 12. 13 • Java design-time environment • Runs on any platform • Thin client for browsing Metadata • Java runtime environment • Runs on any platform • Orchestrates the execution of data flows • Metadata repository • Pluggable on many RDBMS • Ready for deployment • Modular and extensible metadata Design-Time Metadata Management Runtime Agent Data Flow Conductor Service Interfaces and Developer APIs User Interfaces Thin Client Data Flow Generator Knowledge Module Interpreter Knowledge Modules Master Repository Work Repositories Runtime Repositories Data Flow Generator Runtime Session Interpreter Data Flow Operator Designer Architecture: Conceptual View
  13. 13. 14 Architecture: Component View ODI Design-Time Environment Development Servers and Applications Design-time Repositories Code Execution Execution Log Return Codes Agent Data Flow Conductor CRM Legacy ER P Data Warehouse Files / XML User Interfaces Administrators Designers Topology/Security Metadata/Rules Development ESB Production Servers and Applications ODI Runtime Environment Runtime Repository Return Codes Code Execution Log Execution Metadata Navigator Production CRM Legacy ER P Data Warehouse Files / XML ESB User Interfaces Administrators Operators Thin Client Data Stewarts Topology/Security Execution Log Metadata Lineage Agent Data Flow Conductor Scenarios and Projects Releases
  14. 14. 15 Oracle Data Integrator 6 steps to Production
  15. 15. 16 ODI Design-Time Environment ODI Runtime Environment User Interfaces Overview: 6 steps to Production 1. Retrieve/Enrich metadata 2. Design transformations 3. Orchestrate data flows 4. Generate/Deploy data flows 5. Monitor executions 6. Analyze impact / data lineage Development Development Servers and Applications Agent Data Flow Conductor CRM Legacy ERP Data Warehouse Files / XML User Interfaces Administrators Designers ESB Design-time RepositoriesDesign-time Repositories Production Production Servers and Applications Agent Data Flow Conductor CRM Legacy ERP Data Warehouse Files / XML Operator Metadata Navigator ESB Runtime Repository
  16. 16. 17 1. Reverse-engineer Metadata • Automatic • Customizable • 40+ technologies supported 2. Enrich Metadata • Documentation • Declarative rules for Data Integrity • Cross-technologies references Design-Time Environment ODI Designer Design-time Repositories Development Servers and Applications CRM Legacy ERP Data Warehouse Files / XML ESB Retrieve/Enrich Metadata1
  17. 17. 18 Oracle Data Integrator “Interface” Declarative Design 1 Define What You Want 3 Automatically Generate Data flows 2 Define How to Do It: Select Template Bulk Load • Changed Data Capture • Incremental Update • Slowly Changing Dimension Design Transformations2
  18. 18. 19 1. Sequence Transformations 2. Leverage OracleDI Tools • Data Quality Processes • Files/Archives Management • Send/Receive Emails • Web Services Invokation • Event Detection • Create your Own Tools 3. Use Control Structures • Loops • Conditions • Error Handling 3 Orchestrate Data Flows
  19. 19. 20 1. Create Scenarios • Compile Data Flows for Run-time 2. Version the Data Flows • Advanced Version Management 3. Deploy to Production Design-time Repositories Scenarios and Projects Releases Runtime Repository Generate and Deploy Data Flows4
  20. 20. 21 • View sessions running in real- time • Review generated code • Detailed run-time statistics • Restart failed sessions 5 Monitor Executions
  21. 21. 22 Analyze impact / data lineage • Maintain a large number of data flows in a complex environment • Web-based end-to-end data lineage 1. Understand your data flows 2. Follow the path of data 3. Drill-down to transformations ? 6
  22. 22. 23 Oracle Data Integrator ODI or ESB?
  23. 23. 24 What tool is best suited for task X? Requirement ES B ODI Recommende d Latency / Volume Synchronous Integration  ESB Asynchronous Integration with routing and transformation  ESB Asynchronous Integration for Active Data Warehousing (mini-batch)  ODI Batch Integration with High Volume  ODI Transformations In-memory XSLT Transformations (XML to XML)   ESB Transformations in App Server  ESB Transformations in Database (E-LT)   ODI Integration Topology Data Warehouse Loading (E-LT)  ODI JMS to JMS  ESB JMS to DB/App with routing and transformation (real-time or synchronous)  ESB JMS to DB/App with bulk transformation (mini-batch)  ODI DB/App to DB/App (batch or mini-batch with CDC)  ODI DB/App to DB/App (synchronous or real-time with CDC Adapters)  ESB …
  24. 24. 25 Oracle Data Integrator ESB and ODI in real-life scenarios Data Volume Processing Data Latency Message by Message Mini Batches Large Volume (over 1M) Synchronous (immediate) Asynchronous Batch (over 2 hours) Oracle Enterprise Service Bus
  25. 25. 26 Oracle Data Integrator Extended Capabilities
  26. 26. 27 Extended Capabilities • Master Data Management enabled • Common Format Designer • Automated generation of canonical format and transformations • Built-in Data Integrity • Real-time enabled • Changed Data Capture • Message Oriented Integration (JMS) • SOA enabled • Generation of Data Services • Generation of Transformation Services • Extensibility • Knowledge Modules Framework • Scripting Languages • Open Tools
  27. 27. 28 Oracle Data Integrator Master Data Management Enabled
  28. 28. 29 MDM Enabled: Canonical Format Design • Use in conjunction with packaged MDM solution • Design and Populate Canonical Format 1. Use existing metadata artifacts to design MDM application (entities, fields, relationships) 2. Generate and maintain Master Data structure 3. Generate and deploy transformations using metadata artifacts CRM Enterprise Service Bus SCM Legacy ERP Master Data
  29. 29. 30 MDM Enabled: Built-in Data Integrity • Data Integrity Firewall • Auditing, cleansing and recycling 1. Declare constraints at table level 2. Design mappings and check flow integrity 3. Audit, cleanse or recycle rejected records Message Id Name City Duplicated Record 001 John Doe New York Duplicated Record 022 John Doe Boston Invalid City Reference 230 Albert Fresh Maris
  30. 30. 31 Oracle Data Integrator Real-time Enabled
  31. 31. 32 Real-time enabled: Changed Data Capture • Publish and Subscribe CDC Framework • Database logs • Triggers • Third-tier solutions • Ensures “read” transaction integrity across multiple tables 1. Design or generate Mappings 2. Select Journalized Data Only 3. Start Journals CDC
  32. 32. 33 Real-time enabled: Message Oriented Integration • Connect to Publish and Subscribe JMS Message Providers • Ensure messages delivery with transaction integrity • High-volume bulk transformations 1. Design complex bulk transformations mixing Queues, Databases and Applications 2. Use JMS Queues and topics as sources or targets JMS Provider (MOM, ESB) CDC Subscribe Publish
  33. 33. 34 Oracle Data Integrator SOA Enabled
  34. 34. 35 SOA Enabled: Data Access Services • Generate and share data access services 1. Generate and deploy data services 2. Test data services 3. Leverage data services in your SOA infrastructure SOA Infrastructure Services Data Access Transform Business Business Processes ESB
  35. 35. 36 SOA Enabled: Data Flow Services • Expose transformations as Web Services 1. Orchestrate data flows 2. Publish data flows as web services in your SOA infrastructure SOA Infrastructure Services Data Access Transform Business Business Processes ESB Bulk Transf .
  36. 36. 37 Oracle Data Integrator Extensible Framework
  37. 37. 38 Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse Engineer Metadata Reverse Journalize Load Check Integrate Services Pluggable Knowledge Modules Architecture CDC Sources Staging Tables Error Tables Target Tables W S W S W S Extensibility: Knowledge Modules • 120+ KMs out-of-the-box  Tailor to existing best practices  Ease administration work  Reduce cost of ownership • Customizable and extensible KM Interpreter KM’s Meta Code Metadata Executed Code
  38. 38. 39 Extensibility: Scripting Framework • Extend data flows with scripting procedures • Leverage all database languages • SQL, PL/SQL, Transact SQL, etc. • Use Operating Systems shell scripts • Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc. • Choose from compatible Bean Scripting Framework languages • Java, JavaScript, Jython (Java Python), Perl, etc.
  39. 39. 40 Extensibility: Open Tools • Extend ODI tools • Add your own tools to the Design Palette 1. Implement OdiOpenToolAbstract Java Interface 2. Register Open Tool in ODI Designer 3. Use Open Tool in your design environment
  40. 40. 41 Popular Usage Scenarios
  41. 41. 42 E-LT for Data Warehouse Create Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI  Heterogeneous sources and targets  Incremental load  Slowly changing dimensions  Data integrity and consistency  Changed data capture  Data lineage Data Warehouse Cube Cube Cube ---- ---- ---- ---- Operational Analytics Metadata Load Transform Capture Changes Incremental Update Data Integrity Aggregate Export
  42. 42. 43 SOA Initiative Establish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI  Invoke external services for data integration  Deploy data services  Deploy transformation services  Integrate data and transformation services in your SOA infrastructure Services Data Access Transformation Others ---- ---- ---- ---- Operational Metadata Generate Data Services Expose Transformation Services Deploy and reuse Services Business Processes
  43. 43. 44 Master Data Management Create Single View of the Truth Synchronize Data with ODI  Use in conjunction with packaged MDM solution  Use as infrastructure for designing your own hub  Create declarative data flows  Capture changes (CDC)  Reconcile and cleanse the data  Publish and share master data  Extend metadata definitions Master Data ---- ---- ---- ---- Metadata Change Data Capture Master Data Load Canonical Format Design Cleansing and Reconciliation Master Data Publishing ---- ---- ---- ---- CDC CDC CDC
  44. 44. 45 Migration Upgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI  Bulk-load historical data to new application  Transform source format to target  Synchronize new and old applications during overlap time  Capture changes in a bi- directional way (CDC) OldApplications NewApplication Metadata Initial bulk load CDC for synchronization Transformation to new application format CDC for loop- back synchronization CDC CDC ---- ---- ---- ----
  45. 45. 46 ODI Enhances Oracle BI Populate Warehouse with High Performance ODI Oracle Business Intelligence Suite EE:  Simplified Business Model View  Advanced Calculation & Integration Engine  Intelligent Request Generation  Optimized Data Access Oracle Data Integrator:  Populate Enterprise Data Warehouse  Optimized Performance for Load and Transform  Extensible Pre-packaged E-LT Content Siebel CRM Oracle EBS PeopleSoftSAP/R3 Other Sources Oracle Data Integrator E-LT Metadata E-LT Agent Oracle BI Enterprise Data Warehouse Oracle BI Suite EE Oracle BI Server Oracle BI Presentation Server Answers Interactive Dashboards Publisher Delivers Bulk E-LT
  46. 46. 47 ODI Enhances Oracle SOA Suite Add Bulk Data Transformation to BPEL Process Oracle SOA Suite:  BPEL Process Manager for Business Process Orchestration Oracle Data Integrator:  Efficient Bulk Data Processing as Part of Business Process  Interact via Data Services and Transformation Services Oracle SOA Suite Business Activity Monitoring Web Services Manager Business Rules Engine Enterprise Service Bus BPEL Process Manager Bulk Data Processing Oracle Data Integrator E-LT Metadata E-LT Agent
  47. 47. 48 ODI Enhances Oracle SOA Suite Populate BAM Active Data Cache Efficiently Oracle SOA Suite:  Business Activity Monitoring for Real-time Insight Oracle Data Integrator:  High Performance Loading of BAM’s Active Data Cache  Pre-built and Integrated Data Warehouse Oracle SOA Suite BPEL Process Manager Web Services Manager Business Rules Engine Enterprise Service Bus Bulk and Real-Time Data Processing SAP/R3 PeopleSoft Me ss ag e Qu eu es CDC Business Activity Monitoring Active Data Cache Event Engine Report Cache Event Monitoring Web Applications Oracle Data Integrator MetadataAgent
  48. 48. 49 Roadmap and Direction
  49. 49. 50 Oracle Data Integrator: Roadmap • Focus Areas for Next Major Release • Deep Integration with Fusion Middleware • Runtime, Design time, Security, Administration, Events • Functional Integration with Oracle Warehouse Builder • Runtime Integration, Metadata Sharing, Knowledge Module Sharing • Deployment of ODI for Embedded Data Integration • OracleBI Enterprise Edition, Data Hubs, Application Migrations • Enhanced Usability and Debuggability • Wizards, New Views, User-definable Debugging • Improved Support for Native Oracle Database Features • Oracle OLAP
  50. 50. 51 ODI Statement of Direction • Statement of Direction • • Key Points of Direction • Commitment to heterogeneous systems support • Including: DB2, Teradata, Netezza, Hyperion, etc. • Commitment to Fusion design principles • Including: J2EE compliance, container portability • Commitment to best-of-class E-LT performance • Across platforms, batch & realtime, high complexity