SlideShare a Scribd company logo
1 of 40
Leading Practices and Insights for Managing Data Integration Initiatives May 7, 2010
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
About Allin ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
About Optima ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Data Integration Overview 03/09/11 Definition  - combining data residing in different sources and providing users with a unified view of these data Mediated Schema Example Data Warehouse Example
Business / Technical Drivers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Business / Technical Drivers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Key Variables / Considerations 03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Approaches / Strategies 03/09/11
03/09/11 ,[object Object],Approaches / Strategies
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],Approaches
Application Specific Solutions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 Data Propagation ,[object Object],[object Object],[object Object],[object Object]
Data Federation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Integration Tools 03/09/11
03/09/11 ETL ETL: Extract, Transform and Load ,[object Object],[object Object],[object Object],[object Object]
ETL Integrated Architecture XYZ Corp Systems  Data Extraction &  Integration Business Process Layer Information Management Presentation Layer Accessible throughout the organization Distribution Ad-hoc Analysis Tools Core Business Systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Architecture ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Reporting Data   Storage Transformation Subject Areas Transformation Extract, Transformation, & Load (ETL) Layer The “Unified Business Model” and Information Management Analytics and Reporting Tools OLAP Cubes &  Predictive Models ,[object Object],[object Object],[object Object],[object Object],Exception Notifications Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 EAI ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 EAI Architecture
03/09/11 Tool Comparison ,[object Object]
03/09/11 ETL Tools - Key Features ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Extraction &  Integration Data   Storage Transformation Transformation Extract, Transformation, & Load (ETL) Layer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 ETL Tools – Key Features  (continued) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Extraction &  Integration Data   Storage Transformation Transformation Extract, Transformation, & Load (ETL) Layer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Challenges 03/09/11
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Challenges
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Challenges  (continued)
03/09/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Integration Challenges  (continued)
Case Studies 03/09/11
Business Case 1 03/09/11 Service Company – Revenue: $200m, Size: 300 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 03/09/11 Business Case 2 Education Company – Revenue: $500m, Size: 300 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
03/09/11 Business Case 3 Healthcare Company – Revenue: $150m, Size: 200 FTEs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions 03/09/11
The Value of a Data Quality Effort  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
The Value of a Data Quality Effort  ,[object Object],[object Object],[object Object],[object Object],03/09/11
Importance of Governance ,[object Object],Size 03/09/11 Incorrect hardware or software 7% 2% Failure to define objectives 17% Unfamiliarity with scope and complexity 17% Lack of communication 20% Inadequate Project Management 32% Other 5%
Data Stewardship Data stewards act as the conduit between IT and the business and accept accountability for the data management process.  ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Stewards play a the central role in the management of data across the organization and in assuring its usefulness for the business. 03/09/11 Data stewards become the “public face” for data and have the following responsibilities: IT Business Business Data  Stewards
Success Factors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Success Factors  (continued) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],03/09/11
Appendices 03/09/11
03/09/11 ETL Vendors ETL Vendors ETL Tools Microsoft   SQL Server Integration Services  Oracle   Oracle Warehouse Builder (OWB) SAP Business Objects Data Integrator & Data Services  IBM IBM Information Server (Datastage) IBM Data Manager/Decision Stream (Cognos) SAS Institute  SAS Data Integration Studio Informatica   PowerCenter Ab Initio Co>Operating System Information Builders Data Migrator Adeptia Adeptia Integration Server CastIron Systems OmniConnect Platform Pitney Bowes Business Insight DataFlow Manager Pervasive Data Integrator Elixir  Elixir Repertoire Javlin   Clover ETL Pentaho   Pentaho Data Integration  Talend  Talend Open Studio
03/09/11 ETL / EAI - Tool Strengths ETL  EAI  Excels at bulk data movement  Limited in data movement capabilities  Provides for complex transformations, aggregation from multiple sources and sophisticated business rules.  Offer less sophisticated transformation and extraction functions  Assumes data delays.  Operates in real time  Are batch-oriented, making them fast and simple for one-time projects and testing  Work better with continuously interacting systems  Offers little in the way of workflow  Workflow-oriented at the core  Works primarily at the session layer  Works primarily at the transport layer

More Related Content

What's hot

Web Mining Presentation Final
Web Mining Presentation FinalWeb Mining Presentation Final
Web Mining Presentation Final
Er. Jagrat Gupta
 
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Alan McSweeney
 

What's hot (20)

Web Mining Presentation Final
Web Mining Presentation FinalWeb Mining Presentation Final
Web Mining Presentation Final
 
Big data, Big decision
Big data, Big decisionBig data, Big decision
Big data, Big decision
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
Data Integration, Access, Flow, Exchange, Transfer, Load And Extract Architec...
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake Overview
 
Business Data Lake Best Practices
Business Data Lake Best PracticesBusiness Data Lake Best Practices
Business Data Lake Best Practices
 
Master Data Management
Master Data ManagementMaster Data Management
Master Data Management
 
From Data Lakes to the Data Fabric: Our Vision for Digital Strategy
From Data Lakes to the Data Fabric: Our Vision for Digital StrategyFrom Data Lakes to the Data Fabric: Our Vision for Digital Strategy
From Data Lakes to the Data Fabric: Our Vision for Digital Strategy
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
DataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data ArchitectureDataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data Architecture
 
Data Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future OutlookData Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future Outlook
 
Business Intelligence Architecture
Business Intelligence ArchitectureBusiness Intelligence Architecture
Business Intelligence Architecture
 
The data quality challenge
The data quality challengeThe data quality challenge
The data quality challenge
 
Data integration
Data integrationData integration
Data integration
 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for Dinner
 
Measuring Data Quality with DataOps
Measuring Data Quality with DataOpsMeasuring Data Quality with DataOps
Measuring Data Quality with DataOps
 
Introducing the Snowflake Computing Cloud Data Warehouse
Introducing the Snowflake Computing Cloud Data WarehouseIntroducing the Snowflake Computing Cloud Data Warehouse
Introducing the Snowflake Computing Cloud Data Warehouse
 
Strategic Business Requirements for Master Data Management Systems
Strategic Business Requirements for Master Data Management SystemsStrategic Business Requirements for Master Data Management Systems
Strategic Business Requirements for Master Data Management Systems
 

Similar to Managing Data Integration Initiatives

t2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevityt2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevity
Jonathan Hamilton Solórzano
 
Datawarehousing
DatawarehousingDatawarehousing
Datawarehousing
sumit621
 
Rev_3 Components of a Data Warehouse
Rev_3 Components of a Data WarehouseRev_3 Components of a Data Warehouse
Rev_3 Components of a Data Warehouse
Ryan Andhavarapu
 
What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...
kzayra69
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Nathan Bijnens
 

Similar to Managing Data Integration Initiatives (20)

Chapter 2 - Enterprise Application Integration.pdf
Chapter 2 - Enterprise Application Integration.pdfChapter 2 - Enterprise Application Integration.pdf
Chapter 2 - Enterprise Application Integration.pdf
 
DW 101
DW 101DW 101
DW 101
 
t2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevityt2_4-architecting-data-for-integration-and-longevity
t2_4-architecting-data-for-integration-and-longevity
 
Datawarehousing & DSS
Datawarehousing & DSSDatawarehousing & DSS
Datawarehousing & DSS
 
ETL Market Webcast
ETL Market WebcastETL Market Webcast
ETL Market Webcast
 
Why Data Virtualization? An Introduction by Denodo
Why Data Virtualization? An Introduction by DenodoWhy Data Virtualization? An Introduction by Denodo
Why Data Virtualization? An Introduction by Denodo
 
GROPSIKS.pptx
GROPSIKS.pptxGROPSIKS.pptx
GROPSIKS.pptx
 
Datawarehousing
DatawarehousingDatawarehousing
Datawarehousing
 
A Comparitive Study Of ETL Tools
A Comparitive Study Of ETL ToolsA Comparitive Study Of ETL Tools
A Comparitive Study Of ETL Tools
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....
 
Rev_3 Components of a Data Warehouse
Rev_3 Components of a Data WarehouseRev_3 Components of a Data Warehouse
Rev_3 Components of a Data Warehouse
 
Business Analytics System
Business Analytics SystemBusiness Analytics System
Business Analytics System
 
What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...
 
An Integrated ERP with Web Portal
An Integrated ERP with Web Portal An Integrated ERP with Web Portal
An Integrated ERP with Web Portal
 
Data Warehouse Basic Guide
Data Warehouse Basic GuideData Warehouse Basic Guide
Data Warehouse Basic Guide
 
SW Architecture Monolithic to SOA
SW Architecture Monolithic to SOASW Architecture Monolithic to SOA
SW Architecture Monolithic to SOA
 
Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)Data Mesh in Azure using Cloud Scale Analytics (WAF)
Data Mesh in Azure using Cloud Scale Analytics (WAF)
 
Data warehouse presentaion
Data warehouse presentaionData warehouse presentaion
Data warehouse presentaion
 
Integrating SIS’s with Salesforce: An Accidental Integrator’s Guide
Integrating SIS’s with Salesforce: An Accidental Integrator’s GuideIntegrating SIS’s with Salesforce: An Accidental Integrator’s Guide
Integrating SIS’s with Salesforce: An Accidental Integrator’s Guide
 
An Integrated ERP With Web Portal
An Integrated ERP With Web PortalAn Integrated ERP With Web Portal
An Integrated ERP With Web Portal
 

Managing Data Integration Initiatives

  • 1. Leading Practices and Insights for Managing Data Integration Initiatives May 7, 2010
  • 2.
  • 3.
  • 4.
  • 5. Data Integration Overview 03/09/11 Definition - combining data residing in different sources and providing users with a unified view of these data Mediated Schema Example Data Warehouse Example
  • 6.
  • 7.
  • 8.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 16.
  • 17.
  • 18.
  • 20.
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 28.
  • 29.
  • 30.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 39. 03/09/11 ETL Vendors ETL Vendors ETL Tools Microsoft  SQL Server Integration Services Oracle  Oracle Warehouse Builder (OWB) SAP Business Objects Data Integrator & Data Services  IBM IBM Information Server (Datastage) IBM Data Manager/Decision Stream (Cognos) SAS Institute SAS Data Integration Studio Informatica  PowerCenter Ab Initio Co>Operating System Information Builders Data Migrator Adeptia Adeptia Integration Server CastIron Systems OmniConnect Platform Pitney Bowes Business Insight DataFlow Manager Pervasive Data Integrator Elixir Elixir Repertoire Javlin  Clover ETL Pentaho  Pentaho Data Integration Talend Talend Open Studio
  • 40. 03/09/11 ETL / EAI - Tool Strengths ETL EAI Excels at bulk data movement Limited in data movement capabilities Provides for complex transformations, aggregation from multiple sources and sophisticated business rules. Offer less sophisticated transformation and extraction functions Assumes data delays. Operates in real time Are batch-oriented, making them fast and simple for one-time projects and testing Work better with continuously interacting systems Offers little in the way of workflow Workflow-oriented at the core Works primarily at the session layer Works primarily at the transport layer

Editor's Notes

  1. 3
  2. 3
  3. Bulk Extract – utilizes copy management tools or unload utilities to extract all or a subset of the operational relational database. The data which has been extracted may be transformed to the format used by the target on the host or target server . The DBMS system load tools are then used in order to refresh the database target. File Compare – process compares the newly extracted operational data to the previous version. After that, a set of incremental change records is created and are applied as updates to the target server within the scheduled process. Change Data Propagation – captures and records the changes to the file as part of the application change process. Techniques that can be used include triggers, log exits, log post processing or DBMS extensions. A file of incremental changes is created to contain the captured changes.
  4. Data stewardship involves taking responsibility for data elements for their end-to-end usage across the enterprise .