Enterprise Information Integration Technology: Architectures, Uses and Abuses Mark Madsen http://ThirdNature.net [email_ad...
Intro & Outline <ul><li>Why we need new tools for on-demand integration </li></ul><ul><li>What are the alternatives? </li>...
Problems Requiring New Approaches <ul><li>The technical problems driving us to extend our integration tools: </li></ul><ul...
What We’ve Been Building <ul><li>The original view: </li></ul><ul><li>Data warehouse as a system or application. </li></ul...
<ul><li>The new view: </li></ul><ul><li>Data warehouse as a platform. </li></ul><ul><li>Access to both historical  and cur...
EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
ETL: Extract, Transform and Load ETL   Engine The good points Connectivity Transformation Read-write access Metadata The b...
Enterprise Application Integration <ul><li>Multiple models </li></ul><ul><li>Multiple implementations </li></ul><ul><li>Ev...
EII: Enterprise Information Integration EII Server Databases  Documents  Flat Files  XML  Services  ERP  Applications Sour...
EII Architecture: Process-based EII View APIs xform Publish an ETL process as a service – the output of the process is the...
EII Architecture: Model-based View is defined as a model over sources Most EII products work this way Front-to-back model ...
How EII Works 1. User’s client sends a request which goes to the EII server interface Applications Databases XML/Services ...
How EII Works 2. The EII server parses the request and creates query fragments to send to the sources Applications Databas...
How EII Works 3. Each source system receives its query fragment, processes it and returns the result Applications Database...
How EII Works 4. EII server assembles the final result (which may mean additional processing) and the result is sent back ...
How You Work With EII Tools EII Server <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul>...
How You Work With EII Tools EII Server <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul>...
How You Work With EII Tools <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul>EII Server EII ...
How You Work With EII Tools <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul><li>Map sou...
What Sorts of Problems Can We Solve? <ul><li>Applications where you are being asked to provide any of the following: </li>...
Adding On-demand BI Capabilities Source Environments Data Warehouse ETL Historical data Current data ODS BI Applications
Drill-through to Source Detail Source Environments Data Warehouse ETL Drill-through accesses detail data via virtual table...
Consolidated Views Data Mart Source Environments ODS ETL Data Mart EII Server ETL Data is available via the EII server, pr...
Performance Management and BI Source Environments EII Server ETL Data Warehouse BI Applications ODS Dashboards BPM BAM Ope...
Compliance and Regulatory Reporting <ul><li>Many compliance and reporting solutions deal with process, content and trackin...
Portals, Composite Applications and SOA <ul><li>Typical Portal scenario: </li></ul><ul><li>Disparate data </li></ul><ul><l...
Portals, Composite Applications and SOA <ul><li>EII provides a “data services” layer </li></ul><ul><li>Standards-based int...
Portals, Composite Applications and SOA Data Services (EII) Transaction Services (EAI) Databases  Documents  Flat Files  X...
Rapid Prototyping <ul><li>If you begin building a data services layer, it can make the prototyping of new data-intensive a...
Just Remember It’s a Prototype
Going Further: the Virtual Data Warehouse
Other Questionable Ideas <ul><li>Getting around pesky rules and procedures: </li></ul><ul><ul><li>Like being direct query ...
EII Isn’t a Miracle Cure <ul><li>It can help you take shortcuts you shouldn’t. </li></ul><ul><li>Some other shortcomings y...
Some Guidance: When Does EII Fit? <ul><li>There are some basic criteria and tradeoffs to consider: </li></ul><ul><ul><li>D...
Upcoming SlideShare
Loading in...5
×

Data Federation/EII Uses And Abuses

3,960

Published on

As the need for "right-time" information becomes more prevalent, organizations are looking to new information delivery methods such as Enterprise Information Integration (EII). It's more than a distributed query, but where do vendors over-promise and under-deliver? This presentation will explain EII, how it's different from other integration technologies, and the underlying mechanisms. The goal is to outline areas where EII is a good fit and areas where other tools may be more appropriate.

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
  • Here is a discussion on what TRUE data virtualization is all about. Don't confuse this with simple, traditional data federation, which is a subset of data virtualization.

    http://tdwi.org/Articles/2011/06/15/Virtualization-and-Data-Integration-Issues.aspx#

    Some information on Informatica's data virtualization solution:
    http://www.informatica.com/Pages/data_virtualization_index.aspx
    http://www.informatica.com/products_services/data_services/Pages/index.aspx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,960
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide
  • Bio stuff: Background, been through some of these decisions
  • Data Federation/EII Uses And Abuses

    1. 1. Enterprise Information Integration Technology: Architectures, Uses and Abuses Mark Madsen http://ThirdNature.net [email_address]
    2. 2. Intro & Outline <ul><li>Why we need new tools for on-demand integration </li></ul><ul><li>What are the alternatives? </li></ul><ul><li>EII definition and models </li></ul><ul><li>How EII tools work </li></ul><ul><li>Ways you can use and abuse the tools </li></ul><ul><li>Some usage criteria </li></ul>
    3. 3. Problems Requiring New Approaches <ul><li>The technical problems driving us to extend our integration tools: </li></ul><ul><ul><li>Data currency </li></ul></ul><ul><ul><li>Data source diversity </li></ul></ul><ul><ul><li>Data type diversity </li></ul></ul><ul><ul><li>Delivery via different APIs </li></ul></ul><ul><ul><li>On-demand data consistency </li></ul></ul><ul><ul><li>Regulation and data security </li></ul></ul>
    4. 4. What We’ve Been Building <ul><li>The original view: </li></ul><ul><li>Data warehouse as a system or application. </li></ul><ul><li>Cleansed historical data on a snapshot basis. </li></ul><ul><li>Physical storage of all the data to be accessed. </li></ul><ul><li>End user tools and infrastructure combined. </li></ul>SQL Warehouse Database ETL ODS Mart Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Warehouse Clients Dashboards OLAP Productivity BAM/BPM Reporting DM Data Mining
    5. 5. <ul><li>The new view: </li></ul><ul><li>Data warehouse as a platform. </li></ul><ul><li>Access to both historical and current data. </li></ul><ul><li>Multiple storage methods, possibly distributed. </li></ul><ul><li>Multiple access methods. </li></ul><ul><li>Data usage decoupled from underlying platform. </li></ul>What We Need to Build Now SQL Warehouse Database ETL ? ? Mart ODS ? ? Content Store Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Consumers Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications
    6. 6. EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
    7. 7. ETL: Extract, Transform and Load ETL Engine The good points Connectivity Transformation Read-write access Metadata The bad points Requires a physical target Latency Distributed query limitations Complexity Database Targets Databases Documents Flat Files XML Services ERP Applications Source Environments
    8. 8. Enterprise Application Integration <ul><li>Multiple models </li></ul><ul><li>Multiple implementations </li></ul><ul><li>Evolution toward open standards-based services and SOA </li></ul>Hub Point Bus
    9. 9. EII: Enterprise Information Integration EII Server Databases Documents Flat Files XML Services ERP Applications Source Environments SQL SOAP WS-* REST File Virtual Models Consuming Environments Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL ERP Applications
    10. 10. EII Architecture: Process-based EII View APIs xform Publish an ETL process as a service – the output of the process is the “view” Approach some ETL vendors have for on-demand access Back-to-front model Applications Databases XML/Services
    11. 11. EII Architecture: Model-based View is defined as a model over sources Most EII products work this way Front-to-back model Applications Databases XML/Services EII View APIs
    12. 12. How EII Works 1. User’s client sends a request which goes to the EII server interface Applications Databases XML/Services EII Server EII View SQL
    13. 13. How EII Works 2. The EII server parses the request and creates query fragments to send to the sources Applications Databases XML/Services EII Server EII View SQL
    14. 14. How EII Works 3. Each source system receives its query fragment, processes it and returns the result Applications Databases XML/Services EII Server SQL
    15. 15. How EII Works 4. EII server assembles the final result (which may mean additional processing) and the result is sent back to the client Applications Databases XML/Services EII Server EII View SQL
    16. 16. How You Work With EII Tools EII Server <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul><li>Map source data into the integrated view </li></ul><ul><li>Publish view </li></ul><ul><li>Tune query* </li></ul><ul><li>Query via the published interfaces </li></ul>
    17. 17. How You Work With EII Tools EII Server <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul><li>Map source data into the integrated view </li></ul><ul><li>Publish view </li></ul><ul><li>Tune query* </li></ul><ul><li>Query via the published interfaces </li></ul>
    18. 18. How You Work With EII Tools <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul>EII Server EII Virtual View <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul><li>Map source data into the integrated view </li></ul><ul><li>Publish view </li></ul><ul><li>Tune query* </li></ul><ul><li>Query via the published interfaces </li></ul>
    19. 19. How You Work With EII Tools <ul><li>Define data sources </li></ul><ul><li>Define integrated view </li></ul><ul><li>Map source data into the integrated view </li></ul><ul><li>Publish view </li></ul><ul><li>Query via the published interfaces </li></ul><ul><li>Tune query* </li></ul>EII Server EII Virtual View SQL SOAP WS-* REST File
    20. 20. What Sorts of Problems Can We Solve? <ul><li>Applications where you are being asked to provide any of the following: </li></ul><ul><ul><li>More up-to-date data </li></ul></ul><ul><ul><li>Data in alternate formats </li></ul></ul><ul><ul><li>Data in non-relational formats </li></ul></ul><ul><ul><li>Data for portals, web sites or composite applications </li></ul></ul>
    21. 21. Adding On-demand BI Capabilities Source Environments Data Warehouse ETL Historical data Current data ODS BI Applications
    22. 22. Drill-through to Source Detail Source Environments Data Warehouse ETL Drill-through accesses detail data via virtual tables in the warehouse ODS BI Applications
    23. 23. Consolidated Views Data Mart Source Environments ODS ETL Data Mart EII Server ETL Data is available via the EII server, providing a virtual consolidation mart BI Applications
    24. 24. Performance Management and BI Source Environments EII Server ETL Data Warehouse BI Applications ODS Dashboards BPM BAM Operational Reporting
    25. 25. Compliance and Regulatory Reporting <ul><li>Many compliance and reporting solutions deal with process, content and tracking documents. </li></ul><ul><li>Typical characteristics: </li></ul><ul><ul><li>Historical data </li></ul></ul><ul><ul><li>Non-BI data </li></ul></ul><ul><ul><li>Multiple source applications </li></ul></ul><ul><ul><li>Documents and content </li></ul></ul><ul><ul><li>Needs vary over time </li></ul></ul>
    26. 26. Portals, Composite Applications and SOA <ul><li>Typical Portal scenario: </li></ul><ul><li>Disparate data </li></ul><ul><li>Heterogeneous sources </li></ul><ul><li>Point integration </li></ul><ul><li>Minimal reuse </li></ul>Databases Documents Flat Files XML Services ERP Applications Source Environments
    27. 27. Portals, Composite Applications and SOA <ul><li>EII provides a “data services” layer </li></ul><ul><li>Standards-based interfaces </li></ul><ul><li>Single view of source data </li></ul><ul><li>Single point of integration </li></ul><ul><li>Reuse of data as a service </li></ul>Data Services (EII) Databases Documents Flat Files XML Services ERP Applications Source Environments
    28. 28. Portals, Composite Applications and SOA Data Services (EII) Transaction Services (EAI) Databases Documents Flat Files XML Services ERP Applications Source Environments
    29. 29. Rapid Prototyping <ul><li>If you begin building a data services layer, it can make the prototyping of new data-intensive applications go much faster. </li></ul><ul><ul><li>No need to physically model, extract and store data for a prototype application. </li></ul></ul><ul><ul><li>Simple to set up </li></ul></ul><ul><li>Examples: </li></ul><ul><ul><li>Adding a subject area </li></ul></ul><ul><ul><li>Adding a star schema </li></ul></ul><ul><ul><li>Adding data to a portal </li></ul></ul>
    30. 30. Just Remember It’s a Prototype
    31. 31. Going Further: the Virtual Data Warehouse
    32. 32. Other Questionable Ideas <ul><li>Getting around pesky rules and procedures: </li></ul><ul><ul><li>Like being direct query access cutoffs </li></ul></ul><ul><ul><li>Or security </li></ul></ul><ul><ul><li>Or development controls regarding data access </li></ul></ul><ul><li>You can even treat an EII view like a data source for copying data… </li></ul>
    33. 33. EII Isn’t a Miracle Cure <ul><li>It can help you take shortcuts you shouldn’t. </li></ul><ul><li>Some other shortcomings you may face: </li></ul><ul><ul><li>Metadata sharing and interoperability </li></ul></ul><ul><ul><li>Reliability (failure modes, not the tools) </li></ul></ul><ul><ul><li>Performance considerations </li></ul></ul>
    34. 34. Some Guidance: When Does EII Fit? <ul><li>There are some basic criteria and tradeoffs to consider: </li></ul><ul><ul><li>Data currency vs. latency </li></ul></ul><ul><ul><li>Diversity of data sources </li></ul></ul><ul><ul><li>Data cleansing & transformation </li></ul></ul><ul><ul><li>Predictability of performance </li></ul></ul><ul><ul><li>Access to the same data is needed via different interfaces </li></ul></ul><ul><ul><li>Non-relational sources </li></ul></ul><ul><ul><li>Frequency of access </li></ul></ul><ul><ul><li>Data volumes </li></ul></ul><ul><ul><li>And more… </li></ul></ul>

    ×