• Save
Data Federation/EII Uses And Abuses
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Data Federation/EII Uses And Abuses

  • 5,955 views
Uploaded on

As the need for "right-time" information becomes more prevalent, organizations are looking to new information delivery methods such as Enterprise Information Integration (EII). It's more than a......

As the need for "right-time" information becomes more prevalent, organizations are looking to new information delivery methods such as Enterprise Information Integration (EII). It's more than a distributed query, but where do vendors over-promise and under-deliver? This presentation will explain EII, how it's different from other integration technologies, and the underlying mechanisms. The goal is to outline areas where EII is a good fit and areas where other tools may be more appropriate.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Here is a discussion on what TRUE data virtualization is all about. Don't confuse this with simple, traditional data federation, which is a subset of data virtualization.

    http://tdwi.org/Articles/2011/06/15/Virtualization-and-Data-Integration-Issues.aspx#

    Some information on Informatica's data virtualization solution:
    http://www.informatica.com/Pages/data_virtualization_index.aspx
    http://www.informatica.com/products_services/data_services/Pages/index.aspx
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
5,955
On Slideshare
5,786
From Embeds
169
Number of Embeds
4

Actions

Shares
Downloads
0
Comments
1
Likes
4

Embeds 169

http://clickstream.blogspot.com 142
http://www.slideshare.net 18
http://www.linkedin.com 7
https://www.linkedin.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Bio stuff: Background, been through some of these decisions

Transcript

  • 1. Enterprise Information Integration Technology: Architectures, Uses and Abuses Mark Madsen http://ThirdNature.net [email_address]
  • 2. Intro & Outline
    • Why we need new tools for on-demand integration
    • What are the alternatives?
    • EII definition and models
    • How EII tools work
    • Ways you can use and abuse the tools
    • Some usage criteria
  • 3. Problems Requiring New Approaches
    • The technical problems driving us to extend our integration tools:
      • Data currency
      • Data source diversity
      • Data type diversity
      • Delivery via different APIs
      • On-demand data consistency
      • Regulation and data security
  • 4. What We’ve Been Building
    • The original view:
    • Data warehouse as a system or application.
    • Cleansed historical data on a snapshot basis.
    • Physical storage of all the data to be accessed.
    • End user tools and infrastructure combined.
    SQL Warehouse Database ETL ODS Mart Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Warehouse Clients Dashboards OLAP Productivity BAM/BPM Reporting DM Data Mining
  • 5.
    • The new view:
    • Data warehouse as a platform.
    • Access to both historical and current data.
    • Multiple storage methods, possibly distributed.
    • Multiple access methods.
    • Data usage decoupled from underlying platform.
    What We Need to Build Now SQL Warehouse Database ETL ? ? Mart ODS ? ? Content Store Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Consumers Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications
  • 6. EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
  • 7. ETL: Extract, Transform and Load ETL Engine The good points Connectivity Transformation Read-write access Metadata The bad points Requires a physical target Latency Distributed query limitations Complexity Database Targets Databases Documents Flat Files XML Services ERP Applications Source Environments
  • 8. Enterprise Application Integration
    • Multiple models
    • Multiple implementations
    • Evolution toward open standards-based services and SOA
    Hub Point Bus
  • 9. EII: Enterprise Information Integration EII Server Databases Documents Flat Files XML Services ERP Applications Source Environments SQL SOAP WS-* REST File Virtual Models Consuming Environments Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL ERP Applications
  • 10. EII Architecture: Process-based EII View APIs xform Publish an ETL process as a service – the output of the process is the “view” Approach some ETL vendors have for on-demand access Back-to-front model Applications Databases XML/Services
  • 11. EII Architecture: Model-based View is defined as a model over sources Most EII products work this way Front-to-back model Applications Databases XML/Services EII View APIs
  • 12. How EII Works 1. User’s client sends a request which goes to the EII server interface Applications Databases XML/Services EII Server EII View SQL
  • 13. How EII Works 2. The EII server parses the request and creates query fragments to send to the sources Applications Databases XML/Services EII Server EII View SQL
  • 14. How EII Works 3. Each source system receives its query fragment, processes it and returns the result Applications Databases XML/Services EII Server SQL
  • 15. How EII Works 4. EII server assembles the final result (which may mean additional processing) and the result is sent back to the client Applications Databases XML/Services EII Server EII View SQL
  • 16. How You Work With EII Tools EII Server
    • Define data sources
    • Define integrated view
    • Map source data into the integrated view
    • Publish view
    • Tune query*
    • Query via the published interfaces
  • 17. How You Work With EII Tools EII Server
    • Define data sources
    • Define integrated view
    • Map source data into the integrated view
    • Publish view
    • Tune query*
    • Query via the published interfaces
  • 18. How You Work With EII Tools
    • Define data sources
    • Define integrated view
    EII Server EII Virtual View
    • Define data sources
    • Define integrated view
    • Map source data into the integrated view
    • Publish view
    • Tune query*
    • Query via the published interfaces
  • 19. How You Work With EII Tools
    • Define data sources
    • Define integrated view
    • Map source data into the integrated view
    • Publish view
    • Query via the published interfaces
    • Tune query*
    EII Server EII Virtual View SQL SOAP WS-* REST File
  • 20. What Sorts of Problems Can We Solve?
    • Applications where you are being asked to provide any of the following:
      • More up-to-date data
      • Data in alternate formats
      • Data in non-relational formats
      • Data for portals, web sites or composite applications
  • 21. Adding On-demand BI Capabilities Source Environments Data Warehouse ETL Historical data Current data ODS BI Applications
  • 22. Drill-through to Source Detail Source Environments Data Warehouse ETL Drill-through accesses detail data via virtual tables in the warehouse ODS BI Applications
  • 23. Consolidated Views Data Mart Source Environments ODS ETL Data Mart EII Server ETL Data is available via the EII server, providing a virtual consolidation mart BI Applications
  • 24. Performance Management and BI Source Environments EII Server ETL Data Warehouse BI Applications ODS Dashboards BPM BAM Operational Reporting
  • 25. Compliance and Regulatory Reporting
    • Many compliance and reporting solutions deal with process, content and tracking documents.
    • Typical characteristics:
      • Historical data
      • Non-BI data
      • Multiple source applications
      • Documents and content
      • Needs vary over time
  • 26. Portals, Composite Applications and SOA
    • Typical Portal scenario:
    • Disparate data
    • Heterogeneous sources
    • Point integration
    • Minimal reuse
    Databases Documents Flat Files XML Services ERP Applications Source Environments
  • 27. Portals, Composite Applications and SOA
    • EII provides a “data services” layer
    • Standards-based interfaces
    • Single view of source data
    • Single point of integration
    • Reuse of data as a service
    Data Services (EII) Databases Documents Flat Files XML Services ERP Applications Source Environments
  • 28. Portals, Composite Applications and SOA Data Services (EII) Transaction Services (EAI) Databases Documents Flat Files XML Services ERP Applications Source Environments
  • 29. Rapid Prototyping
    • If you begin building a data services layer, it can make the prototyping of new data-intensive applications go much faster.
      • No need to physically model, extract and store data for a prototype application.
      • Simple to set up
    • Examples:
      • Adding a subject area
      • Adding a star schema
      • Adding data to a portal
  • 30. Just Remember It’s a Prototype
  • 31. Going Further: the Virtual Data Warehouse
  • 32. Other Questionable Ideas
    • Getting around pesky rules and procedures:
      • Like being direct query access cutoffs
      • Or security
      • Or development controls regarding data access
    • You can even treat an EII view like a data source for copying data…
  • 33. EII Isn’t a Miracle Cure
    • It can help you take shortcuts you shouldn’t.
    • Some other shortcomings you may face:
      • Metadata sharing and interoperability
      • Reliability (failure modes, not the tools)
      • Performance considerations
  • 34. Some Guidance: When Does EII Fit?
    • There are some basic criteria and tradeoffs to consider:
      • Data currency vs. latency
      • Diversity of data sources
      • Data cleansing & transformation
      • Predictability of performance
      • Access to the same data is needed via different interfaces
      • Non-relational sources
      • Frequency of access
      • Data volumes
      • And more…