Data Federation/EII Uses And Abuses

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    Bio stuff: Background, been through some of these decisions

    4 Favorites

    Data Federation/EII Uses And Abuses - Presentation Transcript

    1. Enterprise Information Integration Technology: Architectures, Uses and Abuses Mark Madsen http://ThirdNature.net [email_address]
    2. Intro & Outline
      • Why we need new tools for on-demand integration
      • What are the alternatives?
      • EII definition and models
      • How EII tools work
      • Ways you can use and abuse the tools
      • Some usage criteria
    3. Problems Requiring New Approaches
      • The technical problems driving us to extend our integration tools:
        • Data currency
        • Data source diversity
        • Data type diversity
        • Delivery via different APIs
        • On-demand data consistency
        • Regulation and data security
    4. What We’ve Been Building
      • The original view:
      • Data warehouse as a system or application.
      • Cleansed historical data on a snapshot basis.
      • Physical storage of all the data to be accessed.
      • End user tools and infrastructure combined.
      SQL Warehouse Database ETL ODS Mart Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Warehouse Clients Dashboards OLAP Productivity BAM/BPM Reporting DM Data Mining
      • The new view:
      • Data warehouse as a platform.
      • Access to both historical and current data.
      • Multiple storage methods, possibly distributed.
      • Multiple access methods.
      • Data usage decoupled from underlying platform.
      What We Need to Build Now SQL Warehouse Database ETL ? ? Mart ODS ? ? Content Store Databases Documents Flat Files XML Queues ERP Applications Source Environments Data Consumers Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL Data Mining Applications
    5. EDR: Enterprise Data Replication EDR Server Order Entry CRM Fulfillment Inventory EDR EDR
    6. ETL: Extract, Transform and Load ETL Engine The good points Connectivity Transformation Read-write access Metadata The bad points Requires a physical target Latency Distributed query limitations Complexity Database Targets Databases Documents Flat Files XML Services ERP Applications Source Environments
    7. Enterprise Application Integration
      • Multiple models
      • Multiple implementations
      • Evolution toward open standards-based services and SOA
      Hub Point Bus
    8. EII: Enterprise Information Integration EII Server Databases Documents Flat Files XML Services ERP Applications Source Environments SQL SOAP WS-* REST File Virtual Models Consuming Environments Databases Dashboards OLAP Productivity BAM/BPM Reporting ETL ERP Applications
    9. EII Architecture: Process-based EII View APIs xform Publish an ETL process as a service – the output of the process is the “view” Approach some ETL vendors have for on-demand access Back-to-front model Applications Databases XML/Services
    10. EII Architecture: Model-based View is defined as a model over sources Most EII products work this way Front-to-back model Applications Databases XML/Services EII View APIs
    11. How EII Works 1. User’s client sends a request which goes to the EII server interface Applications Databases XML/Services EII Server EII View SQL
    12. How EII Works 2. The EII server parses the request and creates query fragments to send to the sources Applications Databases XML/Services EII Server EII View SQL
    13. How EII Works 3. Each source system receives its query fragment, processes it and returns the result Applications Databases XML/Services EII Server SQL
    14. How EII Works 4. EII server assembles the final result (which may mean additional processing) and the result is sent back to the client Applications Databases XML/Services EII Server EII View SQL
    15. How You Work With EII Tools EII Server
      • Define data sources
      • Define integrated view
      • Map source data into the integrated view
      • Publish view
      • Tune query*
      • Query via the published interfaces
    16. How You Work With EII Tools EII Server
      • Define data sources
      • Define integrated view
      • Map source data into the integrated view
      • Publish view
      • Tune query*
      • Query via the published interfaces
    17. How You Work With EII Tools
      • Define data sources
      • Define integrated view
      EII Server EII Virtual View
      • Define data sources
      • Define integrated view
      • Map source data into the integrated view
      • Publish view
      • Tune query*
      • Query via the published interfaces
    18. How You Work With EII Tools
      • Define data sources
      • Define integrated view
      • Map source data into the integrated view
      • Publish view
      • Query via the published interfaces
      • Tune query*
      EII Server EII Virtual View SQL SOAP WS-* REST File
    19. What Sorts of Problems Can We Solve?
      • Applications where you are being asked to provide any of the following:
        • More up-to-date data
        • Data in alternate formats
        • Data in non-relational formats
        • Data for portals, web sites or composite applications
    20. Adding On-demand BI Capabilities Source Environments Data Warehouse ETL Historical data Current data ODS BI Applications
    21. Drill-through to Source Detail Source Environments Data Warehouse ETL Drill-through accesses detail data via virtual tables in the warehouse ODS BI Applications
    22. Consolidated Views Data Mart Source Environments ODS ETL Data Mart EII Server ETL Data is available via the EII server, providing a virtual consolidation mart BI Applications
    23. Performance Management and BI Source Environments EII Server ETL Data Warehouse BI Applications ODS Dashboards BPM BAM Operational Reporting
    24. Compliance and Regulatory Reporting
      • Many compliance and reporting solutions deal with process, content and tracking documents.
      • Typical characteristics:
        • Historical data
        • Non-BI data
        • Multiple source applications
        • Documents and content
        • Needs vary over time
    25. Portals, Composite Applications and SOA
      • Typical Portal scenario:
      • Disparate data
      • Heterogeneous sources
      • Point integration
      • Minimal reuse
      Databases Documents Flat Files XML Services ERP Applications Source Environments
    26. Portals, Composite Applications and SOA
      • EII provides a “data services” layer
      • Standards-based interfaces
      • Single view of source data
      • Single point of integration
      • Reuse of data as a service
      Data Services (EII) Databases Documents Flat Files XML Services ERP Applications Source Environments
    27. Portals, Composite Applications and SOA Data Services (EII) Transaction Services (EAI) Databases Documents Flat Files XML Services ERP Applications Source Environments
    28. Rapid Prototyping
      • If you begin building a data services layer, it can make the prototyping of new data-intensive applications go much faster.
        • No need to physically model, extract and store data for a prototype application.
        • Simple to set up
      • Examples:
        • Adding a subject area
        • Adding a star schema
        • Adding data to a portal
    29. Just Remember It’s a Prototype
    30. Going Further: the Virtual Data Warehouse
    31. Other Questionable Ideas
      • Getting around pesky rules and procedures:
        • Like being direct query access cutoffs
        • Or security
        • Or development controls regarding data access
      • You can even treat an EII view like a data source for copying data…
    32. EII Isn’t a Miracle Cure
      • It can help you take shortcuts you shouldn’t.
      • Some other shortcomings you may face:
        • Metadata sharing and interoperability
        • Reliability (failure modes, not the tools)
        • Performance considerations
    33. Some Guidance: When Does EII Fit?
      • There are some basic criteria and tradeoffs to consider:
        • Data currency vs. latency
        • Diversity of data sources
        • Data cleansing & transformation
        • Predictability of performance
        • Access to the same data is needed via different interfaces
        • Non-relational sources
        • Frequency of access
        • Data volumes
        • And more…

    + mark madsenmark madsen, 3 years ago

    custom

    2630 views, 4 favs, 1 embeds more stats

    As the need for "right-time" information becomes mo more

    More info about this document

    CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

    Go to text version

    • Total Views 2630
      • 2488 on SlideShare
      • 142 from embeds
    • Comments 0
    • Favorites 4
    • Downloads 0
    Most viewed embeds
    • 142 views on http://clickstream.blogspot.com

    more

    All embeds
    • 142 views on http://clickstream.blogspot.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories