• Like
  • Save
Data Federation
Upcoming SlideShare
Loading in...5
×
 

Data Federation

on

  • 5,168 views

This briefing provides a high level overview regarding the implications and efficacy of data federation.

This briefing provides a high level overview regarding the implications and efficacy of data federation.

Statistics

Views

Total Views
5,168
Views on SlideShare
5,127
Embed Views
41

Actions

Likes
3
Downloads
0
Comments
0

4 Embeds 41

http://www.semantech-inc.com 19
http://agileintelligence.blogspot.com 18
http://www.slideshare.net 3
http://semantech-inc.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Data Federation Data Federation Presentation Transcript

  • Semantech Inc. 2008 - Lecture Series Federated Data Architecture (Logical Data Integration) Presented by Stephen Lahanas Principal Consultant, Semantech Inc. Feb 14th, 2008 Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Presentation Objectives
      • To define what is Federated Data Architecture.
      • To highlight current best practice solutions that match this architectural approach.
      • To Illustrate the concepts in the context of a real world case study.
      • To illustrate concept/s in context with federal and commercial sector IT modernization efforts.
    I Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  •  
    • Characteristics of Data Federation
      • It is a design philosophy.
      • It is a data architecture ‘pattern.’
      • It is an integration approach.
      • It can be merged within a Lifecycle Methodology.
      • It is user-centric.
      • It is flexible.
      • It is rapid – both in development & system response.
      • It is focused on capability rather than technical orthodoxy.
      • It is designed for performance & pragmatism.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • A Definition of Data Federation
      • Data Federation represents a pragmatic solution for ‘loosely coupled’ enterprise integration and perhaps more importantly enterprise and multi-enterprise interoperability. Federated data architectures support the exploitation of multiple disparate authoritative data sources within the context of a logically integrated or orchestrated view of the enterprise. Federated architectures are by definition data fusion solutions.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Best Practices & Architectural Principles
      • User involvement & Ontology Automation
      • Metadata Orchestration Layer
      • Data Performance Engineering
      • Identity Federation & Multi-level Security
      • SOA Framework Exploitation (Enterprise Service Bus)
      • Common Messaging Exchange (format) Exploitation
      • Data Governance Framework (patterned upon federated coordination of authoritative systems as opposed to the configuration of one, single source data repository).
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  •  
  • “ Federated data orchestration is already a best practice on the Internet, in nature and in the management of other knowledge resources. The exponential proliferation of data over the next two decades will make single source / warehouse solutions even less practical.” Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  • Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • How does this approach impact the enterprise?
      • Data It reduces the risk involved in “Big Bang” data warehouse / datamart implementations.
      • It reduces the risk involved in “Big Bang” data warehouse / datamart implementations.
      • It allows for more pragmatic transitions of legacy capability to modernized solutions.
      • It helps to maintain vital knowledge capital associated with the system /data experts. This extra time can then be used to ensure that follow-on systems, services or consolidations move forward without losing corporate knowledge.
      • It allows for more flexibility in the face of complex integration / interoperability scenarios or budgetary constraints.
      • It helps to maintain a close connection to current data systems end-users.
  •  
    • As with any new approach, there is some confusion…
      • “ Data Federation doesn’t support enterprise data standardization.” Yes and no. In most cases large scale data strictly governance solutions have failed. Why? Because they lacked flexibility, speed and user input. Data federation gives us the chance to move quickly but also begin the holistic definition process with end user feedback.
      • “ Data Federation” doesn’t lend itself to optimal DBMS performance (query response). Not true, logical integrations of federated sources perform far better than data warehouse counterparts. Why, the optimization is ABSTRACTED from the source data. This flexibility allows value add to Authoritative Data Sources (ADSs) w/o redesign to all systems. The design occurs once in one place in an metadata driven optimization layer.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  • Case Study Part 1: (USAF Financial System) Subject: A global enterprise, comprised of multiple semi-autonomous divisions; The United States Air Force. Problem: Manage an operating budget of nearly $40 billion affecting the programs, administration and combat operations of personnel and facilities worldwide. Not only is the amount staggering, but consider that these dollars support more than 400,000 civilian, military and contractor employees working around the clock in more than 100 global locations. The need was immediate – capability was expected within a year. Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Case Study Part 2
    • The Architectural Challenge
    • U.S. Air Force needed to integrate financial data from disparate systems (more than 20), and provide geographically dispersed financial managers with the tools to manage day-to-day operations.
        • Senior leaders at all levels required a real-time snapshot
        • A Financial data management system capable of scaling initially from a few hundred users to more than 15,000 users
    • System had to be operational 24/7 in every time zone worldwide
    • System users at all levels needed instant access in order to make timely decisions
    • With limited resources, the USAF needed to keep development & deployment costs for any solution to an absolute minimum
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Case Study Part 3
    • Realization – Only an Agile Architecture would work…
    • This solution would push the limits of typical data warehousing solutions.
    • Users needed ad-hoc control of their queries. A few canned, stock reports would be insufficient.
    • The system’s security model required unprecedented access controls since it dealt with highly sensitive financial data.
    • Data had to be available for analysis within minutes to worldwide users, and the system required scalability without information flow disruption.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  • Case Study Part 4 How does this exploit Federated Data Orchestration? The solution allows for the continued operation of legacy authoritative systems while they are being replaced / modernized / migrated to the AF ERP solutions (DEAMS & ECSS). This represents federated source data. The solution deploys a layer of federated data and metadata for optimization in a centralized location – thus mirroring the source environment without performance constraints of directly sourcing data from the BI layer. True federation occurs at a minimum of three tiers as the solution interacts with domain data from across the USAF, this represents “logical integration.” Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Case Study Part 5
    • Current Federated Architecture Solution Performance
    • Averaging more than 600,000+ ad hoc queries per month
    • 3 TB - will double over the next 3 years
    • 2 Billion+ rows of data
    • 15,000+ users world-wide
    • 8.29M queries in FY06 – 95% ad hoc
    • 1.2M queries in September ’06
    • Ad hoc results: 80% in 10 seconds or less
    • 99.7% up time, “follow the sun” globally
    • At least 3 other systems were developed in attempts to replace this solution (attempts to conform with formal data warehouse strategies), none of them met user expectations for accuracy, query capability or performance.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Separate Organizations – ‘Federated Workflows’
      • Many enterprises consist of multiple organizations providing data or services to one another, in separate uncoordinated workflows culminating in eventual Business Intelligence reports.
      • One of the possible goals of any data architecture modernization endeavor is to illustrate and coordinate all of the workflows related to the eventual provision of enterprise-wide analytics.
      • Federated workflows can be enabled through federated but coordinated data sources.
    Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  • By definition, any Common Operating Picture (COP) represents a fusion of disparate or federated sources. It is designed more for real-time awareness and dynamic analytics. The optimization cache can be used for historical reference as well – the key is not trying to replacing everything else with this solution all at once… Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
  • The reality is that nearly all integration within and across SOA environments is based upon data exchange and will ultimately be demonstrated through data exploitation. The data architecture, business architecture and services all must be logically orchestrated for SOA to work as expected. Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com
    • Conclusion
      • Federated Data Architecture is more than merely avoiding the use of traditional Data Warehouse techniques. It represents the next generation approach to sophisticated data integration and involves a variety of tools and techniques.
      • Most importantly though, it provides a more rapid and pragmatic way to solve highly complex enterprise data issues. Enterprises can realistically expect to deploy comprehensive solutions in months, not years…
    I Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com