Data Federation

Semantech Inc. 2008 - Lecture Series Federated Data Architecture (Logical Data Integration) Presented by Stephen Lahanas Principal Consultant, Semantech Inc. Feb 14th, 2008 Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

Presentation Objectives

To define what is Federated Data Architecture.

To highlight current best practice solutions that match this architectural approach.

To Illustrate the concepts in the context of a real world case study.

To illustrate concept/s in context with federal and commercial sector IT modernization efforts.

Characteristics of Data Federation

It is a design philosophy.

It is a data architecture ‘pattern.’

It is an integration approach.

It can be merged within a Lifecycle Methodology.

It is user-centric.

It is flexible.

It is rapid – both in development & system response.

It is focused on capability rather than technical orthodoxy.

It is designed for performance & pragmatism.

A Definition of Data Federation

Data Federation represents a pragmatic solution for ‘loosely coupled’ enterprise integration and perhaps more importantly enterprise and multi-enterprise interoperability. Federated data architectures support the exploitation of multiple disparate authoritative data sources within the context of a logically integrated or orchestrated view of the enterprise. Federated architectures are by definition data fusion solutions.

Best Practices & Architectural Principles

User involvement & Ontology Automation

Metadata Orchestration Layer

Data Performance Engineering

Identity Federation & Multi-level Security

SOA Framework Exploitation (Enterprise Service Bus)

Common Messaging Exchange (format) Exploitation

Data Governance Framework (patterned upon federated coordination of authoritative systems as opposed to the configuration of one, single source data repository).

“ Federated data orchestration is already a best practice on the Internet, in nature and in the management of other knowledge resources. The exponential proliferation of data over the next two decades will make single source / warehouse solutions even less practical.” Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

How does this approach impact the enterprise?

Data It reduces the risk involved in “Big Bang” data warehouse / datamart implementations.

It reduces the risk involved in “Big Bang” data warehouse / datamart implementations.

It allows for more pragmatic transitions of legacy capability to modernized solutions.

It helps to maintain vital knowledge capital associated with the system /data experts. This extra time can then be used to ensure that follow-on systems, services or consolidations move forward without losing corporate knowledge.

It allows for more flexibility in the face of complex integration / interoperability scenarios or budgetary constraints.

It helps to maintain a close connection to current data systems end-users.

As with any new approach, there is some confusion…

“ Data Federation doesn’t support enterprise data standardization.” Yes and no. In most cases large scale data strictly governance solutions have failed. Why? Because they lacked flexibility, speed and user input. Data federation gives us the chance to move quickly but also begin the holistic definition process with end user feedback.

“ Data Federation” doesn’t lend itself to optimal DBMS performance (query response). Not true, logical integrations of federated sources perform far better than data warehouse counterparts. Why, the optimization is ABSTRACTED from the source data. This flexibility allows value add to Authoritative Data Sources (ADSs) w/o redesign to all systems. The design occurs once in one place in an metadata driven optimization layer.

Case Study Part 1: (USAF Financial System) Subject: A global enterprise, comprised of multiple semi-autonomous divisions; The United States Air Force. Problem: Manage an operating budget of nearly $40 billion affecting the programs, administration and combat operations of personnel and facilities worldwide. Not only is the amount staggering, but consider that these dollars support more than 400,000 civilian, military and contractor employees working around the clock in more than 100 global locations. The need was immediate – capability was expected within a year. Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

Case Study Part 2

The Architectural Challenge

U.S. Air Force needed to integrate financial data from disparate systems (more than 20), and provide geographically dispersed financial managers with the tools to manage day-to-day operations.

Senior leaders at all levels required a real-time snapshot

A Financial data management system capable of scaling initially from a few hundred users to more than 15,000 users

System had to be operational 24/7 in every time zone worldwide

System users at all levels needed instant access in order to make timely decisions

With limited resources, the USAF needed to keep development & deployment costs for any solution to an absolute minimum

Case Study Part 3

Realization – Only an Agile Architecture would work…

This solution would push the limits of typical data warehousing solutions.

Users needed ad-hoc control of their queries. A few canned, stock reports would be insufficient.

The system’s security model required unprecedented access controls since it dealt with highly sensitive financial data.

Data had to be available for analysis within minutes to worldwide users, and the system required scalability without information flow disruption.

Case Study Part 4 How does this exploit Federated Data Orchestration? The solution allows for the continued operation of legacy authoritative systems while they are being replaced / modernized / migrated to the AF ERP solutions (DEAMS & ECSS). This represents federated source data. The solution deploys a layer of federated data and metadata for optimization in a centralized location – thus mirroring the source environment without performance constraints of directly sourcing data from the BI layer. True federation occurs at a minimum of three tiers as the solution interacts with domain data from across the USAF, this represents “logical integration.” Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

Case Study Part 5

Current Federated Architecture Solution Performance

Averaging more than 600,000+ ad hoc queries per month

3 TB - will double over the next 3 years

2 Billion+ rows of data

15,000+ users world-wide

8.29M queries in FY06 – 95% ad hoc

1.2M queries in September ’06

Ad hoc results: 80% in 10 seconds or less

99.7% up time, “follow the sun” globally

At least 3 other systems were developed in attempts to replace this solution (attempts to conform with formal data warehouse strategies), none of them met user expectations for accuracy, query capability or performance.

Separate Organizations – ‘Federated Workflows’

Many enterprises consist of multiple organizations providing data or services to one another, in separate uncoordinated workflows culminating in eventual Business Intelligence reports.

One of the possible goals of any data architecture modernization endeavor is to illustrate and coordinate all of the workflows related to the eventual provision of enterprise-wide analytics.

Federated workflows can be enabled through federated but coordinated data sources.

By definition, any Common Operating Picture (COP) represents a fusion of disparate or federated sources. It is designed more for real-time awareness and dynamic analytics. The optimization cache can be used for historical reference as well – the key is not trying to replacing everything else with this solution all at once… Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

The reality is that nearly all integration within and across SOA environments is based upon data exchange and will ultimately be demonstrated through data exploitation. The data architecture, business architecture and services all must be logically orchestrated for SOA to work as expected. Copyright 2008, Semantech Inc. – All Rights Reserved http://www.semantech-inc.com

Conclusion

Federated Data Architecture is more than merely avoiding the use of traditional Data Warehouse techniques. It represents the next generation approach to sophisticated data integration and involves a variety of tools and techniques.

Most importantly though, it provides a more rapid and pragmatic way to solve highly complex enterprise data issues. Enterprises can realistically expect to deploy comprehensive solutions in months, not years…