The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson , Washington University, St. Louis G. Leptoukh , NASA GSFC Spring AGU, May 29, 2008, Ft. Lauderdale

R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson , Washington University, St. Louis

G. Leptoukh , NASA GSFC

DataFed in a Nutshell: A Federation of autonomous, distributed data providers Performs non-intrusive wrapping of data into web services Provides service-based analysis services and tools General Experience with DataFed: It is an agile virtual data system can deliver info products to diverse users Third-party mediation can homogenize distributed data on the fly Since 2005, DataFed is used by EPA and in research DataFed Motivated by GEOSS DataFed development is guided by the meme of GEOSS

Five practices for agile, seamless data federation: Space-Time Query for standardized access to all data (WCS) Data Wrappers for turning heterogeneous data into web services Data Mediators for transforming data into ‘Views’ Mashups for connecting autonomous application DataSpaces for shared metadata by the users, for the users

Space-Time Query for standardized access to all data (WCS)

Data Wrappers for turning heterogeneous data into web services

Data Mediators for transforming data into ‘Views’

Mashups for connecting autonomous application

DataSpaces for shared metadata by the users, for the users

Parameter-Space-Time Query Using OGC WCS Data Access Protocol Regardless of the data location, data type and format, the parameter-space-time query is the same the return is in user selectable format from the offerings Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF Coverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFF Coverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-table Grid Image Station Data Parameter Bounding Box Time Range Out Format

Regardless of the data location, data type and format,

the parameter-space-time query is the same

the return is in user selectable format from the offerings

DataFed wrappers are non-intrusive, third party Third Party Data Wrappers Heterogeneous input data >>> Homogeneous (WCS) Query

DataFed wrappers are non-intrusive, third party

Mediated User-Data Interface Mediator turns data into Views Mediated Integration is a flexible design pattern for System of Systems Client-Server design is demanding: User carries the burden of integration Query Data Views

SOAP RDF Mashup Workflow Mashups: Loose Coupling of Autonomous Applications DataFed – Wiki -- GoogleEarth

DataSpaces for Datasets GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Catalog User Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users Services Community DataSpaces

GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users views report Services Community DataSpaces

-+ GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users views report Services Community DataSpaces

Wiki ‘DataSpaces’ Creating and Sharing Metadata Community Catalog - Find Dataset Describe Dataset Discuss Dataset ESIP Communal Wiki Semantic Wiki: Structured (RDF and Unstructured Content Open, Standard Matadata - RDF Ready for Export/Harvesting by Registries, Catalogs

Semantic Wiki: Structured (RDF and Unstructured Content

Open, Standard Matadata - RDF

Ready for Export/Harvesting by Registries, Catalogs

Sharing Best Practices: GEO Best Practice Wiki

Developments and Challenges: Favorable Engineering Developments: A Core network for Air Quality data sharing is emerging. Standards are available for sharing previously unstructured data Third-party mediation can homogenize the distributed data Agile SOA-based systems can deliver info products to diverse users Since 2005, one such IS, DataFed is used by EPA and in research However: Service interfaces are still uneven; networks are still fragile The utility of social networking in science is not understood Users can not provide feedback to upstream providers Many cultural, legal and other barriers hamper progress

Favorable Engineering Developments:

A Core network for Air Quality data sharing is emerging.

Standards are available for sharing previously unstructured data

Third-party mediation can homogenize the distributed data

Agile SOA-based systems can deliver info products to diverse users

Since 2005, one such IS, DataFed is used by EPA and in research

However:

Service interfaces are still uneven; networks are still fragile

The utility of social networking in science is not understood

Users can not provide feedback to upstream providers

Many cultural, legal and other barriers hamper progress

ESIP Coordination Application