Your SlideShare is downloading. ×
  • Like
2008-05-27: Spring AGU, DataFed Best Practices
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

2008-05-27: Spring AGU, DataFed Best Practices

  • 450 views
Published

 

Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
450
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • This talk is about the federated.. In particular our experineces with datafed over past 4 years of data homog and networking

Transcript

  • 1. The Federated Data System DataFed: Experiences in Data Homogenization and Networking
    • R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson , Washington University, St. Louis
    • G. Leptoukh , NASA GSFC
    Spring AGU, May 29, 2008, Ft. Lauderdale
  • 2. DataFed in a Nutshell: A Federation of autonomous, distributed data providers Performs non-intrusive wrapping of data into web services Provides service-based analysis services and tools General Experience with DataFed: It is an agile virtual data system can deliver info products to diverse users Third-party mediation can homogenize distributed data on the fly Since 2005, DataFed is used by EPA and in research DataFed Motivated by GEOSS DataFed development is guided by the meme of GEOSS
  • 3. Five practices for agile, seamless data federation:
    • Space-Time Query for standardized access to all data (WCS)
    • Data Wrappers for turning heterogeneous data into web services
    • Data Mediators for transforming data into ‘Views’
    • Mashups for connecting autonomous application
    • DataSpaces for shared metadata by the users, for the users
  • 4. Parameter-Space-Time Query Using OGC WCS Data Access Protocol
    • Regardless of the data location, data type and format,
    • the parameter-space-time query is the same
    • the return is in user selectable format from the offerings
    Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF Coverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFF Coverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-table Grid Image Station Data Parameter Bounding Box Time Range Out Format
  • 5.
    • DataFed wrappers are non-intrusive, third party
    Third Party Data Wrappers Heterogeneous input data >>> Homogeneous (WCS) Query
  • 6. Mediated User-Data Interface Mediator turns data into Views Mediated Integration is a flexible design pattern for System of Systems Client-Server design is demanding: User carries the burden of integration Query Data Views
  • 7. SOAP RDF Mashup Workflow Mashups: Loose Coupling of Autonomous Applications DataFed – Wiki -- GoogleEarth
  • 8. DataSpaces for Datasets GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Catalog User Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users Services Community DataSpaces
  • 9. GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users views report Services Community DataSpaces
  • 10. -+ GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall , Feb 2008 by R. Husar, March 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Decision Maker Policy Analyst Informs find links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users views report Services Community DataSpaces
  • 11. Wiki ‘DataSpaces’ Creating and Sharing Metadata Community Catalog - Find Dataset Describe Dataset Discuss Dataset ESIP Communal Wiki
    • Semantic Wiki: Structured (RDF and Unstructured Content
    • Open, Standard Matadata - RDF
    • Ready for Export/Harvesting by Registries, Catalogs
  • 12. Sharing Best Practices: GEO Best Practice Wiki
  • 13. Developments and Challenges:
    • Favorable Engineering Developments:
    • A Core network for Air Quality data sharing is emerging.
    • Standards are available for sharing previously unstructured data
    • Third-party mediation can homogenize the distributed data
    • Agile SOA-based systems can deliver info products to diverse users
    • Since 2005, one such IS, DataFed is used by EPA and in research
    • However:
    • Service interfaces are still uneven; networks are still fragile
    • The utility of social networking in science is not understood
    • Users can not provide feedback to upstream providers
    • Many cultural, legal and other barriers hamper progress
  • 14. ESIP Coordination Application
  • 15.  
  • 16.