Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NASA HDF and HDF-EOS Status - Use in EOSDIS


Published on

EOSDIS Status 2003

Published in: Technology
  • Be the first to comment

  • Be the first to like this

NASA HDF and HDF-EOS Status - Use in EOSDIS

  1. 1. NASA HDF and HDF-EOS Status Use in EOSDIS HDF & HDF-EOS Workshop September 23-25, 2003 Silver Spring, MD Richard Ullman
  2. 2. EOSDIS EOSDIS is a system to acquire, archive, manage and distribute NASA’s Earth observation data. EOSDIS data sources: “Standard products” from EOS satellite instruments. Field Measurement programs Legacy data sets EOSDIS consists of the geographically distributed EOSDIS Core System (ECS), a network of EOS production systems (called SIPS) as well as other production and archive systems. EOSDIS is managing and distributing data from: EOS missions: 3050 Terabytes Pre-EOS missions: UARS, SeaWIFS, TOMS-EP, TOPEX/Poseidon and TRMM, and ESE legacy data (284 Terabytes.)
  3. 3. EOSDIS User Base Distinct ECS and Non-ECS Users Accessing DAACs By Domain Foreign Other 12% U.S. Education 7% Other 36% Foreign Commercial 5% University of Washington 2% Columbia University 2% Pennsylvania State University University of Colorado Foreign Education 2% U.S. Government 1% U.S. Other 1% Total Distinct Users: 1,821,563 U.S. Commercial 36% Distinct Foreign ECS and Non-ECS Users Accessing DAACs by Country Top 10 14% Other 88% France 7% Texas A & M University Stanford University Canada 16% Other 36% University of Minnesota University of Maryland Distinct Users: 122,556 University of Illinois Harvard University Australia 7% Japan 6% Spain 3% Mexico 3% Distinct Users: 335,597 Italy 6% Germany 5% Netherlands 5% United Kingdom 6% Based on Email Addresses of Users October 1, 2002 - August 31, 2003
  4. 4. Standard Data Format Goals Support Core System Development Provide interface standard for passing of data Allow Dataset-Independent Development and Services Provide Instrument-Independent Services Geolocation Services: row/column to/from latitude/longitude Subsetting and subsampling by parameter, row/col, lat/long
  5. 5. Standard Data Format Goals Facilitate scientific data storage, exchange, access, analysis and discovery. Make Data Easily Accessible Make data self describing Provide instrument-independent data structures Use widely available standards and software Metadata, Geolocation, Annotations, Organization
  6. 6. HDF Format and software for scientific data Stores images, arrays, tables, etc. Emphasis on storage and I/O efficiency Free and commercial software support Emphasis on standards Users from many engineering & scientific fields
  7. 7. HDF-EOS A “profile” of HDF Profiles are sets of standard conventions for application to a particular community of users HDF-EOS conventions Consistent metadata location and format Consistent application of geolocation information Point, Swath, Grid objects HE5 additional objects: atmospheric profile, zonal The EOS profile of HDF is supported by a library that calls the underlying HDF library Similar API philosophy HDF-EOS files are readable by standard HDF libraries but without EOS specific services. Can facilitate standard services on data Example subsetting
  8. 8. HDF-EOS History September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities December 1994, Began definition of HDF profile for EOS ( HDF-EOS) June 1996, HDF-EOS v1.0 using HDF-4 (HE-4) November 2000, HD-EOS v5.0 using HDF-5 (HE-5) Upgrades to both HE-4 and HE-5 occur as needed - about every 6 months
  9. 9. HDF-EOS Present ESDIS development of HDF-EOS is complete. Remaining work is primarily corrective and adaptive maintenance. Current version HE-4 is 2.9 on HDF 4.1r5 Current version HE-5 is 5.1.5 on HDF 5-1.4.4 Nearly all ECS holdings are in HE-4 Instrument science teams are encouraged to consider migration to HE5 to avoid future obsolescence but there is no stated requirement for this migration. Many tools for use with EOS data rely on HDF-EOS. Many developed and maintained by instrument science teams and DAACs. See See also DAAC websites Some commercial tools include HDF-EOS read capability MatLab, IDL, ENVI, PCI Geomatics (Landsat), others
  10. 10. EOS Profile Experience HDF was new to most in the EOS program when the format was chosen for EOSDIS standard products. Initial reluctance by EOS product developers yielded to appreciation for self-description and data organization flexibility. Flexibility of HDF allows tailoring to custom needs and desires of producers. Still learning impact of HDF on end users Anecdotal evidence is that many users do not use data in HDFEOS, but instead “unpack” the data for use in their own application.
  11. 11. EOS Profile Experience “Weak” profile is a hindrance to general data interoperability Increases difficulty for development of general services for data. Discourages adoption of standard by general tool vendors. Discourages reuse of format by external product developers. Hinders end-user ability to easily use data.
  12. 12. New directions for Data Standards at NASA Earth Science Enterprise NASA has directed the formation of a standards working group for recommending adoption of ESE wide standards for data and data systems interfaces. Standards adoption will stress implementation and operational experience. Standards will be “community based” and will likely differ for different kinds of measurements or applications.
  13. 13. Workshop The HDF and HDF-EOS workshop is the annual meeting of the HDF community. Goals: Educate participants in the use of HDF and HDF-EOS Share progress Help prioritize future development of the standard Logistics Tutorials (yesterday) Consulting (please talk with NCSA or L3 staff about private consultation) “Plenary” - this morning will be in “lecture” format Presentations - presentations will be in combined oral abstract with poster sessions. Discussion - discussion topics are meant to be free-form and open and are intended to develop understanding of the community’s needs with respect to HDF and HDF-EOS.