• Like

Using HDF5 Archive Information Package to preserve HDF-EOS2 data

  • 311 views
Uploaded on

This one-year research project, funded by NOAA Climate Program Office (CPO) Scientific Data Stewardship (SDS), provides a solution to migrate data to a single standards-based archive format. …

This one-year research project, funded by NOAA Climate Program Office (CPO) Scientific Data Stewardship (SDS), provides a solution to migrate data to a single standards-based archive format. Specifically, we investigate on how to store NASA ECS data and metadata into HDF5 Archival Information Packages (AIP). To achieve this, the HDF4 to HDF5 conversion tool has been enhanced so that converted ECS data can be read through the NetCDF4/CDM interface. In addition, metadata tools will be developed that convert ECS collection and granule level metadata to NOAA's collection level and NARA's METS standard. The enhanced HDF4 to HDF5 conversion tool has been released in May 2008 and it includes new functionality as the converted ECS data can be read through the NetCDF4 interface. We have tested 33 typical HDF-EOS2 swath, grid and point products at the National Snow and Ice Data Center (NSIDC). We also demonstrate the initial effort of the work to develop METS compliant metadata from granule metadata held in NASA's Earth Observing System (EOS) Data and Information System (EOSDIS) Core System (ECS).

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
311
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Primary Schema Extension Schema
  • Deliverable is the enhanced h4toh5 conversion tool
  • Clearly mention ‘orthogonal’, ‘two-dimensional longitude and latitude’
  • http://svs.gsfc.nasa.gov/vis/a000000/a002900/a002935/index.html
    NASA/Goddard Space Flight CenterScientific Visualization Studio

Transcript

  • 1. Storing EOS data using HDF5 Archival Information Package MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ) 10/16/2008 HDF and HDF-EOS Workshop XII 1
  • 2. Definitions of some terms • CDM(Common Data Model) • Unification of the data models of OPeNDAP, netCDF and HDF5 • CF-Conventions • Convention of metadata providing spatial properties, among climate research community… • CLASS (Comprehensive Large Array-data Stewardship System) • Electronic library of NOAA environmental data 10/16/2008 HDF and HDF-EOS Workshop XII 2
  • 3. Prerequisite • METS(Metadata Encoding & Transmission Standard) • Standard for encoding structural metadata • ISO-19115 • International Schema for describing geographic information • File-level Metadata • Metadata about the individual file or granule • Dataset-level Metadata • Metadata that applies to each and every granule/file in the whole data set(product) 10/16/2008 HDF and HDF-EOS Workshop XII 3
  • 4. HDF5 Archive Information Package HDF5 Data file Metadata file Primary Schema METS Extension Schema |<mets> |---<dmdSec>----------------<MODS> |---<amdSec>--------------|--<techMD> | |--<rightsMD> | |--<sourceMD> |----<fileGrp> |----<structMap> HDF5 AIP Components p://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf 10/16/2008 HDF and HDF-EOS Workshop XII 4
  • 5. NOAA SDS Program CDM/NetCDF4 ISO-19115 NCDC:CLASS ECS to ISO19115 HDF5-AIP NetCDF4 / HDF5 Data METS NetCDF4/HDF5-data 10/16/2008 ECS to METS NSIDC/ECS Metadata H4to H5 NSIDC/ ECS HDF and HDF-EOS Workshop XII HDF4-data 5
  • 6. Enhanced H4toH5 conversion tool • Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data $ ./h4toh5 –eos –nc4 input.he2 output.nc4 $ ./h4toh5 –eos –nc4 input.he2 output.nc4 • Official release (2.0) can be found at http://hdfgroup.org/h4toh5/ 10/16/2008 HDF and HDF-EOS Workshop XII 6
  • 7. Challenges to do the conversion • Retrieve geo-location information from HDFEOS2 data • Conform to NetCDF4 data model in the existing H4toH5 conversion tool • …… 10/16/2008 HDF and HDF-EOS Workshop XII 7
  • 8. Challenges: Handle EOS - Grid • Grid lacks geolocation fields Data [4][12] [4][8] Lon[4][8] • Use predefined projections Lon[12] • • • • Geographic Geographic Sinusoidal Sinusoidal Polar stereographic … • New converter creates geolocation fields • HDF-EOS2 API GDij2ll() 10/16/2008 HDF and HDF-EOS Workshop XII 8
  • 9. Challenges: Handle EOS - Swath • The size of geolocation fields can be different from data fields • New converter has to handle geolocation fields correctly 10/16/2008 HDF and HDF-EOS Workshop XII 9
  • 10. Challenges in conforming to NetCDF4 New longitude has three columns Longitude has three columns Data field • Follow CF conventions field has two columns • Create two variables: NewLongitude and NewLatitude • Add to the data field an attribute coordinates=“NewLongitude NewLatitude” • Keep the original Latitude and Longitude 10/16/2008 HDF and HDF-EOS Workshop XII 10
  • 11. Now some examples to show NetCDF4 files converted from EOS2 10/16/2008 HDF and HDF-EOS Workshop XII 11
  • 12. A netCDF-4 file converted from EOS2 data at NSIDC 10/16/2008 HDF and HDF-EOS Workshop XII 12
  • 13. A netCDF-4 file converted from EOS2 data at NSIDC 10/16/2008 HDF and HDF-EOS Workshop XII 13
  • 14. Deliverables and future work • Deliverables 1. Enhanced HDF4 to HDF5 conversion tool http://hdfgroup.org/h4toh5/ 2. A validation tool to verify the correctness of the conversion Will be released soon! • Potential future work 1. Reduce the output file size 2. Improve the performance 10/16/2008 HDF and HDF-EOS Workshop XII 14
  • 15. Acknowledgement This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA. 10/16/2008 HDF and HDF-EOS Workshop XII 15