This one-year research project, funded by NOAA Climate Program Office (CPO) Scientific Data Stewardship (SDS), provides a solution to migrate data to a single standards-based archive format. Specifically, we investigate on how to store NASA ECS data and metadata into HDF5 Archival Information Packages (AIP). To achieve this, the HDF4 to HDF5 conversion tool has been enhanced so that converted ECS data can be read through the NetCDF4/CDM interface. In addition, metadata tools will be developed that convert ECS collection and granule level metadata to NOAA's collection level and NARA's METS standard. The enhanced HDF4 to HDF5 conversion tool has been released in May 2008 and it includes new functionality as the converted ECS data can be read through the NetCDF4 interface. We have tested 33 typical HDF-EOS2 swath, grid and point products at the National Snow and Ice Data Center (NSIDC). We also demonstrate the initial effort of the work to develop METS compliant metadata from granule metadata held in NASA's Earth Observing System (EOS) Data and Information System (EOSDIS) Core System (ECS).

1. Storing EOS data using HDF5
Archival Information Package
MuQun Yang (The HDF Group)
Choonghwan Lee (The HDF Group)
Ruth Duerr (NSIDC )
10/16/2008
HDF and HDF-EOS Workshop XII
1

2. Definitions of some terms
• CDM(Common Data Model)
• Unification of the data models of OPeNDAP,
netCDF and HDF5
• CF-Conventions
• Convention of metadata providing spatial
properties, among climate research community…
• CLASS (Comprehensive Large Array-data
Stewardship System)
• Electronic library of NOAA environmental data
10/16/2008
HDF and HDF-EOS Workshop XII
2

3. Prerequisite
• METS(Metadata Encoding & Transmission
Standard)
• Standard for encoding structural metadata
• ISO-19115
• International Schema for describing geographic
information
• File-level Metadata
• Metadata about the individual file or granule
• Dataset-level Metadata
• Metadata that applies to each and every
granule/file in the whole data set(product)
10/16/2008
HDF and HDF-EOS Workshop XII
3

4. HDF5 Archive Information Package
HDF5
Data file
Metadata file
Primary Schema
METS
Extension Schema
|<mets>
|---<dmdSec>----------------<MODS>
|---<amdSec>--------------|--<techMD>
|
|--<rightsMD>
|
|--<sourceMD>
|----<fileGrp>
|----<structMap>
HDF5 AIP Components
p://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf
10/16/2008
HDF and HDF-EOS Workshop XII
4

5. NOAA SDS Program
CDM/NetCDF4
ISO-19115
NCDC:CLASS
ECS to
ISO19115
HDF5-AIP
NetCDF4 /
HDF5 Data
METS
NetCDF4/HDF5-data
10/16/2008
ECS to
METS
NSIDC/ECS
Metadata
H4to
H5
NSIDC/ ECS
HDF and HDF-EOS Workshop XII
HDF4-data
5

6. Enhanced H4toH5 conversion tool
• Convert HDF-EOS2 data to NetCDF4-compliant HDF5
data
$ ./h4toh5 –eos –nc4 input.he2 output.nc4
$ ./h4toh5 –eos –nc4 input.he2 output.nc4
• Official release (2.0) can be found at
http://hdfgroup.org/h4toh5/
10/16/2008
HDF and HDF-EOS Workshop XII
6

7. Challenges to do the conversion
• Retrieve geo-location information from HDFEOS2 data
• Conform to NetCDF4 data model in the existing
H4toH5 conversion tool
• ……
10/16/2008
HDF and HDF-EOS Workshop XII
7

8. Challenges: Handle EOS - Grid
• Grid lacks geolocation fields Data [4][12]
[4][8]
Lon[4][8]
• Use predefined projections Lon[12]
•
•
•
•
Geographic
Geographic
Sinusoidal
Sinusoidal
Polar stereographic
…
• New converter creates geolocation fields
• HDF-EOS2 API GDij2ll()
10/16/2008
HDF and HDF-EOS Workshop XII
8

9. Challenges: Handle EOS - Swath
• The size of geolocation fields can be different from data
fields
• New converter has to handle geolocation fields
correctly
10/16/2008
HDF and HDF-EOS Workshop XII
9

10. Challenges in conforming to NetCDF4
New longitude has three columns
Longitude has three columns
Data field
• Follow CF conventions field has two columns
• Create two variables: NewLongitude and
NewLatitude
• Add to the data field an attribute
coordinates=“NewLongitude NewLatitude”
• Keep the original Latitude and Longitude
10/16/2008
HDF and HDF-EOS Workshop XII
10

11. Now some examples
to show NetCDF4 files
converted from EOS2
10/16/2008
HDF and HDF-EOS Workshop XII
11

12. A netCDF-4 file converted from EOS2 data at NSIDC
10/16/2008
HDF and HDF-EOS Workshop XII
12

13. A netCDF-4 file converted from EOS2 data at NSIDC
10/16/2008
HDF and HDF-EOS Workshop XII
13

14. Deliverables and future work
• Deliverables
1. Enhanced HDF4 to HDF5 conversion tool
http://hdfgroup.org/h4toh5/
2. A validation tool to verify the correctness of the
conversion
Will be released soon!
• Potential future work
1. Reduce the output file size
2. Improve the performance
10/16/2008
HDF and HDF-EOS Workshop XII
14

15. Acknowledgement
This work was supported under NOAA Scientific Stewardship Program
grant number NA07OAR4310286. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of NOAA.
10/16/2008
HDF and HDF-EOS Workshop XII
15