HDF-EOS Overview and Status
Emergent Information Technologies, Inc.
September 19, 2000
What is HDF-EOS?
• An extension to HDF, built with HDF objects.
• A standard for data structures for remote
sensing data and in-situ measurements with:
– Tightly coupled geolocation information
– Access based on geolocation
• Incorporates ECS metadata attributes.
• Used for many EOS standard products.
HDF-EOS File Contents
• Global HDF attributes
– Core Metadata - searchable attributes, copied in
– Archive Metadata - non-searchable attributes
• Any number of HDF objects specifying
HDF-EOS objects (Grid, Point, Swath)
– Structural metadata - links geolocation to
• Additional HDF objects.
HDF-EOS Grid Structure
• Usage - Data which is organized by regular
geographic spacing, specified by projection
– Any number of 2-D to 8-D data arrays per
structure, one per data type (e.g. temperature)
– Geolocation information contained in projection
formula, coupled by structural metadata.
– Any number of Grid structures per file allowed.
Grid Data: 1 Km. Digital Elevation Model
HDF-EOS Swath Structure
• Usage - Data which is organized by time,
monotonic geolocation, track parameter.
Spacing can be irregular.
– Geolocation information stored explictly in
Geolocation Field (2-D array)
– Data stored in 2-D or 3-D arrays
– Time stored in 1-D or 2-D array, TAI is standard.
– Geolocation/science data connected by structural
HDF-EOS Point Structure
• Usage - Data which is specified temporally
and/or spatially, but with no particular
– Tables (Vdatas) used to store science data at a
– Up to eight levels of data allowed. Structural
metadata specifies relationship between levels.
HDF-EOS Interface Schematic
ECS Metadata Interface
Low Level HDF
Access - Open, close objects/files
Definition - Set key features of data sets
Basic I/O - Read write data
Index I/O - Link tables (Point only)
Inquiry - Return information about data
Subset - Read data from specific region
Metadata access (SDP toolkit)
Current Archive Holdings
• GSFC DAAC: MODIS L1, Atmos./Ocean
– 50 TB, 105,000 granules, 300 GB/day
– will be ~1.3 TB / day with addition of MODIS
L2 and EOS Aqua data.
• EDC DAAC: Landsat/ASTER/MODIS Land
– 20 TB, 106,000 granules, 274 GB/day
• Langely DAAC: MISR
– 9 TB, 39,000 granules, 88 GB/day
• HDF library - ftp hdf.ncsa.uiuc.edu
• HDF-EOS V2.6 library (HDF4 based)
• SDP Toolkit library - same location as HDF-EOS
• Platform Support
– DEC, HP, IBM, Sun, SGI, Windows, Linux (soon)
• Language Support - C, C++, F77, F90
• HDF4-based library will be supported indefinitely
HDF5 - Based Development
• NCSA has introduced HDF5
– A complete rewrite of HDF4, a new user
– Simplified data model
– Breaks 2 GB file size limit
– Potential for parallel I/O, multi-threading
HDF5 - Based Development
• Development overview
– Maintain current HDF-EOS interface to
– Allow all HDF5 features through interface
– Build HDF-EOS (4 to 5) conversion function
– Data production executable can be built with
either or both HDF4 and HDF5 - based library
– Support heterogeneous ECS archive (future)
Grid, Point, Swath
Definition, Access, I/O, Inquiry, Subset
• H5EOSView - EOSView for HDF5
• Java Browser - Browser for both HDF4 and
5 based granules
• HDF-EOS 4 to 5 converter, HDF-EOS 2.x
to HDF-EOS 3.x
• Jan., 2000 - Beta Version of library
– H5EOSView, Sun, SGI, C support
• Oct. 2000 - Beta2 Version
– Add F77, F90, DEC support
– Local Attributes, Profile Interface,
– HDF-EOS 4 to 5 converter and Java Browser
• Jan. 2001 - SCF version
– add HP, IBM, Windows, Linux (maybe a few
months later), C++ support
• DAAC version (ECS Drop 6B) - Summer
– Archives can contain both HDF4 and HDF5 based granules.
– ECS subsystems don’t crack granules, but
regression testing needed.
– 2 GB size limit is an issue.