This document provides an overview of HDF-EOS, which is an extension to HDF that defines standard data structures for remote sensing and in-situ data with tightly coupled geolocation information. It describes the core components of HDF-EOS files, including Grid, Swath, and Point structures, and provides examples. It also outlines the development of an HDF5-based version to overcome limitations of the HDF4-based library and allow for larger files.

1. HDF-EOS Overview and Status
Larry Klein
Emergent Information Technologies, Inc.
September 19, 2000
larry@eos.hitc.com

2. What is HDF-EOS?
• An extension to HDF, built with HDF objects.
• A standard for data structures for remote
sensing data and in-situ measurements with:
– Tightly coupled geolocation information
– Access based on geolocation
• Incorporates ECS metadata attributes.
• Used for many EOS standard products.
2

3. HDF-EOS File Contents
• Global HDF attributes
– Core Metadata - searchable attributes, copied in
database
– Archive Metadata - non-searchable attributes
• Any number of HDF objects specifying
HDF-EOS objects (Grid, Point, Swath)
– Structural metadata - links geolocation to
science data
• Additional HDF objects.
3

4. HDF-EOS Grid Structure
• Usage - Data which is organized by regular
geographic spacing, specified by projection
parameters.
• Structure
– Any number of 2-D to 8-D data arrays per
structure, one per data type (e.g. temperature)
– Geolocation information contained in projection
formula, coupled by structural metadata.
– Any number of Grid structures per file allowed.
4

5. Grid Data Example
5

6. Grid Data: 1 Km. Digital Elevation Model
South Pole
6

7. HDF-EOS Swath Structure
• Usage - Data which is organized by time,
monotonic geolocation, track parameter.
Spacing can be irregular.
• Structure
– Geolocation information stored explictly in
Geolocation Field (2-D array)
– Data stored in 2-D or 3-D arrays
– Time stored in 1-D or 2-D array, TAI is standard.
– Geolocation/science data connected by structural
7
metadata

8. Swath Data Example
8

9. HDF-EOS Point Structure
• Usage - Data which is specified temporally
and/or spatially, but with no particular
organization.
• Structure
– Tables (Vdatas) used to store science data at a
particular Lat/Long/Height.
– Up to eight levels of data allowed. Structural
metadata specifies relationship between levels.
9

10. Point Data Example
Lat
61.12
45.31
38.50
38.39
30.00
37.45
18.00
43.40
34.03
32.45
33.30
42.15
35.05
34.12
46.32
47.36
39.44
21.25
44.58
41.49
25.45
Lon
-149.48
-122.41
-77.00
-90.15
-90.05
-122.26
-76.45
-79.23
-118.14
-96.48
-112.00
-71.07
-106.40
-77.56
-87.25
-122.20
-104.59
-78.00
-93.15
-87.37
-80.11
Temp(C)
15.00
17.00
24.00
27.00
22.00
25.00
27.00
30.00
25.00
32.00
30.00
28.00
30.00
28.00
30.00
32.00
31.00
28.00
32.00
28.00
19.00
10
Dewpt(C)
5.00
5.00
7.00
11.00
7.00
10.00
4.00
14.00
4.00
8.00
10.00
7.00
9.00
9.00
8.00
15.00
16.00
7.00
13.00
9.00
3.00

11. HDF-EOS Interface Schematic
User Application
HDF-EOS Interface
Grid
Point
ECS Metadata Interface
Swath
Low Level HDF
File I/O
11

12. User Interface
•
•
•
•
•
•
•
Access - Open, close objects/files
Definition - Set key features of data sets
Basic I/O - Read write data
Index I/O - Link tables (Point only)
Inquiry - Return information about data
Subset - Read data from specific region
Metadata access (SDP toolkit)
12

13. Current Archive Holdings
• GSFC DAAC: MODIS L1, Atmos./Ocean
– 50 TB, 105,000 granules, 300 GB/day
– will be ~1.3 TB / day with addition of MODIS
L2 and EOS Aqua data.
• EDC DAAC: Landsat/ASTER/MODIS Land
– 20 TB, 106,000 granules, 274 GB/day
• Langely DAAC: MISR
– 9 TB, 39,000 granules, 88 GB/day
13

14. Availability
• HDF library - ftp hdf.ncsa.uiuc.edu
• HDF-EOS V2.6 library (HDF4 based)
– http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
• SDP Toolkit library - same location as HDF-EOS
• Platform Support
– DEC, HP, IBM, Sun, SGI, Windows, Linux (soon)
• Language Support - C, C++, F77, F90
• HDF4-based library will be supported indefinitely
14

15. HDF5 - Based Development
• NCSA has introduced HDF5
– A complete rewrite of HDF4, a new user
interface.
– Simplified data model
– Breaks 2 GB file size limit
– Potential for parallel I/O, multi-threading
15

16. HDF5 - Based Development
HDF-EOS 3.0
• Development overview
– Maintain current HDF-EOS interface to
maximum extent
– Allow all HDF5 features through interface
– Build HDF-EOS (4 to 5) conversion function
– Data production executable can be built with
either or both HDF4 and HDF5 - based library
– Support heterogeneous ECS archive (future)
16

17. Design Schematic
User Application
HDF-EOS Interface
Grid, Point, Swath
ECS Metadata
Interface
Common Functions
Definition, Access, I/O, Inquiry, Subset
*Implemented
if required
HDF4
Interface*
HDF5
Interface
Combined
HDF4/5
Interface*
17
Build
Options

18. Applications
• H5EOSView - EOSView for HDF5
• Java Browser - Browser for both HDF4 and
5 based granules
• HDF-EOS 4 to 5 converter, HDF-EOS 2.x
to HDF-EOS 3.x
18

19. Schedule
• Jan., 2000 - Beta Version of library
– H5EOSView, Sun, SGI, C support
• Oct. 2000 - Beta2 Version
– Add F77, F90, DEC support
– Local Attributes, Profile Interface,
Chunking/Tiling
– HDF-EOS 4 to 5 converter and Java Browser
19

20. Schedule (cont.)
• Jan. 2001 - SCF version
– add HP, IBM, Windows, Linux (maybe a few
months later), C++ support
• DAAC version (ECS Drop 6B) - Summer
2001.
– Archives can contain both HDF4 and HDF5 based granules.
– ECS subsystems don’t crack granules, but
regression testing needed.
– 2 GB size limit is an issue.
20