1. HDF-EOS APIs, tools, etc.
Larry Klein, Abe Taaheri, and Cid Praderas
L-3 Communications
Government Services, Inc.
November 30, 2005
1

2. What is HDF-EOS ?
• HDF (Hierarchical Data Format) is the format
chosen by NASA as the standard for the EOS
program
• HDF-EOS is an extension to HDF which
standardizes storage of and access to common Earth
Sciences data structures
– By providing a single interface to data structures
common to earth science
– By providing a container for EOS inventory,
archive and product specific metadata
– By establishing a consistent relationship between
geolocation and science data.
2

3. HDF-EOS Grid Structure
• Usage - Data which is organized
by regular geographic spacing,
specified by projection parameters.
• Structure
– Any number of 2-D to 8-D data arrays per structure, one
per data type (e.g. temperature)
– Geolocation information contained in projection formula,
coupled by structural metadata.
– Any number of Grid structures per file allowed.
3

4. HDF-EOS Swath Structure
• Data which is organized by time, or other track
parameter.
• Spacing can be irregular.
• Structure
– Geolocation information stored explicitly in
Geolocation Field (2-D array)
– Data stored in 2-D or 3-D arrays
– Time stored in 1-D or 2-D array, TAI is standard.
– Geolocation/science data connected by structural
metadata
4

5. Swath Data Example
5

6. HDF-EOS Point Structure
• Data is specified temporally and/or spatially,
but with no particular organization
• Structure
– Tables used to store science data at a particular
Lat/Long/Height
– Up to eight levels of data allowed. Structural
metadata specifies relationship between levels.
6

7. Zonal Average Structure
• Generalized array structure with no
geolocation linkage
• Data can be organized by time or track
parameter
• Data spacing can be irregular
• Structure
– Data stored in multidimensional arrays
– Time stored in 1-D or 2-D array
7

8. HDF-EOS 2
• HDF4 - based, storage format for EOS standard
products.
• Used operationally by MODIS, MISR, ASTER,
Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by
NCSA/NASA for the fore-seeable future.
8

9. HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF4 with
a different user interface.
– First released in 2000.
– Format used by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the
maximum extent possible.
– Supports same data structures
– Added prefix ‘HE5_’ to HDF-EOS 2 functions.
– Doesn’t preclude HDF5 functionality.
9

10. Top Level of HDF-EOS 5 File
Root -- “/”
Root -- “/”
HDFEOS
HDFEOS
INFORMATION
INFORMATION
HDFEOS
HDFEOS
ECS METADATA
ECS METADATA
ADDITIONAL
ADDITIONAL
STRUC. METADATA
STRUC. METADATA
ARRAY SWATH
ARRAY SWATH
GRID
GRID
POINT
POINT
Global (File)
Global (File)
Attributes
Attributes
10

11. Basic HDF-EOS Functions
• Access - Open, create, close objects/files
• Definition - Set key parameters of data sets
• Input/Output - Read/Write data
• Inquiry - Return information about data
• Subset - Read data from specified region
11

12. File/Object Access
•
•
•
•
•
GRID:
HE5_GDopen/he5_gdopen
HE5_GDcreate/he5_gdcreate
HE5_GDattach/he5_gdattach
HE5_GDdetach/he5_gddetach
HE5_GDclose/he5_gdclose
12

13. File/Object Access
•
•
•
•
•
POINT:
HE5_PTopen/he5_ptopen
HE5_PTcreate/he5_ptcreate
HE5_PTattach/he5_ptattach
HE5_PTdetach/he5_ptdetach
HE5_PTclose/he5_ptclose
13

14. File/Object Access
•
•
•
•
•
SWATH:
HE5_SWopen/he5_swopen
HE5_SWcreate/he5_swcreate
HE5_SWattach/he5_swattach
HE5_SWdetach/he5_swdetach
HE5_SWclose/he5_swclose
14

15. File/Object Access
•
•
•
•
•
ZA:
HE5_ZAopen/he5_zaopen
HE5_ZAcreate/he5_zacreate
HE5_ZAattach/he5_zaattach
HE5_ZAdetach/he5_zadetach
HE5_ZAclose/he5_zaclose
15

16. Definition
GRID:
• HE5_GDdefdim/he5_gddefdim
• HE5_GDdefproj/he5_gddefproj
• HE5_GDdeforigin/he5_gddeforigin
• HE5_GDdefpixreg/he5_gd_defpreg
16

17. Definition
POINT:
• HE5_PTdeflevel/he5_ptdeflevel
• HE5_PTdeflinkage/he5_ptdeflinkage
17

18. Definition
•
•
•
•
•
SWATH:
HE5_SWdefdim/he5_swdefdim
HE5_SWdefdimmap/he5_swdefmap
HE5_SWdefidxmap/he5_swdefimap
HE5_SWdefgeofield/he5_swdefgfld
HE5_SWdefdatafield/he5_swdefdfld
18

19. Definition
•
•
•
•
ZA:
HE5_ZAdefdim/he5_zadefdim
HE5_ZAdefine/he5_zadefine
HE5_ZAdefcomp/he5_zadefcomp
HE5_ZAdefchunk/he5_zadefchunk
19

20. Input/Output
GRID:
• HE5_GDwritefield/he5_gdwrfld
• HE5_GDreadfield/he5_gdrdfld
• HE5_GDwriteattr/he5_gdwrattr
• HE5_GDreadattr/he5_gdrdattr
20

21. Input/Output
POINT:
• HE5_PTwritelevel/he5_ptwritelevel
• HE5_PTreadlevel/he5_ptreadlevel
• HE5_PTwriteattr/he5_ptwrattr
• HE5_PTreadattr/he5_ptrdattr
21

22. Input/Output
SWATH:
• HE5_SWwritefield/he5_swwrfld
• HE5_SWreadfield/he5_swrdfld
• HE5_SWwriteattr/he5_swwrattr
• HE5_SWreadattr/he5_swrdattr
22

23. Input/Output
ZA:
• HE5_ZAwrite/he5_zawrite
• HE5_ZAread/he5_zaread
• HE5_ZAwriteattr/he5_zawrattr
• HE5_ZAreadattr/he5_zardattr
23

24. Inquiry
•
•
•
•
•
GRID:
HE5_GDinqdims/he5_gdinqdims
HE5_GDdiminfo/he5_gddiminfo
HE5_GDprojinfo/he5_gdprojinfo
HE5_GDorigininfo/he5_gdorigininfo
HE5_GDpixreginfo/he5_gdpreginfo
24

25. Inquiry
POINT:
• HE5_PTlevelinfo/he5_ptlevelinfo
• HE5_PTinqattrs/he5_ptinqattrs
• HE5_PTattrinfo/he5_ptattrinfo
25

26. Inquiry
SWATH:
• HE5_SWinqdatafields/he5_swinwdflds
• HE5_SWinqgeofields/he5_swinqgflds
• HE5_SWinqattrs/he5_swinqattrs
• HE5_SWattrinfo/he5_swattrinfo
26

27. Inquiry
ZA:
• HE5_ZAinquire/he5_zainquire
• HE5_ZAinfo/he5_zainfo
• HE5_ZAinqattrs/he5_zainqattrs
• HE5_ZAattrinfo/he5_zaattrinfo
27

28. Programming Model
• Open the HDF-EOS5 file
• Create or Attach to a specified
Grid/Swath/Point/ZA object
• Define key object features and/or
perform necessary operations (with
the data sets, attributes, etc)
• Detach from the Grid/Swath/Point/ZA
object
• Close the HDF-EOS5 file
28

29. HDF-EOS Functionality
•
•
•
•
•
•
Basic File I/O
Swath Interface
Grid Interface
Point Interface
Profile Interface
Generalized Array
Interface (Zonal Avg.
(HDF5))
• Subsetting
• Fill Values
• Compression (gzip,
szip, shuffling)
• Chunking/Tiling
• Aliases
• Global (File), Group
& Local Attributes
• Threadsafe Version
29

30. Applications
• HE5View (HDF5 - based browser)
• EOSView (HDF4 - based browser)
• HDFView (with HDF-EOS Plug-in), Java EOS
Browser (HDF-EOS 2 and 5 access)
• heconvert (converts HDF4 - based Grid/Point/Swath
structures to HDF-EOS 5 equivalents)
– Compliments HDF4 -> HDF5 conversion tool
– Not tested on all products
• HEG (HDF-EOS to GeoTIFF Converter/Subsetter)
30

31. HDFView
•
•
•
•
•
HDFView: A Java-based browser from NCSA for
HDF4 and HDF5 – based data.
HDF-EOS plug-in extends HDFView to browse
HDF-EOS data in both versions.
– One reason: EOSView is becoming obsolete
The extension is a plug-in module and does not
change NCSA HDFView functionality. All copy,
save, editing, image display, etc. for HDF4 and 5
objects remains.
Identical look and feel when displaying HDF and
HDF-EOS objects.
Users can access a complete breakdown of HDFEOS objects. e.g. grid information, projection
information, dimensions, datafields, and attributes
are displayed.
31

32. HDFView (Cont.)
• Release 2.2 October 2005
- A separate HDF-EOS plug-in: Users install HDFView
2.2 and integrates HDF-EOS plug-in into it.
- Ability to do "Save as" for HDF-EOS5: modify
individual values in grid or swath data. Delete entire
grids or swaths. Save the result.
- Separate help window for HDF-EOS objects
• Release 2.3 January 2006
- Point Object viewing capability
- Geolocating pixels for swath and grid images
 Future Plans:
- Ability to modify input HDF-EOS2 granules.
- Ability to cut/paste objects, modify/delete attributes.
32
- Ability to create new grids/swaths

33. HDFView showing ASTER L1B swath
33

34. HDF-EOS to GeoTIFF Converter
(HEG)
• A utility that converts EOSDIS data from HDF-EOS
format to common Geographical Information System
(GIS)-compatible formats.
• Two versions available:
– A downloadable desktop version
– Access through NASA archive online storage (Data
Pools)
– Implemented also in prototypes for Stand-alone Data
Pool, OGC (Open Geospatial Consortium) for Web
Services Chaining, and DOWS (Deploy OGC Web
Services) on the DAACs
• Details of internal file structures are transparent to users.
34

35. HEG Functionality
• Subsetting (spatial, field, band)
• Performs conversion to a single-band or multiband GeoTIFF’s
• HDF-EOS Swath to HDF-EOS Grid and Swath
• HDF-EOS Swath or Grid to GeoTIFF
• HDF-EOS Swath or Grid to generic Binary (with
metadata file)
• Stitching (mosaicing) + subsetting + Reprojecting
• Running from Command Line (batch jobs) or with
Java-based GUI
• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,
GEO, SIN)
• Subsampling (Subsample stacks)
35

36. HEG Functionality
• Metadata preservation/creation.
• Currently supports MODIS, MISR, ASTER, AIRS and
AMSR-E products on TERRA and AQUA (>110
products), AURA/HDF-EOS5 support in progress
• Allows file selection from users local storage
• Operable on Sun, SGI, Win, Linux (64-bit soon), MAC
• Integrated into ECS Data Pool
– Reduces the transfer time of HDF-EOS data sets (if
subset is requested)
– Provides the end-user with the exact file required by
their application
• User Interface
– Portable, written in Java.
– Not dependent on COTS (eg. IDL). 36

37. Availability
• Access to libraries and applications:
http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
• Email
–
–
–
–
larry.klein@l-3com.com
Abe_Taaheri-NR@raytheon.com
Cid_Praderas-NR@raytheon.com
Landover_PGSTLKIT@raytheon.com
37

38. Mt. Etna Eruption: MISR L1B Swaths: Stitched
38

39. Mt. Etna: Stitched, Subsetted ASTER L1B
39

40. MODIS L1B: Reprojected, Co-Registered with
ASTER, La Plata Tornado
40

41. La Plata Tornado Damage: ASTER L1B
reprojected/stitched/subsetted
41

42. MLS L2GP-H2O Product (2004, day 243),
Geographic Projection, Full Globe
42

43. MLS L2GP-H2O Product,
PS Projection, Northern Hemisphere
43

44. 2004 Hurricane Francis, MODIS
(MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)
44

45. AST_L1B, 2003 California Fires
Geo. projection, Multi-Band GeoTIFF,
(3 bands: Visibile Near Infrared (VNIR))
….. and Subsetted
Two Swaths Stitched …..
45