3. Overview
•
The HDF has been selected by the EOSDIS
Project as the format of choice for standard
product distribution of EOS data
•
Under NASA contracts (ECS, EMD, EEB, EED)
we developed HDF-EOS, and performed other
development and maintenance activities
Page 3
4. Overview
– Libraries
HDF-EOS 2 & 5
SDP & MTD Toolkits
– Tools
HDF-EOS2 to GeoTIFF Converter (HEG)
HDF-EOS plug-in for THG HDFView
HDF-EOS2 to HDF-EOS5 conversion
– User support
Helpdesk for Toolkit/HDF-EOS/HEG
Page 4
6. Why HDF?
• HDF was selected for EOS data since
– Many of the HDF defined datatypes map well
to EOS datatypes
Raster images, multi-dimensional arrays,
and text blocks
– HDF provides efficient data management
Storage & performance
Page 6
7. HDF-EOS Datatypes
• Some other EOS datatypes do not map directly to
HDF datatypes, particularly
̶ geolocated datatypes
Therefore, some additions to traditional HDF are
required to fully support these datatypes.
Page 7
8. HDF-EOS Datatypes
To bridge the gap between the needs of
EOS data products and the capabilities of
HDF, 4 new EOS specific datatypes were
introduced:
Point, Swath, and Grid
Zonal Average (hdf-eos5)
Page 8
9. HDF-EOS Datatypes
• Point : Unorginzed data (spatial or temporal)
that has associated geolocation
information
• Swath: Time-ordered data such as satellite
swaths (time-ordered series of scanlines),
or
profilers (time-ordered series of profiles).
• Grid: Data stored in a rectilinear array based on
a well defined and explicitly
supported
projection.
• ZA:
Swath like datatype without geolocation
mapping
Page 9
10. HDF4 File Contents – User View
Objects & Relationships
Object Data
User Metadata
Page 10
11. HDF4 File Contents – Format View
variable
name = variable_name
rank
type
storagetype
1
Vgroup
name = variable_name
class = Var0.0
1
1
Object Data
1
1
1
0...1
1
SD
SDD
1
0...1
data
0…*
byte order,
chunked storage,
compression, …
attribute
name = attribute_name
HDF4 Mapping Project Update - Ruth Aydt
1
1
0...1
NT
1
1
1
1
1
1
NDG
0…*
Vdata
name = attribute_name
class = Attr0.0
Page 11
12. HDF/HDF-EOS are Complicated!
• HDF is not efficient for Users since they need to
learn
- how to use HDF & HDF-EOS libraries
- how to process the data correctly
• Licenses for commercial tools are expensive
The HDF Group & Raytheon have developed
some tools to help users
Page 12
13. HDF & HDF-EOS Tools
For most users HDF readers or NetCDF/CF
conversion applications are enough
- hdp, h5dump, eos2dump(for lat/lon), HDFView
- h4tonccf, h4cf, OpeNDAP Handlers
But some need more processing and tools
appropriate for EOS data
• HDF-EOS to GeoTIFF Conversion Tool (HEG)
• HDF-EOS plug-in for HDFView
Page 13
15. HEG
Name says HEG is:
A utility to convert EOSDIS data from
HDF-EOS2 to GeoTIFF format.
But in practice:
It can do more than just conversion to
GeoTIFF
Page 15
16. HEG Functionality
• Conversion
– HDF-EOS (Swath or Grid ) to a single-band or
multi-band GeoTIFF’s
– HDF-EOS Swath to HDF-EOS Grid --> GeoTIFF
OR
– ASTER Swath is written directly into GeoTIFF
– HDF-EOS Swath or Grid to generic Binary (with
metadata file)
Page 16
17. HEG Functionality
• Subsetting
– spatial: using Swath/Grid corner lat/lon or pixel
numbers
– field: Grid, Swath, and Field selection
– band: Selection of data layers in 3-D and 4-D
datasets ( 3-D to 6-D MISR SOM datasets )
Page 17
25. HEG Data/Platform Support
• Currently supports
MODIS, MISR, ASTER, AIRS, AMSR-E
products on TERRA and AQUA
– more than 210 products
• Operable on Sun, Windows, Linux, MAC
Page 25
26. HEG Versions
Versions
Available:
Stand-alone:
A downloadable desktop version
with a User Interface
̶ Portable, written in Java.
̶ Not dependent on COTS (eg. IDL).
Page 26
27. HEG Versions
Integrated into ECS Data Pool:
Access through NASA archive online
storage
̶ Reduces the transfer time of HDF-EOS data
sets (if subset is requested)
̶ Provides the end-user with the exact file
required by their application
Page 27
36. HDFView with HDF-EOS Plug-in
–Extends HDFView
to browse HDF-EOS data in both versions (2 & 5).
–Is a plug-in module
does not change HDFView functionality
–Identical look and feel
when displaying HDF and HDF-EOS objects
–Access to complete breakdown
of HDF-EOS objects.
e.g. grid info, projection info, dims, datafields, and
attributes
Page 36
Look a bit more closely at the types of information that are in an HDF4 file… all of these are important.The object data typically takes up most of the space in an HDF4 file and is the most difficult to represent in Text format.
If we look “under the hood” things are more complicated than they appear from the HDFview display…. The HDF4 library typically hides the complexity.
The car’s engine is kind of like the object data in the HDF4 file… challenging to represent and explain.
Bits of data for a single object can be spread throughout the file, compressed, etc.
Can we represent this in a way that something other than the HDF4 library can find the right bits & reconstruct the data?
