Status of HDF-EOS,

Related Software, and Tools
Abe Taaheri, Raytheon IIS
HDF & HDF-EOS Workshp XIII
Riverdale, MD
November 4, 2009
Page 1
TOOLKIT / HDF-EOS Support
•

Maintenance support is funded on EMD to EED
Bridge (EEB) contract through Dec 2010

•

Support will continue with EOSDIS Evolution &
Development (EED) award in 2010

•

Current support (One full-time developer) :








HDF-EOS2 & 5
SDP/MTD Toolkit
HDF-EOS plug-in for HDFView
HDF-EOS2 to GeoTIFF Converter (HEG)
Other Toolkit/HDF-EOS related software
Toolkit/HDF-EOS/HEG helpdesk
Related web sites maintenance

Page 2
ECS Support of HDF-EOS
• Both HDF4 and HDF5 based flavors of HDF-EOS are part
of ECS baseline.

• OS’s: Solaris (9, 10), Linux ( 32 and 64-bit ),
Mac OS X (Power PC and Mac Intel),
Windows XP (MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView)

• Added full Cygwin support in July 2009 release
• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++,
gfortran (Mac Intel)

Page 3
Downloadable Software

• SDP/MTD Toolkit (Science Data Processing Tools)
• HDF-EOS2 (HDF4 based)
• HDF-EOS5 (HDF5 based)
• heconvert (converts HDF4 - based
Grid/Point/Swath structures to HDF-EOS 5
equivalents)
Page 4
Downloadable Applications
• HEG
- GeoTIFF conversion
- Spatial subsetting, Field Subsetting
- reprojection
- stitching, sub-sampling and more ….

• HDFView Plug-in, Java-based browser
Plug-in
(HDF4, HDF5, HDF-EOS 2 and 5 access)
• HE5View* (HDF5 - based browser)
HE5View
• EOSView* (HDF4 - based browser)
EOSView
* Dropped support for OS updates on July 2006
Page 5
Software/Tools in detail

• What are:
TOOLKIT, HDF-EOS ?

Page 6
Toolkit
•

Science Data Production (SDP) Toolkit :
collection of tools used by
- Science software developers who produce
code to process instrument data.
- HDF-EOS users who need to
• Access metadata in HDF-EOS files
• Perform Time/Date conversion
• Access Digital Elevation Model Data
• Access Ancillary Data, such as Digital Chart
of the World database (DCW), Olson World
Ecosystem files, DEM data files, etc.
Page 7
Toolkit (cont.)
•

Tools provided in SDP Toolkit:
-

AA
CBP
CSC
CUC
DEM
EPH
GCT
IO
MEM
MET
PC
SMF
TD

(Ancillary Data Access)
(CELESTIAL Body Position)
(Coordinate System Conversion)
(Constant and Unit Conversions)
(Digital Elevation Model Access)
(Ephemeris Data Access)
(Geo Coordinate Transformation)
(Input Output (File I/O))
(Memory Management)
(Metadata Access)
(Process Control)
(Status Message File (Error/Status))
(Time and Date Conversion)
Page 8
Toolkit (cont.)

•

Tools provided in MTD Toolkit:
- MET
- TD

(Metadata Access)
(Time and Date Conversion)

Page 9
HDF-EOS
• HDF-EOS is an extension to HDF which

standardizes storage of and access to common
Earth Sciences data structures

– By providing a single interface to data
structures common to earth science

– By providing a container for EOS inventory,
archive and product specific metadata

– By establishing a consistent relationship
between geolocation and science data.

Page 10
HDF-EOS 2
• HDF4 - based, storage format for EOS standard
products.

• Used operationally by MODIS, MISR, ASTER,
Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures

Page 11
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF-EOS2
with a different user interface.

– Used operationally by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the
maximum extent possible.
– Supports same data structures (and more)
– Added prefix ‘HE5_’ to HDF-EOS 2 functions

Page 12
Software Releases
Toolkit/HDF-EOS:
• Last release – July 2009
– TOOLKIT 5.2.16
– HDF-EOS 2.16, HDF- EOS5.1.12
with HDF4.2r4, HDF5-1.8.3, Szip2.1

• Next release – June 2010
– TOOLKIT 5.2.17
– HDF-EOS 2.17, HDF-EOS5.1.13
with new releases of HDF4, 5 (5-1.8.X)
Page 13
Software Releases (Cont.)
HEG (HDF-EOS2 to GeoTIFF Converter):
• Last release – August 2008
– Version 2.9
• Next release – November 2009
– Version 2.10

Page 14
Software Releases (Cont.)
HDF-EOS Plug-in for HDFView:
• Last release – September 2008
– Version 2.4

• Next release –
– For HDFView Version 2.5, December 2009

Page 15
Software Releases (Cont.)
HECONVERT (HDF-EOS2 to HDF-EOS5
Converter):
• Early releases – 2001, 2004
– Version ?
• Last release – Dec. 2008
– Version 2.0
• Next release – Not Planned
Page 16
Major Development/Fixes
Last Release (July 2009)

TOOLKIT:
– XML metadata writing capabilty for inventory
metadata in SDP toolkit (MLS)
– ODL fix (WriteLable) for potential core dump
while writing long input pointer metadata
attributes (AIRS)
– Support TOOLKIT for CYGWIN
– pccheck utility fix for problems resulting with
empty space at a line beginning in PCF file
Page 17
Major Development/Fixes (cont.)
HDF-EOS5:
– Support for reading HDF-EOS5 files with netCDF

– Correcting wrong tiling and compression info

in the structure metadata
– Use of unlimited dimension in HDF-EOS5 (Users
Guide clarification)
– Correction of output attribute data types in
attribute inquiry routine
– Fix for writing int8 data correctly to output hdf5
file with C calls. There was no such a
problem with Fortran calls
Page 18
Major Development/Fixes (cont.)
HDF-EOS2:

– Fixed PTdeftimeperiod to return an error
when stop time < start time
– Fix for correct internal compression method
detection for the products that are
compressed internally after the production,
possibly using hrepack (structure metadata
has no record of compression info)
– Numerous fixes for HEConvert (version 2.0 release)
Page 19
Availability
• Access to libraries, applications, Documentation :
TOOLKIT:
http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

HDFView:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/
HDFView_hdfeos_plugin.html

HEG:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
Page 20
Availability

• Questions, Comments, Requests:
 Abe_Taaheri@raytheon.com
 Landover_PGSTLKIT@raytheon.com
 http://www.hdfeos.org/ : HDF-EOS Forum

Page 21
Thank You!

Page 22

Status of HDF-EOS, Related Software, and Tools

  • 1.
    Status of HDF-EOS, RelatedSoftware, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshp XIII Riverdale, MD November 4, 2009 Page 1
  • 2.
    TOOLKIT / HDF-EOSSupport • Maintenance support is funded on EMD to EED Bridge (EEB) contract through Dec 2010 • Support will continue with EOSDIS Evolution & Development (EED) award in 2010 • Current support (One full-time developer) :        HDF-EOS2 & 5 SDP/MTD Toolkit HDF-EOS plug-in for HDFView HDF-EOS2 to GeoTIFF Converter (HEG) Other Toolkit/HDF-EOS related software Toolkit/HDF-EOS/HEG helpdesk Related web sites maintenance Page 2
  • 3.
    ECS Support ofHDF-EOS • Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris (9, 10), Linux ( 32 and 64-bit ), Mac OS X (Power PC and Mac Intel), Windows XP (MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView) • Added full Cygwin support in July 2009 release • Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++, gfortran (Mac Intel) Page 3
  • 4.
    Downloadable Software • SDP/MTDToolkit (Science Data Processing Tools) • HDF-EOS2 (HDF4 based) • HDF-EOS5 (HDF5 based) • heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) Page 4
  • 5.
    Downloadable Applications • HEG -GeoTIFF conversion - Spatial subsetting, Field Subsetting - reprojection - stitching, sub-sampling and more …. • HDFView Plug-in, Java-based browser Plug-in (HDF4, HDF5, HDF-EOS 2 and 5 access) • HE5View* (HDF5 - based browser) HE5View • EOSView* (HDF4 - based browser) EOSView * Dropped support for OS updates on July 2006 Page 5
  • 6.
    Software/Tools in detail •What are: TOOLKIT, HDF-EOS ? Page 6
  • 7.
    Toolkit • Science Data Production(SDP) Toolkit : collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS users who need to • Access metadata in HDF-EOS files • Perform Time/Date conversion • Access Digital Elevation Model Data • Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc. Page 7
  • 8.
    Toolkit (cont.) • Tools providedin SDP Toolkit: - AA CBP CSC CUC DEM EPH GCT IO MEM MET PC SMF TD (Ancillary Data Access) (CELESTIAL Body Position) (Coordinate System Conversion) (Constant and Unit Conversions) (Digital Elevation Model Access) (Ephemeris Data Access) (Geo Coordinate Transformation) (Input Output (File I/O)) (Memory Management) (Metadata Access) (Process Control) (Status Message File (Error/Status)) (Time and Date Conversion) Page 8
  • 9.
    Toolkit (cont.) • Tools providedin MTD Toolkit: - MET - TD (Metadata Access) (Time and Date Conversion) Page 9
  • 10.
    HDF-EOS • HDF-EOS isan extension to HDF which standardizes storage of and access to common Earth Sciences data structures – By providing a single interface to data structures common to earth science – By providing a container for EOS inventory, archive and product specific metadata – By establishing a consistent relationship between geolocation and science data. Page 10
  • 11.
    HDF-EOS 2 • HDF4- based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures Page 11
  • 12.
    HDF-EOS 5 • Basedon HDF5, a complete rewrite of HDF-EOS2 with a different user interface. – Used operationally by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures (and more) – Added prefix ‘HE5_’ to HDF-EOS 2 functions Page 12
  • 13.
    Software Releases Toolkit/HDF-EOS: • Lastrelease – July 2009 – TOOLKIT 5.2.16 – HDF-EOS 2.16, HDF- EOS5.1.12 with HDF4.2r4, HDF5-1.8.3, Szip2.1 • Next release – June 2010 – TOOLKIT 5.2.17 – HDF-EOS 2.17, HDF-EOS5.1.13 with new releases of HDF4, 5 (5-1.8.X) Page 13
  • 14.
    Software Releases (Cont.) HEG(HDF-EOS2 to GeoTIFF Converter): • Last release – August 2008 – Version 2.9 • Next release – November 2009 – Version 2.10 Page 14
  • 15.
    Software Releases (Cont.) HDF-EOSPlug-in for HDFView: • Last release – September 2008 – Version 2.4 • Next release – – For HDFView Version 2.5, December 2009 Page 15
  • 16.
    Software Releases (Cont.) HECONVERT(HDF-EOS2 to HDF-EOS5 Converter): • Early releases – 2001, 2004 – Version ? • Last release – Dec. 2008 – Version 2.0 • Next release – Not Planned Page 16
  • 17.
    Major Development/Fixes Last Release(July 2009) TOOLKIT: – XML metadata writing capabilty for inventory metadata in SDP toolkit (MLS) – ODL fix (WriteLable) for potential core dump while writing long input pointer metadata attributes (AIRS) – Support TOOLKIT for CYGWIN – pccheck utility fix for problems resulting with empty space at a line beginning in PCF file Page 17
  • 18.
    Major Development/Fixes (cont.) HDF-EOS5: –Support for reading HDF-EOS5 files with netCDF – Correcting wrong tiling and compression info in the structure metadata – Use of unlimited dimension in HDF-EOS5 (Users Guide clarification) – Correction of output attribute data types in attribute inquiry routine – Fix for writing int8 data correctly to output hdf5 file with C calls. There was no such a problem with Fortran calls Page 18
  • 19.
    Major Development/Fixes (cont.) HDF-EOS2: –Fixed PTdeftimeperiod to return an error when stop time < start time – Fix for correct internal compression method detection for the products that are compressed internally after the production, possibly using hrepack (structure metadata has no record of compression info) – Numerous fixes for HEConvert (version 2.0 release) Page 19
  • 20.
    Availability • Access tolibraries, applications, Documentation : TOOLKIT: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/ HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html Page 20
  • 21.
    Availability • Questions, Comments,Requests:  Abe_Taaheri@raytheon.com  Landover_PGSTLKIT@raytheon.com  http://www.hdfeos.org/ : HDF-EOS Forum Page 21
  • 22.