Hdf eos status-workshp_xi_nov_2007


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Hdf eos status-workshp_xi_nov_2007

  1. 1. Current Status of HDF-EOS, Maintenance, Developments, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshop XI November 2007 Page 1
  2. 2. TOOLKIT / HDF-EOS Support • • Maintenance support is funded on EMD Evolution through May 2008 One year extension until May 2009 • Current staff - One full-time developer for maintaining:  SDP/MTD Toolkit  HDF-EOS2 & 5  HDF-EOS plug-in for HDFView  HDF-EOS2 to GeoTIFF Converter (HEG)  Other Toolkit/HDF-EOS related software  Toolkit/HDF-EOS/HEG helpdesk - One half-time developer for HEG maintenance Page 2
  3. 3. ECS Support of HDF-EOS • Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris (9, 10), Irix6.5 (64, n32), Windows XP, Linux (including 64-bit Opteron and Itanium), Mac OS X (MTD Toolkit, HDF-EOS2 & 5) (Dropped support for Solaris 8, HP, HP11, Dec, IBM for next release) • adding Mac Intel support for 2007 release and full Cygwin support for 2008 • Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++ Page 3
  4. 4. Downloadable Software and Applications • SDP/MTD Toolkit (Science Data Processing Tools) • HDF-EOS2 (HDF4 based) • HDF-EOS5 (HDF5 based) • HE5View* (HDF5 - based browser) • EOSView* (HDF4 - based browser) • HDFView, Java-based browser (HDF4, HDF5, HDFEOS 2 and 5 access) • heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) • HEG (GeoTIFF conversion, Spatial subsetting, Field Subsetting, reprojection, stitching, sub-sampling, etc.) * Dropped support for OS updates on July 2006 Page 4
  5. 5. Software Releases Toolkit/HDF-EOS: • Last release – March 2006 TOOLKIT 5.2.14, HDF-EOS 2.14, HDF-EOS5.1.10 with HDF4.2r1, HDF5-1.6.5-post5, Szip2.0 • Next release – November 2007 TOOLKIT 5.2.15, HDF-EOS 2.15, HDF-EOS5.1.11 with new releases of HDF4, HDF5, and SZIP HDF-EOS Plug-in for HDFView: • Last release – May 2006 – Version 2.3 • Next release – Dec. 2007 – HDFView2.4 was released October 31 by The HDF Group Page 5
  6. 6. Software Releases (Cont.) HEG (HDF-EOS2 to GeoTIFF Converter): • Last release – July 2007 – Version 2.8 • Next release – May 2008 – Version 2.9 Page 6
  7. 7. SDP Toolkit • Science Data Production (SDP) Toolkit is a collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS users who need to • Access metadata in HDF-EOS files • Perform Time/Date conversion • Access Digital Elevation Model Data • Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc. Page 7
  8. 8. SDP Toolkit (cont.) • Tools provided in (SDP, MTD) Toolkit: - AA (Ancillary Data Access) - CBP (CELESTIAL Body Position) - CSC (Coordinate System Conversion) - CUC (Constant and Unit Conversions) - DEM (Digital Elevation Model Access) - EPH (Ephemeris Data Access) - GCT (Geo Coordinate Transformation) - IO (Input Output (File I/O)) - MEM (Memory Management) - MET (Metadata Access) - PC (Process Control) - SMF (Status Message File (Error/Status)) - TD (Time and Date Conversion) Page 8
  9. 9. What is HDF-EOS ? • HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures –By providing a single interface to data structures common to earth science –By providing a container for EOS inventory, archive and product specific metadata –By establishing a consistent relationship between geolocation and science data. Page 9
  10. 10. HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future. Page 10
  11. 11. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface. –First released in 2000. –Format used by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. –Supports same data structures –Added prefix ‘HE5_’ to HDF-EOS 2 functions. Page 11
  12. 12. HDFView • • • • • • HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data. HDF-EOS plug-in extends HDFView to browse HDFEOS data in both versions. The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. Identical look and feel when displaying HDF and HDF-EOS objects. Users can access a complete breakdown of HDFEOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed. Current Support: SUN, SGI, LINUX, Mac, WINDOWS platforms Page 12
  13. 13. Swath, Grid, and Point Objects Page 13
  14. 14. AMSR-E and OMI Point objects Page 14
  15. 15. AMSR-E Swath New Features in Next Release Version 2.4 Page 15
  16. 16. HDF-EOS to GeoTIFF Converter (HEG) HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: – A downloadable desktop version – Access through NASA archive online storage (Data Pools) – Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs Page 16
  17. 17. HEG Functionality • Conversion: – HDF-EOS (Swath or Grid ) to a single-band or multi- • • • • band GeoTIFF’s – HDF-EOS Swath to HDF-EOS Grid and Swath – HDF-EOS Swath or Grid to generic Binary (with metadata file) Subsetting (spatial, field, band) Stitching (mosaicing) + subsetting + Reprojecting Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers) Subsampling (Subsample stacks) • Running from Command Line (batch jobs) or with Java-based GUI Page 17
  18. 18. HEG Functionality • Metadata preservation/creation. • Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), • Allows file selection from users local storage • Operable on Sun, SGI, Win, Linux, MAC • Integrated into ECS Data Pool – Reduces the transfer time of HDF-EOS data sets (if subset is requested) – Provides the end-user with the exact file required by their application • User Interface – Portable, written in Java. – Not dependent on COTS (eg. IDL). Page 18
  19. 19. Mt. Etna Eruption: MISR L1B Stitched Blocks Page 19
  20. 20. 2004 Hurricane Francis, MODIS MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5 Page 20
  21. 21. AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF, (3 bands: Visible Near Infrared (VNIR)) ….. and Subsetted Two Swaths Stitched ….. Page 21
  22. 22. Major Development/Fixes Next Release: November 2007  SZIP Insatallation: Default decoder, Choice for encoder  Choice for configuring HDF4 with/without SZIP  Choice for disabling HDF4’s internal netCDF  Choice for installing Toolkit/HDF-EOS in 32 or 64-bit modes in 64-bit Linux platforms  Support for new releases of HDF4/HDF5/SZIP (HDF4.2r2, HDF5-1.6.6, SZIP-2.1)  Installation fix for a problem caused by limits.h in some Linux platforms  Auto Configuration capability (like HDF) added for both HDF-EOS2 and HDF-EOS5  Support for Mac Intel platform. Drop of support for Sun5.8, Dec, HP, HP 11, IBM  Fixes for Lat/Lon to pixel conversions in DEM tools for Linux to get results similar to those in Unix platforms Page 22
  23. 23. Major Development/Fixes (cont.) HDF-EOS Plug-in for HDFView: Current Support: SUN, SGI, LINUX, WINDOWS, and MAC platforms Last Release: Version 2.3 – Display Point Objects – Display pixel's lat/lon for swath and grid images – Scientific notation in Table data Next Release: Version 2.4 – Auto installation of plug-in into HDFView running selfextracting archives. – Display image for selected pixels in the viewed tables – Display Lat/Lon for a selected pixel in the data field tables – Mac OSX fixes for one-button mice – Better display of Plug-in User’s Guide – New icons for activating Lat/Lon display for Swath & Grid Page 23
  24. 24. Major Development/Fixes (cont.) HEG: Last Release (Version 2.8) – Added Support for: MOD02SSH, MYD02SSH, MOD14A1.v005, MYD14A1.005 MOD29P1D.V005, MOD08_M3.v005, MOD11_L2.v005 AMSR-E Snow and Ice, AMSR-E Ocean – MISR Bug fixes & enhancements:  Added “No-reprojection” for ouput in SOM projection  Added “Nearest Neighbour” for sampling type  Fixed problem with missing metadata for HDF-EOS outputs – Added Albers Projection – New "Subsetter" functionality for Parameters (aka Fields). This results in: • Output to HDF-EOS SWATH from input HDF-EOS SWATH • No alteration of input data because of projection operation.24 Page
  25. 25. HDF-EOS/Toolkit Future Plans: • • • • Auto Configuration for TOOLKIT Full Support for Cygwin platform Support for HDF5-1.8 Performance improvement, if possible, for reading HDF format ephemeris/attitude files in Toolkit • Performance improvement in HDF-EOS5 by redesigning some functions • More functionalities in HDF-EOS Plug-in for HDFView: Format conversion: HDF-EOS2 -> HDF-EOS5 Improved Help Window Ability to modify input HDF-EOS2 granules. Ability to cut/paste objects, modify/delete attributes. Ability to create new grids/swaths Page 25
  26. 26. Availability • Access to libraries, applications, Documentation: TOOLKIT: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/ HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html • Email – Abe_Taaheri@raytheon.com – Landover_PGSTLKIT@raytheon.com Page 26
  27. 27. Page 27
  28. 28. Major Development/Fixes Last Release HDF-EOS5:  Added a routine to return list of aliases for a field  Used HDF5’s H5T_C_S1 datatype for string type  Added Fortran wrapper for SWwritedatameta() routine  Replaced symbol “H5E_NONE_MAJOR” with other symbols for better description of errors  Ported to Solaris 9 and Solaris 10  Fixed POINT object for: linkage between levels - updating levels writing Forward and Backward Linkage data char field inquiry Page 28
  29. 29. Major Development/Fixes (cont.) Last Release  Fixed subsetting problem in HDF-EOS5 Grid (correction to conversion of DMS degree to Decimal degree).  Set LD_LIBRARY_PATH for SZIP/HDF5 shared libraries in installation and environment setting scripts TOOLKIT/HDF-EOS2:  Ported to Solaris 9 and Solaris 10  Gdapi.c changes for adding more significant digits after the decimal point to projection parameters if needed.  chkeph utility fixes for reading binary ephemeris files (files with different endiannes than the machine)  SGI 7.4.2 compiler support  orbsim fixes for Linux to create files similar to those in UNIX platforms (Random number generation fixes) Page 29