0
HDF-EOS Development
Current Status and Schedule
Larry Klein, Shen Zhao, Abe Taaheri and
Ray Milburn
L-3 Communications
Gov...
What is HDF-EOS ?
• HDF (Hierarchical Data Format) is a disk-based
format chosen by NASA as the standard for the EOS
progr...
HDF-EOS Grid Structure
• Usage - Data which is organized
by regular geographic spacing,
specified by projection parameters...
HDF-EOS Swath Structure
• Data which is organized by time, or other track
parameter.
• Spacing can be irregular.
• Structu...
Swath Data Example

5
HDF-EOS Point Structure
• Data is specified temporally and/or spatially,
but with no particular organization
• Structure
–...
Zonal Average Structure
• Generalized array structure with no
geolocation linkage
• Data can be organized by time or track...
HDF-EOS 2
• HDF4 - based, storage format for EOS standard
products.
• Used operationally by MODIS, MISR, ASTER,
Landsat, A...
HDF-EOS 5
• Based on HDF5, a complete rewrite of
HDF4 with a different user interface.
– First released in 2000.
– Format ...
HDF-EOS Functionality
• Basic File I/O
• Fill Values
• Compression (gzip,
szip (Oct. 03))
• Chunking/Tiling
• Swath Interf...
HDF-EOS 5
• HDF-EOS 5 is used by EOS Aura producers
and users. Launch Jan. 04.
• Assume that HDF-EOS 2 users will convert
...
Development Since December, 02
• Added the zonal average interface ZAapi.c.
• Added two new functions HE5_SWinqdfldalias()...
ECS Support of HDF-EOS
• Both flavors of HDF-EOS are part of ECS
baseline.
• OS’s: Solaris, Irix, HP, AIX, DEC, Windows
NT...
Applications
•
•
•
•

HE5View (HDF5 - based browser)
EOSView (HDF4 - based browser)
Java EOS Browser (HDF-EOS 2 and 5 acce...
HDFView
• HDFView: A Java-based browse tool from NCSA
for HDF4 and HDF5 – based data.
• We are currently extending HDFView...
HDFView (Cont.)
• The look and feel of HDFView will be identical
when displaying HDF and HDF-EOS objects.
• Users will acc...
HDF-EOS to GeoTIFF
Converter(HEG)
• A utility that converts EOSDIS data from HDFEOS format to common Geographical Informat...
HEG Functionality
• Performs conversion to a single-band or
multi-band GeoTIFF’s
• HDF-EOS Swath to HDF-EOS Grid
• HDF-EOS...
HEG Functionality
• Reprojection (USGS, MODIS Integerized
Sinusoidal, EASEGrid)
• Subsampling (Subsample stacks)
• Boundar...
HEG (cont.)
•
•
•
•

Java-based GUI
Command line interface
Operable on Sun, SGI, Windows, Linux
Integrated into ECS Data P...
OpenGIS Interface
• Ionic’s Open GIS Consortium (OGC) Open
Web Service (OWS), Web Coverage
Service (WCS), Version 3.1.
• U...
Availability
• Access to libraries and applications:
http://newsroom.gsfc.nasa.gov/sdptoolkit/to
olkit.html
• Email
–
–
–
...
Top Level of HDF-EOS 5 File
Root -- “/”
HDFEOS
INFORMATION

HDFEOS

STRUC. METADATA
ADDITIONAL
ARRAY

SWATH

GRID

POINT

...
Swath Structure
SwathName

Global Attribute
<SwathName>:<AttrName>

Group Attribute
<DataFields>:<AttrName>
Local Attribut...
Upcoming SlideShare
Loading in...5
×

HDF-EOS Development: Current Status and Tools

53

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
53
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "HDF-EOS Development: Current Status and Tools"

  1. 1. HDF-EOS Development Current Status and Schedule Larry Klein, Shen Zhao, Abe Taaheri and Ray Milburn L-3 Communications Government Services, Inc. September 24, 2003 1
  2. 2. What is HDF-EOS ? • HDF (Hierarchical Data Format) is a disk-based format chosen by NASA as the standard for the EOS program • HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures – By providing a single interface to data structures common to earth science – By providing a container for EOS inventory, archive and product specific metadata – By establishing a consistent relationship between geolocation and science data. 2
  3. 3. HDF-EOS Grid Structure • Usage - Data which is organized by regular geographic spacing, specified by projection parameters. • Structure – Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) – Geolocation information contained in projection formula, coupled by structural metadata. – Any number of Grid structures per file allowed. 3
  4. 4. HDF-EOS Swath Structure • Data which is organized by time, or other track parameter. • Spacing can be irregular. • Structure – Geolocation information stored explicitly in Geolocation Field (2-D array) – Data stored in 2-D or 3-D arrays – Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural metadata 4
  5. 5. Swath Data Example 5
  6. 6. HDF-EOS Point Structure • Data is specified temporally and/or spatially, but with no particular organization • Structure – Tables used to store science data at a particular Lat/Long/Height – Up to eight levels of data allowed. Structural metadata specifies relationship between levels. 6
  7. 7. Zonal Average Structure • Generalized array structure with no geolocation linkage • Data can be organized by time or track parameter • Data spacing can be irregular • Structure – Data stored in multidimensional arrays – Time stored in 1-D or 2-D array 7
  8. 8. HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • Current archive holdings: ~1.5 PB, 30M granules • HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future. 8
  9. 9. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different user interface. – First released in 2000. – Format to be used by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures – Added prefix ‘HE5_’ to HDF-EOS 2 functions. – Doesn’t preclude HDF5 functionality. 9
  10. 10. HDF-EOS Functionality • Basic File I/O • Fill Values • Compression (gzip, szip (Oct. 03)) • Chunking/Tiling • Swath Interface • Grid Interface • Point Interface • Profile Interface • Global (File), Group & Local Attributes • External Data Files • Subsetting • Threadsafe Version • Generalized Array Interface (Zonal Avg. (HDF5)) 10
  11. 11. HDF-EOS 5 • HDF-EOS 5 is used by EOS Aura producers and users. Launch Jan. 04. • Assume that HDF-EOS 2 users will convert when PI’s determine that the time is right. 11
  12. 12. Development Since December, 02 • Added the zonal average interface ZAapi.c. • Added two new functions HE5_SWinqdfldalias() and HE5_SWinqgfldalias() to retrieve information about data/geolocation fields and aliases in Swath. • Added a new function HE5_EHinqglbdatatype() to retrieve the data type information from a Global File attribute. • Szip compression (next release). • Next release - mid-Oct. linked with HDF5-1.6.1. 12
  13. 13. ECS Support of HDF-EOS • Both flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris, Irix, HP, AIX, DEC, Windows NT/98, Linux (64-bit soon), Mac OS X (soon) • Compilers: FORTRAN 77/90, C, C++. • Size limit per granule is 2 GBytes 13
  14. 14. Applications • • • • HE5View (HDF5 - based browser) EOSView (HDF4 - based browser) Java EOS Browser (HDF-EOS 2 and 5 access) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) – Compliments HDF4 -> HDF5 conversion tool – Not tested on all products 14
  15. 15. HDFView • HDFView: A Java-based browse tool from NCSA for HDF4 and HDF5 – based data. • We are currently extending HDFView to browse HDF-EOS data in both versions. – One reason: EOSView is becoming obsolete • The extension is a plug-in module and will not change functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects will remain. 15
  16. 16. HDFView (Cont.) • The look and feel of HDFView will be identical when displaying HDF and HDF-EOS objects. • Users will access a complete breakdown of HDFEOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes will be displayed. • A beta release is scheduled for end of Oct-03. 16
  17. 17. HDF-EOS to GeoTIFF Converter(HEG) • A utility that converts EOSDIS data from HDFEOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: – A downloadable desktop version – Access through NASA archive online storage (Data Pools) • Details of internal file structures are transparent to users. 17
  18. 18. HEG Functionality • Performs conversion to a single-band or multi-band GeoTIFF’s • HDF-EOS Swath to HDF-EOS Grid • HDF-EOS Swath or Grid to GeoTIFF • HDF-EOS Swath or Grid to generic binary (with metadata file) • Subsetting (spatial, temporal, parameter) • Stitching (mosaicing) 18
  19. 19. HEG Functionality • Reprojection (USGS, MODIS Integerized Sinusoidal, EASEGrid) • Subsampling (Subsample stacks) • Boundary smoothing • Metadata preservation/creation. • Currently supports MODIS, MISR, and ASTER products (>70 products) (AIRS proposed) • Allows file selection from users local storage 19
  20. 20. HEG (cont.) • • • • Java-based GUI Command line interface Operable on Sun, SGI, Windows, Linux Integrated into ECS Data Pool – Reduces the transfer time of HDF-EOS data sets (if subset is requested) – Provides the end-user with the exact file required by their application • User Interface – Portable, written in Java. – Initially operable on Sun, SGI, Linux. – Not dependent on COTS (eg. IDL). 20
  21. 21. OpenGIS Interface • Ionic’s Open GIS Consortium (OGC) Open Web Service (OWS), Web Coverage Service (WCS), Version 3.1. • Uses HEG • Will expose EOS products to tools on global Open GIS network. • Integration into the ECS Data Pool is planned for 2004 21
  22. 22. Availability • Access to libraries and applications: http://newsroom.gsfc.nasa.gov/sdptoolkit/to olkit.html • Email – – – – larry.klein@l-3com.com ray.milburn@l-3com.com shen.zhao@l-3com.com abe.taaheri@l-3com.com 22
  23. 23. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL ARRAY SWATH GRID POINT Global (File) Attributes . 23
  24. 24. Swath Structure SwathName Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales Group CoLatitude Shaded Objects are implemented in a fixed way so the user doesn’t have direct access via the interface Attribute Data Set 24
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×