PRACTICAL METHODS FOR
MAKING HDF DATA USABLE IN
NETCDF-BASED TOOLS
Chris Lynnes and Fan Fang
Goddard Earth Sciences
Data a...
Panoply
IDV

GrADS

Ferret

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

2
2
"And then a
miracle occurs..."

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

Unidata IDV

3

3
IDV with HDF-EOS5 “self-describing format”

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

4
4
IDV with HDF-EOS5 “self-describing format”

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

4
4
Panoply w/HDF5

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

5
5
Ferret and GrADS with HDF-EOS5

*Can be opened if user defines dimensions and uses xdfopen
HDF Workshop
Thursday, November...
Solution #1: Convert to netCDF
• On-the-fly conversion to netCDF-3
• Custom C++ code
• Acquiring the URL executes the conv...
Panoply w/converted AIRS L3

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

8
8
IDV w/Converted AIRS L2
Surface Air Temperature

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

9
9
But...
• Custom C++ code is expensive to maintain
• Z dimension scale values are hard to find in AIRS
data
–
–
–
–

vdatas...
Solution #2: Lats4D Conversion
• Essentially a GrADS script from Arlindo DaSilva, NASA/GSFC/
GMAO
• Same on-the-fly mechan...
TRMM Monthly (May 2009) in Panoply
Lats4D Conversion + Subset

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

12
12
Solution #3: OGC Web Coverage Service
• OGC WCS with netCDF/CF-1 profile
–
–
–
–

Supports subsetting
But: “prefers” inter...
AIRS L3 via WCS in Panoply

wget -O airs_wcs.nc 'http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsAIRSL3?VERSION=1.0.0&SERVICE=...
Solution #4: OPeNDAP
• HDF handlers
– Released HDF5 version supports L3 grid
– New HDF4 and HDF5 version supports
• HDF4
•...
OPeNDAP Usability Results
IDV Panoply Ferret GrADS
✓

Comments
exc. AIRX3C2M

✓

latest version only

AIRS L3

✓

✓

MODIS...
OMI L3 NearUV AAOD via OPeNDAP
in Panoply

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

17
17
(but watch out for fast rendering in IDV)

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

18
18
IDV w/fast rendering off

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

19
19
Panoply w/MODIS L2 via new OPeNDAP
HDF4 handler

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

20
20
Panoply w/AIRS L2 via new OPeNDAP
HDF4 Handler

• Cool..but who projected the data and how?
HDF Workshop
Thursday, Novembe...
Solution #5: GrADS Data Server
• DAP protocol, but data read by GrADS
– Sort of like an OPeNDAP with a GrADS handler
– If ...
MERRA via GDS in Panoply

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

23
23
Solution #6?: netcdf library
MERRA HDF Subset from Lats4D in Panoply

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009
...
netcdf libraries
• netcdf-java library can read some HDF files
– But not HDF-EOS...
• Tried adding lat/long dimscales to a...
Panoply, using netcdf-java
w/ MLS L3, HDF-EOS5

HDF Workshop
Thursday, November 5, 2009

5 Nov 2009

26
26
Can we make netcdf libraries read
HDF-EOS?
• Might require joint effort between UCAR, THG
and tool developers to develop a...
Trade Space

OPeNDAP

Cons
Costly
Data size expansion
L3/L4 grid only
Slow
Interpolation
Costly
Reprojects L2 swath

netcd...
Future of HDF-netCDF Tool
Interoperability: Key Challenges
• Level 1b data (calibrated radiance)
– How should we fit the c...
Upcoming SlideShare
Loading in …5
×

Practical Methods for Making HDF Data Usable by NetCDF-based Tools

568 views

Published on

A number of powerful NetCDF-based tools exist for visualizing and analyzing remote sensing data, such as IDV, McIDAS-V, Panoply, Ferret and GrADS. As a result, the GES DISC has been deploying a number of different strategies to make its native HDF, HDF-EOS2 and HDF-EOS5 data available in usable forms to these NetCDF tools. The most elaborate of these is an online, on-the-fly Web Service to convert HDF and HDF-EOS2 data into NetCDF-CF1. The end result is highly effective in making the data usable, even Level 2 swath data, which can be visualized in IDV. Unfortunately, the code is relatively convoluted and inflexible due to the difficulty of locating values in the HDF input file to describe the vertical dimension in CF1. (This could be made easier if HDF dimscales could be written through the HDF-EOS2 interface.) A similar approach has been to serve certain datasets through a Web Coverage Server developed by George Mason University, which is subject to many of the same issues. A third approach has been to use OPeNDAP to make data available to NetCDF-based tools. The HDF5 handler has proven to be effective for Level 3 gridded data in this respect at very low cost. These tools can access remote data directly through OPeNDAP, or the HDF5 data can be downloaded as a local NetCDF file. However, this is limited to HDF5 and does not work for Level 2 swath data. Recently, we have begun deploying a fourth approach, the Lats4D script running within GrADS. Though slower than our custom C++ converter, it supports (in principle) conversion of any Level 3 data that can be read in through GrADS, allowing for dimension definition through GrADS control files. Experimenting with Lats4D also led to the serendipitous discovery that NetCDF-based tools based on the netcdf-java library (e.g., IDV and Panoply) can actually read certain HDF files. If this capability could be generalized, the expense of netCDF conversion could be eliminated. Alas, the hierarchies written by HDF-EOS block this capability; we welcome ideas and contributions from the HDF and NetCDF communities to devising a solution to this impasse.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
568
On SlideShare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Practical Methods for Making HDF Data Usable by NetCDF-based Tools

  1. 1. PRACTICAL METHODS FOR MAKING HDF DATA USABLE IN NETCDF-BASED TOOLS Chris Lynnes and Fan Fang Goddard Earth Sciences Data and Information Services Center HDF Workshop Thursday, November 5, 2009 5 Nov 2009 1
  2. 2. Panoply IDV GrADS Ferret HDF Workshop Thursday, November 5, 2009 5 Nov 2009 2 2
  3. 3. "And then a miracle occurs..." HDF Workshop Thursday, November 5, 2009 5 Nov 2009 Unidata IDV 3 3
  4. 4. IDV with HDF-EOS5 “self-describing format” HDF Workshop Thursday, November 5, 2009 5 Nov 2009 4 4
  5. 5. IDV with HDF-EOS5 “self-describing format” HDF Workshop Thursday, November 5, 2009 5 Nov 2009 4 4
  6. 6. Panoply w/HDF5 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 5 5
  7. 7. Ferret and GrADS with HDF-EOS5 *Can be opened if user defines dimensions and uses xdfopen HDF Workshop Thursday, November 5, 2009 5 Nov 2009 6 6
  8. 8. Solution #1: Convert to netCDF • On-the-fly conversion to netCDF-3 • Custom C++ code • Acquiring the URL executes the conversion and downloads the file HDF Workshop Thursday, November 5, 2009 5 Nov 2009 7 7
  9. 9. Panoply w/converted AIRS L3 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 8 8
  10. 10. IDV w/Converted AIRS L2 Surface Air Temperature HDF Workshop Thursday, November 5, 2009 5 Nov 2009 9 9
  11. 11. But... • Custom C++ code is expensive to maintain • Z dimension scale values are hard to find in AIRS data – – – – vdatas w/non-standard names external documents therefore: hard-coded in C++ class if only we could attach standard Z dimscales to the SDS • New class for each dataset HDF Workshop Thursday, November 5, 2009 5 Nov 2009 10 10
  12. 12. Solution #2: Lats4D Conversion • Essentially a GrADS script from Arlindo DaSilva, NASA/GSFC/ GMAO • Same on-the-fly mechanism as NetCDF converter • If you can read it into GrADS, you can convert it (usually) – Dimensions can be defined in GrADS control file – Limited to L3 and L4 gridded products – Handles both HDF-EOS and HDF • Also supports subsetting HDF Workshop Thursday, November 5, 2009 5 Nov 2009 11 11
  13. 13. TRMM Monthly (May 2009) in Panoply Lats4D Conversion + Subset HDF Workshop Thursday, November 5, 2009 5 Nov 2009 12 12
  14. 14. Solution #3: OGC Web Coverage Service • OGC WCS with netCDF/CF-1 profile – – – – Supports subsetting But: “prefers” interpolation to user’s resolution Requests are difficult for users to submit Works better for machine-to-machine interfaces • DataFed • DLR – Using GMU-developed WCS Server HDF Workshop Thursday, November 5, 2009 5 Nov 2009 13 13
  15. 15. AIRS L3 via WCS in Panoply wget -O airs_wcs.nc 'http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsAIRSL3?VERSION=1.0.0&SERVICE=WCS &REQUEST=getCoverage&FORMAT=NetCDF&COVERAGE=AIRX3STD:ascending:TotCldLiqH2O_A &BBOX=-179.5,-89.5,179.5,89.5&TIME=2007-04-07/2007-04-07/P1D&CRS=CRS:84&RESX=1.0&RESY=1.0' HDF Workshop Thursday, November 5, 2009 5 Nov 2009 14 14
  16. 16. Solution #4: OPeNDAP • HDF handlers – Released HDF5 version supports L3 grid – New HDF4 and HDF5 version supports • HDF4 • L2 swath HDF Workshop Thursday, November 5, 2009 5 Nov 2009 15 15
  17. 17. OPeNDAP Usability Results IDV Panoply Ferret GrADS ✓ Comments exc. AIRX3C2M ✓ latest version only AIRS L3 ✓ ✓ MODIS L3 ✓ ✓ OMI L3 ✓ ✓ ✓ ✓ TOMS L3 ✓ ✓ ✓ AIRS L2 ✓ ✓ ✓ ✓ x ✓ x x x ✓ x HIRDLS L2 MLS L2 MODIS L2 ✓ OMI L2 ✓ HDF Workshop Thursday, November 5, 2009 ✓ ✓ 5 Nov 2009 so far: MOD04, MOD07 exc. OMIBRO, OMHCO, OMOCLO, OMUVB 16 16
  18. 18. OMI L3 NearUV AAOD via OPeNDAP in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 17 17
  19. 19. (but watch out for fast rendering in IDV) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 18 18
  20. 20. IDV w/fast rendering off HDF Workshop Thursday, November 5, 2009 5 Nov 2009 19 19
  21. 21. Panoply w/MODIS L2 via new OPeNDAP HDF4 handler HDF Workshop Thursday, November 5, 2009 5 Nov 2009 20 20
  22. 22. Panoply w/AIRS L2 via new OPeNDAP HDF4 Handler • Cool..but who projected the data and how? HDF Workshop Thursday, November 5, 2009 5 Nov 2009 21 21
  23. 23. Solution #5: GrADS Data Server • DAP protocol, but data read by GrADS – Sort of like an OPeNDAP with a GrADS handler – If you can read it in in GrADS... • Includes time aggregation – (Not always a good thing) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 22 22
  24. 24. MERRA via GDS in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 23 23
  25. 25. Solution #6?: netcdf library MERRA HDF Subset from Lats4D in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 24 24
  26. 26. netcdf libraries • netcdf-java library can read some HDF files – But not HDF-EOS... • Tried adding lat/long dimscales to an HDF-EOS2 grid, to no avail – Except for an MLS L3 HDF-EOS5 • (not a standard product) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 25 25
  27. 27. Panoply, using netcdf-java w/ MLS L3, HDF-EOS5 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 26 26
  28. 28. Can we make netcdf libraries read HDF-EOS? • Might require joint effort between UCAR, THG and tool developers to develop and debug... • OR, solution might emerge from HDF-EOS5 / netCDF-4 Augmentation effort • ...BUT – could make a lot of conversions unnecessary – would make data usage much easier for users HDF Workshop Thursday, November 5, 2009 5 Nov 2009 27 27
  29. 29. Trade Space OPeNDAP Cons Costly Data size expansion L3/L4 grid only Slow Interpolation Costly Reprojects L2 swath netcdf library L3/L4 only Slow HDF-EOS Custom C++ Conversion Lats4D Conversion WCS Pros L2 swath + L3/L4 grid Fast Easy to add HDF + HDF-EOS Subsetting L2 swath + L3/L4 grid Efficient Inexpensive GrADS Data Server HDF + HDF-EOS HDF Workshop Thursday, November 5, 2009 Enables many tools Obsoletes conversion 5 Nov 2009 28 28
  30. 30. Future of HDF-netCDF Tool Interoperability: Key Challenges • Level 1b data (calibrated radiance) – How should we fit the channel/wavelength dimension into CF1? • Level 2 in native swath coordinates • netCDF library direct access to HDF-EOS • Non-self-describing implementations of HDF – e.g., NPP HDF Workshop Thursday, November 5, 2009 5 Nov 2009 29 29

×