Tools to improve the usability of NASA HDF Data

709 views

Published on

Published in: Technology, Art & Photos
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
709
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Tools to improve the usability of NASA HDF Data

  1. 1. The HDF Group Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012 HDF/HDF-EOS Workshop XV 1 www.hdfgroup.org
  2. 2. Tutorial Outline Introduction 5 Tools 1. H4CF Conversion Toolkit (h4cf) 2. h4toh5 Conversion Tool (h4toh5) 3. HDF-EOS5 to netCDF-4 Converter (eos52nc4) 4. HDF-EOS5 Augmentation Tool (aug_eos5) 5. HDF-EOS2 Dumper (eos2dump) Demo April 17, 2012 HDF/HDF-EOS Workshop XV 2 www.hdfgroup.org
  3. 3. Who Said Users Don’t Matter? April 17, 2012 HDF/HDF-EOS Workshop XV 3 www.hdfgroup.org
  4. 4. Usability of NASA HDF Data MATLAB Panoply Panoply April 17, 2012 IDL NASA HDF/HDF-EOS Data IDV IDV AA school teacher’s view school teacher’s expectation HDF/HDF-EOS Workshop XV 4 www.hdfgroup.org
  5. 5. The 5 Tools Improve the usability of NASA Data Supported by The HDF Group Funded by NASA / Raytheon http://hdfeos.org/software/tool.php April 17, 2012 HDF/HDF-EOS Workshop XV 5 www.hdfgroup.org
  6. 6. 5 Tools that Improve Usability Panoply / IDV NETCDF-3 h4cf h4toh5 eos2 dump eos2 hdf4 ASCII eos5 NETCDF-4 April 17, 2012 eos52nc4 aug_eos5 HDF/HDF-EOS Workshop XV MATLAB IDL NCL 6 www.hdfgroup.org
  7. 7. H4CF Conversion Toolkit Panoply / IDV NETCDF-3 h4cf eos2 hdf4 Follow NETCDF-4 CF convention Both Library and Tools http://hdfeos.org/software/h4cflib April 17, 2012 HDF/HDF-EOS Workshop XV 7 www.hdfgroup.org
  8. 8. H4CF Conversion Toolkit Two ways to build: 1. Use configure script. 2. Edit Makefile template. April 17, 2012 HDF/HDF-EOS Workshop XV 8 www.hdfgroup.org
  9. 9. H4CF Conversion Library • Variables (datasets) list • Attributes (metadata) list • All Vars & Attrs follow CF conventions • Develop your own CF application! April 17, 2012 HDF/HDF-EOS Workshop XV 9 www.hdfgroup.org
  10. 10. H4CF Library APIs Example // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map<string, string> file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list<var*> pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); April 17, 2012 HDF/HDF-EOS Workshop XV 10 www.hdfgroup.org
  11. 11. H4CF Conversion Tools 2 Tools based on H4CF APIs: cdldumper h4tonccf April 17, 2012 HDF/HDF-EOS Workshop XV 11 www.hdfgroup.org
  12. 12. cdldumper • Usage: cdldumper filename.hdf • Dump file in Common Data Language. • Useful for quick check and debugging purpose. April 17, 2012 HDF/HDF-EOS Workshop XV 12 www.hdfgroup.org
  13. 13. cdldumper Demo Number of attributes=1 _FillValue=255 April 17, 2012 HDF/HDF-EOS Workshop XV 13 www.hdfgroup.org
  14. 14. cdldumper Demo Number of attributes=4 _FillValue=255 %./cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf orig_name netCDF AMSR_E_L3_5DaySnow_V09_20050126.hdf { long_name dimensions: coordinates XDim_Northern_Hemisphere = 721, XDim_Southern_… variables: ubyte Flags_NorthernPentad(YDim_Northern_Hemis… Flags_NorthernPentad:_FillValue = 255; Flags_NorthernPentad:origname = "/Northern… Flags_NorthernPentad:long_name = "/Northern… Flags_NorthernPentad:coordinates = "latitude… April 17, 2012 HDF/HDF-EOS Workshop XV 14 www.hdfgroup.org
  15. 15. h4tonccf • Usage: h4tonccf filename.hdf [filename.nc] • generates either netcdf-3 or netcdf-4 file. • follows netcdf April 17, 2012 classic model. HDF/HDF-EOS Workshop XV 15 www.hdfgroup.org
  16. 16. h4tonccf Demo April 17, 2012 HDF/HDF-EOS Workshop XV 16 www.hdfgroup.org
  17. 17. Don’t Follow CF Conventions Blindly [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = -15000.0 [1] 0.01 * 5000 + (-15000) = -14950 [2] 0.01 * (5000 – (-15000)) = 200 April 17, 2012 HDF/HDF-EOS Workshop XV 17 www.hdfgroup.org
  18. 18. -14950 Kelvin? April 17, 2012 HDF/HDF-EOS Workshop XV 18 www.hdfgroup.org
  19. 19. MODIS Scale / Offset Correction April 18, 2012 HDF/HDF-EOS Workshop XV 19 www.hdfgroup.org
  20. 20. Converted File Size Comparison 656M Netcdf-3 128M Netcdf-4 72M HDF-EOS2 April 17, 2012 HDF/HDF-EOS Workshop XV 9X 20 www.hdfgroup.org
  21. 21. Limitation – No Support Yet MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc. April 17, 2012 HDF/HDF-EOS Workshop XV 21 www.hdfgroup.org
  22. 22. h4toh5 Conversion Tool Convert any hdf4 to hdf5. http://hdfgroup.org/h4toh5 h4toh5 NETCDF-4 April 17, 2012 eos2 Usage: h4toh5 –eos –nc4 in.hdf out.nc Static Binary w/ –eos & -nc4: http://hdfeos.org/software/h4toh5/bin/ HDF/HDF-EOS Workshop XV 22 www.hdfgroup.org
  23. 23. h4tonccf vs. h4toh5 h4tonccf Handle non-eos2 hdf4 Follow CF conventions Output netcdf classic Output netcdf enhanced April 17, 2012 Yes h4toh5 w/ -eos –nc4 No Yes No Yes No No Yes HDF/HDF-EOS Workshop XV 23 www.hdfgroup.org
  24. 24. eos52nc4 • • • • • converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc http://hdfeos.org/software/convert_hdfeos5.php eos5 NETCDF-4 eos52nc4 April 17, 2012 HDF/HDF-EOS Workshop XV 24 www.hdfgroup.org
  25. 25. aug_eos5 • augments hdf-eos5 to netcdf-4. • Both HDF-EOS5 and NetCDF4 can access it. • Usage: aug_eos5 [-i|-f map] test.h5 • modifies the input file directly. • does not follow CF conventions. • http://hdfeos.org/software/aug_hdfeos5.php eos5 NETCDF-4 aug_eos5 April 17, 2012 HDF/HDF-EOS Workshop XV 25 www.hdfgroup.org
  26. 26. Augmentation Concept aug_eos5 HDF-EOS5 API NETCDF-4 API hdf-eos5 augmentation netcdf4 hdf5 HDF5 API April 17, 2012 HDF/HDF-EOS Workshop XV 26 www.hdfgroup.org
  27. 27. Why NetCDF-4 Doesn’t Work [nLevels][nTimes] hdf-eos5 April 17, 2012 HDF/HDF-EOS Workshop XV NETCDF-4 API 27 www.hdfgroup.org
  28. 28. What aug_eos5 Does [nLevels][nTimes] hdf-eos5 April 17, 2012 HDF/HDF-EOS Workshop XV NETCDF-4 API 28 www.hdfgroup.org
  29. 29. Options in aug_eos5 1. Default option • -i option 2.-f map_file option April 17, 2012 HDF/HDF-EOS Workshop XV 29 www.hdfgroup.org
  30. 30. Default option in aug_eos5 Grid • XDim, YDim: calculated • Other dims: default fill value Swath / ZA • All dims: default fill value April 17, 2012 HDF/HDF-EOS Workshop XV 30 www.hdfgroup.org
  31. 31. Better -i option in aug_eos5 • Grid • XDim, YDim: calculated • Other dims: 1,2,…,n • Swath / ZA • All dims: April 17, 2012 1,2,…,n HDF/HDF-EOS Workshop XV 31 www.hdfgroup.org
  32. 32. Best -f option in aug_eos5 • Needs an input map file. • Input file specifies 3 rules: 1. What to be filled in 2. How to fill in 3. Where to get the values • Most useful and flexible option April 17, 2012 HDF/HDF-EOS Workshop XV 32 www.hdfgroup.org
  33. 33. aug_eos5 Input File Format Each line in input file contains 1. How: 0,1,2 2. What: dim name 3. Where: data file or existing variable in file. 4. (Optional) Multiple grid/swath/za case only: • April 17, 2012 grid/swath/za name:<name> HDF/HDF-EOS Workshop XV 33 www.hdfgroup.org
  34. 34. aug_eos5 Input File Example Copy the data Read file data.txt How 0 1 2 What Where nTimes Time nLevels data.txt nCandidates Fill 1,2,…, n 95% cases – Copy the data April 17, 2012 HDF/HDF-EOS Workshop XV 34 www.hdfgroup.org
  35. 35. Multiple Swath Case How Multiple What Where swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates swath name: BrO column 2 nTimes 1 nLevels data2.txt April 17, 2012 HDF/HDF-EOS Workshop XV 35 www.hdfgroup.org
  36. 36. aug_eos5 vs eos52nc4 aug_eos5 hdfeos-5 Yes No netcdf-4 April 17, 2012 eos52nc4 Yes Yes HDF/HDF-EOS Workshop XV 36 www.hdfgroup.org
  37. 37. HIRDLS/MLS Team Use aug_eos5 So You Can! April 17, 2012 HDF/HDF-EOS Workshop XV 37 www.hdfgroup.org
  38. 38. HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. eos2 eos2 dump ASCII http://hdfeos.org/zoo MATLAB IDL NCL April 17, 2012 HDF/HDF-EOS Workshop XV 38 www.hdfgroup.org
  39. 39. Important Tool for Zoo Maintenance April 17, 2012 HDF/HDF-EOS Workshop XV 39 www.hdfgroup.org
  40. 40. Why Tools Need HDF-OES2 Dumper HDF-EOS2 dumper Lat / Lon in ASCII Lat / Lon LaRC MISR SOM Projection HDF-EOS2 April 17, 2012 MATLAB IDL NCL HDF/HDF-EOS Workshop XV 40 www.hdfgroup.org
  41. 41. HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line April 17, 2012 HDF/HDF-EOS Workshop XV 41 www.hdfgroup.org
  42. 42. eos2dump Latitude Example 90 80 eos2dump –c filename.hdf > latitude.txt 70 60 50 40 What: 30 Format: one value per line (default) 20 10 0 -10 -20 -30 1 lat April 17, 2012 HDF/HDF-EOS Workshop XV 42 www.hdfgroup.org
  43. 43. eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. 180 175 170 165 160 155 150 145 140 135 130 125 120 115 110 April 17, 2012 HDF/HDF-EOS Workshop XV 43 www.hdfgroup.org
  44. 44. Got Tools? April 17, 2012 HDF/HDF-EOS Workshop XV 44 www.hdfgroup.org
  45. 45. Enjoy the NASA HDF Data! Panoply / IDV NETCDF-3 h4cf h4toh5 eos2 NASA HDF hdf4 data users eos2 dump ASCII eos5 NETCDF-4 April 17, 2012 eos52nc4 aug_eos5 HDF/HDF-EOS Workshop XV MATLAB IDL NCL 45 www.hdfgroup.org
  46. 46. … and observe the Earth! April 17, 2012 HDF/HDF-EOS Workshop XV 46 www.hdfgroup.org
  47. 47. The HDF Group Thank you ! April 17, 2012 HDF/HDF-EOS Workshop XV 47 www.hdfgroup.org
  48. 48. Acknowledgements This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration. April 17, 2012 HDF/HDF-EOS Workshop XV 48 www.hdfgroup.org
  49. 49. The HDF Group Questions/comments? April 17, 2012 HDF/HDF-EOS Workshop XV 49 www.hdfgroup.org

×