NASA HDF/HDF-EOS Data for Dummies (and Developers)

4,613 views

Published on

This is an introductory slide for accessing NASA HDF/HDF-EOS data for beginners. NASA distributes many Earth Science data in HDF/HDF-EOS file format and new users struggle to understand the file format and use the NASA HDF/HDF-EOS data properly. This brief presentation will help new users to understand the basic concepts about the HDF/HDF-EOS and to know the available tools that can access the NASA data easily.

Published in: Technology, Art & Photos
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,613
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
141
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • I believe it’s good to have.
  • I believe it’s good to have.
  • I believe it’s good to have.
  • I believe it’s good to have.
  • I believe it’s good to have.
  • I believe it’s good to have.
  • When it comes to the network access of data, people think of this manner.OPeNDAP is an open implementation of easy data access through visualization tools like IDV.Our work bridges the existing OPeNDAP framework and HDF so that visualization is possible over the Internet.
  • For people who are not familiar with OPeNDAP, I’ll explain how OPeNDAP works with HDF.OPeNDAP is an open implementation of easy data access through visualization tools like IDV.Our work bridges the existing OPeNDAP framework and HDF so that visualization is possible over the Internet.
  • The wider the bottom, it takes longer to reach the top.Development effort, computationally expensive.For each layer, they have their own agenda.
  • The wider the bottom, it takes longer to reach the top.Development effort, computationally expensive.For each layer, they have their own agenda.
  • The wider the bottom, it takes longer to reach the top.Development effort, computationally expensive.For each layer, they have their own agenda.
  • The wider the bottom, it takes longer to reach the top.Development effort, computationally expensive.For each layer, they have their own agenda.
  • The wider the bottom, it takes longer to reach the top.Elevator model
  • The wider the bottom, it takes longer to reach the top.Elevator model
  • If you are going to produce data, in HDF5.The wider the bottom, it takes longer to reach the top.Elevator modelThe HDF Group may help you with CF issues. Suggestions…
  • Make CF-convention consistent.
  • Add ICESAT-2 supported. Data Products Limitation.Add levels of support – web browser.General HDF5 – what kind of support is needed. ICESat-2Not Stable2) The nature of data products3) CF convention (still followed)
  • Correct it to Swath.Subsetting – Key deom – subsetting with IDV.
  • Put limitation merge.Other clients
  • Put limitation merge.Other clients
  • Put limitation merge.Other clients
  • Put limitation merge.Other clients
  • Put limitation merge.Other clients
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • Make animation – screen shots in between texts.
  • Make animation – screen shots in between texts.
  • Show MATLAB doesn’t have any limit.Common limiation.This limiation can be scarificed.Powerful users of MATLAB / IDL can visualize.Check name clashing, removed special characters ---- CF-friendly by default.If data products are good for CF-compliant, things will work immediately.
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • We have summarized the top 7 reasons why some HDF-EOS will fail.We also listed some sample files that will not work with HDF4 handler.Here are screenshots of website.Make animation – screen shots in between texts.
  • At the end of the slide, how many people are familiar with netCDF-4?
  • May skip this slide quickly: not mention data model.Several popular tools that are used widely by Earth Science community.
  • The shared dimensions can be easily used by Earth Science applications to specify the coordinate variables.
  • Group hierarchy and User-defined datatypes are key concepts to addTo the enhanced model.After this slide, ask the audiences how many people heard of CF conventions, how many people would like to know how CF attributes can affectthe visualization results.
  • NASA HDF/HDF-EOS Data for Dummies (and Developers)

    1. 1. The HDF Group NASA HDF/HDF-EOS Data for Dummies (and Developers) Joe Lee The HDF Group July 19, 2012 ESIP 2012 Summer Meeting 1 www.hdfgroup.org
    2. 2. How many dummies in this room? I’ve never used NASA HDF data. (What is HDF…?) July 19, 2012 ESIP 2012 Summer Meeting 2 www.hdfgroup.org
    3. 3. I was and I am still a dummy because NASA HDF is complex! Even after 5 years of work at The HDF Group, I still have something to July 19, 2012 ESIP 2012 Summer Meeting learn. 3 www.hdfgroup.org
    4. 4. I still remember the day one as a NASA HDF dummy so I’d like to share • common mistakes • very useful tips for handling NASA HDF data products from the perspective of new users like you. July 19, 2012 ESIP 2012 Summer Meeting 4 www.hdfgroup.org
    5. 5. Let’s feel the pain of a new user. Analyze Drought in Madison,WI using NASA EOS data. July 16 – Images of the Day – earthobservatory.org (6/25-7/10) July 19, 2012 ESIP 2012 Summer Meeting 5 www.hdfgroup.org
    6. 6. Possible tasks for a new user Compare the current drought period (6/25-7/10) data against the average data of the same period in last 10 years in terms of: • Rainfall • Ozone • Normalized Difference Vegetation Index (NDVI) • Wildfire and visualize the data! Can a new user do it easily with NASA satellite data? July 19, 2012 ESIP 2012 Summer Meeting 6 www.hdfgroup.org
    7. 7. The answer will be no because the user should master the skill of handling many different NASA data products in HDF. July 19, 2012 ESIP 2012 Summer Meeting 7 www.hdfgroup.org
    8. 8. Rainfall task example A simple internet search will eventually lead to GES DISC TRMM web page that distributes data in HDF . July 19, 2012 ESIP 2012 Summer Meeting 8 www.hdfgroup.org
    9. 9. Same for Ozone, NDVI, and Wildfire A simple internet search will eventually lead to various NASA data centers web page that distributes data in HDF. HDF is simply July 19, 2012 indispensable. ESIP 2012 Summer Meeting 9 .. www.hdfgroup.org
    10. 10. So what is HDF? I downloaded the .hdf file but no application can open. PDF HDF It’s a file format for scientific data. It needs an application like HDFView to open it. July 19, 2012 ESIP 2012 Summer Meeting 10 www.hdfgroup.org
    11. 11. Why does NASA use HDF? Why didn’t NASA use these well-known file formats? • ASCII • Spreadsheet (MS Excel) • Database (MySQL, Oracle) • Image (JPEG, GeoTIFF, GIF) • XML because HDF is efficient data management solution in terms of space (storage) and time (performance). July 19, 2012 ESIP 2012 Summer Meeting 11 www.hdfgroup.org
    12. 12. Is HDF also efficient for users? Not necessarily because users may need to learn • new tools and APIs • how to process the data correctly Yet, there’s no “NASA HDF for Dummies” book. Instead, help is always available through hdfgroup.org and hdfeos.org. July 19, 2012 ESIP 2012 Summer Meeting 12 www.hdfgroup.org
    13. 13. What’s inside NASA HDF? A typical NASA HDF file has • attributes (metadata) describing the file contents itself • coordinate variables (time, location – lat/lon/height) • data variable (fire, water, air, land) July 19, 2012 ESIP 2012 Summer Meeting 13 www.hdfgroup.org
    14. 14. NASA HDF has four faces NASA data centers provide files in • HDF4 (.h4, .hdf) • HDF-EOS2 (he2, he) • HDF5 (.h5) • HDF-EOS5 (he5) They are all different although they share same HDF prefix. July 19, 2012 ESIP 2012 Summer Meeting 14 www.hdfgroup.org
    15. 15. HDF4 vs HDF5 HDF4 is old. File starts with ^N^C^S^A. HDF5 is new. File starts with 211HDF. not compatible. They are You need different tools and APIs to access them. HDF4: Hopen() HDF5: H5open() July 19, 2012 ESIP 2012 Summer Meeting 15 www.hdfgroup.org
    16. 16. NASA HDF vs. HDF-EOS HDF- EOS is HDF + GCTP. GCTP: General Cartographic Transformation Package Library HDF: has latitude/longitude dataset. HDF-EOS: has no latitude/longitude dataset. July 19, 2012 ESIP 2012 Summer Meeting 16 www.hdfgroup.org
    17. 17. Why does HDF-EOS includes GCTP? Save space for geo-location (map) information data by storing only a few parameters in the special July 19, 2012 StructMetadata file attribute. ESIP 2012 Summer Meeting 17 www.hdfgroup.org
    18. 18. HDF-EOS is based on HDF API (1) so HDF APIs can read all HDF-EOS objects. no easy way However, there’s to figure out geo-location information with HDF APIs alone. exception: geographic projection / regular grid July 19, 2012 ESIP 2012 Summer Meeting 18 www.hdfgroup.org
    19. 19. HDF-EOS API vs. HDF API Thus, use HDF-EOS APIs for HDF-EOS products instead of HDF APIs to resolve lat/lon. HDF-EOS2: GDopen() HDF4: Hopen() HDF-EOS5:HE5GDopen() HDF5:H5Open() HDF-EOS APIs start with (HE5) July 19, 2012 ESIP 2012 Summer Meeting GD/SW. 19 www.hdfgroup.org
    20. 20. What are GD/SW in HDF-EOS APIs? GD: Grid Level 3 data global SW: Swath Level 2 data regional HDF-EOS API is specific to Earth Science domain. HDF API is for any domain such as finance and biology. July 19, 2012 ESIP 2012 Summer Meeting 20 www.hdfgroup.org
    21. 21. HDF-EOS API is based on HDF API (2) so HDF APIs can modify all HDF-EOS objects. Data producers can add new data using HDF API later. HDF-EOS APIs cannot access such data. July 19, 2012 ESIP 2012 Summer Meeting 21 www.hdfgroup.org
    22. 22. NASA HDF format summary If the NASA file ends with .he2 .hdf, .h4 HDF4 API .he5 HDF-EOS5 API .h5 July 19, 2012 Use HDF5 API HDF-EOS2 API ESIP 2012 Summer Meeting 22 www.hdfgroup.org
    23. 23. NASA HDF format review Quiz: If file ends with .he2 but has some additional objects that HDF-EOS2 cannot access, what a user should do? Answer: Open and read twice. First, with HDF-EOS2 API. Second, with HDF4 API for additional objects. July 19, 2012 ESIP 2012 Summer Meeting 23 www.hdfgroup.org
    24. 24. So far, we talked about formats only Show me the real example. July 19, 2012 ESIP 2012 Summer Meeting 24 www.hdfgroup.org
    25. 25. Read HDF-EOS2 Grid data in C #include <HdfEosDef.h> int main(int argc, char **argv) { /* Open 'MCD43C4.A2000049.005.2006271205532.hdf' using grid API */ if ((gridfile1 = GDopen("MCD43C4.A2000049.005.2006271205532.hdf", DFACC_RDONLY)) == -1) { } /* Open a grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((grid1 = GDattach(gridfile1, "MCD_CMG_BRDF_0.05Deg")) == -1) { } /* Retrieve information about 'Nadir_Reflectance_Band1' datafield */ if ((GDfieldinfo(grid1, "Nadir_Reflectance_Band1", &datafield1rank, datafield1dimsize, &datafield1type, datafield1dimname)) == -1) { } /* Allocate buffer for 'Nadir_Reflectance_Band1' */ if ((datafield1data = malloc(sizeof(int16) * 3600 * 7200)) == NULL) { } /* Read data from 'Nadir_Reflectance_Band1' */ if ((GDreadfield(grid1, "Nadir_Reflectance_Band1", NULL, NULL, NULL, datafield1data)) == -1) { } /* Close the grid named 'MCD_CMG_BRDF_0.05Deg' */ if ((GDdetach(grid1)) == -1) { } /* Close 'MCD43C4.A2000049.005.2006271205532.hdf' */ if ((GDclose(gridfile1)) == -1) { } return 0; } July 19, 2012 ESIP 2012 Summer Meeting 25 www.hdfgroup.org
    26. 26. Read HDF-EOS5 Swath data in FORTRAN c Open 'OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5' using swath c API swathfile1 = he5_swopen $ ('OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5', $ HE5F_ACC_RDONLY) c Open a swath named 'ColumnAmountAerosol' swath1 = he5_swattach(swathfile1, 'ColumnAmountAerosol') c Retrieve information about 'AerosolIndexUV' datafield status = he5_swfldinfo(swath1, 'AerosolIndexUV', datafield1rank, $ datafield1dimsize, datafield1type, datafield1dimname, $ datafield1maxdimname) c Select necessary range of elements datafield1start(1) = 0 datafield1stride(1) = 1 datafield1edge(1) = 60 datafield1start(2) = 0 datafield1stride(2) = 1 datafield1edge(2) = 1644 c Read data from 'AerosolIndexUV' status = he5_swrdfld(swath1, 'AerosolIndexUV', datafield1start, $ datafield1stride, datafield1edge, datafield1data) c Dump data from 'AerosolIndexUV' do i = 1, 16 do j = 1, 6 write(*,*), datafield1data(j,i) enddo write(*,*) enddo July 19, 2012 ESIP 2012 Summer Meeting 26 www.hdfgroup.org
    27. 27. Complete Examples for Dummies Complete and more examples: http://hdfeos.org/examples/library.php Don’t miss how to read lat/lon: http://hdfeos.org/examples/c_grid_lonlat.php July 19, 2012 ESIP 2012 Summer Meeting 27 www.hdfgroup.org
    28. 28. HDF/HDF-EOS API look too difficult Is there a simple API like netCDF? July 19, 2012 ESIP 2012 Summer Meeting 28 www.hdfgroup.org
    29. 29. H4CF Conversion Library APIs // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map<string, string> file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list<var*> pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); July 19, 2012 ESIP 2012 Summer Meeting 29 www.hdfgroup.org
    30. 30. H4CF Conversion Library • works on both HDF4 and HDF-EOS2. • Variables (datasets) list • Attributes (metadata) list • All Vars & Attrs follow CF conventions • Develop your own CF application! July 19, 2012 ESIP 2012 Summer Meeting 30 www.hdfgroup.org
    31. 31. What is CF conventions? • Metadata conventions for earth science data. • Sharing of files created with the NetCDF APIs, but not specifically to netCDF. • The CF conventions are now increasingly gaining acceptance. • URL: http://cf-pcmdi.llnl.gov/ Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 31 www.hdfgroup.org
    32. 32. Don’t follow CF conventions blindly. [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = -15000.0 [1] 0.01 * 5000 + (-15000) = -14950 [2] 0.01 * (5000 – (-15000)) = 200 July 19, 2012 ESIP 2012 Summer Meeting 32 www.hdfgroup.org
    33. 33. -14950 Kelvin? July 19, 2012 ESIP 2012 Summer Meeting 33 www.hdfgroup.org
    34. 34. MODIS Scale / Offset Correction July 19, 2012 ESIP 2012 Summer Meeting 34 www.hdfgroup.org
    35. 35. Limitation – No Support Yet MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc. July 19, 2012 ESIP 2012 Summer Meeting 35 www.hdfgroup.org
    36. 36. Is H5CF Conversion Library available? • Unfortunately, no. • Compared to HDF4/HDF-EOS2, HDF5/HDF-EOS5 products are not diverse. • HDF-EOS5 products follow NASA AURA file guidelines and use only geographic projections. • New HDF5 products will follow CFconventions. July 19, 2012 ESIP 2012 Summer Meeting 36 www.hdfgroup.org
    37. 37. How about programming dummies? What is FORTRAN? What is C? What is C++? … July 19, 2012 ESIP 2012 Summer Meeting 37 www.hdfgroup.org
    38. 38. Why do I need APIs to read data? Do you use PDF APIs to read PDF document? • For most users, PDF reader application is enough. • Same for most HDF users. HDF reader application is enough. July 19, 2012 ESIP 2012 Summer Meeting 38 www.hdfgroup.org
    39. 39. Read HDF data in ASCII Use generic HDF “dump” tools • hdp for HDF-EOS2 and HDF4 files • h5dump for HDF-EOS5 and HDF5 files They are only good for attributes and data. Cannot extract lat/lon from HDF-EOS Grid July 19, 2012 ESIP 2012 Summer Meeting 39 www.hdfgroup.org
    40. 40. HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. eos2 eos2 dump ASCII http://hdfeos.org/zoo MATLAB IDL NCL July 19, 2012 ESIP 2012 Summer Meeting 40 www.hdfgroup.org
    41. 41. Important Tool for Zoo Maintenance July 19, 2012 ESIP 2012 Summer Meeting 41 www.hdfgroup.org
    42. 42. Why Tools Need HDF-EOS2 Dumper HDF-EOS2 dumper Lat / Lon in ASCII Lat / Lon LaRC MISR SOM Projection HDF-EOS2 July 19, 2012 MATLAB IDL NCL ESIP 2012 Summer Meeting 42 www.hdfgroup.org
    43. 43. HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line July 19, 2012 ESIP 2012 Summer Meeting 43 www.hdfgroup.org
    44. 44. eos2dump Latitude Example 90 80 eos2dump –c filename.hdf > latitude.txt 70 60 50 40 What: 30 Format: one value per line (default) 20 10 0 -10 -20 -30 1 lat July 19, 2012 ESIP 2012 Summer Meeting 44 www.hdfgroup.org
    45. 45. eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. 180 175 170 165 160 155 150 145 140 135 130 125 120 115 110 July 19, 2012 ESIP 2012 Summer Meeting 45 www.hdfgroup.org
    46. 46. How about eos5dump? • It’s not necessary because all HDF-EOS5 Grid use geographic projection. • You can easily generate lat/lon using the parameters stored in StructMetadata. • E.g., 360 / x-dimension, 180 / y-dimension July 19, 2012 ESIP 2012 Summer Meeting 46 www.hdfgroup.org
    47. 47. How about visualization? 0000111020230232 3234234299234234 2342343292343493 … ASCII data is boring! July 19, 2012 ESIP 2012 Summer Meeting 47 www.hdfgroup.org
    48. 48. HDFView THE GUI Tool equivalent to Acrobat Reader. • It can visualize data as image. • It can even edit some data. • Critical drawback: no map! • Why? Again, it’s a general purpose tool. Not for Earth science users only. July 19, 2012 ESIP 2012 Summer Meeting 48 www.hdfgroup.org
    49. 49. HDFView HDF-EOS Plug-in An additional download for HDFView. • It can visualize HDF-EOS2 and HDF-EOS5. • It supports map. July 19, 2012 ESIP 2012 Summer Meeting 49 www.hdfgroup.org
    50. 50. HDFView HDF-EOS Plug-in screenshot Close but no cigar – it works only for HDF-EOS! There are many generic HDF data products. July 19, 2012 ESIP 2012 Summer Meeting 50 www.hdfgroup.org
    51. 51. Then, what should dummies do? Any visualization tool that works for both HDF and HDF-EOS? July 19, 2012 ESIP 2012 Summer Meeting 51 www.hdfgroup.org
    52. 52. Use a script language tool such as • NCL • IDL • MATLAB with eos2dump tool July 19, 2012 ESIP 2012 Summer Meeting 52 www.hdfgroup.org
    53. 53. Rich Dummy Poor Dummy MATLAB and IDL licenses are expensive! July 19, 2012 ESIP 2012 Summer Meeting 53 www.hdfgroup.org
    54. 54. Solution for Poor Dummy Try NCL. • Free yet powerful. • http://hdfeos.org/zoo page has complete codes for visualizing most NASA data products in NCL. • ncl-talk@ucar.edu mailing list will rock you. July 19, 2012 ESIP 2012 Summer Meeting 54 www.hdfgroup.org
    55. 55. Solution for Rich Dummy Visit Comprehensive Examples page. • http://hdfeos.org/zoo page has complete codes for visualizing most NASA data products in IDL and MATLAB. July 19, 2012 ESIP 2012 Summer Meeting 55 www.hdfgroup.org
    56. 56. Poor and Lazy Dummy I hate programming. Any free visualization tool that works for both HDF/HDF-EOS? July 19, 2012 ESIP 2012 Summer Meeting 56 www.hdfgroup.org
    57. 57. Recall: H4CF Conversion Library • works on both HDF4 and HDF-EOS2. • Variables (datasets) list • Attributes (metadata) list • All Vars & Attrs follow CF conventions • Develop your own CF application! July 19, 2012 ESIP 2012 Summer Meeting 57 www.hdfgroup.org
    58. 58. Solution: h4tonccf • Usage: h4tonccf filename.hdf [filename.nc] • generates either netcdf-3 or netcdf-4 file. • follows netcdf classic model. • is based on H4CF Conversion Library. Use free netcdf visualization tool like Panoply and IDV after conversion. July 19, 2012 ESIP 2012 Summer Meeting 58 www.hdfgroup.org
    59. 59. h4tonccf Demo July 19, 2012 ESIP 2012 Summer Meeting 59 www.hdfgroup.org
    60. 60. Converted File Size Comparison 656M Netcdf-3 128M Netcdf-4 72M HDF-EOS2 July 19, 2012 ESIP 2012 Summer Meeting 9X 60 www.hdfgroup.org
    61. 61. Same Limitation – No Support Yet MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc. July 19, 2012 ESIP 2012 Summer Meeting 61 www.hdfgroup.org
    62. 62. How about h5tonccf? • Recall: no H5CF Conversion Library exists. • The latest IDV and Panoply may open and visualize HDF-EOS5 directly. • However, no luck with HDF5. July 19, 2012 ESIP 2012 Summer Meeting 62 www.hdfgroup.org
    63. 63. No hope for Poor and Lazy Dummy? I hate programming. I want free visualization tool that works for both HDF/HDF-EOS. Am I hopeless? July 19, 2012 ESIP 2012 Summer Meeting 63 www.hdfgroup.org
    64. 64. Solution for the hopeless: OPeNDAP • Why OPeNDAP? • What is HDF Handler? • How important CF conventions? July 19, 2012 ESIP 2012 Summer Meeting 64 www.hdfgroup.org
    65. 65. Access HDF via FTP Aqua/Aura Users IDL AIRS/OMI FTP Local HDF Files July 19, 2012 FTP Clients FTP Servers ESIP 2012 Summer Meeting HDF Files 65 www.hdfgroup.org
    66. 66. OPeNDAP access Typical remoteconcept problem View Data (User) FTP/HTTP Remote Data (HDF4/5) Visualization Tools tools (e.g., GrADS) (gradsdap) Handler (hdf4/5_handler) Client Library (libnc_dap) Server (Hyrax) Data Access Protocol (DAP) via Internet July 19, 2012 ESIP 2012 Summer Meeting 66 www.hdfgroup.org
    67. 67. Access remote data easily July 19, 2012 ESIP 2012 Summer Meeting 67 www.hdfgroup.org
    68. 68. without worrying about data plans. July 19, 2012 ESIP 2012 Summer Meeting 68 www.hdfgroup.org
    69. 69. Visualize remote data easily, too. July 19, 2012 ESIP 2012 Summer Meeting 69 www.hdfgroup.org
    70. 70. HDF handler is key Aqua/Aura HDF HANDLER Users IDV Hyrax NetCDF Java AIRS/OMI DAP Visualization Tools July 19, 2012 OPeNDAP Clients OPeNDAP Servers ESIP 2012 Summer Meeting HDF Files 70 www.hdfgroup.org
    71. 71. because if handler doesn’t follow CF No Visualization Bonus! July 19, 2012 ESIP 2012 Summer Meeting 71 www.hdfgroup.org
    72. 72. If handler gives up extra bonus handler development is easier. DAP DAP Server HDF / HDF-EOS July 19, 2012 ESIP 2012 Summer Meeting 72 www.hdfgroup.org
    73. 73. What’s involved in extra bonus Earth Scientist Visualization DAP Client DAP DAP Server CF Conventions HDF / HDF-EOS July 19, 2012 ESIP 2012 Summer Meeting 73 www.hdfgroup.org
    74. 74. How handler developers feel July 19, 2012 ESIP 2012 Summer Meeting 74 www.hdfgroup.org
    75. 75. Because we care all NASA products. AURA MISR ? AQUA TERRA NPP July 19, 2012 ESIP 2012 Summer Meeting 75 www.hdfgroup.org
    76. 76. Developers’ Dream … all data are created equal, that they shall be visualized by all client tools easily… July 19, 2012 ESIP 2012 Summer Meeting 76 www.hdfgroup.org
    77. 77. What if all people agreed… development might have been much easier. Earth Scientist Visualization DAP Client DAP DAP Server Check once. July 19, 2012 UDF (Imaginary) ESIP 2012 Summer Meeting 77 www.hdfgroup.org
    78. 78. Lessons from handler development Communicate and work together among different groups. Standardize everything from data products to visualization tools. Follow conventions and standards. July 19, 2012 ESIP 2012 Summer Meeting 78 www.hdfgroup.org
    79. 79. Redesign of HDF5 handler Goal: Support all NASA HDF5 products. Modularize for future product support. Follow CF conventions robustly. Product July 19, 2012 ESIP 2012 Summer Meeting 79 www.hdfgroup.org
    80. 80. Why did we redesign? In 2007, HDF5 handler AURA Grid was enough for July 19, 2012 ESIP 2012 Summer Meeting 80 www.hdfgroup.org
    81. 81. Why did we redesign? In 2011, wasn’t enough for AURA Grid July 19, 2012 AURA Swath ESIP 2012 Summer Meeting AURA ZA 81 GSSTF www.hdfgroup.org
    82. 82. Why did we redesign? In 2011, wasn’t enough for AURA Grid July 19, 2012 AURA Swath ESIP 2012 Summer Meeting AURA ZA 82 GSSTF www.hdfgroup.org
    83. 83. and more… so we gave up! ICESat-2 Aquarius GOSAT SMAP July 19, 2012 ESIP 2012 Summer Meeting 83 www.hdfgroup.org
    84. 84. We rebuilt one that can scale up July 19, 2012 ESIP 2012 Summer Meeting 84 www.hdfgroup.org
    85. 85. by modularizing the design HDF5 handler CF Generic HDFEOS5 HDF5 GOSAT Aquarius SMAP with a July 19, 2012 GRID SWATH ZA big room for future products. ESIP 2012 Summer Meeting 85 www.hdfgroup.org
    86. 86. Key updates on HDF5 handler version 2.0.0 CF-convention Support Range of NASA Data Products Mulitiple Grids/Swath/ZA Configuration Options Support for Future Products July 19, 2012 1.5.x Primitive Robust Limited All No Yes Static Build Run-time Rigid Flexible ESIP 2012 Summer Meeting 86 www.hdfgroup.org
    87. 87. HDF-EOS5 products supported • • • • • July 19, 2012 OMI Swath, Grid MLS Swath HIRDLS Swath, ZA TES Swath, Grid GSSTF ESIP 2012 Summer Meeting 87 www.hdfgroup.org
    88. 88. HDF5 products supported • • • • July 19, 2012 Aquarius Grid MEaSUREs SeaWiFS DeepBlue SBUV2 Ozone (O3) Profile SMAP (Simulation) ESIP 2012 Summer Meeting 88 www.hdfgroup.org
    89. 89. HIRDLS Zonal Average July 19, 2012 ESIP 2012 Summer Meeting 89 www.hdfgroup.org
    90. 90. New TES Swath July 19, 2012 ESIP 2012 Summer Meeting 90 www.hdfgroup.org
    91. 91. MEaSUREs GSSTF July 19, 2012 ESIP 2012 Summer Meeting 91 www.hdfgroup.org
    92. 92. Aquarius Grid (Simulation) July 19, 2012 ESIP 2012 Summer Meeting 92 www.hdfgroup.org
    93. 93. MEaSUREs SeaWiFS DeepBlue July 19, 2012 ESIP 2012 Summer Meeting 93 www.hdfgroup.org
    94. 94. Ozone Profile (BUV) July 19, 2012 ESIP 2012 Summer Meeting 94 www.hdfgroup.org
    95. 95. SMAP Swath (Simulation) July 19, 2012 ESIP 2012 Summer Meeting 95 www.hdfgroup.org
    96. 96. SMAP Swath (Simulation) July 19, 2012 ESIP 2012 Summer Meeting 96 www.hdfgroup.org
    97. 97. No CF magic for some products Tool / CF Limitation • GOSAT/acos: 1-D trajectory • ICESat-2 • OMI L2G: Lat/Lon with 3-D Java-NetCDF Client Limitation • Dataset w/ long string July 19, 2012 ESIP 2012 Summer Meeting 97 www.hdfgroup.org
    98. 98. It doesn’t mean handler failure because other tools like can still access and visualize data. July 19, 2012 ESIP 2012 Summer Meeting 98 www.hdfgroup.org
    99. 99. MATLAB and IDL will work too. HDF5 Examples under zoo will work. • Visit http://hdfeos.org/zoo • Replace local file access with OPeNDAP URL access.  MATLAB: Try new OPeNDAP functions. • The rest will be same. July 19, 2012 ESIP 2012 Summer Meeting 99 www.hdfgroup.org
    100. 100. Read if you can’t see http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3OMTO3e_2012m0416_v0032012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029] 2012m0418t020608.he5 lon, Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003- 77.375 lat, 38.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375 _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_ O3.lat=38.375], O3_Data_Fields_ColumnAmount July 19, 2012 309.6 ESIP 2012 Summer Meeting 100 www.hdfgroup.org
    101. 101. or hear from Siri? July 19, 2012 ESIP 2012 Summer Meeting 101 www.hdfgroup.org
    102. 102. HDF4 OPeNDAP handler update • Corrected the handling the scale/offset for MODIS products. • Added the support for Lambert azimuthal equal-area (LAMAZ). • Added the support for Space-oblique Mercator projection (SOM). • Other minor bug fixes. July 19, 2012 ESIP 2012 Summer Meeting 102 www.hdfgroup.org
    103. 103. Key updates on HDF4 handler version 3.9.4 Some MODIS Scale/offset LAMAZ projection support SOM projection support July 19, 2012 3.9.3 Wrong Corrected No Yes No Yes ESIP 2012 Summer Meeting 103 www.hdfgroup.org
    104. 104. MODIS scale / offset correction July 19, 2012 ESIP 2012 Summer Meeting 104 www.hdfgroup.org
    105. 105. Support for LAMAZ projection July 19, 2012 ESIP 2012 Summer Meeting 105 www.hdfgroup.org
    106. 106. Support for SOM projection Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512]; … } Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; … } July 19, 2012 ESIP 2012 Summer Meeting 106 www.hdfgroup.org
    107. 107. HDF-EOS2 products supported GES DISC • AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC • Many MODIS products NSIDC • AMSR_E / NISE / MODIS LaRC • MISR / MOPITT / some CERES (TRMM) July 19, 2012 ESIP 2012 Summer Meeting 107 www.hdfgroup.org
    108. 108. HDF4 products supported GES DISC • TRMM L1B, L2B Swath • TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) • SeaWiFS / MODIST / MODISA • CZCS / OCTS July 19, 2012 ESIP 2012 Summer Meeting 108 www.hdfgroup.org
    109. 109. HDF4 products supported CERES • • • • • • • July 19, 2012 CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG ESIP 2012 Summer Meeting 109 www.hdfgroup.org
    110. 110. HDF4 handler limitations • HDF-EOS2 + additional HDF4 objects • In some products, some additional HDF4 objects will be ignored. not listed • Other HDF4 products in previous slides are not tested and may not be visualized by Java OPeNDAP clients. July 19, 2012 ESIP 2012 Summer Meeting 110 www.hdfgroup.org
    111. 111. Other limitations CF conventions Limitation • _FillValue inside latitude and longitude Java-NetCDF Limitation • Java Heap Memory July 19, 2012 ESIP 2012 Summer Meeting 111 www.hdfgroup.org
    112. 112. Summary: we’ve updated handlers HDFEOS2 HDF4 OPeNDAP HDF Handlers HDF5 HDFEOS5 … and will continue. July 19, 2012 ESIP 2012 Summer Meeting 112 www.hdfgroup.org
    113. 113. GES DISC Uses Handlers July 19, 2012 ESIP 2012 Summer Meeting 113 www.hdfgroup.org
    114. 114. Please try HDF4/HDF5 handlers New hdf5_handler 2.0.0 is released! Source codes and RPMs: http://opendap.org Demo HDF5/HDF4 servers: http://hdfeos.org/zoo/hdf4_handler/ http://hdfeos.org/zoo/hdf5_handler/ July 19, 2012 ESIP 2012 Summer Meeting 114 www.hdfgroup.org
    115. 115. and give us feedback. July 19, 2012 ESIP 2012 Summer Meeting 115 www.hdfgroup.org
    116. 116. Got HDF? I’m not a dummy any more. July 19, 2012 ESIP 2012 Summer Meeting 116 www.hdfgroup.org
    117. 117. Enjoy observing the Earth! July 19, 2012 ESIP 2012 Summer Meeting 117 www.hdfgroup.org
    118. 118. Topics for advanced dummies That’s it? How about netCDF-4 interoperability? July 19, 2012 ESIP 2012 Summer Meeting 118 www.hdfgroup.org
    119. 119. Clarification –netCDF format • netCDF-3 format • Simple self-describing data format based on netCDF classic data model • netCDF-4 format • Uses HDF5 as a storage layer • Exploits • Compression, chunking, parallel-IO • Group hierarchy, user-defined data types, etc. • Supports both netCDF enhanced and netCDF classic data models • Interoperability with netCDF-4 format in this talk Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 119 www.hdfgroup.org
    120. 120. Clarification – netCDF packages • netCDF software packages • netCDF-C • Support both netCDF-3 and netCDF-4 formats • C++/Fortran Wrappers • netCDF-Java • Support both netCDF-3 and netCDF-4 formats • The implementation of the Common Data Model • netCDF version 4 • Generally mean the version 4 package of the netCDF-C library (Not only support netCDF-4 format) Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 120 www.hdfgroup.org
    121. 121. Why netCDF-4 • Big user community • User-friendly data models • Tools • Home-grown and third-party visualization and analysis tools - ncdump, ncgen, IDV, Panoply, Ferret etc. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 121 www.hdfgroup.org
    122. 122. NetCDF Classic Model Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 122 www.hdfgroup.org
    123. 123. NetCDF enhanced model Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 123 www.hdfgroup.org
    124. 124. Conversion Creating a new netCDF-4 file from HDF5 file. July 19, 2012 ESIP 2012 Summer Meeting 124 www.hdfgroup.org
    125. 125. eos52nc4 • • • • • converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc http://hdfeos.org/software/convert_hdfeos5.php eos5 NETCDF-4 eos52nc4 July 19, 2012 ESIP 2012 Summer Meeting 125 www.hdfgroup.org
    126. 126. Augmentation Making an HDF5 file look like a netCDF-4 file. July 19, 2012 ESIP 2012 Summer Meeting 126 www.hdfgroup.org
    127. 127. aug_eos5 • augments hdf-eos5 to netcdf-4. • Both HDF-EOS5 and NetCDF4 can access it. • Usage: aug_eos5 [-i|-f map] test.h5 • modifies the input file directly. • does not follow CF conventions. • http://hdfeos.org/software/aug_hdfeos5.php eos5 NETCDF-4 aug_eos5 July 19, 2012 ESIP 2012 Summer Meeting 127 www.hdfgroup.org
    128. 128. Augmentation Concept aug_eos5 HDF-EOS5 API NETCDF-4 API hdf-eos5 augmentation netcdf4 hdf5 HDF5 API July 19, 2012 ESIP 2012 Summer Meeting 128 www.hdfgroup.org
    129. 129. Why NetCDF-4 Doesn’t Work [nLevels][nTimes] hdf-eos5 July 19, 2012 ESIP 2012 Summer Meeting NETCDF-4 API 129 www.hdfgroup.org
    130. 130. What aug_eos5 Does [nLevels][nTimes] hdf-eos5 July 19, 2012 ESIP 2012 Summer Meeting NETCDF-4 API 130 www.hdfgroup.org
    131. 131. Options in aug_eos5 1. Default option • -i option 2.-f map_file option July 19, 2012 ESIP 2012 Summer Meeting 131 www.hdfgroup.org
    132. 132. Default option in aug_eos5 Grid • XDim, YDim: calculated • Other dims: default fill value Swath / ZA • All dims: default fill value July 19, 2012 ESIP 2012 Summer Meeting 132 www.hdfgroup.org
    133. 133. Better -i option in aug_eos5 • Grid • XDim, YDim: calculated • Other dims: 1,2,…,n • Swath / ZA • All dims: July 19, 2012 1,2,…,n ESIP 2012 Summer Meeting 133 www.hdfgroup.org
    134. 134. Best -f option in aug_eos5 • Needs an input map file. • Input file specifies 3 rules: 1. What to be filled in 2. How to fill in 3. Where to get the values • Most useful and flexible option July 19, 2012 ESIP 2012 Summer Meeting 134 www.hdfgroup.org
    135. 135. aug_eos5 Input File Format Each line in input file contains 1. How: 0,1,2 2. What: dim name 3. Where: data file or existing variable in file. 4. (Optional) Multiple grid/swath/za case only: • July 19, 2012 grid/swath/za name:<name> ESIP 2012 Summer Meeting 135 www.hdfgroup.org
    136. 136. aug_eos5 Input File Example Copy the data Read file data.txt How 0 1 2 What Where nTimes Time nLevels data.txt nCandidates Fill 1,2,…, n 95% cases – Copy the data July 19, 2012 ESIP 2012 Summer Meeting 136 www.hdfgroup.org
    137. 137. Multiple Swath Case How Multiple What Where swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates swath name: BrO column 2 nTimes 1 nLevels data2.txt July 19, 2012 ESIP 2012 Summer Meeting 137 www.hdfgroup.org
    138. 138. aug_eos5 vs eos52nc4 aug_eos5 hdfeos-5 Yes No netcdf-4 July 19, 2012 eos52nc4 Yes Yes ESIP 2012 Summer Meeting 138 www.hdfgroup.org
    139. 139. HIRDLS/MLS Team Use aug_eos5 So You Can! July 19, 2012 ESIP 2012 Summer Meeting 139 www.hdfgroup.org
    140. 140. The HDF Group Thank you! July 19, 2012 ESIP 2012 Summer Meeting 140 www.hdfgroup.org
    141. 141. Acknowledgements This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration. July 19, 2012 ESIP 2012 Summer Meeting 141 www.hdfgroup.org
    142. 142. The HDF Group Questions/comments? help@hdfgroup.org July 19, 2012 ESIP 2012 Summer Meeting 142 www.hdfgroup.org

    ×