Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Implementation of HDF-EOS5 and HDF5 into NCL

577 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Implementation of HDF-EOS5 and HDF5 into NCL

  1. 1. Implementation of HDFEOS5 and HDF5 in NCL Wei Huang, Dave Brown, Mary Haley, Rick Brownrigg CISL/NCAR Dennis Shea, Adam Phillips CGD/NCAR huangwei@ucar.edu
  2. 2. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information 2/17/2014 CISL/NCAR
  3. 3. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization 2/17/2014 CISL/NCAR
  4. 4. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source 2/17/2014 CISL/NCAR
  5. 5. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) 2/17/2014 CISL/NCAR
  6. 6. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) 2/17/2014 CISL/NCAR
  7. 7. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics 2/17/2014 CISL/NCAR
  8. 8. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics ◦ Hundreds of example scripts, lots of sample data 2/17/2014 CISL/NCAR
  9. 9. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics ◦ Hundreds of example scripts, lots of sample data ◦ PyNIO, Python interface for same NCL file I/O 2/17/2014 CISL/NCAR
  10. 10. HDF-EOS5 Implementation  Start with HDF-EOS Code ◦ Reconstruct function structure ◦ Create new HDF-EOS5 functions ◦ Variables follow the NetCDF data model 2/17/2014 CISL/NCAR
  11. 11. HDF-EOS5 Implementation  Start with HDF-EOS Code ◦ Reconstruct function structure ◦ Create new HDF-EOS5 functions ◦ Variables follow the NetCDF data model  How NCL open HDF-EOS5 files ◦ Command: f = addfile(filename, “r”)  Filename has the form: body.suffix     NetCDF has suffix: nc, cdf, netcdf Grib has: gr, gr1, grb, grb1, grib, gr2, grb2, grib2 HDF has: hdf, h4 HDF-EOS has: hdfeos, he2, he4 2/17/2014 CISL/NCAR
  12. 12. HDF-EOS5 Implementation (continue)  HDF-EOS5 has suffix: he5 ◦ Function addfile check file suffix, and when suffix is he5, it knows to move function pointers put to HDF-EOS5 functions ◦ HDF-EOS5 uses HDF5, but we do not need to deal with HDF5 directly 2/17/2014 CISL/NCAR
  13. 13. HDF-EOS5 Sample 2/17/2014 CISL/NCAR
  14. 14. HDF-EOS5 Sample (continue) 2/17/2014 CISL/NCAR
  15. 15. HDF-EOS5 Sample (continue) Reflectivity of the ground pixel 2/17/2014 CISL/NCAR
  16. 16. More HDF-EOS5 info  Visit NCL web site:  http://www.ncl.ucar.edu/Applications/HDF.shtml  For Sample Plots and Sample Scripts Currently at Beta Version  May provide user binary on request  2/17/2014 CISL/NCAR
  17. 17. 2/17/2014 CISL/NCAR
  18. 18. HDF5 Implementation  HDF5 is handled differently in NCL from HDF in many ways ◦ Not just simply copy/modify from existing HDF code 2/17/2014 CISL/NCAR
  19. 19. HDF5 Implementation  HDF5 is handled differently in NCL from HDF in many ways ◦ Not just simply copy/modify from existing HDF code  HDF5 Tools are useful: ◦ h5dump, h5ls, hdfview ◦ Help us understand data structural/attribute/value, and coding 2/17/2014 CISL/NCAR
  20. 20. HDF5 Implementation (continue)  Read Data on Demand ◦ Data structures are saved in memory, but not data itself (sometimes, it is impossible) ◦ Data are read when in need 2/17/2014 CISL/NCAR
  21. 21. HDF5 Implementation (continue)  First, HDF5 Has a Group (Root Group) 2/17/2014 CISL/NCAR
  22. 22. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… 2/17/2014 CISL/NCAR
  23. 23. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… 2/17/2014 CISL/NCAR
  24. 24. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… ◦ Links ◦ Groups 2/17/2014 CISL/NCAR
  25. 25. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… ◦ Links ◦ Groups  Let us have a visual view 2/17/2014 CISL/NCAR
  26. 26. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  27. 27. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  28. 28. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  29. 29. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  30. 30. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  31. 31. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  32. 32. Sample Script to Read HDF5 and Some Print-out  f = addfile(“MOD13A2A.h5", "r”) ◦ ;print(f) ;NCL treats “;” as comments thereafter ◦ printVarSummary(f) 2/17/2014 CISL/NCAR
  33. 33. Sample Script to Read HDF5 and Some Print-out (continue) Variable: f filename: MOD13A2A path: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATA GROUPTYPE = MASTERGROUP …. 2/17/2014 CISL/NCAR
  34. 34. Sample Script to Read HDF5 and Some Print-out (continue) dimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 1200 2/17/2014 CISL/NCAR
  35. 35. Sample Script to Read HDF5 and Some Print-out (continue) variables: group </HDF4_DIMGROUP> group </MODIS_Grid_16DAY_1km_VI> HDF4_OBJECT_NAME : MODIS_Grid_16DAY_1km_VI HDF4_OBJECT_TYPE : Vgroup HDF4_REF_NUM : 2 2/17/2014 CISL/NCAR
  36. 36. Sample Script to Read HDF5 and Some Print-out (continue) short 1_km_16_days_EVI ( YDim:MODIS_Grid_16DAY_1km_VI, XDim:MODIS_Grid_16DAY_1km_VI ) HDF4_REF_NUM : 6 _FillValue : -3000 add_offset : 0 add_offset_err : long_name : 0 1 km 16 days EVI scale_factor : 10000 scale_factor_err : units : 0 EVI valid_range : ( -2000, 10000 ) …... 2/17/2014 CISL/NCAR
  37. 37. Sample Script to Read HDF5 and Some Print-out (continue) f = addfile(“MOD13A2A.h5", "r”) printVarSummary(f) Variable: f (0) filename: MOD13A2A path: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATA GROUPTYPE …. = MASTERGROUP dimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 1200 variables: group </HDF4_DIMGROUP> 2/17/2014 CISL/NCAR
  38. 38. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps) 2/17/2014 CISL/NCAR
  39. 39. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps) Variable: grps Type: string Total Size: 16 bytes 4 values Number of Dimensions: 1 Dimensions and sizes: [4] Coordinates: (0) /HDF4_DIMGROUP (1) /MODIS_Grid_16DAY_1km_VI (2) /MODIS_Grid_16DAY_1km_VI/Data Fields (3) /MODIS_Grid_16DAY_1km_VI/Grid Attributes 2/17/2014 CISL/NCAR
  40. 40. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2) 2/17/2014 CISL/NCAR
  41. 41. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2) Variable: grps2 Type: string Total Size: 8 bytes 2 values Number of Dimensions: 1 Dimensions and sizes: [2] Coordinates: (0) /MODIS_Grid_16DAY_1km_VI/Data Fields (1) /MODIS_Grid_16DAY_1km_VI/Grid Attributes 2/17/2014 CISL/NCAR
  42. 42. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1) 2/17/2014 CISL/NCAR
  43. 43. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1) Variable: v1 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2 Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x[XDim:MODIS_Grid_16DAY_1km_VI | 1200] Coordinates: Number Of Attributes: 13 _FillValue : -1000 long_name : 1 km 16 days red reflectance …… 2/17/2014 CISL/NCAR
  44. 44. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance” v2 = f->$vn$ printVarSummary(v2) 2/17/2014 CISL/NCAR
  45. 45. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance” v2 = f->$vn$ printVarSummary(v2) Variable: v2 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2 Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x [XDim:MODIS_Grid_16DAY_1km_VI | 1200] Coordinates: Number Of Attributes: 13 …… _FillValue : -1000 long_name : 1 km 16 days blue reflectance …… 2/17/2014 CISL/NCAR
  46. 46. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  47. 47. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  48. 48. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  49. 49. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  50. 50. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  51. 51. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  52. 52. Another Sample Script to Read HDF5 fn = "K1VHR_23SEP2008_2330_L02_SST.h5” f = addfile(fn, "r”) print(f) lon1d = f->/SST/SST_Composite.Longitude lat1d = f->/SST/SST_Composite.Latitude sst1d = f->/SST/SST_Dataset.SST 2/17/2014 CISL/NCAR
  53. 53. Print-out filename: K1VHR_23SEP2008_2330_L02_SST file global attributes: dimensions: DIM_000 = 36381 variables: group </PRODUCT_INFORMATION> group </PRODUCT_METADATA> group </PRODUCT_METADATA/PRODUCT_DETAILS> group </PRODUCT_METADATA/PROJECTION_PARAMETERS> group </SST> compound <SST_Composite> (Latitude, Longitude, SST) (DIM_000) compound <SST_Dataset> (Latitude, Longitude, SST) (DIM_000) group </SST/GP_PARAM_INFO> 2/17/2014 CISL/NCAR
  54. 54. Plot for Compound Dataset 2/17/2014 CISL/NCAR
  55. 55. Add This Lines to Generate above Plot dimlon = dimsizes(lon1d) res@mpLimitMode = "Corners” dimlat = dimsizes(lat1d) res@mpLeftCornerLatF = lat1d(0) sst1d@long_name = "SST” res@mpLeftCornerLonF = lon1d(0) sst1d@units = "K” res@mpRightCornerLatF = lat1d(dimlat(0) - 1) sst1d@_FillValue = -999. res@mpRightCornerLonF = lon1d(dimlon(0) - 1) wks = gsn_open_wks ("x11",”K1VHR”) res@sfXArray = lon1d res res@sfYArray = lat1d = True res@cnFillOn = True res@cnLinesOn plot = gsn_csm_contour_map (wks, sst1d, res) = False res@cnLineLabelsOn = False res@cnFillMode = "RasterFill” res@cnRasterSmoothingOn = True res@gsnSpreadColors = True res@gsnAddCyclic = False res@lbLabelAutoStride = True res@tiMainString = fn 2/17/2014 CISL/NCAR
  56. 56. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go” 2/17/2014 CISL/NCAR
  57. 57. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go”  NCL with HDF-EOS5 is in Beta Version ◦ Will be released early 2010 ◦ User may ask for binary 2/17/2014 CISL/NCAR
  58. 58. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go”  NCL with HDF-EOS5 is in Beta Version ◦ Will be released early 2010 ◦ User may ask for binary  NCL with HDF5 is in Alpha Version ◦ Will be released Later 2010 ◦ Try to be a friendly user for early beta test 2/17/2014 CISL/NCAR
  59. 59. Conclusion  HDF-EOS5 is handled well in NCL ◦ ◦ ◦ ◦ ◦ Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta Version 2/17/2014 CISL/NCAR
  60. 60. Conclusion  HDF-EOS5 is handled well in NCL ◦ ◦ ◦ ◦ ◦  Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta Version Lots of can be done with HDF5 in NCL ◦ Some satellite data tested ◦ Currently in Alpha Version 2/17/2014 CISL/NCAR
  61. 61. 2/17/2014 CISL/NCAR
  62. 62. 2/17/2014 CISL/NCAR
  63. 63. A Full NCL Script load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl” load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl” f = addfile("NISESSMI.h5", "r”) grps1 = getfilegroups(f, "/Southern Hemisphere/", 0) vs = grps1(0)+"/Extent” v1 = f->$vs$ grps2 = getfilegroups(f, "/Northern Hemisphere/", 0) vn = grps2(0)+"/Extent” v2 = f->$vn$ wks = gsn_open_wks ("ps","NISESSMI”) gsn_define_colormap(wks,"rainbow”) 2/17/2014 CISL/NCAR
  64. 64. A Full NCL Script (Continue) res = True res@tiMainString = "NISESSMI.h5” res@cnFillOn = True res@cnLinesOn = False res@cnLineLabelsOn = False res@gsnSpreadColors = True res@lbLabelAutoStride= True res@lbOrientation = "Vertical” res@cnFillMode = "RasterFill" ; faster res@cnRasterSmoothingOn = True res@tiMainString = vs plot1 = gsn_csm_contour (wks, tofloat(v1), res) res@tiMainString = vn plot2 = gsn_csm_contour (wks, tofloat(v2), res) 2/17/2014 CISL/NCAR

×