• Like

Implementation of HDF-EOS5 and HDF5 into NCL

  • 54 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
54
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Implementation of HDFEOS5 and HDF5 in NCL Wei Huang, Dave Brown, Mary Haley, Rick Brownrigg CISL/NCAR Dennis Shea, Adam Phillips CGD/NCAR huangwei@ucar.edu
  • 2. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information 2/17/2014 CISL/NCAR
  • 3. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization 2/17/2014 CISL/NCAR
  • 4. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source 2/17/2014 CISL/NCAR
  • 5. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) 2/17/2014 CISL/NCAR
  • 6. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) 2/17/2014 CISL/NCAR
  • 7. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics 2/17/2014 CISL/NCAR
  • 8. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics ◦ Hundreds of example scripts, lots of sample data 2/17/2014 CISL/NCAR
  • 9. What is NCL ◦ NCL stands for NCAR Command Language  check http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, or open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) ◦ Newly added HDF5 and HDF-EOS5 (in 2009) ◦ High quality graphics ◦ Hundreds of example scripts, lots of sample data ◦ PyNIO, Python interface for same NCL file I/O 2/17/2014 CISL/NCAR
  • 10. HDF-EOS5 Implementation  Start with HDF-EOS Code ◦ Reconstruct function structure ◦ Create new HDF-EOS5 functions ◦ Variables follow the NetCDF data model 2/17/2014 CISL/NCAR
  • 11. HDF-EOS5 Implementation  Start with HDF-EOS Code ◦ Reconstruct function structure ◦ Create new HDF-EOS5 functions ◦ Variables follow the NetCDF data model  How NCL open HDF-EOS5 files ◦ Command: f = addfile(filename, “r”)  Filename has the form: body.suffix     NetCDF has suffix: nc, cdf, netcdf Grib has: gr, gr1, grb, grb1, grib, gr2, grb2, grib2 HDF has: hdf, h4 HDF-EOS has: hdfeos, he2, he4 2/17/2014 CISL/NCAR
  • 12. HDF-EOS5 Implementation (continue)  HDF-EOS5 has suffix: he5 ◦ Function addfile check file suffix, and when suffix is he5, it knows to move function pointers put to HDF-EOS5 functions ◦ HDF-EOS5 uses HDF5, but we do not need to deal with HDF5 directly 2/17/2014 CISL/NCAR
  • 13. HDF-EOS5 Sample 2/17/2014 CISL/NCAR
  • 14. HDF-EOS5 Sample (continue) 2/17/2014 CISL/NCAR
  • 15. HDF-EOS5 Sample (continue) Reflectivity of the ground pixel 2/17/2014 CISL/NCAR
  • 16. More HDF-EOS5 info  Visit NCL web site:  http://www.ncl.ucar.edu/Applications/HDF.shtml  For Sample Plots and Sample Scripts Currently at Beta Version  May provide user binary on request  2/17/2014 CISL/NCAR
  • 17. 2/17/2014 CISL/NCAR
  • 18. HDF5 Implementation  HDF5 is handled differently in NCL from HDF in many ways ◦ Not just simply copy/modify from existing HDF code 2/17/2014 CISL/NCAR
  • 19. HDF5 Implementation  HDF5 is handled differently in NCL from HDF in many ways ◦ Not just simply copy/modify from existing HDF code  HDF5 Tools are useful: ◦ h5dump, h5ls, hdfview ◦ Help us understand data structural/attribute/value, and coding 2/17/2014 CISL/NCAR
  • 20. HDF5 Implementation (continue)  Read Data on Demand ◦ Data structures are saved in memory, but not data itself (sometimes, it is impossible) ◦ Data are read when in need 2/17/2014 CISL/NCAR
  • 21. HDF5 Implementation (continue)  First, HDF5 Has a Group (Root Group) 2/17/2014 CISL/NCAR
  • 22. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… 2/17/2014 CISL/NCAR
  • 23. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… 2/17/2014 CISL/NCAR
  • 24. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… ◦ Links ◦ Groups 2/17/2014 CISL/NCAR
  • 25. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)  Second, a Group Has:  ◦ Attributes  Name, Type, Value, …… ◦ Datasets  Attributes  Name, Type, Dimensions, Value, …… ◦ Links ◦ Groups  Let us have a visual view 2/17/2014 CISL/NCAR
  • 26. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 27. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 28. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 29. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 30. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 31. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 32. Sample Script to Read HDF5 and Some Print-out  f = addfile(“MOD13A2A.h5", "r”) ◦ ;print(f) ;NCL treats “;” as comments thereafter ◦ printVarSummary(f) 2/17/2014 CISL/NCAR
  • 33. Sample Script to Read HDF5 and Some Print-out (continue) Variable: f filename: MOD13A2A path: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATA GROUPTYPE = MASTERGROUP …. 2/17/2014 CISL/NCAR
  • 34. Sample Script to Read HDF5 and Some Print-out (continue) dimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 1200 2/17/2014 CISL/NCAR
  • 35. Sample Script to Read HDF5 and Some Print-out (continue) variables: group </HDF4_DIMGROUP> group </MODIS_Grid_16DAY_1km_VI> HDF4_OBJECT_NAME : MODIS_Grid_16DAY_1km_VI HDF4_OBJECT_TYPE : Vgroup HDF4_REF_NUM : 2 2/17/2014 CISL/NCAR
  • 36. Sample Script to Read HDF5 and Some Print-out (continue) short 1_km_16_days_EVI ( YDim:MODIS_Grid_16DAY_1km_VI, XDim:MODIS_Grid_16DAY_1km_VI ) HDF4_REF_NUM : 6 _FillValue : -3000 add_offset : 0 add_offset_err : long_name : 0 1 km 16 days EVI scale_factor : 10000 scale_factor_err : units : 0 EVI valid_range : ( -2000, 10000 ) …... 2/17/2014 CISL/NCAR
  • 37. Sample Script to Read HDF5 and Some Print-out (continue) f = addfile(“MOD13A2A.h5", "r”) printVarSummary(f) Variable: f (0) filename: MOD13A2A path: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATA GROUPTYPE …. = MASTERGROUP dimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 1200 variables: group </HDF4_DIMGROUP> 2/17/2014 CISL/NCAR
  • 38. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps) 2/17/2014 CISL/NCAR
  • 39. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps) Variable: grps Type: string Total Size: 16 bytes 4 values Number of Dimensions: 1 Dimensions and sizes: [4] Coordinates: (0) /HDF4_DIMGROUP (1) /MODIS_Grid_16DAY_1km_VI (2) /MODIS_Grid_16DAY_1km_VI/Data Fields (3) /MODIS_Grid_16DAY_1km_VI/Grid Attributes 2/17/2014 CISL/NCAR
  • 40. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2) 2/17/2014 CISL/NCAR
  • 41. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2) Variable: grps2 Type: string Total Size: 8 bytes 2 values Number of Dimensions: 1 Dimensions and sizes: [2] Coordinates: (0) /MODIS_Grid_16DAY_1km_VI/Data Fields (1) /MODIS_Grid_16DAY_1km_VI/Grid Attributes 2/17/2014 CISL/NCAR
  • 42. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1) 2/17/2014 CISL/NCAR
  • 43. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1) Variable: v1 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2 Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x[XDim:MODIS_Grid_16DAY_1km_VI | 1200] Coordinates: Number Of Attributes: 13 _FillValue : -1000 long_name : 1 km 16 days red reflectance …… 2/17/2014 CISL/NCAR
  • 44. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance” v2 = f->$vn$ printVarSummary(v2) 2/17/2014 CISL/NCAR
  • 45. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance” v2 = f->$vn$ printVarSummary(v2) Variable: v2 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2 Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x [XDim:MODIS_Grid_16DAY_1km_VI | 1200] Coordinates: Number Of Attributes: 13 …… _FillValue : -1000 long_name : 1 km 16 days blue reflectance …… 2/17/2014 CISL/NCAR
  • 46. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 47. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 48. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 49. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 50. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 51. HDF5 Implementation (continue) 2/17/2014 CISL/NCAR
  • 52. Another Sample Script to Read HDF5 fn = "K1VHR_23SEP2008_2330_L02_SST.h5” f = addfile(fn, "r”) print(f) lon1d = f->/SST/SST_Composite.Longitude lat1d = f->/SST/SST_Composite.Latitude sst1d = f->/SST/SST_Dataset.SST 2/17/2014 CISL/NCAR
  • 53. Print-out filename: K1VHR_23SEP2008_2330_L02_SST file global attributes: dimensions: DIM_000 = 36381 variables: group </PRODUCT_INFORMATION> group </PRODUCT_METADATA> group </PRODUCT_METADATA/PRODUCT_DETAILS> group </PRODUCT_METADATA/PROJECTION_PARAMETERS> group </SST> compound <SST_Composite> (Latitude, Longitude, SST) (DIM_000) compound <SST_Dataset> (Latitude, Longitude, SST) (DIM_000) group </SST/GP_PARAM_INFO> 2/17/2014 CISL/NCAR
  • 54. Plot for Compound Dataset 2/17/2014 CISL/NCAR
  • 55. Add This Lines to Generate above Plot dimlon = dimsizes(lon1d) res@mpLimitMode = "Corners” dimlat = dimsizes(lat1d) res@mpLeftCornerLatF = lat1d(0) sst1d@long_name = "SST” res@mpLeftCornerLonF = lon1d(0) sst1d@units = "K” res@mpRightCornerLatF = lat1d(dimlat(0) - 1) sst1d@_FillValue = -999. res@mpRightCornerLonF = lon1d(dimlon(0) - 1) wks = gsn_open_wks ("x11",”K1VHR”) res@sfXArray = lon1d res res@sfYArray = lat1d = True res@cnFillOn = True res@cnLinesOn plot = gsn_csm_contour_map (wks, sst1d, res) = False res@cnLineLabelsOn = False res@cnFillMode = "RasterFill” res@cnRasterSmoothingOn = True res@gsnSpreadColors = True res@gsnAddCyclic = False res@lbLabelAutoStride = True res@tiMainString = fn 2/17/2014 CISL/NCAR
  • 56. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go” 2/17/2014 CISL/NCAR
  • 57. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go”  NCL with HDF-EOS5 is in Beta Version ◦ Will be released early 2010 ◦ User may ask for binary 2/17/2014 CISL/NCAR
  • 58. Summary  No need to become an expert on HDFEOS5 or HDF5 interfaces ◦ User can “check” and “go”  NCL with HDF-EOS5 is in Beta Version ◦ Will be released early 2010 ◦ User may ask for binary  NCL with HDF5 is in Alpha Version ◦ Will be released Later 2010 ◦ Try to be a friendly user for early beta test 2/17/2014 CISL/NCAR
  • 59. Conclusion  HDF-EOS5 is handled well in NCL ◦ ◦ ◦ ◦ ◦ Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta Version 2/17/2014 CISL/NCAR
  • 60. Conclusion  HDF-EOS5 is handled well in NCL ◦ ◦ ◦ ◦ ◦  Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta Version Lots of can be done with HDF5 in NCL ◦ Some satellite data tested ◦ Currently in Alpha Version 2/17/2014 CISL/NCAR
  • 61. 2/17/2014 CISL/NCAR
  • 62. 2/17/2014 CISL/NCAR
  • 63. A Full NCL Script load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl” load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl” f = addfile("NISESSMI.h5", "r”) grps1 = getfilegroups(f, "/Southern Hemisphere/", 0) vs = grps1(0)+"/Extent” v1 = f->$vs$ grps2 = getfilegroups(f, "/Northern Hemisphere/", 0) vn = grps2(0)+"/Extent” v2 = f->$vn$ wks = gsn_open_wks ("ps","NISESSMI”) gsn_define_colormap(wks,"rainbow”) 2/17/2014 CISL/NCAR
  • 64. A Full NCL Script (Continue) res = True res@tiMainString = "NISESSMI.h5” res@cnFillOn = True res@cnLinesOn = False res@cnLineLabelsOn = False res@gsnSpreadColors = True res@lbLabelAutoStride= True res@lbOrientation = "Vertical” res@cnFillMode = "RasterFill" ; faster res@cnRasterSmoothingOn = True res@tiMainString = vs plot1 = gsn_csm_contour (wks, tofloat(v1), res) res@tiMainString = vn plot2 = gsn_csm_contour (wks, tofloat(v2), res) 2/17/2014 CISL/NCAR