Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS

362
views

Published on

HDF AND HDF-EOS WORKSHOP XVII …

HDF AND HDF-EOS WORKSHOP XVII
Joe Lee

Published in: Technology, Education

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
362
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Hi, my name is Joe Lee and I am going to present how to meet the CF conventions with NcML for NASA HDF and HDF-EOS products.
  • In this room, is there anyone who doesn’t know anything about the CF conventions?
    Great, you’re a very lucky person because I’m going to tell you the most important, essential part of the CF conventions so that your brain will not be damaged by the 116 page long, Complex and Frustrating conventions.

    In my opinion, meeting the CF conventions fully is as difficult as team USA winning the worldcup.
    Therefore, focusing on the following 4 key conventions will be enough in most cases.
    Bald, Fat, Short, Poor
  • So, why does the CF conventions matter?
    Practically speaking, if a data product that you produce doesn’t meet them, your netcdf tool doesn’t work.
    For example, IDV, a very nice yet free NetCDF visualization tool, rejects your data product immediately.
    Simply, the IDV will say “you are not my type.”
  • A few months ago, NASA ocean biology group has provided us their new beta NetCDF-4 products for evaluation.
    They are currently serving data in HDF4 but they want to change the format to NetCDF-4.
    Please note that NetCDF-4 uses HDF5 underneath and NcML can work with HDF and HDF-EOS.
  • It’s an extensible markup language but I treat it as extreme makeover language for HDF data.
  • Transcript

    • 1. The HDF Group www.hdfgroup.orgJuly 8, 2014 2014 Summer ESIP Federation Meeting How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS Hyo-Kyung Joe Lee and Ted Habermann The HDF Group 1
    • 2. www.hdfgroup.org2014 Summer ESIP Federation Meeting The CF Conventions 2 Big and complex! Just focus on a few key conventions: • coordinate variables & attributes (bald) • valid_range / _FillValue (fat) • scale_factor / add_offset (short) • units ($ vs. ₵)
    • 3. www.hdfgroup.org2014 Summer ESIP Federation Meeting Why the key CF conventions matter 3 If a data product doesn’t follow them, your NetCDF visualization tool like IDV is useless.
    • 4. www.hdfgroup.org2014 Summer ESIP Federation Meeting But if you follow them carefully… 4 you can visualize data instantly & correctly.
    • 5. www.hdfgroup.org2014 Summer ESIP Federation Meeting OBPG L3 NetCDF-4 Example 5 The previous screenshots are created from a NASA OBPG beta product. The product doesn’t work with IDV as is. But we made the product work by augmenting data with NcML.
    • 6. www.hdfgroup.org2014 Summer ESIP Federation Meeting NcML 6 NetCDF Markup Language XML representation of netCDF metadata It can be an input file for editing metadata.
    • 7. www.hdfgroup.org2014 Summer ESIP Federation Meeting Why NcML? 7 Isn’t CDL enough? No. NcML has more features and flexibility. Plus, NcML works with THREDDS Data Server.
    • 8. www.hdfgroup.org2014 Summer ESIP Federation Meeting What OBPG NC4 missed - #1 8 Coordinate variables are not provided. Dataset { String palette[rgb = 3]; Int16 /geophysical_data/sst[rows = 4320][columns = 8640]; Byte /geophysical_data/sst_qual[rows = 4320][columns = 8640]; Float32 /geophysical_data/Lat[rows= 4320]; Float32 /geophysical_data/Lon[columns = 8640]; } testAll/A20030602003090.L3m_MO_AT109_SST_4.nc4; The CF convention expects lat/lon variables.
    • 9. www.hdfgroup.org2014 Summer ESIP Federation Meeting Add new coordinate variables in NcML. 9 <netcdf location="c:/tomcat/webapps/thredds/share/testdata/cdmUnitTest/it/content/thr edds/public/testdata/A20030602003090.L3m_MO_AT109_SST_4.nc4" xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"> <dimension name="_geophysical_data_Lat" length="4320"/> <dimension name="_geophysical_data_Lon" length="8640"/> <group name="geophysical_data"> <variable name="Lat" shape="_geophysical_data_Lat" type="float"> <values start="90.0" increment="-0.0416"/> <attribute type="string" name="units" value="degrees_north" /> </variable> <variable name="Lon" shape="_geophysical_data_Lon" type="float"> <values start="-180.0" increment="0.0416"/> <attribute type="string" name="units" value="degrees_east" /> </variable> …
    • 10. www.hdfgroup.org2014 Summer ESIP Federation Meeting Now IDV can visualize data but… 10 No data on Sea w/ Range = [-1, -1]? Examine units.
    • 11. www.hdfgroup.org2014 Summer ESIP Federation Meeting What OBPG NC4 missed - #2 11 “units” attribute has “degrees-C” value. Accepted values in the CF Convention: Celsius / celsius / degree_centigrade / degC / degreeC / degree_C / degree_c / deg_C / deg_c Remove “s-” from “degrees-C”.
    • 12. www.hdfgroup.org2014 Summer ESIP Federation Meeting Modify attribute value with NcML. 12 … <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> </variable> … Before: After:
    • 13. www.hdfgroup.org2014 Summer ESIP Federation Meeting What OBPG NC4 missed - #3 13 The actual fill value is -1 according to HDFView. Attributes { /geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; } HDFView
    • 14. www.hdfgroup.org2014 Summer ESIP Federation Meeting Modify _FillValue attribute with NcML. 14 … <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type=“short" name=“_FillValue" value=“-1" /> </variable> … Now every value is a fill value – “missing” in IDV.
    • 15. www.hdfgroup.org2014 Summer ESIP Federation Meeting What OBPG NC4 missed - #4 15 valid_min / valid_max doesn’t match the range of actual data. Attributes { /geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; } No scale/offset attributes are defined either.
    • 16. www.hdfgroup.org2014 Summer ESIP Federation Meeting Add scale/offset attributes with NcML. 16 <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type="float" name="scale_factor" value="0.000717185"/> <attribute type="float" name="add_offset" value="-2.0"/> <attribute type="float" name="valid_min" value="-5.0"/> <attribute type="float" name="valid_max" value="50.0"/> <attribute type="short" name="_FillValue" value="-1"/> </variable> Please note that all types match as float. _FillValue’s type matches the data type.
    • 17. www.hdfgroup.org2014 Summer ESIP Federation Meeting Finally, OBPG met the CF conventions. 17
    • 18. www.hdfgroup.org2014 Summer ESIP Federation Meeting But tools behave differently. 18 The same NcML doesn’t work for GoDiva2. NC4 “Group” is the reason. GoDiva2: lat/lon outside group IDV/Panopy: lat/lon inside group Is CF/NcML ready for “Group”?
    • 19. www.hdfgroup.org2014 Summer ESIP Federation Meeting GoDiva2 19
    • 20. www.hdfgroup.org2014 Summer ESIP Federation Meeting Panoply 20
    • 21. www.hdfgroup.org2014 Summer ESIP Federation Meeting Acknowledgement 21 This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

    ×