The HDF Group

Interoperability with
netCDF-4
Kent Yang, Larry Knox, Elena Pourmal
The HDF Group
The 15th HDF and HDF-EOS ...
Outline
• Background
• netCDF-4
• CF

• Use cases

• Experience with HDF-EOS5 products
• Experience with JPSS products
• C...
Clarification –netCDF format
• netCDF-3 format
• Simple self-describing data format based on
netCDF classic data model

• ...
Clarification – netCDF packages
• netCDF software packages
• netCDF-C
• Support both netCDF-3 and netCDF-4 formats
• C++/F...
Why netCDF-4
• Big user community
• User-friendly data models
• Tools
• Home-grown and third-party visualization and
analy...
NetCDF Classic Model

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

6

www.hdfgroup.org
NetCDF enhanced model

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

7

www.hdfgroup.org
CF conventions
• Metadata conventions for earth science data.
• Sharing of files created with the NetCDF APIs, but
not spe...
• In this tutorial, we only review the key CF
attributes that affect the access of NASA and
other Earth Science HDF and HD...
Key CF data description attributes
Attribute

Description

Units

A string that represents the quantity of measurement. A
...
Reduction of dataset size
Attribute

Description

scale_factor

If present for a variable, the data are to be multiplied b...
“Units” for coordinate variables
• Horizontal
• Latitude – “degrees_north”
• Longitude – “degrees_east”

• Vertical
• Pres...
“coordinates” attribute
• List the associated coordinate variable names
of the variable
• An example
•

Variable: Temperat...
The HDF Group

Interoperability of HDF5
with netCDF-4
General Information

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

14

...
Review Concepts
• netCDF classic model
• Shared dimension

• netCDF enhanced model
• Group hierarchy

Apr. 17-19, 2012

HD...
Use cases to access HDF5 via netCDF-4
1. General HDF5
• Follow neither netCDF data models nor CF conventions

2. netCDF-4 ...
How to demonstrate
• Simple HDF5 files
• netCDF tools
• netCDF C
• ncdump

• netCDF Java
• IDV

Apr. 17-19, 2012

HDF/HDF-...
• There are some limitations for netCDF4 to
access HDF5 files

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

18

www.hdfgroup...
General HDF5
• Add phony dimension names to variables
• Generally cannot be opened by IDV

Apr. 17-19, 2012

HDF/HDF-EOS W...
netCDF-4 HDF5
• HDF5 that follows netCDF enhanced model
• ncdump can pick up the dimension information
• cannot be opened ...
netCDF-4 CF HDF5
• HDF5 that follows netCDF enhanced model
and CF conventions
• cannot be opened by IDV

Apr. 17-19, 2012
...
• We will use several demos to show the
differences for the last two cases
• netCDF(classic) HDF5
• netCDF(classic) CF HDF...
Demo dataset

Latitude/Longitude
10,12.5,15,17.5,20,22.5,

……
42.5, 45.0, 47.5
Data:
1,2,3,4,5,6,7,8,9,10,
11,12,13,14,15,...
Use case 4 and 5 IDV demo 1
CF category
Units for latitude and
longitude

Apr. 17-19, 2012

netCDF classic ( Case 4)
Units...
Use case 4 and 5 IDV demo 1
CF category

netCDF classic ( Case 4)

netCDF classic CF (Case 5)

Units for latitude and
long...
Use case 4 and 5 IDV demo 2
CF category

netCDF classic (Case 4)

netCDF classic CF (Case 5)

_FillValue

No _FillValue at...
Use case 4 and 5 IDV demo 2
CF category

netCDF classic (Case 4)

netCDF classic CF (Case 5)

_FillValue

Treat _FillValue...
Use case 4 and 5 IDV demo 3
CF category
scale_factor and
add_offset

scale_factor = 10.0
add_offset = 1000.0

netCDF class...
Use case 4 and 5 IDV demo 3
CF category
scale_factor and
add_offset

Apr. 17-19, 2012

netCDF classic(Case 4)
Doesn‟t appl...
Use case 4 and 5 IDV demo 4
CF category
coordinates

Apr. 17-19, 2012

netCDF classic(Case 4)
No „coordinates‟ attribute

...
Use case 4 and 5 IDV demo 4
CF category
Coordinates

Apr. 17-19, 2012

netCDF classic(Case 4)
May not pick up the correct
...
netCDF HDF5 vs netCDF CF HDF5
• CF attributes are key to make IDV
correctly display the data

Apr. 17-19, 2012

HDF/HDF-EO...
Summary of use cases
• There are some limitations for netCDF4 to access HDF5 files
Use Cases

ncdump

IDV

General HDF5

C...
The HDF Group

Interoperability of HDF5
with netCDF-4
Experience with HDF-EOS5

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV
...
netCDF4 to access HDF-EOS5 files
• Augmentation
• One file can be accessed by both EOS5 and netCDF-4
• Accessed by netCDF4...
An HDF-EOS5 file structure
HDFEOS
GRIDS

CloudFractionAndPressure
Data Fields
CloudFraction
CloudPressure

Because of the ...
An example: Augment an HDF-EOS5 Grid
• The HDF-EOS5 saves the
coordinate information
XDim and YDim in an
equation
HDFEOS
G...
• How does the augmented HDF-EOS5 file
follow CF conventions?
• The Aura teams(HIRDLS etc.) add key CF
attributes when cre...
Use cases to access HDF5 via netCDF-4
1. General HDF5
• Follow neither netCDF data models nor CF conventions

2. netCDF-4 ...
The HDF Group

Interoperability of HDF5
with netCDF-4
Experience with JPSS

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

40
...
Applications for JPSS files

• Many potentially useful applications are
netCDF based
• Structure of JPSS files allows for ...
JPSS obstacles to using netCDF-4 tools
1. Limitations of netCDF-4 – HDF5 may have
objects unknown to netCDF-4 (Use case 1)...
JPSS obstacles to using netCDF-4 tools
1. Limitations of netCDF-4 - objects unknown to
netCDF-4
2. Files are not netCDF Cl...
JPSS obstacles to using netCDF-4 tools
1. Limitations of netCDF-4 - objects unknown to
netCDF-4
2. Not netCDF Classic Mode...
Modification of JPSS files to overcome obstacles

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

45

www.hdfgroup.org
JPSS file structure
/
/All_Data
VIIRS-M3-SDR_All
Raw data
/Data_Products
VIIRS-M3-SDR
References to raw data

Groups
Apr. ...
JPSS file structure
1. Hide problem objects

/
/All_Data
VIIRS-M3-SDR_All
Raw data
/Data_Products
VIIRS-M3-SDR
References ...
JPSS file structure
2. Hide structure that does not conform to
Classic Model

/
/All_Data
VIIRS-M3-SDR_All
Raw data
/Data_...
JPSS file structure
3. Import external information from product profiles
/
/All_Data
VIIRS-M3-SDR_All Group

Raw data data...
Tool for obstacles 1 – 3: H5augjpss

1. Hides problem objects
2. Makes structure conform to Classic Model
3. Imports exter...
Key CF data description attributes

Variable
Latitude
Longitude
Radiance

Apr. 17-19, 2012

Required Attribute
Units
Units...
Summary
Obstacles to making JPSS files readable by
netCDF-4 can be addressed by:
• hiding file structure and objects unkno...
The HDF Group

Interoperability of HDF5
with netCDF-4
Current Status and Future Directions

Apr. 17-19, 2012

HDF/HDF-EOS ...
Current Status
• Opportunities for interoperability primarily
involve reading HDF5 produced files using
netCDF-4 and netCD...
Future Directions

Continue working with Unidata to reduce
obstacles to HDF5/netCDF-4 interoperability

Apr. 17-19, 2012

...
The HDF Group

Thank You!

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

56

www.hdfgroup.org
Acknowledgements
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by...
The HDF Group

Questions/comments?

Apr. 17-19, 2012

HDF/HDF-EOS Workshop XV

58

www.hdfgroup.org
Upcoming SlideShare
Loading in...5
×

Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products

387

Published on

Published in: Technology, Art & Photos
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
387
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • At the end of the slide, how many people are familiar with netCDF-4?
  • May skip this slide quickly: not mention data model.Several popular tools that are used widely by Earth Science community.
  • The shared dimensions can be easily used by Earth Science applications to specify the coordinate variables.
  • Group hierarchy and User-defined datatypes are key concepts to addTo the enhanced model.After this slide, ask the audiences how many people heard of CF conventions, how many people would like to know how CF attributes can affectthe visualization results.
  • May quickly go through this slide if audiences know CF attributes._FillVaule only specifies one distinct missing or undefined value. Using valid_min, valid_max or valid_range multiple distinct missing or undefined values can be filtered out
  • Be careful when using these two attributes. Joe will share with you MODIS examples.
  • Also make sure to use the exact format. Some tools are picky.
  • At the end of slides, just mention other CF requirement: The variable and attribute names can only have number, letter and underscore. No other characters are allowed.
  • HDFView: t_int.h5ncdump –h t_int.h5 Not workingTdset.h5 ncdump –h tdset.h5 working
  • Demos:Lat/Lon: screenshots and IDV demoUnits that don’t follow CF, IDV cannot open the file.Units that follow CF, IDV can.
  • Demos:Lat/Lon: screenshots and IDV demoUnits that don’t follow CF, IDV cannot open the file.Units that follow CF, IDV can.
  • Demos:_FillValue: 1) Use HDFView to show the data, 2) IDV first shows the data without the fillvalue attribute 3) Add _FillValue attribute at HDFView. Change the HDF5 dataset name to another name.4) Save the file(not to another HDF5 file) 5) Open with the IDV again, this time the _FillValue is not printed.
  • Demos:_FillValue: 1) Use HDFView to show the data, 2) IDV first shows the data without the fillvalue attribute 3) Add _FillValue attribute at HDFView. Change the HDF5 dataset name to another name.4) Save the file(not to another HDF5 file) 5) Open with the IDV again, this time the _FillValue is not printed.
  • Demos:Show the scale factor and add_offset. The initial value and the final value.2.Show the wrong attribute name and the right attribute name3.. Demo:file with the wrong attribute names(1,2,3…)file with the right attribute names(1010,1020, etc.)
  • Demos:Show the scale factor and add_offset. The initial value and the final value.2.Show the wrong attribute name and the right attribute name3.. Demo:file with the wrong attribute names(1,2,3…)file with the right attribute names(1010,1020, etc.)
  • Demos:Screenshots:Let the audience note the “coordinates” attribute.4. Show IDV, don’t display the value plot(IDV has a bug), just show the vertical coordinate
  • Demos:Screenshots:Let the audience note the “coordinates” attribute.4. Show IDV, don’t display the value plot(IDV has a bug), just show the vertical coordinate
  • The netCDF-4 requires the association of dimensions with coordinate variables.
  • Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products

    1. 1. The HDF Group Interoperability with netCDF-4 Kent Yang, Larry Knox, Elena Pourmal The HDF Group The 15th HDF and HDF-EOS Workshop April 17-19, 2012 Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 1 www.hdfgroup.org
    2. 2. Outline • Background • netCDF-4 • CF • Use cases • Experience with HDF-EOS5 products • Experience with JPSS products • Current Status and future directions Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 2 www.hdfgroup.org
    3. 3. Clarification –netCDF format • netCDF-3 format • Simple self-describing data format based on netCDF classic data model • netCDF-4 format • Uses HDF5 as a storage layer • Exploits • Compression, chunking, parallel-IO • Group hierarchy, user-defined data types, etc. • Supports both netCDF enhanced and netCDF classic data models • Interoperability with netCDF-4 format in this talk Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 3 www.hdfgroup.org
    4. 4. Clarification – netCDF packages • netCDF software packages • netCDF-C • Support both netCDF-3 and netCDF-4 formats • C++/Fortran Wrappers • netCDF-Java • Support both netCDF-3 and netCDF-4 formats • The implementation of the Common Data Model • netCDF version 4 • Generally mean the version 4 package of the netCDF-C library (Not only support netCDF-4 format) Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 4 www.hdfgroup.org
    5. 5. Why netCDF-4 • Big user community • User-friendly data models • Tools • Home-grown and third-party visualization and analysis tools - ncdump, ncgen, IDV, Panoply, Ferret etc. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 5 www.hdfgroup.org
    6. 6. NetCDF Classic Model Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 6 www.hdfgroup.org
    7. 7. NetCDF enhanced model Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 7 www.hdfgroup.org
    8. 8. CF conventions • Metadata conventions for earth science data. • Sharing of files created with the NetCDF APIs, but not specifically to netCDF. • The CF conventions are now increasingly gaining acceptance. • URL: http://cf-pcmdi.llnl.gov/ Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 8 www.hdfgroup.org
    9. 9. • In this tutorial, we only review the key CF attributes that affect the access of NASA and other Earth Science HDF and HDF-EOS data via popular visualization tools. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 9 www.hdfgroup.org
    10. 10. Key CF data description attributes Attribute Description Units A string that represents the quantity of measurement. A variable with no units attribute is assumed to be dimensionless. long_name A descriptive name that indicates a variable‟s content. standard_name A standard name that references a description of a variable‟s content in the standard name table of CF conventions. _FillValue A value used to represent missing or undefined data. valid_min Smallest valid value of a variable. valid_max Largest valid value of a variable. valid_range Smallest and largest valid values of a variable. Use these attributes if possible, especially use _FillValue, valid_min,valid_max if you have missing value(s). Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 10 www.hdfgroup.org
    11. 11. Reduction of dataset size Attribute Description scale_factor If present for a variable, the data are to be multiplied by this factor after the data are read by an application. add_offset If present for a variable, this number is to be added to the data after it is read by an application. If both scale_factor and add_offset attributes are present, the data are first scaled before the offset is added. The equation that describes the usage of scale_factor and add_offset is: Final_data_value = “scale_factor” * Raw_data_value + “add_offset”; Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 11 www.hdfgroup.org
    12. 12. “Units” for coordinate variables • Horizontal • Latitude – “degrees_north” • Longitude – “degrees_east” • Vertical • Pressure – “hPa” • Height(depth) – “Meter” (m) or “kilometer”(km) • Time • seconds(minutes etc.) since a time point • An example • “seconds since 1992-10-8 15:15:42.5 -6:00” Use these attributes with the CF values if possible, without following CF conventions for these attributes, some tools cannot properly visualize the data. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 12 www.hdfgroup.org
    13. 13. “coordinates” attribute • List the associated coordinate variable names of the variable • An example • Variable: Temperature • • Associated Coordinate variables: “latitude”,“longitude”,”pressure” coordinates = “latitude longitude pressure” Include this attribute if possible. For some data products, this is the key attribute to specific the coordinates of a variable. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 13 www.hdfgroup.org
    14. 14. The HDF Group Interoperability of HDF5 with netCDF-4 General Information Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 14 www.hdfgroup.org
    15. 15. Review Concepts • netCDF classic model • Shared dimension • netCDF enhanced model • Group hierarchy Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 15 www.hdfgroup.org
    16. 16. Use cases to access HDF5 via netCDF-4 1. General HDF5 • Follow neither netCDF data models nor CF conventions 2. netCDF-4 HDF5 • Follow netCDF enhanced data model 3. netCDF-4 CF HDF5 • Follow netCDF enhanced model and CF conventions 4. netCDF (classic) HDF5 • Follow netCDF classic model 5. netCDF (classic) CF HDF5 • Follow netCDF classic model and CF conventions Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 16 www.hdfgroup.org
    17. 17. How to demonstrate • Simple HDF5 files • netCDF tools • netCDF C • ncdump • netCDF Java • IDV Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 17 www.hdfgroup.org
    18. 18. • There are some limitations for netCDF4 to access HDF5 files Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 18 www.hdfgroup.org
    19. 19. General HDF5 • Add phony dimension names to variables • Generally cannot be opened by IDV Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 19 www.hdfgroup.org
    20. 20. netCDF-4 HDF5 • HDF5 that follows netCDF enhanced model • ncdump can pick up the dimension information • cannot be opened by IDV Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 20 www.hdfgroup.org
    21. 21. netCDF-4 CF HDF5 • HDF5 that follows netCDF enhanced model and CF conventions • cannot be opened by IDV Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 21 www.hdfgroup.org
    22. 22. • We will use several demos to show the differences for the last two cases • netCDF(classic) HDF5 • netCDF(classic) CF HDF5 • ncdump can dump all demo files Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 22 www.hdfgroup.org
    23. 23. Demo dataset Latitude/Longitude 10,12.5,15,17.5,20,22.5, …… 42.5, 45.0, 47.5 Data: 1,2,3,4,5,6,7,8,9,10, 11,12,13,14,15,16 Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 23 www.hdfgroup.org
    24. 24. Use case 4 and 5 IDV demo 1 CF category Units for latitude and longitude Apr. 17-19, 2012 netCDF classic ( Case 4) Units is “degrees” HDF/HDF-EOS Workshop XV netCDF classic CF (Case 5) Latitude: “degrees_north” Longitude: “degrees_east” 24 www.hdfgroup.org
    25. 25. Use case 4 and 5 IDV demo 1 CF category netCDF classic ( Case 4) netCDF classic CF (Case 5) Units for latitude and longitude Cannot open the file Correctly display the data Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 25 www.hdfgroup.org
    26. 26. Use case 4 and 5 IDV demo 2 CF category netCDF classic (Case 4) netCDF classic CF (Case 5) _FillValue No _FillValue attribute Have _FillValue Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 26 www.hdfgroup.org
    27. 27. Use case 4 and 5 IDV demo 2 CF category netCDF classic (Case 4) netCDF classic CF (Case 5) _FillValue Treat _FillValue as the real data Correctly filter out the _FillValue Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 27 www.hdfgroup.org
    28. 28. Use case 4 and 5 IDV demo 3 CF category scale_factor and add_offset scale_factor = 10.0 add_offset = 1000.0 netCDF classic(Case 4) Attribute names for scale_factor and add_offset don‟t follow CF conventions netCDF classic CF(Case 5) Have correct scale_factor and add_offset attribute names Attributes: Attributes: wrong_scale_name = 10.0 Wrong_offset_name = 1000.0 scale_factor = 10.0 add_offset = 1000.0 1,2,3,4,5,6,7,8,9,10, 11,12,13,14,15,16 Scale Offset 1010,1020,1030, …… Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 28 www.hdfgroup.org
    29. 29. Use case 4 and 5 IDV demo 3 CF category scale_factor and add_offset Apr. 17-19, 2012 netCDF classic(Case 4) Doesn‟t apply the scale_factor and add_offset HDF/HDF-EOS Workshop XV netCDF classic CF(Case 5) Correctly apply scale_factor and add_offset 29 www.hdfgroup.org
    30. 30. Use case 4 and 5 IDV demo 4 CF category coordinates Apr. 17-19, 2012 netCDF classic(Case 4) No „coordinates‟ attribute HDF/HDF-EOS Workshop XV netCDF classic CF(Case 5) Have „coordinates‟ 30 www.hdfgroup.org
    31. 31. Use case 4 and 5 IDV demo 4 CF category Coordinates Apr. 17-19, 2012 netCDF classic(Case 4) May not pick up the correct coordinate HDF/HDF-EOS Workshop XV netCDF classic CF(Case 5) Correctly pick up the right coordinate 31 www.hdfgroup.org
    32. 32. netCDF HDF5 vs netCDF CF HDF5 • CF attributes are key to make IDV correctly display the data Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 32 www.hdfgroup.org
    33. 33. Summary of use cases • There are some limitations for netCDF4 to access HDF5 files Use Cases ncdump IDV General HDF5 Can view (with phony dimensions) Generally cannot view netCDF-4 HDF5 Can view Generally cannot view netCDF-4 CF HDF5 Can view Generally cannot view netCDF Classic HDF5 Can view Can view some files but the visualization may not be right netCDF Classic CF HDF5 Can view Can view Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 33 www.hdfgroup.org
    34. 34. The HDF Group Interoperability of HDF5 with netCDF-4 Experience with HDF-EOS5 Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 34 www.hdfgroup.org
    35. 35. netCDF4 to access HDF-EOS5 files • Augmentation • One file can be accessed by both EOS5 and netCDF-4 • Accessed by netCDF4 • netCDF data model should be followed netCDF4 HDF-EOS5 HDF5 Augmentation HDF5 HDF-EOS5 file Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 35 www.hdfgroup.org
    36. 36. An HDF-EOS5 file structure HDFEOS GRIDS CloudFractionAndPressure Data Fields CloudFraction CloudPressure Because of the group hierarchy, we can only augment the HDF-EOS5 file by following the netCDF enhanced model Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 36 www.hdfgroup.org
    37. 37. An example: Augment an HDF-EOS5 Grid • The HDF-EOS5 saves the coordinate information XDim and YDim in an equation HDFEOS GRIDS • The tool retrieves the CloudFractionAndPressure values of XDim and YDim Data Fields CloudFraction[XDim][YDim] • It creates coordinate CloudPressure[XDim][YDim] variables XDim and YDim XDim with the raw values YDim • Then it associates the coordinate variables with the data variables • Then netCDF-4 can follow the netCDF enhanced model to access the HDFEOS5 data www.hdfgroup.org
    38. 38. • How does the augmented HDF-EOS5 file follow CF conventions? • The Aura teams(HIRDLS etc.) add key CF attributes when creating the original HDF-EOS5 file. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 38 www.hdfgroup.org
    39. 39. Use cases to access HDF5 via netCDF-4 1. General HDF5 • Follow neither netCDF data models nor CF conventions 2. netCDF-4 HDF5 • Follow netCDF enhanced data model 3. netCDF-4 CF HDF5 • Follow netCDF enhanced model and CF conventions 4. netCDF (classic) HDF5 • Follow netCDF classic model 5. netCDF (classic) CF HDF5 • Follow netCDF classic model and CF conventions Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 39 www.hdfgroup.org
    40. 40. The HDF Group Interoperability of HDF5 with netCDF-4 Experience with JPSS Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 40 www.hdfgroup.org
    41. 41. Applications for JPSS files • Many potentially useful applications are netCDF based • Structure of JPSS files allows for effective modification – data is separated from objects that are unknown to netCDF-4 Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 41 www.hdfgroup.org
    42. 42. JPSS obstacles to using netCDF-4 tools 1. Limitations of netCDF-4 – HDF5 may have objects unknown to netCDF-4 (Use case 1) • References • Multi-dimensional attributes • Chunked datasets (variables) with unlimited maximum size Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 42 www.hdfgroup.org
    43. 43. JPSS obstacles to using netCDF-4 tools 1. Limitations of netCDF-4 - objects unknown to netCDF-4 2. Files are not netCDF Classic Model conformant (Use case 4) a) Group structure b) Important information including dimensions in separate xml file c) Geolocation data in separate file or group Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 43 www.hdfgroup.org
    44. 44. JPSS obstacles to using netCDF-4 tools 1. Limitations of netCDF-4 - objects unknown to netCDF-4 2. Not netCDF Classic Model conformant 3. Key CF Attributes (Use Case 5) • Latitude Longitude • Measurement units • Valid_min valid_max (determined by data type and fill values) • Scale factors Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 44 www.hdfgroup.org
    45. 45. Modification of JPSS files to overcome obstacles Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 45 www.hdfgroup.org
    46. 46. JPSS file structure / /All_Data VIIRS-M3-SDR_All Raw data /Data_Products VIIRS-M3-SDR References to raw data Groups Apr. 17-19, 2012 Datasets HDF/HDF-EOS Workshop XV 46 www.hdfgroup.org
    47. 47. JPSS file structure 1. Hide problem objects / /All_Data VIIRS-M3-SDR_All Raw data /Data_Products VIIRS-M3-SDR References to raw data Groups Apr. 17-19, 2012 Datasets HDF/HDF-EOS Workshop XV 47 www.hdfgroup.org
    48. 48. JPSS file structure 2. Hide structure that does not conform to Classic Model / /All_Data VIIRS-M3-SDR_All Raw data /Data_Products VIIRS-M3-SDR References to raw data Groups Apr. 17-19, 2012 Datasets HDF/HDF-EOS Workshop XV 48 www.hdfgroup.org
    49. 49. JPSS file structure 3. Import external information from product profiles / /All_Data VIIRS-M3-SDR_All Group Raw data datasets Geolocation datasets Dimension name and length and other attributes from product profiles /Data_Products VIIRS-M3-SDR Reference datasets Groups Apr. 17-19, 2012 Datasets XV HDF/HDF-EOS Workshop attributes 49 www.hdfgroup.org
    50. 50. Tool for obstacles 1 – 3: H5augjpss 1. Hides problem objects 2. Makes structure conform to Classic Model 3. Imports external information Files are modified! Copy to preserve original • Final obstacle: CF compliance Manual additions with HDFView or h5edit Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 50 www.hdfgroup.org
    51. 51. Key CF data description attributes Variable Latitude Longitude Radiance Apr. 17-19, 2012 Required Attribute Units Units Coordinates add_offset scale_factor valid_min valid_max Type string string string float float ushort ushort HDF/HDF-EOS Workshop XV Value degrees_north degrees_east Latitude Longitude -0.08 2.8339462E-4 0 65527 51 www.hdfgroup.org
    52. 52. Summary Obstacles to making JPSS files readable by netCDF-4 can be addressed by: • hiding file structure and objects unknown to netCDF-4. • importing information to interpret the data from external files. Product specific information for CF compliance is currently added with HDFView or h5edit. Unknown objects that are hidden can be unhidden with saved object address. No tool is planned to remove imported data. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 52 www.hdfgroup.org
    53. 53. The HDF Group Interoperability of HDF5 with netCDF-4 Current Status and Future Directions Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 53 www.hdfgroup.org
    54. 54. Current Status • Opportunities for interoperability primarily involve reading HDF5 produced files using netCDF-4 and netCDF-4 based tools. • Issues are periodically identified and some have been solved. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 54 www.hdfgroup.org
    55. 55. Future Directions Continue working with Unidata to reduce obstacles to HDF5/netCDF-4 interoperability Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 55 www.hdfgroup.org
    56. 56. The HDF Group Thank You! Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 56 www.hdfgroup.org
    57. 57. Acknowledgements This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 57 www.hdfgroup.org
    58. 58. The HDF Group Questions/comments? Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 58 www.hdfgroup.org
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×