Interoperable data have been a long-time goal in many scientific communities. The recent growth in analysis, visualization and mash-up applications that expect data stored in a standardized manner has brought the interoperability issue to the fore. On the other hand, producing interoperable data is often regarded as a sideline task in a typical research team for which resources are not readily available. The HDF Group is developing a software tool aimed at lessening the burden of creating data in standards-compliant, interoperable HDF5 files. The tool, named HDF Product Designer, lowers the threshold needed to design such files by providing a user interface that combines the rich HDF5 feature set with applicable metadata conventions. Users can quickly devise new HDF5 files while at the same time seamlessly incorporating the latest best practices and conventions from their community. That is what the term interoperability in the first mile means: enabling generation of interoperable data in HDF5 files from the onset of their production. The tool also incorporates collaborative features, allowing team approach in the file design, as well as easy transfer of best practices as they are being developed. The current state of the tool and the plans for future development will be presented. Constructive input from interested parties is always welcome.

1. The HDF Group
www.hdfgroup.orgDecember 17, 2014 American Geophysical Union Fall Meeting
HDF Product Designer:
Interoperability in the First Mile
H. Joe Lee (hyoklee@hdfgroup.org),
Aleksandar Jelenak, and Ted Habermann
The HDF Group

2. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Repurposing
Data
Collection

3. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection

4. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
Experts Non-Experts

5. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
# Users
Experts Non-Experts

6. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
# Users
Experts Non-Experts
Standards and
Conventions

7. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
# Users
Experts Non-Experts
? Standards and
Conventions

8. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
To facilitate collaborative design of
interoperable and standards-compliant
data products in HDF5 as early as possible
in the mission development process.

9. www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
Data
Collection
To facilitate collaborative design of
interoperable and standards-compliant
data products in HDF5 as early as possible
in the mission development process.
Interoperability in the First Mile

10. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
Mission Requirements
• Science objectives
• Data processing
• Data discovery & distribution
• Data documentation
• User engagement,
preparedness, feedback

11. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
Interoperability
Mission Requirements
• Standards
• Conventions
• Best Practices
• Metadata
• Software Tools
• netCDF4, CF
• Science objectives
• Data processing
• Data discovery & distribution
• Data documentation
• User engagement,
preparedness, feedback

12. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
HDF FeaturesInteroperability
Mission Requirements
• Datatypes
• Groups
• Attributes
• Scale/offset
• Dimension scales
• Compression
• Chunking
• Standards
• Conventions
• Best Practices
• Metadata
• Software Tools
• netCDF4, CF
• Science objectives
• Data processing
• Data discovery & distribution
• Data documentation
• User engagement,
preparedness, feedback

13. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
HDF FeaturesInteroperability
Mission Requirements
HDF Product
Designer
• Datatypes
• Groups
• Attributes
• Scale/offset
• Dimension scales
• Compression
• Chunking
• Standards
• Conventions
• Best Practices
• Metadata
• Software Tools
• netCDF4, CF
• Science objectives
• Data processing
• Data discovery & distribution
• Data documentation
• User engagement,
preparedness, feedback

14. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
Individuals

15. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
Individuals
Teams

16. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
Individuals
Teams
Projects
Programs

17. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Desktop
Client

18. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Desktop
Client
Restful
Service
(Tornado/Py
thon)

19. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store
(PostgreSQL)
Desktop
Client
Restful
Service
(Tornado/Py
thon)
HDF5
Server

20. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store
(PostgreSQL)
Desktop
Client
Restful
Service
(Tornado/Py
thon)
HDF5
JSON
HDF4 MAP
XML
NcML HDF5
Server
Flexible
Input

21. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store
(PostgreSQL)
Desktop
Client
HDF5
JSON
Fortran IDL MATLAB Python
Restful
Service
(Tornado/Py
thon)
CSV
(Excel)
HDF5
JSON
HDF4 MAP
XML
NcML HDF5
Server
HDF5 File
Template
Flexible
Input
Flexible Output

22. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Convention Support
• Initial:
• NetCDF User’s Guide (NUG)
• Attribute Convention for Data Discovery (ACDD)
• Object Convention for Data Discovery (OCDD)
• Climate and Forecast (CF)
• HDF-EOS
• Implementation:
• Conventions for groups and variables
• Convention for connected variables (e.g.
coordinate dimensions)
• Compliance checkers on entire file
• Support for community components

23. www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Conclusion
• enable individuals and mission teams to
design products quickly and easily
• enable collaboration at many levels
• promote data management best practices
HDF Product Designer is being built using a
flexible architecture to support multiple front
and back ends. It will: