This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.

1. Copyright 2017, The HDF Group.
The HDF Group
How can we help you?
1

2. Collaboration 2
Self-Oriented Products:
For your use and publication
No user’s guide / no metadata
Custom software to access data
Reusable Products:
Outreach to potential users in advance.
Write a good user’s guide.
Add searchable metadata.
Test with existing software.
Interoperable Products:
Think beyond your community
Metadata for use and understanding
Test all software that you can envision.

3. Product Designer Hub
3
Data Store
(users, projects, file
structures, metadata)
Desktop Client
(Easy Install)
Python IDL MATLAB
Restful Server
Export
Options
HDF5
JSON
HDF4
MAP
NcML
(Remote)
HDF online
(files, data)
HDF5
Template
OPeNDAP
DMR
CDLHDF5
Excel
Text (CSV)
DB
HDF
JSON
Import Options
Product
Specifications

4. 4At HDFEOS.org, we LOVE Earth data!
• Solutions for NASA HDF-EOS data
• 150 visitors per day
• 847 registered forum users
• Goal: Make Earth data access easy.

5. hdfeos.org 5
100+ Tool and Library examples:
ArcGIS, Google Earth, Panoply, IDV, NCO,
GrADS, ...
GDAL, C, Java, Python, MATLAB, R, Scala,
IDL, NcML, …
300+ NASA HDF Product examples
MATLAB/Python/IDL/NCL visualization
Other Cloud / Bigdata solutions
Spark+Hadoop / Elastic Search / Amazon Web
Service

6. HDF Tools 6
Distribution Discovery AnalysisArchive
Repurposing
Product
Designer
Data Structures
Metadata &
Convention
Support
ISO, HDF-EOS,
ACDD, CF
HDFView (HDF-
EOS plugin)
HDFCompass
HDF-EOS Dumper
IDL/ENVI, MATLAB,
Panoply, IDV, Ferret,
Python, User support
HDF4 Mapper OPeNDAP
Handlers,
H5Server,
ncISO,
THREDDS
Climate-Forecast
(CF)
Conventions
netCDF API
HDF-EOS
HDF4 to HDF5
Library / Tools
Metadata / Data
Augmentation
Product (Re-
)Designer
HDF4, HDF5 &
HDF-EOS
Libraries
IDL
Supporting the entire data life cycle
Product
Generation
Mission
Planning and
Design

7. HDF5 Tools 7
GUI Text
Built in Java Built in C
Separate release Part of the library release
HDF4, HDF5, NetCDF, etc. HDF5 only
HDFView HDF5 Command-line tools

8. HDFView - VIIRS Day/Night band files 8
• floating
windows
• multiple files
and datasets,
• bigger
datasets,
• faster and
consumes less
memory.

9. HDF5 command-line tools 9
Tool Purpose
h5dump, h5ls Display content of HDF5 file
h5diff Compares structure and data of two HDF5 files
h5copy Copies the whole of parts of the HDF5 file
h5repack Repacks HDF5 file (can change storage layout,
compression and other properties; can use
custom compression via dynamically loaded
filters)

10. HDF4 to HDF5 conversion tool
Although we continue to maintain HDF4, we do not plan on adding new features to
HDF4, and recommend users move to HDF5 to take advantage of the improved
features and performance of HDF5. This standard tool can be used to:
• Convert any HDF4 files to HDF5 files using HDF4 to HDF5 Mapping Specification
• Package URL: https://support.hdfgroup.org/products/hdf5_tools/h4toh5/
• Example HDF5 files converted from the NASA HDF4 and HDF-EOS2 products:
http://hdfeos.org/zoo/h4toh5/h4toh5-def-demo.php
• Guaranteed to preserve data content and data structures
• New release during the summer of 2017

11. HDF4 to CF conversion toolkit
Need to connect HDF4 data to the netCDF Library?
• A desktop version of the HDF4 OPeNDAP handler.
• Apply to NASA HDF-EOS2 and some NASA HDF4 products.
• The output files can be either netCDF-3 or netCDF-4 classic files and the
CF conventions are enforced.
• Package URL: http://hdfeos.org/software/h4cflib.php
• Example HDF5 files converted from the NASA HDF4 and HDF-EOS2
products: http://hdfeos.org/zoo/h4cf/index.php

12. More Information: http://hdfeos.org/software/aug_eos5.php
HDF5
HDF-EOS5
HDF5
netCDF4
Augmentation
HDF-EOS5 file
HDF-EOS5 Augmentation Tool
• Make HDF-EOS5 files accessable by netCDF4 using the netCDF data model.

13. HDF5 tools for working with NPP files
ADL 4.2 Workshop
13
January 8, 2014
• nagg allows reorganizing already downloaded data
• h5edit provides command line editing of HDF5 file
metadata
• h5augjpss (experimental) provides options to import
metadata from product profiles and remove obstacles
for netCDF-4 tools

14. Example: aggregate 36 granules from 9 files 14
nagg –n 36 –t SVM07 GMODO-
SVM07_npp_d20121028_*.h5

15. 15HDF-EOS Examples Future Directions:
NASA Developer Portal + CMR
Jupyter Notebook Python Examples
• Search & query collection and granules from Common Metadata
Repository (CMR).
• Access data using OPeNDAP using the query result.
Future: CMR Ingestion and Testing Example
• Integration with HDF Product Designer?

16. Future Work 16
• Product Designer
• h4h5diff
• OPeNDAP WOS Support
• HDF5 Handler improvements
• DAAC User Needs?
• Join the discussion at the HDF-EOS Forum

17. Sustainable Archives? 17
• ODL to Groups/Attributes
• Metadata to ISO
• Conventions to HDF-EOS (?) / CF
• Migration from HDF4 to HDF5
• Data to the Cloud?
• HDF4 chunks/bytestreams > objects?