The document describes the HDF5 file format used to store processed data products from the National Polar-orbiting Operational Environmental Satellite System (NPOESS) and NPOESS Preparatory Program (NPP). HDF5 is used to organize product data and metadata within groups and datasets in a consistent manner. This includes organizing multi-dimensional scientific data arrays, scaling integers for storage, and including quality flags and geolocation data. Product profiles provide metadata to interpret the scientific data values and quality flags.

1. NPP/ NPOESS
Product Data Format
Richard E. Ullman
NOAA/NESDIS/IPO • NASA/GSFC/NPP
Algorithm Division • System Engineering
Data/Information Architecture richard.
ullman@nasa.gov
NPOESS•INTEGRATED PROGRAM OFFICE

2. Scope
•
•
The file profile described in this presentation applies to
all NPOESS delivered environmental remote sensing
products of the following types:
–
–
–
–
–
–
SDR • Sensor Data Record
TDR • Temperature Data Record
EDR • Environmental Data Record
IP • Intermediate Product
ARP • Application Related Product
GEO • Geolocation
This profile does not apply to other types such as:
– RDR • Raw Data Record
– DDR • Data Delivery Record
– Mission notices, status notices, documentation, software
deliveries, etc
NPOESS•INTEGRATED PROGRAM OFFICE
2
The HDF Workshop 6, 2007
Nov

3. HDF5 for NPOESS
•
•
•
•
•
•
Hierarchical Data Format 5 (HDF5) is the format for delivery of
processed products from the National Polar-orbiting Operational
Environmental Satellite System (NPOESS) and for the NPOESS
Preparatory Program (NPP).
HDF5 is a general purpose library and file format for storing
scientific data. Two primary objects:
• Dataset, a multidimensional array of data elements
• Group, a structure for organizing objects
Efficient storage and I/O, including parallel I/O.
Free, open source software, multiple platforms.
Data stored in HDF5 is used in many fields from computational fluid
dynamics to film making.
Data can be stored in HDF5 in an endless variety of ways, so it is
important to standardize how NPOESS product data is organized in
HDF5.
NPOESS•INTEGRATED PROGRAM OFFICE
3
The HDF Workshop 6, 2007
Nov

4. Format Strengths
•
Straight HDF5.
•
Consistent HDF5 group structure
•
Allows for flexible temporal aggregation
– No need for additional libraries.
– Organization for each product is the same as all
others.
– Data “payload” is always in a product group within
All_Data group.
– Granules are appended by extending dataset
dimension.
NPOESS•INTEGRATED PROGRAM OFFICE
4
The HDF Workshop 6, 2007
Nov

5. Format Challenges
•
•
•
•
•
Geolocation appears in a separate product
group and may be in separate HDF5 file.
Field metadata, used to interpret data (similar to
netCDF CF) are in separate product profile file.
Quality flags must be parsed before they can be
interpreted.
Information needed for un-scaling scaled
integers is not obvious.
HDF5 indirect reference link API, used to link
metadata to the data in NPOESS’ use is
complex and not supported by all analysis COTS
implementations.
NPOESS•INTEGRATED PROGRAM OFFICE
5
The HDF Workshop 6, 2007
Nov

6. Information
Model UML
Diagram
NPOESS•INTEGRATED PROGRAM OFFICE
6
The HDF Workshop 6, 2007
Nov

7. An Example Product Group
•
In this example product group:
–
–
–
–
–
–
–
Five datasets constitute the product.
There are two common dimensions.
There are three congruent datasets.
Two datasets contain scale and offset values.
One dataset contains quality flags by element.
There are two granules in this aggregation.
Dimension “alongTrack” crosses the “granule
boundary.”
NPOESS•INTEGRATED PROGRAM OFFICE
7
The HDF Workshop 6, 2007
Nov

8. Example extracted from VIIRS Sea
Surface Temperature EDR
Field Name
Description
Data Type
Dimensions
Units
BulkTemp
Sea Surface Bulk
Temperature
16-bit unsigned
integer
[ N*768, 3200 ]
Kelvin /
Unitless
SkinTemp
Sea Surface Skin
Temperature
16-bit unsigned
integer
[ N*768, 3200 ]
Kelvin /
Unitless
QF1_VIIRSSSTEDR
Land/Water
Background
1-bit
[N*768, 3200 ]
Unitless
SST Skin Quality
2-bit
Unitless
SST Bulk Quality
2-bit
Unitless
Aerosol Correction
3-bit
Unitless
Bulk SST Scale
32-bit float
Bulk SST Offset
32-bit float
Skin SST Scale
32-bit float
Skin SST Offset
32-bit float
SSTBulkFactors
SSTSkinFactors
[ N*2]
Unitless
Kelvin
[ N*2]
Unitless
Kelvin
N in Dimension is number of granules
NPOESS•INTEGRATED PROGRAM OFFICE
8
The HDF Workshop 6, 2007
Nov

9. Example Product Group
NPOESS Product Group
alo
Tr ng
ac
k
crossTrack
Granule 0
BulkTemp
Granule 0
SSTBulkFactors
Granule 1
SkinTemp
Granule 1
S
O
S
O
Granule 0
QF1_VIIRSSSTEDR
Granule 1
NPOESS•INTEGRATED PROGRAM OFFICE
Granule 0
Granule 1
SSTSkinFactors
S
O
S
O
Granule 0
Granule 1
9
The HDF Workshop 6, 2007
Nov

10. Dimensions
•
•
•
•
•
Dimensions are defined for each field.
Fields are related by congruency and common
dimensions.
Common dimensions are given the same name.
One dimension crosses the granule boundary.
When multiple granules are “aggregated” the
“granule boundary” dimension is extended.
Dimension names and attributes are provided in
the product profile.
NPOESS•INTEGRATED PROGRAM OFFICE
10
The HDF Workshop 6, 2007
Nov

11. Scaled Integer Storage
•
•
•
•
•
For storage efficiency floating point data values may be
stored as scaled integers.
To re-generate the data value, the dataset element must
be multiplied by a supplied scale factor and an integer
offset added.
The scale factor and offset are provided, one pair for
each granule as a separate dataset.
The scale and offset value is the same for all granules
produced with a given version of an algorithm - not
dynamic scaling.
The fact that a dataset is a scaled value and the
association between the data dataset and the scale
factor dataset is contained in the product profile.
NPOESS•INTEGRATED PROGRAM OFFICE
11
The HDF Workshop 6, 2007
Nov

12. Quality Flags by Element
•
•
•
Most NPOESS products contain multiple indicators of
quality on an element by element basis.
Quality flags are associated by congruency (shared
dimension) with a data array.
Multiple Flags of less than 8-bits are “packed” into
structures aligned on 8-bit boundaries.
1
1
Quality Flags
0
1
0
1-bit
2-bit
2-bit
0
0
3-bit
1
NPOESS•INTEGRATED PROGRAM OFFICE
12
The HDF Workshop 6, 2007
Nov

13. Geolocation
•
•
•
Geolocation products are constructed using the same
conventions as SDRs and EDRs.
Geolocation datasets have a congruence relationship
with the same dimensions as the datasets to which they
apply.
The association between a data product with its
geolocation product is made on one of two ways:
– The geolocation product may be packaged as a separate
product group within the same HDF5 file.
– The name of a separate geolocation product file may be stored
in the N_GEO_Ref attribute on the root HDF group.
– Choice of “as a product group” or “as a separate file” is made
upon order from the NPOESS IDPS.
NPOESS•INTEGRATED PROGRAM OFFICE
13
The HDF Workshop 6, 2007
Nov

14. Common Geolocation Fields
for VIIRS Products
Field Name
Comments
Dimensions
Units
Data Type
StartTime
since epoch 1/1/1958 [per scan or swath]
microseconds 64-bit signed integer
MidTime
since epoch 1/1/1958 [per scan or swath]
microseconds 64-bit signed integer
SCPosition
ECR coordinates
[per scan or swath]
meters
SCVelocity
ECR coordinates
[per scan or swath] meters/second
32-bit float
32-bit float
Latitude
[per cell]
degrees
32-bit float
Longitude
[per cell]
degrees
32-bit float
SolarZenithAngle
[per cell]
degrees
32-bit float
SolarAzimuthAngle
[per cell]
degrees
32-bit float
SensorZenithAngle
[per cell]
degrees
32-bit float
SensorAzimuthAngle
[per cell]
degrees
32-bit float
[per cell]
meters
32-bit float
[per cell]
meters
32-bit float
Height
SatelliteRange
geoid or terrain
NPOESS•INTEGRATED PROGRAM OFFICE
14
The HDF Workshop 6, 2007
Nov

15. Product Profiles
•
XML documents provide definition of product
fields.
– Product Profile is delivered as part of the product
documentation.
– Contains metadata such as units of measure,
dimension names, legend entries, etc
– A separate profile per product, but each conforms to
the same NPOESS Product document type definition
(dtd) and XML schema definition (xsd).
– A style sheet is provided that can render the profile
for a web browser.
– Example:VIIRI_IIT_II D
RIxml
NPOESS•INTEGRATED PROGRAM OFFICE
15
The HDF Workshop 6, 2007
Nov

16. 17 field attributes in XML
product profile (1..9)
Attribute Name
Type
DataType
String
Description
String
Set of
Boolean
set of
string
String
Set of
string
Set of
number
Set of
string
Set of
number
Dimension_GranuleBoundary
Dimension_Name
Field_Name
FillValue_Name
FillValue_Value
LegendEntry_Name
LegendEntry_Value
Comments
String format is:
“%d-bit %s”, where %d is the number of bits and %s is one of:
o
signed integer
o
unsigned integer
o
floating point
o
<blank> (a bitfield)
A descriptive text
True (1) indicates that this is dimension extends when granules
are appended.
Name match indicates that this dimension is congruent with the
same dimension names in other datasets in this product group.
The name of the HDF5 dataset that contains the field values.
Data type matches type of dataset.
Data type matches type of dataset.
NPOESS•INTEGRATED PROGRAM OFFICE
16
The HDF Workshop 6, 2007
Nov

17. 17 field attributes in XML
product profile (10..17)
Attribute Name
Type
Comments
MeasurementUnits
String
Consistent with SI naming and Unidata’s “udunits” package
NumberOfDimensions
Integer
Integer greater than zero.
NumberOfFillValues
Integer
NumberOfLegendEntries
Integer
RangeMax
Number
RangeMin
Number
Scaled
Boolean
ScaleFactorName
String
If zero, then no FillValue_Name and FillValue_Value attributes
are present. Fill Values are used for primary data fields only.
If zero, then no LegendEntry_Name and LegendEntry_Value
attributes are present. Legend entries are used for quality fields
only.
Maximum expected value of field elements in the product, not
just this dataset instance. Data type matches type of dataset.
Minimum expected value of field elements in the product, not
just this instance. Data type matches type of dataset.
True indicates that the dataset is scaled. Note that fill values are
in the dataset type and so must be tested before un-scaling.
the name of the HDF5 dataset that contains scaling coefficients.
To un-scale the elements, first multiply the scaled element by
the first element and then add the second element. If the dataset
is not scaled, Scale_AttributeName will not exist.
NPOESS•INTEGRATED PROGRAM OFFICE
17
The HDF Workshop 6, 2007
Nov