1. Transition from HDF4 to
HDF5: Issues
Robert E. McGrath
NCSA
University of Illinois at Urbana-Champaign
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
1
HDF
2. Overall Goal
A discussion of compatibility, conversion,
and interoperability issues for HDF4 and
HDF5, and HDFEOS.
For more details see the white paper at:
http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
2
HDF
3. Conclusions (Preview)
Transition from HDF4 to HDF5 is not
necessarily transparent or effortless.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
3
HDF
4. Conclusions: HDF
• Transition from HDF4 to HDF5 is not
transparent: in most cases source code will
have to be written or rewritten.
• Many tools will support both HDF4 and
HDF5.
• Default translation from HDF4 to HDF5
will be supported as a utility and library.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
4
HDF
5. Conclusions: HDF-EOS
• Transition from HDFEOS2 to HDFEOS3 is
not transparent: in most cases source code
will have to be written or rewritten
• Translation of ‘pure’ HDF-EOS will be
supported
• Translation of ‘hybrid’ HDF-EOS files is
very difficult.
• Not possible to support V.2 and V.3 in a
single program
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
5
HDF
6. What Does Compatibility Mean?
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
6
HDF
7. What does “compatibility” mean?
• Ability to “do the same things” with new
software
– I.e., “compatibility” is relative to some purpose
– Many dimensions and shades of compatibility
– No golden solution sufficient for every use
• Assume HDF4 users want to continue doing
what they are already doing, but move to
HDF5
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
7
HDF
8. Some Types of Compatibility
Compatibility
Issue
Conceptual
Description
Format
Software
Programming
model
Features
Interfaces
The overall conceptual approach:
the formal and informal models.
Storage layout and other format
details.
Compatibility of the software
The assumptions and approach
to programming.
The set of features supported.
The programming interfaces.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
8
HDF
9. Compatibility of HDF and HDF-EOS
Compatibility Issue HDF4/HDF5 HDFEOS2/HDFEOS3
Conceptual
Yes, H5 is a Yes
superset
Format
No
No
Software
Programming model No?
Yes
Features
Some
Yes, V3 is a superset
Interfaces
No
No
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
9
HDF
10. Interoperation
• My definition of “interoperation”:
one program or tool may use HDF4 or
HDF5 or both
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
10
HDF
11. NCSA HDF4 and HDF5
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
11
HDF
12. Interoperability: Any program may use either HDF4 or 5
or both:
Not Supported
dataset1
dataset2
image1
image2
Single Combined HDF Library
HDF 4 Library
HDF 5 Library
Stored as
HDF4 Objects
Stored as
HDF5 Objects
HDF4 file
HDF5 file
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
12
HDF
13. Interoperability: Specific Application or tool may use
either HDF4 or 5 or both:
Up to Application and Tool Developers
dataset1
dataset2
image1
image2
Application Specific Library
specific
HDF 4 Library
HDF 5 Library
Stored as
HDF4 Objects
Stored as
HDF5 Objects
HDF4 file
HDF5 file
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
13
HDF
14. HDF-EOS Version 2 and Version 3
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
14
HDF
15. Interoperability:
Any program may using HDF-EOS Objects with either
HDF4 or HDF5 or both:
Not supported
HDFEOS Library
Grid1
Grid2
HDF 4 Library
Grid3
Grid4
HDF 5 Library
Stored as
HDF4 Objects
Stored as
HDF5 Objects
HDF4 file
HDF5 file
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
15
HDF
16. Interoperability: One application may use HDF-EOS
Objects with either HDF4 or HDF5 or both:
Not supported
Grid1
Grid2
HDFEOS V2 Library
HDF 4 Library
Grid3
Grid4
HDFEOS V3 Library
HDF 5 Library
Stored as
HDF4 Objects
Stored as
HDF5 Objects
HDF4 file
HDF5 file
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
16
HDF
17. Translation
• Convert HDF4 objects (file) to HDF5
objects (file)
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
17
HDF
18. Translation of HDF4 to HDF5
Supported: NCSA mapping and h4toh5 tool and library
HDF4 Library
HDF5 Library
/
Stored as
HDF4 Objects
Stored as Analogous
HDF5 Objects
HDF5 File
HDF4 File
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
18
HDF
19. Translation of HDF-EOS V2 to V3
Supported
HDF EOS V. 2.x Library
Grid1
Grid2
HDF EOS V. 3 Library
Grid2
Grid1
SAME
HDFEOS
Objects
HDFEOS
Objects
HDF4 Library
HDF5 Library
Stored as
HDF4 Objects
Stored as
HDF5 Objects
/
HDF5 file
HDF4 file
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
19
HDF
20. “Hybrid” HDF-EOS files
• HDF-EOS files may contain native HDF
objects (images, arrays, etc.) as well as
HDF-EOS objects (Grid, etc.)
• Almost all HDF-EOS files are ‘hybrids’.
• Very difficult to automatically translate
hybrid files.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
20
HDF
21. “Hybrid” HDF-EOS File
Science Application Code
HDFEOS
Other
NCSA HDF
Components of
HDFEOS Objects
stored in HDF
Objects
Other HDF
Objects Stored
Directly Through
HDF
One HDF file, objects
mixed together.
Very difficult
to tell which
objects are part
of HDF-EOS
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
21
HDF
22. Conclusions
Transition from HDF4 to HDF5 is not
necessarily transparent or effortless.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
22
HDF
23. Conclusions: HDF
• Transition from HDF4 to HDF5 is not
transparent: in most cases source code will
have to be rewritten.
• Many tools will support both HDF4 and
HDF5.
• Default translation from HDF4 to HDF5
will be supported as a utility and library.
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
23
HDF
24. Conclusions: HDF-EOS
• Transition from HDFEOS2 to HDFEOS3 is
not transparent: in most cases source code
will have to be rewritten
• Translation of ‘pure’ HDF-EOS will be
supported
• Translation of ‘hybrid’ HDF-EOS files is
very difficult.
• Not possible to support V.2 and V.3 in a
single program
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
24
HDF
25. For More Information
• NCSA White paper: “Transition From HDF4 to HDF5”:
http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf
• “Guidelines for Mapping HDF4 to HDF5 Objects”,
http://hdf.ncsa.uiuc.edu/HDF5/papers/H4-to-H5MappingGuidelines.pdf
• h4toh5 tool: Soon
University of Illinois at Urbana-Champaign
9/19/2000
McGrath 9/19/2000
25
HDF