This tutorial is designed for anyone who needs to work with data stored in HDF5 files. It will cover functionality and useful features of the HDF5 utilities, which include h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. The tutorial will also introduce recently changes and new features of the utilities. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files. Some basic features and new changes of HDFView will be presented. Details of recent development in HDF-Java products will be discussed in a separate presentation.

1. The HDF Group
HDF5 Tools
Peter Cao
The HDF Group
The 13th HDF and HDF-EOS Workshop
November 3-5, 2009
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
1
www.hdfgroup.org

2. Outline
•
•
November 3-5, 2009
An introduction to HDFView
HDF5 command line tools
HDF/HDF-EOS Workshop XIII
2 www.hdfgroup.org

3. What is HDFView?
HDFView
HDF Application
Browser/Editor
Written in Java
For more information and download, visit
www.hdfgroup.org/hdf-java-html
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
3
www.hdfgroup.org

4. HDFView Demo
• Browse objects in an HDF5 file
• Show file content
• Change file content
Advanced features and new development such as plugin, region
references, full support of hdf5 1.8 will be in a separate talk.
HDFView is for general use. For specific purpose or more advanced
features, you use other tools like MATLAB or IDL, or other earth
science related tools such as HEG, NCL, or GrADS. At the same time,
welcome ideas for features.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
4
www.hdfgroup.org

5. HDFView 2.6
•
•
•
•
•
Release planned in December, 2009
Build with HDF5 1.8.4
Fix bugs
Add new features
Add support for 64-bit mac.
Drop mac PowerPC
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
5
www.hdfgroup.org

6. HDF5 Command Line Tools
•
Look or check file content
h5dump, h5diff, h5ls, h5check, etc.
•
Change file content or create new file
h5repack, h5import, h5jam, h5copy, etc.
•
Convert file content
h4toh5, h5toh4, gif2h5, h52gif, etc.
Only show you the basic features of some tools
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
6
www.hdfgroup.org

7. Where to Start?
Tools are distributed with the library release and located under bin/
1. Use “-h” option for help
2. Consult online help
http://www.hdfgroup.org/hdf5tools.html
3. Email help desk
help@hdfgroup.org
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
7 www.hdfgroup.org

8. h5dump
Dumps contents of HDF5 file to stdout
ASCII text
• XML
• Binary
•
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
8 www.hdfgroup.org

9. h5dump: Object Headers
What’s in the file?
h5dump -H SDS.h5
HDF5 "SDS.h5" {
GROUP "/" {
GROUP "Floats" {
DATASET "FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
}
}
DATASET "IntArray" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }
}
}
}
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
9 www.hdfgroup.org

10. h5dump: Dataset
What’s the content?
h5dump -d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30
}
}
}
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
10 www.hdfgroup.org

11. h5diff
Show
differences
between…
two objects
or
two files
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
11 www.hdfgroup.org

12. h5diff
Any change in the data process?
h5diff SDS.h5 SDS2.h5
Dataset: </IntArray>
5 differences found
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
12 www.hdfgroup.org

13. h5diff: Report Differences
What are the changes?
h5diff -r SDS.h5 SDS2.h5 /IntArray
Dataset:
position
</IntArray> and </IntArray>
IntArray IntArray difference
-----------------------------------------[ 0 0 ]
0
10
10
[ 1 0 ]
10
100
90
[ 2 0 ]
20
200
180
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
13 www.hdfgroup.org

14. h5diff: Unsolved Issues
How to compare two files with different
• Structures
• Storage layouts
• Other properties
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
14 www.hdfgroup.org

15. h5repack
Copies
an entire file to
a new file…
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
15 www.hdfgroup.org

16. h5repack: Change Data Layout
Size of a 2D byte array of 1024x1024?
3MB
Why is my file so big?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
16 www.hdfgroup.org

17. h5repack: Change Data Layout
h5repack -i huge.h5 -o good.h5 -l CHUNK=64x1024
Size of good.h5? 1MB
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
17 www.hdfgroup.org

18. h5repack: Compression Filter
Most of my data values are zeros and ones
Can I make it better?
www.hdfgroup.org

19. h5repack: Compression Filter
h5repack -i good.h5 -o better.h5 -f GZIP=6
Size of better.h5? 6KB
Down from 3MB to 6KB, 500 times smaller!!!!
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
19 www.hdfgroup.org

20. h5import
1 GB
1 MB
What should I do?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
20 www.hdfgroup.org

21. h5import
h5import float5x4x2.txt -c First_set.conf -o First_set.h5
1.01, 1.02, 1.03, 1.04
1.11, 1.12, 1.13, 1.14
1.21, 1.22, 1.23, 1.24
1.31, 1.32, 1.33, 1.34
1.41, 1.42, 1.43, 1.44
GROUP "/" {
GROUP "work" {
DATASET "First-set" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) }
DATA {
(0,0,0): 1.01, 1.02, 1.03, 1.04,
(0,1,0): 1.11, 1.12, 1.13, 1.14,
(1,0,0): 1.21, 1.22, 1.23, 1.24,
(1,1,0): 1.31, 1.32, 1.33, 1.34,
(2,0,0): 1.41, 1.42, 1.43, 1.44,
…
}
}
}
}}
Use external datasets
PATH work/First-set
INPUT-CLASS TEXTFP
RANK 3
DIMENSION-SIZES 5 2 4
OUTPUT-CLASS FP
OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS 8 8 -1
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
21 www.hdfgroup.org

22. Only available in1.8, not in 1.6
h5copy
h5mkgrp
h5stat
h5check
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
22 www.hdfgroup.org

23. h5copy
Copies
objects
to files…
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
23 www.hdfgroup.org

24. h5copy: dataset
dset.h5
dset2.h5
/
integers
i1
i2
/
floats
64-bit
f1
f32
f32
f2
h5copy –i dset.h5 –s /floats/f32 –o dset2.h5 –d /f32
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
24 www.hdfgroup.org

25. H5copy: group
dset.h5
dset2.h5
/
floats
integers
i1
i2
/
f32
floats
f32
64-bit
64-bit
f1
f1
f2
f2
h5copy –i dset.h5 –s /floats –o dset2.h5 –d /floats
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
25 www.hdfgroup.org

26. h5mkgrp
dset.h5
/
info
data
test1
test2
h5mkgrp test.h5 /info /data /data/test1 /data/test2
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
26 www.hdfgroup.org

27. h5stat
Prints statistics about HDF5 file
Header size
• Raw data size
• Attributes, datatypes, etc.
•
Helps
Troubleshoot overhead in HDF5 files
• Choose specific object’s properties and storage strategies
•
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
27
www.hdfgroup.org

28. h5stat: Example
>h5stat hdf5_test.h5
Filename: hdf5_test.h5
File information
# of unique groups: 4
# of unique datasets: 13
…
Object header size: (total/unused)
Groups: 288/32
Datasets: 6824/864
…
Dataset storage information:
Total raw data size: 1769009
…
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
28
www.hdfgroup.org

29. Acknowledgements
This work was supported by cooperative agreement
number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the author[s] and do not necessarily reflect
the views of the National Aeronautics and Space
Administration.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
29
www.hdfgroup.org

30. The HDF Group
Thank You!
Questions/comments?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
30
www.hdfgroup.org