This tutorial is designed for anyone who needs to work with data stored in HDF5 files. The tutorial will cover functionality and useful features of the HDF5 utilities h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. We will also introduce a prototype of the new h52jpeg conversion tool and recently released h5perf_serial tool used for performance studies. We will briefly introduce HDFView. Details of the HDFView and HDF-Java will be discussed in a separate tutorial.

1. HDF5 Tools
Peter Cao
The HDF Group
October 15-18, 2008
HDF and HDF-EOS Workshop XII
1

2. Outline
•
•
October 15-18, 2008
A short demo of HDFView
HDF5 command line tools
HDF and HDF-EOS Workshop XII
2

3. What is HDFView?
HDFView
HDF Application
Browser/Editor
Written in Java
For more information and download, visit
www.hdfgroup.org/hdf-java-html
October 15-18, 2008
HDF and HDF-EOS Workshop XII
3

4. HDFView Demo
•
•
•
•
•
Browse objects in an HDF5 file
Show data content in table, image
Make a subset
View attributes and metadata
Create a new HDF5 file and edit data content
October 15-18, 2008
HDF and HDF-EOS Workshop XII
4

5. HDFView 2.5
•
•
•
•
•
•
Release planned in December, 2008
Build with HDF4 2r4 and HDF5 1.8.2
Fix bugs and add new features
Plug memory leaks
Support iRODS (Windows and linux only)
Use jdk 1.6.0_06 or above (for performance)
Drop mac PowerPC
October 15-18, 2008
HDF and HDF-EOS Workshop XII
5

6. HDF5 Command Line Tools
Readers
h5dump, h5diff, h5ls, h5check, h5stat
Writers
h5repack, h5repart, h5import, h5jam,
h5unjam, h5copy, h5mkgrp
Converters
h4toh5, h5toh4, gif2h5, h52gif
October 15-18, 2008
HDF and HDF-EOS Workshop XII
6

7. Where to Start?
Tools located in bin/ directory
1. Use “-h” option for help
2. Consult online help
http://www.hdfgroup.org/hdf5tools.html
3. Email help desk
help@hdfgroup.org
October 15-18, 2008
HDF and HDF-EOS Workshop XII
7

8. h5dump
Dumps contents of HDF5 file to stdout
ASCII text
• XML
• Binary
•
October 15-18, 2008
HDF and HDF-EOS Workshop XII
8

9. h5dump: Object Headers
> h5dump -H SDS.h5
HDF5 "SDS.h5" {
GROUP "/" {
GROUP "Floats" {
DATASET "FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4,
3)}
}
}
DATASET "IntArray" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }
}
}
}
October 15-18, 2008
HDF and HDF-EOS Workshop XII
9

10. h5dump: Dataset
> h5dump –d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30
}
}
}
October 15-18, 2008
HDF and HDF-EOS Workshop XII
10

11. h5dump: Binary Output
-b F, --binary=[MEMORY|FILE|LE|BE]
> h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
0000000
0000030
0000060
0000110
0000140
October 15-18, 2008
00000000
0000000a
00000014
0000001e
00000028
00000001
0000000b
00000015
0000001f
00000029
00000002
0000000c
00000016
00000020
0000002a
00000003
0000000d
00000017
00000021
0000002b
HDF and HDF-EOS Workshop XII
00000004
0000000e
00000018
00000022
0000002c
00000005
0000000f
00000019
00000023
0000002d
11

12. h5diff
Show
differences
between…
two objects
or
two files
October 15-18, 2008
HDF and HDF-EOS Workshop XII
12

13. h5diff
> h5diff SDS.h5 SDS2.h5
Dataset: </IntArray>
5 differences found
October 15-18, 2008
HDF and HDF-EOS Workshop XII
13

14. h5diff: Report Differences
> h5diff SDS.h5 SDS2.h5 -r /IntArray
Dataset:
position
</IntArray> and </IntArray>
IntArray
IntArray
difference
-----------------------------------------------------[ 0 0 ]
0
10
10
[ 1 0 ]
10
100
90
[ 2 0 ]
20
200
180
[ 3 0 ]
30
300
270
[ 4 0 ]
40
400
360
5 differences found
October 15-18, 2008
HDF and HDF-EOS Workshop XII
14

15. h5diff: Unsolved Issues
How to compare two files with different
− Structures
− Storage layouts
− Properties
October 15-18, 2008
HDF and HDF-EOS Workshop XII
15

16. h5repack
Copies a file to a new file with
• Compression filter
• Data layout
• Other user options
October 15-18, 2008
HDF and HDF-EOS Workshop XII
16

17. h5repack: Compression Filter
-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}
For example
> h5repack -i SDS.h5 -o out.h5 -f GZIP=6
October 15-18, 2008
HDF and HDF-EOS Workshop XII
17

18. h5repack: Data Layout
-l {CHUNK| COMPA| CONTI}
For example
> h5repack -i SDS.h5 -o out.h5 -l CHUNK=1000x1000
October 15-18, 2008
HDF and HDF-EOS Workshop XII
18

19. h5repack: File Format Version
h5repack options
File size in bytes
Original file
3,167,264
--latest
2,267,346
--latest –compact=9
2,162,254
--latest –compact=9 –ssize=50:dtype
299,882
The test file was created with HDF5 1.6.5. The root group of the file
contains 100 groups. Each group at the root contains 9 sub groups.
Each sub group contains one compound dataset. The compound
dataset has 40 members of type 32 bit integer
October 15-18, 2008
HDF and HDF-EOS Workshop XII
19

20. h5import
Imports binary/ASCII data into an HDF5 file
h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile
For example
> h5import float5x4x2.txt -c First_set.conf -o First_set.h5
PATH work/First-set
INPUT-CLASS TEXTFP
RANK 3
DIMENSION-SIZES 5 2 4
OUTPUT-CLASS FP
OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS 8 8 -1
October 15-18, 2008
GROUP "/" {
GROUP "work" {
DATASET "First-set" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8,
H5S_UNLIMITED ) }
DATA {
(0,0,0): 1.01, 1.02, 1.03, 1.04,
(0,1,0): 1.11, 1.12, 1.13, 1.14,
(1,0,0): 1.21, 1.22, 1.23, 1.24,
(1,1,0): 1.31, 1.32, 1.33, 1.34,
(2,0,0): 1.41, 1.42, 1.43, 1.44,
…
}
}
}
}}
HDF and HDF-EOS Workshop XII
20

21. h5import: Requested Features
• Make it interchangeable with
− h5dump
− Excel spreadsheet
October 15-18, 2008
HDF and HDF-EOS Workshop XII
21

22. New for 1.6.6+ and 1.8.0+
h5copy
h5mkgrp
h5stat
h5check
October 15-18, 2008
HDF and HDF-EOS Workshop XII
22

23. h5copy
• Copies an object within a file or across files
/
IntArray
/
Floats
FloatArray
FloatArray
October 15-18, 2008
HDF and HDF-EOS Workshop XII
23

24. h5copy –f shallow
DST1
floats
w
llo
ha
fs
-
SRC
64-bit
f32
floats
integers
DST2
i1
i2
f32
64-bit
f1
floats
f2
f32
64-bit
f1
October 15-18, 2008
HDF and HDF-EOS Workshop XII
f2
24

25. h5mkgrp
• Creates a group or group structure
> h5mkgrp test.h5 /info /data /data/test1 /data/test2
/
info
data
test1
October 15-18, 2008
test2
HDF and HDF-EOS Workshop XII
25

26. h5stat
Prints statistics about HDF5 file
−
Header size
− Raw data size
− Attributes, datatypes, etc.
Helps
−
Troubleshoot overhead in HDF5 files
− Choose specific object’s properties and storage strategies
October 15-18, 2008
HDF and HDF-EOS Workshop XII
26

27. h5stat: Example
>h5stat hdf5_test.h5
Filename: hdf5_test.h5
File information
# of unique groups: 4
# of unique datasets: 13
…
Object header size: (total/unused)
Groups: 288/32
Datasets: 6824/864
…
Dataset storage information:
Total raw data size: 1769009
…
October 15-18, 2008
HDF and HDF-EOS Workshop XII
27

28. h5check
• Verifies whether an HDF5 file is compliant with a
specific HDF5 File Format.
• By default, the file is checked against 1.8.x.
• Use -f16 option to check against 1.6.x.
October 15-18, 2008
HDF and HDF-EOS Workshop XII
28

29. h5check
> h5check --verbose=2 example.h5
VALIDATING example.h5 according to library version 1.8.0
FOUND super block signature
VALIDATING the super block at physical address 0...
Validating version 2 superblock...
VALIDATING the object header at logical address 48...
VALIDATING version 2 object header ...
FOUND Version 2 object header signature
VALIDATING version 2 btree at logical address 1864...
VALIDATING version 2 btree header at logical address 1864…
:
No non-compliance errors found
October 15-18, 2008
HDF and HDF-EOS Workshop XII
29

30. h5check
> h5check --verbose=2 –f16 example.h5
VALIDATING example.h5 according to library version 1.6.6
VALIDATING the super block at physical address 0...
Validating version 0/1 superblock...
***Error***
Errors found when checking superblock. Validation stopped.
Disk Format Level 0-File Metadata-->0A-File Signature and Super Block
Superblock v.0/1:Unable to read root symbol table entry
***End of Error messages***
Non-compliance errors found
October 15-18, 2008
HDF and HDF-EOS Workshop XII
30

31. Thank You!
This presentation is based upon work supported in part by a Cooperative
Agreement with NASA under NASA NNX06AC83A. Any opinions, findings,
and conclusions or recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of the National
Aeronautics and Space Administration
October 15-18, 2008
HDF and HDF-EOS Workshop XII
31

32. Questions/comments?
October 15-18, 2008
HDF and HDF-EOS Workshop XII
32