A brief overview of using HDF5 with Python and Andrew Collette's h5py module will be presented, including examples which show how and why Python can be used in the place of HDF5 tools. Extensions to the HDF5 API will be proposed which would further improve the utility of Python/h5py.

1. Substituting HDF5 tools with Python/H5py scripts
Daniel Kahn
Science Systems and Applications Inc.
HDF HDF-EOS Workshop XIV, 28 Sep. 2010
1 of 14

2. What are HDF5 tools?
HDF5 tools are command line programs distributed with the HDF5
library. They allow users to manipulate HDF5 files.
h5dump: dump HDF5 data as ASCII text.
h5import: convert non-HDF5 data to HDF5
h5diff: show differences between HDF5 files.
h5copy: Copy objects between HDF5 files.
h5repack: Copy entire file while changing storage
properties of HDF5 objects.
h5edit: (proposed) add attributes to HDF5 objects.
HDF5 tools have a long history as the first (and for a long time only)
way to manipulate HDF5 files conveniently. I.e. without writing a C or
Java program, or without buying expensive commercial software such
as IDL or Matlab.
2 of 14

3. The tools can be characterized as having three parts:
Text Processing—Evaluate command arguments, process input
text files, match group names.
Tree Walking – Search HDF5 file hierarchy for objects by name.
Object Level Operations – Operate on the objects: copy, diff,
repack, etc.
The tools are simple to use and convenient as they are
distributed with the HDF5 library.
3 of 14

4. Disadvantage of HDF5 tools:
The command line arguments limit tool capability.
Adding new features with command line syntax which is both
readable and does not break the legacy syntax becomes difficult.
Development time for designing and implementing new features is
long (weeks...months).
Use cases must be evaluated, a solution proposed in an RFC, the
proposal must be implemented, new code is distributed in next
release.
4 of 14

5. Here's an example from HDF documentation:
h5copy -v -i "test1.h5" -o "test1.out.h5" -s "/array" -d "/array
But suppose we had multiple datasets named arrayNNN
where N is 0–9. We'd like to write something like:
h5copy -v -i "test1.h5" -o "test1.out.h5" -s "/arrayd+{3}”
So that d+{3} would provide a match to all such objects.
Extending the tool syntax to meet this use case, and then
again for the next use case would be a never ending game of
catch up.
A more flexible substitute is desirable...
5 of 14

6. ...Python?
6 of 14

7. What is Python?
Python is a programming language.
It features dynamic binding of variables, like Perl or shell
scripts, IDL, Matlab, but not C or Fortran.
Unlike Perl, it supports native floating point numbers.
It has scientific array support in the style of IDL or Matlab
(numpy module). Array operations can be programmed using
normal arithmetic operators.
It has access to the HDF5 library (Anderw Collette's
h5py module).
Python is currently the only programming language in wide
spread use to have all these features. They are essential to the
success of the language for easy HDF5 file manipulation.
7 of 14

8. Real world Experience: Learning Python and h5py is quick.
In the summer of 2010 SSAI hired a summer intern.
Equipped with some Perl programming experience the
intern was able to come up to speed on Python, HDF5,
h5py, and numpy within one to two weeks and, over
the summer, develop a specialized file/dataset merging
tool and a dataset conversion tool.
Python and h5py are the best way to introduce HDF5
because it allows the user to concentrate on the H in
HDF5, rather than the C API syntax.
8 of 14

9. Python is well suited to HDF5
Python is well suited to HDF5 because the HDF5 array objects
carry the dimensionality, extent, and element data type
information, just as HDF5 datasets do. The object oriented
nature of Python allows these objects to be manipulated at a
high level. C, by contrast, lacks a scientific array object and
the ability to define object methods.
9 of 14

10. Example: Creating and Writing a Dataset to a New File
Python:
import h5py
import numpy
TestData = numpy.array(range(1,25),dtype='int32').reshape(4,6)
h5py.File("WrittenByH5PY.h5","w")['/TestDataset'] = TestData
Compare to C version:
#include "hdf5.h"
int main() {
hid_t
file_id, dataspace_id, dataset_id; /* identifiers */
herr_t status;
hsize_t dims[2];
const int FirstIndex = 4, SecondIndex = 6;
int
i, j, dset_data[4][6];
for (i = 0; i < 4; i++) /* Initialize the dataset. */
for (j = 0; j < 6; j++)
dset_data[i][j] = i * 6 + j + 1;
dims[0] = FirstIndex;
dims[1] = SecondIndex;
file_id = H5Fcreate("WrittenByC.h5", H5F_ACC_TRUNC, H5P_DEFAULT,H5P_DEFAULT); /* Open an existing file.
*/
dataspace_id = H5Screate_simple(2, dims, NULL);
dataset_id = H5Dcreate(file_id, "/TestDataset", H5T_STD_I32LE, dataspace_id,
H5P_DEFAULT,H5P_DEFAULT,H5P_DEFAULT);
/* Write the dataset. */
status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data);
status = H5Dclose(dataset_id); /* Close the dataset. */
status = H5Fclose(file_id); /* Close the file. */
10 of 14
}

11. And here's the output:
h5dump WrittenByH5PY.h5
HDF5 "WrittenByH5PY.h5" {
GROUP "/" {
DATASET "TestDataset" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 4, 6 ) / ( 4, 6 ) }
DATA {
(0,0): 1, 2, 3, 4, 5, 6,
(1,0): 7, 8, 9, 10, 11, 12,
(2,0): 13, 14, 15, 16, 17, 18,
(3,0): 19, 20, 21, 22, 23, 24
}
}
}
}
11 of 14

12. Python and the Three Pillars of HDF5 Tools
Python is well suited to Text Processing
Python has wide range of string manipulation functions, an easyto-use regular expression module, and list and dictionary (hash
table) objects. No segmentation faults!
Python is well suited to Tree Walking. Recursive
functions and loops over lists are easy to write
Object Level Operations...Not so much.
Object Level Operations (e.g. copy, diff) are challenging to write
efficiently and should be provided as part of the API by the HDF
Group, for example h5o_copy. API functions are available to the
Python programmer via h5py.
12 of 14

13. Why use Python to substitute HDF5 tools?
Python is available now.
Some HDF5 tools are still under development as new use
cases are presented. For example, users have requested a
tool to add attributes to HDF5 files. Such a capability
already exists with h5py:
python -c "import h5py ; fid = h5py.File('FileForAttributeAddition.h5','r+') ;
fid['/TestDataset'].attrs['CmdLine1'] = 'NewValue' ; fid.close()"
It's little ugly, but it is available today.
Python is a full programming language. It can accomplish
tasks which HDF5 tools cannot.
Further Resources:
http://groups.google.com/group/h5py
http://h5py.alfven.org/
13 of 14

14. Recommendations:
Users should consider Python and H5py to accomplish their HDF5
file manipulation projects.
The HDF Group should concentrate on providing efficient
API functions for object level tasks: object copy, dataset
difference, etc.
The HDF Group should avoid complex enhancements to tools
where Python/h5py could be used instead.
An easily searched contributed application repository on the HDF
Group website with user ratings would be very helpful.
14 of 14