This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files. The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday. The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.

1. The HDF Group
HDF Tools Tutorial
Peter Cao, The HDF Group
Jonathan Kim, The HDF Group
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
1
www.hdfgroup.org

2. Tools Tutorial
• HDFView
• HDF5 Command-line tools
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
2
www.hdfgroup.org

3. HDFView
• View file content
• Edit file content
• Show NPOESS/JPSS related features
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
3
www.hdfgroup.org

4. HDFView
• Download HDF-Java 2.7 beta from
http://www.hdfgroup.org/ftp/HDF5/hdf-java-2.7
• Download example files from
ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/WorkshopXIV/HDF5-Files/
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
4
www.hdfgroup.org

5. HDFView
• View file content
•
•
•
•
•
Open OMI-Aura_L3.he5
Open groups (try expand all)
Stretch the metadata view for showing metadata
Double click on “ColumnAmountO3” to show data in table
Use “Open As” to show “ColumnAmountO3” in image
• View a subset
• Open weather.h5
• Select from preview image or selection fields
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
5
www.hdfgroup.org

6. HDFView
• Copy/paste a dataset and a group
• Create a new file: tmp.h5
• Copy/paste “ColumnAmountO3” and show the image from
the new file
• Copy the group “GRIDS” to show the whole group is copied
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
6
www.hdfgroup.org

7. HDFView
• Create a dataset and add an attribute
•
•
•
•
•
Create /d1 (32-bit float, 720x1440)
Copy “ColumnAmountO3” to /d1
Add attribute “IMAGE_MINMAXRANGE = 123.0,447.0”
Export /d1 to text file
Create /d2 and import d1.txt
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
7
www.hdfgroup.org

8. HDFView
• NPOESS/JPSS related features
• Object reference
• Region references
• Quality flags
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
8
www.hdfgroup.org

9. HDFView
• Object reference
1)
2)
3)
4)
5)
6)
Open file: SVI-NPP.h5
Open “VIIRS-I1-SDR_Aggr”
Click on object ID and shows object name
Right mouse to click to open data as image
Open data in table
Select two cells and open them at once
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
9
www.hdfgroup.org

10. HDFView
• Region references
1)
2)
3)
4)
Open “VIIRS-I1-SDR_Gran”
Right mouse to click to open data as image
Open data in table
Select two cells and open at once
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
10
www.hdfgroup.org

11. HDFView
• Quality flags
1) Quality flags
•
•
•
•
Land/Water Background
SST Skin Quality
SST Bulk Quality
Aerosol Correction
-- 1-bit
-- 2-bits
-- 2-bits
-- 3-bits
2) Open QF_VIIRSI1SDR_Array
3) Use "Open As" on QF_VIIRSI1SDR_Array
4) Select first bit
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
11
www.hdfgroup.org

12. HDFView
• Advanced features
1) Import/export images
2) Use HDF-EOS plugin
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
12
www.hdfgroup.org

13. HDF5 Command-line tools
•
•
•
•
•
September 28-30, 2010
h5ls
h5dump
h5repack
h5diff
h5copy
HDF and HDF-EOS Workshop XIV
13
www.hdfgroup.org

14. h5ls
• Show file content list
• h5ls ami.h5
• h5ls –r ami.h5
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
14
www.hdfgroup.org

15. h5dump
•
•
•
•
•
September 28-30, 2010
Structure
Dataset
Binary
XML
Region references
HDF and HDF-EOS Workshop XIV
15
www.hdfgroup.org

16. h5dump
• Structure
• h5dump -H tools_diff1.h5
• h5dump -H ami.h5
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
16
www.hdfgroup.org

17. h5dump
• Dataset
• Content
• h5dump -d /EnSight_model/geometry/parts/part_2/quad4 ami.h5
• Information
• h5dump –p –H -d /EnSight_model/geometry/parts/part_2/quad4 ami.h5
• Subset
• H5dump -d /EnSight_model/geometry/parts/part_2/quad4 -s 10,0 -c 20,2 ami.h5
• <-s and –c must be used after –d>
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
17
www.hdfgroup.org

18. h5dump
• Binary
•
h5dump -d /EnSight_model/geometry/parts/part_2/quad4 -b LE –o out.bin ami.h5
• XML
• h5dump –x ami.h5
• Region
•
•
h5dump -d /Data_Products/VIIRS-I1-SDR/VIIRS-I1-SDR_Gran_0 -s 3 -c 1 SVI-NPP.h5
h5dump –R -d /Data_Products/VIIRS-I1-SDR/VIIRS-I1-SDR_Gran_0 -s 3 -c 1 SVI-NPP.h5
• Packed fields <only 1.8.5+>
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
18
www.hdfgroup.org

19. h5repack
• Remove inaccessible objects / junk spaces
• Change storage layout
• Apply compression filter
Copies a file to a new file
with different storage layouts
and compression filters
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
19
www.hdfgroup.org

20. h5repack
• Remove inaccessible objects
• h5dump tools_junk.h5
• h5repack tools_junk.h5 tmp.h5
• Change layout
• h5dump -H tools_bad_layout.h5
• h5repack tools_bad_layout.h5 tmp.h5
• h5repack -l CHUNK=16x16 tools_bad_layout.h5 tmp.h5
• Change compression
• h5repack -f GZIP=6 tmp.h5 tmp2.h5
• More: use latest version, link storage, shared message, and etc
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
20
www.hdfgroup.org

21. h5diff
Show differences between
two files or two objects
• Dataset
• Whole file
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
21
www.hdfgroup.org

22. h5diff
• Dataset
• h5diff -r tools_diff1.h5 tools_diff2.h5 /g1/uint8 /x
• Whole file
• Run “h5ls –r tools_diff1.h5” and “h5ls -r tools_diff2.h5”
• h5diff tools_diff1.h5 tools_diff2.h5
• h5diff –v tools_diff1.h5 tools_diff2.h5
• Follow symlinks
• h5diff -r --follow-symlinks tools_diff1.h5 tools_diff2.h5 /sl1
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
22
www.hdfgroup.org

23. h5copy
• Dataset
• Group
Copies
objects
to files…
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
23
www.hdfgroup.org

24. h5copy
• Dataset
• h5copy -i tools_input.h5 -o tmp.h5 -s /g1/g2/f -d /float
• Group
• Default: h5copy -i tools_input.h5 -o tmp.h5 -s /g1 -d /g1
• Parent groups: h5copy -p -i tools_input.h5 -o tmp.h5 -s /g1/g2/f -d /g1/g2/f
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
24
www.hdfgroup.org

25. Thank You!
Questions/comments?
September 28-30, 2010
HDF and HDF-EOS Workshop XIV
25
www.hdfgroup.org