Downloaded 15 times
Uploaded byJan Aerts
D Robinson - Using HDF5 to work with large quantities of rich biological data
The document provides an overview of the HDF5 technology stack, which is designed for organizing and storing large quantities of heterogeneous multidimensional biological data. It highlights the advantages of HDF5, including scalability, self-describing files, and support for parallel I/O and compression, while also noting the absence of a built-in query engine. Additionally, it mentions the availability of higher language bindings for various programming languages to facilitate data access.
More Related Content
Similar to D Robinson - Using HDF5 to work with large quantities of rich biological data
D Robinson - Using HDF5 to work with large quantities of rich biological data
- 1. Using HDF5 ToWork With Large Quantities of Rich Biological Data Dana Robinson (derobins @hdfgroup.org) The HDF Group July 13, 2012 BOSC 2012 1
- 2. Today's Goal Is thatyou walk away from this talk with a basic understanding of the HDF5 technology stack. July 13, 2012 BOSC 2012 2
- 3. Where is HDF5used? July 13, 2012 BOSC 2012 3
- 4. What is HDF5? HDF5is a highly scalable way to organize and store heterogeneous, multidimensional data of user-defined types. HDF5 also allows data relationships and context to be stored using annotation and linking. July 13, 2012 BOSC 2012 4
- 5. HDF5 The HDF5 technologysuite includes: • A structured binary file format • An abstract data model for describing your data • A data access library, written in C (w/ bindings for C++, Fortran 95/2003, and Java) July 13, 2012 BOSC 2012 5
- 6. HDF5 has characteristicsof … Directories and Files PDF • standard • hierarchical exchange format • collections of • heterogeneous related information HDF5 information Databases XML • subsetting • self-describing • random access Binary Flat File • extensible • high- types performance • rich metadata July 13, 2012 April 17-19, 2012 BOSC 2012 6
- 7. Advantages of HDF5 •Platform and architecture-independent • Scalable in space and time • File size only limited by OS and filesystem • Data access time (esp. parallel) scales well • Flexible (user-defined types and organization) • Files are self-describing July 13, 2012 BOSC 2012 7
- 8. Advantages of HDF5(2) • High-performance • Parallel I/O via MPI-IO • Supports compression and other filters • Open source (BSD license) • THG committed to provide long-term support July 13, 2012 BOSC 2012 8
- 9. HDF5 Data Objects •Groups • Datatypes • Datasets • Metadata (Attributes) July 13, 2012 BOSC 2012 9
- 10. Example: LCMS Data sample name chromatography parameters ms parameters ms/ms parameters July 13, 2012 BOSC 2012 10
- 11. HDF5 Data Access Unlikemany data storage systems, HDF5 has no built-in query engine or indexes. You will have to write your own data access code, usually using the HDF5 API. July 13, 2012 BOSC 2012 11
- 12. Dataspaces HDF5 has arich set of data subsetting functionality. Example: displaying a thumbnail of a high- resolution image. July 13, 2012 BOSC 2012 12
- 13. Filters and Compression HDF5 supports data filters, including compression, which transform data as it enters or leaves the file. compression filter compressed data uncompressed data in the file in user's buffer Note that HDF5 data objects are filtered individually, not the entire file! July 13, 2012 BOSC 2012 13
- 14. Higher Language Bindings C++ Fortran (95 & 2003) Java .NET Python • C++ & Fortran distributed with library • Java distributed separately • .NET distributed separately, not supported by THG (as-is) • Python (PyTables, h5py) not distributed by THG NOTE: HDF5 bindings are thin wrappers over the C API. • There is no object-oriented interface to HDF5 • Not pure Java, .NET, etc. July 13, 2012 BOSC 2012 14
- 15. Questions? Helpful links THG www.hdfgroup.org Downloads www.hdfgroup.org/HDF5/release/obtain5.html Documentation www.hdfgroup.org/HDF5/doc/index.html Bioinformatics www.hdfgroup.org/projects/bioinformatics/ Tutorials www.hdfgroup.org/HDF5/Tutor/index.html Contact/help desk www.hdfgroup.org/about/contact.html July 13, 2012 BOSC 2012 15
Editor's Notes
- #3 HDF is an ADJECTIVE
- #4 Add Sony Pictures
- #5 The second statement is what we mean by "rich"
- #6 High-level view, point out that the file format is NOT "HDF5" (mention VOL).Gerd is a little unhappy with "structured", but it should be ok for this audience.
- #7 HDF5 has the characteristics of other formats that are outthere.It’s hard to store metadata in a binary flat file and it is not scalable
- #8 Gerd points out that a library is properly a part of the self-describing representation
- #9 High performance can have many meanings
- #11 Again, note that links are named, not objects
- #12 Much more low-level than, say, an RDBMS, though the ease of use of a database can come at a performance cost"easy" access via Python, Gerd'sPowershell snap-in, etc.Can write your own data access API to create queries, etc.
- #13 Need to reword this! "These are calleddataspaces" = bad.
- #16 Add resource links to this slide
- #17 Why should you listen to my talk?
- #18 Note that links are named, not objects!Gerd thinks of names as NAVIGATORS
- #19 Wide variety of integer and floating point types, enum types, etc.Need to point out that variable-length strings have compression issues (fixable, with $$$)
- #20 Might mention sparsity for chunks here.Mike suggests not mentioning chunks, so perhaps that could be replaced with a note about sparse data.