Download to read offline
Uploaded byThe HDF-EOS Tools and Information Center
HDF Updae
The document outlines updates and milestones from the HDF and HDF-EOS Workshop IX, focusing on organizational changes, software updates, and future developments related to the HDF group. Key highlights include the establishment of 'The HDF Group', major software releases, and ongoing projects in various domains such as bioinformatics and hydroinformatics. The document also details enhancements in HDF tools, parallel I/O performance, and the integration of new APIs and utilities.
More Related Content
More from The HDF-EOS Tools and Information Center
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
HDF Updae
- 1. HDF Update Mike Folk NationalCenter for Supercomputing Applications HDF and HDF-EOS Workshop IX December 1, 2005 -1- HDF
- 2. Outline • Organizational info •HDF Software Update • Other Activities of Interest -2- HDF
- 3.
- 4. The HDF Team FrankBaker Christian Chilan Peter Cao Vailin Choi Mike Folk Fang Guo Anne Jennings Barbara Jones Quincey Koziol James Laird Raymond Lu John Mainzer Pedro Nunes Elena Pourmal Binh-minh Ribler Eric Shapiro Rishi Sinha Arash Termehchy Kent Yang And all those wonderful folks out there who contribute ideas, requests, bug reports, code, and support. -4- HDF
- 5. The HDF Groupis Moving HDF -7- HDF
- 6. “The HDF Group”= “THG” -8- HDF
- 7. THG • Why spinoff from U of Illinois? – Creating a sustainable organization – We do more than R&D • THG already exists -9- HDF
- 8. How will THGbe different from the NCSA HDF Group? • • • • • • Business model Location Staff THG – NCSA – UIUC relations Affect on NASA and other affiliation Intellectual property - 10 - HDF
- 9. HDF Software Update -11 - HDF
- 10. Major software milestonessince Oct. 2004 HDF Java 2.1 HDF Web browser plug-in HDF 4.2r1 HDF5 1.6.4 HDF4-to-HDF5 conversion tools 1.2 HDF Java 2.2 HDF5 1.6.5 Nov Dec Jan Feb Mar Apr May Jun 2004 2005 - 12 - Jul Aug Sep Oct Nov HDF
- 11.
- 12. HDF 4.2r1 –February 2005 • Szip compression fixed • Windows – hdiff and hrepack added – Config, build, testing procedures improved • h4fc utility fixed - 14 - HDF
- 13. HDF 4.2r1 –new compilers and platforms • Mac OS X • Linux 2.4 – Fortran IBM xlf v. 8.1 – Absoft f95 v. 8.2 • AMD Opteron • Cray TS IEEE - 15 - – Absoft Fortran f95 v. 9.0 – PGI C and Fortran – Intel C and Fortran HDF
- 14. HDF5-1.6.4 – March2005 • High-Level (HL) library – Some new C APIs added – Fortran APIs added – HL library now built and installed by default • Library built and tested with SZIP 2.0. • Many changes to improve library performance – Especially for variable length types and metadata cache • H5jam – a new utility – Allows a text file to be added to the "user block" at the beginning of an HDF5 file - 16 - HDF
- 15. Platforms to bedropped in future releases • Operating systems – – – – – Solaris 2.8 HPUX B.11.00 Crays T3E and T90 Linux RH 7.* and 8.* Windows 2000 - 17 - • Compilers – We use the latest versions of vendors compilers as they become available and drop the previous ones HDF
- 16. Platforms to beadded • Systems • Compilers – – – – – Solaris 2.10 Cray X1 Cray XT3 NEC SX6 HP 64-bit (HPUX 11.23) – Mac OS 10.4 - 18 - – gcc 4.* – HDF5 Fortran: Leahy, NAG, G95 – MPI-2 HDF
- 17. Coming next: Majorrelease HDF5 1.8 • Windows MPICH support: prototype • Integer to float conversions – Will support integer to float conversions during I/O – http://hdf.ncsa.uiuc.edu/RFC/dtype_conv_overflow/Overflow.html • New error-handling API • Dimension scales – Similar to dimension scales in HDF4 – http://hdf.ncsa.uiuc.edu/RFC/H5DimScales/H5dimscale_Specification_1_ 0-5.pdf - 19 - HDF
- 18. • N-bit compressionfilter – Compact storage for user-defined datatypes. – http://hdf.ncsa.uiuc.edu/RFC/NBitPacking/NBitPacking.html • Offset+size storage filter – Performs a scale and/or offset operation on each data value, truncating the resulting value to a lesser number of bits before storing it. – http://hdf.ncsa.uiuc.edu/RFC/ScaleOffsetCompress/ScaleOffse tCompress.html - 20 - HDF
- 19. • Group revisions –Option to access objects according to creation order – Improved performance for groups containing a large number of objects. – http://hdf.ncsa.uiuc.edu/RFC/ReviseGroups/ • Improved metadata cache – New metadata cache improves performance and memory usage in the library. – Apps that access files with a large number of objects should see significant performance improvement and should use less memory. - 21 - HDF
- 20. • Data transformationfilter – Performs data transformation during I/O operations. – Transform expressed by algebraic formula (e.g. a*x + b) – http://hdf.ncsa.uiuc.edu/HDF5/doc_dev_snapshot/H5_dev/ html/RM_H5P.html#Property-SetDataTransform • Ph5diff – parallel h5diff – Compares two files in an MPI parallel environment. – Compares multiple datasets simultaneously. – http://hdf.ncsa.uiuc.edu/RFC/PH5DIFF/ - 22 - HDF
- 21. • HDFpacket API – – – – Datacollected in “packets” “Horizontal” view, per time step Efficient access to fixed- and variable-length records http://hdf.ncsa.uiuc.edu/RFC/HDF5Packet/Tech_reprt_ HDF5Packet.pdf • Possible: HDFtime_history API – Archival, viewing, analysis – “Vertical” view, per parameter - 23 - HDF
- 22. SZIP integration withHDF4 and HDF5 • Development and integration completed – Includes libraries and tools • SZIP documentation web page – http://hdf.ncsa.uiuc.edu/doc_resource/SZIP/ – Examples and performance studies for HDF5 - 24 - HDF
- 23. Parallel I/O andchunking • Collective I/O – key to improving performance for parallel HDF5 • Current versions only allow collective I/O for regular selection in contiguous storage • Expanding use of collective IO in HDF5 – For regular selection in chunked storage – For irregular selection for both chunked and contiguous storage - 25 - HDF
- 24. Java and othertools - 26 - HDF
- 25. Tools development • HDF4 –hrepack and hdiff performance improved • H4 to H5 Conversion Tools – Updated to HDF4.2r1, HDF5-1.6.4 • H5jam – New tools to add/remove user block in front of file • H5dump – Faster for files with large numbers of objects – Can dump contents of the boot block – Can dump dataset filters, storage layout, fill value • Parallel h5diff – Enables h5diff to run in parallel - 27 - HDF
- 26. HDF Java Products -28 - HDF
- 27. HDFView changes • Supportfor Storage Resource Broker (SRB) – HDF5 object level access to remote files • • • • • Display HDF5 compound datatypes with arrays Create/display HDF5 named datatypes Create links in HDF5 Improve ability to manipulate palette Select row/column for xy plot in the table view - 29 - HDF
- 28. New Functions inJava API • Request an individual object without loading entire structure of file • Send client request to SRB server and receive result from server • Create HDF5 indexing table • Query for HDF5 datasets - 30 - HDF
- 29. HDF Web-browser Plug-in • • • • • • Extendsbrowser to display HDF4/5 files A ‘lite” version of HDFView Analogous to PDF reader Fewer browsing features No editing features Windows Only - 31 - HDF
- 30.
- 31. HDF Web-browser Plug-in •Not an applet – It is downloaded and installed once – An applet is downloaded with each invocation • http://hdf.ncsa.uiuc.edu/plugins/ - 33 - HDF
- 32. HDF-EOS module forHDFView • Developed by HDF-EOS team • Optional module for HDF-EOS files – Reads, displays HDF-EOS grid, swath, etc. – (Generic modules show native HDF5 objects) • Tested with HDFView 2.3 • To do -- get permission to release with HDFView - 34 - HDF
- 33.
- 34. Future work forJava • Add OPeNDAP client support to HDFview – Seamlessly retrieve data from any OPeNDAP server • Support HDF5 Dimension Scales – Recognize geospatial coordinates • Support for HDF5 Indexing – Create indexing table and query HDF5 datasets • H5Gen – Generate HDF5 file from XML file - 36 - HDF
- 35. Other Activities ofInterest - 37 - HDF
- 36. DOE/ASC* “ASC provides theintegrating simulation and modeling capabilities and technologies needed …for future design assessment and certification of nuclear weapons and their components” • Massively parallel computing and I/O • Complex data models and big data • HDF5 a standard format for ASC apps * “Advanced Simulation and Computing Program” - 38 - HDF
- 37.
- 38. Boeing HDF5 for flighttest data • Commercial (Boeing 787) and military planes • 787 active archive – HDFtime_history – 10 TB per flight-test day – Also post-testing data • Must handle raw, real-time data – Variable-length datatypes/records – High speed ingest – HDFpacket API - 40 - HDF
- 39. Boeing High LevelAPI’s • HDFpacket (see above) • HDFtime_history – Structured records for archive, analysis, viewing – “Vertical” view, per parameter - 41 - HDF
- 40. Object encryption tosupport access control • For Boeing • Investigated the role of encryption in developing access control • Developed a prototype, now being tested - 42 - HDF
- 41.
- 42. Projection Indexes inHDF5 • • • • Standardize indexing in HDF5 Make indexes portable Just a prototype See Rishi Sinha’s talk - 44 - HDF
- 43. Product model data -45 - HDF
- 44. Product data exchange– STEP • STEP is an ISO data transfer standard. • Defines characteristics of product throughout its life cycle. • Widely used in design and manufacturing. STE P • Uses EXPRESS data modeling language to describe data. - 46 - HDF
- 45. STEP Limitations • Currentlytext-based format • Requires all the objects to be in memory • Apps starting to produce very large data volumes • EU looking for a binary equivalent for STEP - 47 - HDF
- 46. HDF5 as binaryformat for STEP • EU identified HDF5 as best candidate • Prototype in the works – EXPRESS HDF5 mappings – Convert sample data collections • Workshop at U of Illinois next week. • National Archives also funding HDF study. - 48 - HDF
- 47.
- 48. DNA sequencing workflows •Diverse formats, some proprietary • Highly redundant data • Repeated file processing • Disconnected programs • In-core processing models • Lack of persistence - 50 - HDF
- 49. Multiple Levels ofInformation SNP Score Contig Summaries Discrepancies Contig Qualities Coverage Depth Trace Reads Aligned bases Read quality Contig Percent match - 51 - HDF
- 50. HDF5 as binaryformat for bioinformatics - 52 - HDF
- 51. netCDF and OPeNDAP -54 - HDF
- 52. netCDF-HDF Project • EnhancedNetCDF-4 Interface to HDF5 – Combine features of netCDF and HDF5 – Take advantage of their separate strengths • Collaboration between NCSA and Unidata • Currently in Alpha Release - 55 - HDF
- 53. New OPeNDAP HDF5Project • Four parts – Bring existing prototype into conformance with the DAP2 NASA/ESE RFC – Develop a DAP4 server for HDF5 – Develop server-side utilities to convert DAP4 data responses to an HDF5 file – Investigate an integrated DAP-aware HDF5 library, that could provide seamless access to both local and remote data • Funded by NASA ROSES “Advancing Collaborative Connections for Earth-Sun System Science” - 56 - HDF
- 54.
- 55. Archival formats • RuthDuerr (NSIDC) initiated investigations • How to preserve the content & performance features of complex scientific data formats • At the same time provide the requisite simplicity needed for long term archival storage. • Ruth will speak about this - 58 - HDF
- 56.
- 57. Hydroinformatics • HDF5 asexchange format for hydroinformatics data – Groundswell of interest lately – Sometimes in combination with netCDF 4 – Talk to Mike Folk - 63 - HDF
- 58.
- 59. Thank You - 65- HDF
- 60.
- 61. Information Sources • HDFwebsite – http://hdf.ncsa.uiuc.edu/ • HDF5 Information Center – http://hdf.ncsa.uiuc.edu/HDF5/ • HDF Helpdesk – hdfhelp@ncsa.uiuc.edu • HDF users mailing list – hdfnews@ncsa.uiuc.edu - 67 - HDF