Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Migrating from HDF5 1.6 to 1.8

560 views

Published on

This tutorial targets HDF5 application developers and users who still use HDF5 1.6 releases and anyone who is interested in the HDF5 1.8.x libraries features. We will discuss how applications written for versions 1.6.x and earlier can be seamlessly moved to use the latest HDF5 releases. We will also talk about new features of the 1.8.x HDF5 Library such as redesigned group object, links, creation order, and different performance tuning knobs.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Migrating from HDF5 1.6 to 1.8

  1. 1. Migrating from HDF5 1.6 to HDF5 1.8 October 15, 2008 HDF and HDF-EOS Workshop XII 1
  2. 2. Outline • Status of the HDF5 1.6 and 1.8 releases • Overview of the HDF5 1.8 features • How to move applications to HDF5 1.8 ? October 15, 2008 HDF and HDF-EOS Workshop XII 2
  3. 3. Status of HDF5 releases October 15, 2008 HDF and HDF-EOS Workshop XII 3
  4. 4. Current HDF5 Releases • HDF5 1.8.0 was released in February 2008 • Major update of HDF5 1.6.* series (stable set of features and APIs since 1998) • • • • • New features 200 new APIs Changes to file format Changes to APIs Backward compatible • HDF5 1.8.1 was released in June 2008 • Minor bug fixes • Included Fortran90 APIs for new C functions October 15, 2008 HDF and HDF-EOS Workshop XII 4
  5. 5. Current HDF5 Releases • HDF5 1.6.7 was released in February 2008 • Addressed backward compatibility bug for reading files with corrupted object header information • New maintenance releases will be in November 2008 • HDF5 1.6.8 and 1.8.2 • Minor bug fixes • Tools improvements • Current plans are to support HDF5 1.6 and 1.8 until November 2009 October 15, 2008 HDF and HDF-EOS Workshop XII 5
  6. 6. Information About Current Releases http://www.hdfgroup.org/HDF5 October 15, 2008 HDF and HDF-EOS Workshop XII 6
  7. 7. Goal of the Tutorial • Help with transition to the 1.8 releases • Discuss new features beneficial to applications written for 1.6 releases • Raise awareness about forward/backward compatibility issues with the 1.8 releases • Get feedback from the users who already moved to 1.8 releases October 15, 2008 HDF and HDF-EOS Workshop XII 7
  8. 8. Why New Features? • Need to address some deficiencies in initial design • Examples: • Big overhead in file sizes • Non-tunable metadata cache implementation • Handling of free-space in a file October 15, 2008 HDF and HDF-EOS Workshop XII 8
  9. 9. Why New Features? • Need to address new requirements • Add support for • New types of indexing (object creation order) • Big volumes of variable-length data (DNA sequences) • Simultaneous real-time streams (fast append to one -dimensional datasets) • UTF-8 encoding for objects’ path names • Accessing objects stored in another HDF5 files (external or user-defined links) October 15, 2008 HDF and HDF-EOS Workshop XII 9
  10. 10. What Did We Do in HDF5 1.8? • • • • • • • • Extended File Format Specification Reviewed group implementations Introduced new link object Revamped metadata cache implementation Improved handling of datasets and datatypes Introduced shared object header message Extended error handling Enhanced backward/forward APIs and file format compatibility October 15, 2008 HDF and HDF-EOS Workshop XII 10
  11. 11. What Did We Do in HDF5 1.8? And much more good stuff to make HDF5 Better and Faster October 15, 2008 HDF and HDF-EOS Workshop XII 11
  12. 12. HDF5 File Format Extension October 15, 2008 HDF and HDF-EOS Workshop XII 12
  13. 13. HDF5 File Format Extension • Why: • Address deficiencies of the original file format • Address space overhead in an HDF5 file • Enable new features • What: • New routine that instructs the HDF5 library to create all objects using the latest version of the HDF5 file format (cmp. with the earliest version when object became available, for example, array datatype) October 15, 2008 HDF and HDF-EOS Workshop XII 13
  14. 14. HDF5 File Format Extension Example /* Use the latest version of a file format for each object created in a file */ fapl_id = H5Pcreate(H5P_FILE_ACCESS); H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST); fid = H5Fcreate(…,…,…,fapl_id); or fid = H5Fopen(…,…,fapl_id); October 15, 2008 HDF and HDF-EOS Workshop XII 14
  15. 15. Group Revisions October 15, 2008 HDF and HDF-EOS Workshop XII 15
  16. 16. Better Large Group Storage • Why: • Faster, more scalable storage and access for large groups • What: • New format and method for storing groups with many links October 15, 2008 HDF and HDF-EOS Workshop XII 16
  17. 17. Informal Benchmark • Create a file and a group in a file • Create up to 10^6 groups with one dataset in each group • Compare files sizes and performance of HDF5 1.8.1 using the latest group format with the performance of HDF5 1.8.1 (default, old format) and 1.6.7 • Note: Default 1.8.1 and 1.6.7 became very slow after 700000 groups October 15, 2008 HDF and HDF-EOS Workshop XII 17
  18. 18. Time to Open and Read a Dataset October 15, 2008 HDF and HDF-EOS Workshop XII 18
  19. 19. Time to Close the File October 15, 2008 HDF and HDF-EOS Workshop XII 19
  20. 20. File Size October 15, 2008 HDF and HDF-EOS Workshop XII 20
  21. 21. Access Links by Creation Order • Why: • Allow iteration and lookup of group’s links (children) by creation order as well as by name order • Support NetCDF access model for NetCDF-4 • What: • Option to access objects in group according to relative creation time October 15, 2008 HDF and HDF-EOS Workshop XII 21
  22. 22. Access Links by Creation Order Example /* Track and index creation order of the links */ H5Pset_link_creation_order(gcpl_id, (H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED)); /* Create a group */ gid = H5Gcreate(fid, GNAME, H5P_DEFAULT, gcpl_id, H5P_DEFAULT); October 15, 2008 HDF and HDF-EOS Workshop XII 22
  23. 23. Example: h5dump --group=1 tordergr.h5 HDF5 "tordergr.h5" { GROUP "1" { GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a21" { } GROUP "a22" { } } } GROUP "b" { } GROUP "c" { … October 15, 2008 HDF and HDF-EOS Workshop XII 23
  24. 24. Example: h5dump --sort_by=creation_order HDF5 "tordergr.h5" { GROUP "1" { GROUP "c" { } GROUP "b" { } GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a22" { } GROUP "a21" { } } } October 15, 2008 HDF and HDF-EOS Workshop XII 24
  25. 25. Compact Groups • Why: • Save space and access time for small groups • If groups are small, don’t need B-tree overhead • What: • Alternate storage for groups with few links • Default storage when “latest format” is specified • Library converts to “original” storage (B-tree based) using default or user-specified threshold October 15, 2008 HDF and HDF-EOS Workshop XII 25
  26. 26. Compact Groups • Example • • • • • File with 11,600 groups With original group structure, file size ~ 20 MB With compact groups, file size ~ 12 MB Total savings: 8 MB (40%) Average savings/group: ~700 bytes October 15, 2008 HDF and HDF-EOS Workshop XII 26
  27. 27. Compact Groups Example /* Change storage to “dense” if number of group members is bigger than 16 and go back to compact storage if number of group members is smaller than 12 */ H5Pset_link_phase_change(gcpl_id, 16, 12) /* Create a group */ g_id = H5Gcreate(…,…,…,gcpl_id,…); October 15, 2008 HDF and HDF-EOS Workshop XII 27
  28. 28. Intermediate Group Creation • Why: • Simplify creation of a series of connected groups • Avoid having to create each intermediate group separately, one by one • What: • Intermediate groups can be created when creating an object in a file, with one function call October 15, 2008 HDF and HDF-EOS Workshop XII 28
  29. 29. Intermediate Group Creation • Want to create “/A/B/C/dset1” • “A” exists, but “B/C/dset1” do not / / A A B C dset1 One call creates groups “B” & “C”, then creates “dset1” October 15, 2008 HDF and HDF-EOS Workshop XII 29
  30. 30. Intermediate Group Creation Example /* Create link creation property list */ lcrp_id = H5Pcreate(H5P_LINK_CREATE); /* Set flag for intermediate group creation Groups B and C will be created automatically */ H5Pset_create_intermediate_group(lcrp_id, TRUE); ds_id = H5Dcreate (file_id, "/A/B/C/dset1",…,…, lcrp_id,…,…,); October 15, 2008 HDF and HDF-EOS Workshop XII 30
  31. 31. Link Revisions October 15, 2008 HDF and HDF-EOS Workshop XII 31
  32. 32. What are Links? • Links connect groups to their members • “Hard” links point to a target by address • “Soft” links store the path to a target root group Hard link <address> Soft link “/target dataset” dataset October 15, 2008 HDF and HDF-EOS Workshop XII 32
  33. 33. Links: Before and After • New data model for handling links • Links may have properties (UTF-8 name encoding, creation order indexing, storage property, etc.) Before After Group Group Name and other properties Name Object October 15, 2008 Object HDF and HDF-EOS Workshop XII 33
  34. 34. Anonymous Object • Object can be created without being immediately linked into graph structure • Group, dataset and datatype • See new H5*create_anon APIs Group Object • Use H5O* APIs to manipulate the objects October 15, 2008 HDF and HDF-EOS Workshop XII 34
  35. 35. New: External Links • Why: • Access objects stored in other HDF5 files in a transparent way • What: • Store location of file and path within that file • Can link across files October 15, 2008 HDF and HDF-EOS Workshop XII 35
  36. 36. New: External Links file2.h5 root group file1.h5 “target object” root group <address> “External_link” “file2.h5” “/A/B/C/D/E” group External link object “External_link” in file1.h5 points to the group /A/B/C/D/E in file2.h5 October 15, 2008 HDF and HDF-EOS Workshop XII 36
  37. 37. External Links Example /* Create an external link */ H5Lcreate_external(TARGET_FILE, ”/A/B/C/D/E", source_file_id, ”External_link”, …,…); /* We will use external link to create a group in a target file */ gr_id = H5Gcreate(source_file_id,”External_link/F”,…, …,…,…); /* We can access group “External_link/F” in the source file and group “/A/B/C/D/E/F” in the target file */ October 15, 2008 HDF and HDF-EOS Workshop XII 37
  38. 38. New: User-defined Links • Why: • Allow applications to create their own kinds of links and link operations, such as • Create “hard” external link that finds an object by address • Create link that accesses a URL • Keep track of how often a link is accessed, or other behavior • What: • Applications can create new kinds of links by supplying custom callback functions • Can do anything HDF5 hard, soft, or external links do October 15, 2008 HDF and HDF-EOS Workshop XII 38
  39. 39. Traversing an HDF5 File October 15, 2008 HDF and HDF-EOS Workshop XII 39
  40. 40. Traversing HDF5 File • Why: • Allow applications to iterate through the objects in a group or visit recursively all objects under a group • What: • New APIs to traverse a group hierarchy • New APIs to iterate through a group using different types of indices (name or creation order) • H5Giterate is deprecated in favor of new functions October 15, 2008 HDF and HDF-EOS Workshop XII 40
  41. 41. Traversing HDF5 File Example of some new APIs /* Check if object “A/B” exists in a root group */ H5Lexists(file_id, “A/B”, …); /* Iterate through group members of a root group using name as an index; this function doesn’t recursively follow links into subgroups */ H5Literate(file_id, H5_INDEX_NAME, H5_ITER_INC, &idx, iter_link_cb, &info); /* Visit all objects under the root group; this function recursively follow links into subgroups */ H5Lvisit(file_id, H5_INDEX_NAME, H5_ITER_INC, visit_link_cb, &info); October 15, 2008 HDF and HDF-EOS Workshop XII 41
  42. 42. Traversing HDF5 File • Things to remember • Never use H5Ldelete in any HDF5 iterate or visit call back functions • Always close parent object before deleting a child object October 15, 2008 HDF and HDF-EOS Workshop XII 42
  43. 43. Shared Object Header Messages October 15, 2008 HDF and HDF-EOS Workshop XII 43
  44. 44. Shared Object Header Messages • Why: metadata duplicated many times, wasting space • Example: • You create a file with 10,000 datasets • All use the same datatype and dataspace • HDF5 needs to write this information 10,000 times! Dataset 1 Dataset 2 Dataset 3 datatype datatype datatype dataspace dataspace dataspace data 1 data 2 data 3 October 15, 2008 HDF and HDF-EOS Workshop XII 44
  45. 45. Shared Object Header Messages What: • Enable messages to be shared automatically • HDF5 shares duplicated messages on its own! Dataset 1 Dataset 2 datatype dataspace data 1 October 15, 2008 data 2 HDF and HDF-EOS Workshop XII 45
  46. 46. Shared Messages • Happens automatically • Works with datatypes, dataspaces, attributes, fill values, and filter pipelines • Saves space if these objects are relatively large • May be faster if HDF5 can cache shared messages • Drawbacks • Usually slower than non-shared messages • Adds overhead to the file • Index for storing shared datatypes • 25 bytes per instance • Older library versions can’t read files with shared messages October 15, 2008 HDF and HDF-EOS Workshop XII 46
  47. 47. Two Informal Tests • File with 24 datasets, all with same big datatype • 26,000 bytes normally • 17,000 bytes with shared messages enabled • Saves 375 bytes per dataset • But, make a bad decision: invoke shared messages but only create one dataset… • 9,000 bytes normally • 12,000 bytes with shared messages enabled • Probably slower when reading and writing, too. • Moral: shared messages can be a big help, but only in the right situation! October 15, 2008 HDF and HDF-EOS Workshop XII 47
  48. 48. Error Handling October 15, 2008 HDF and HDF-EOS Workshop XII 48
  49. 49. Extendible Error-handling APIs • Why: Enable application to integrate error reporting with HDF5 library error stack • What: New error handling API • H5Epush - push major and minor error ID on specified error stack • • • • • H5Eprint – print specified stack H5Ewalk – walk through specified stack H5Eclear – clear specified stack H5Eset_auto – turn error printing on/off for specified stack H5Eget_auto – return settings for specified stack traversal October 15, 2008 HDF and HDF-EOS Workshop XII 49
  50. 50. Error-handling Programming Model • Create new class, major and minor error messages • Register messages with the HDF5 library • Manage errors • • • • Use default or create new error stack Push error Print error stack Close stack October 15, 2008 HDF and HDF-EOS Workshop XII 50
  51. 51. Error-handling Example #define ERR_CLS_NAME "Error Test" #define PROG_NAME "Error Program" #define PROG_VERS "1.0” …… #define ERR_MAJ_TEST_MSG "Error in test” #define ERR_MIN_MYFUNC_MSG "Error in my function” …… /* Initialize error information for application */ ERR_CLS = H5Eregister_class(ERR_CLS_NAME, PROG_NAME, PROG_VERS); ERR_MAJ_TEST = H5Ecreate_msg(ERR_CLS, H5E_MAJOR, ERR_MAJ_TEST_MSG); ERR_MIN_MYFUNC = H5Ecreate_msg(ERR_CLS, H5E_MINOR, ERR_MIN_MYFUNC_MSG); …….. /* Unregister major and minor error, and class handles when done */ H5Eunregister_class(ERR_CLS); October 15, 2008 HDF and HDF-EOS Workshop XII 51
  52. 52. Error-handling Example /* This function creates and write a dataset */ static herr_t my_function(hid_t fid) { ……. /* Force this function to fail and make it push error */ H5E_BEGIN_TRY { dataset = H5Dcreate1(FAKE_ID, DSET_NAME, H5T_STD_I32BE, space, H5P_DEFAULT); } H5E_END_TRY; if(dataset < 0) { H5Epush(H5E_DEFAULT, __FILE__, FUNC_my_function, __LINE__, ERR_CLS, ERR_MAJ_IO, ERR_MIN_CREATE, "H5Dcreate failed"); goto error; } /* end if */ …… October 15, 2008 HDF and HDF-EOS Workshop XII 52
  53. 53. Error-handling Example Error Test-DIAG: Error detected in Error Program (1.0) thread 0: #000: error_example.c line 160 in main(): Error stack test failed major: Error in test minor: Error in my function #001: error_example.c line 100 in my_function(): H5Dcreate failed major: Error in IO minor: Error in H5Dcreate HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 0: #002: H5Ddeprec.c line 154 in H5Dcreate1(): not a location ID major: Invalid arguments to routine minor: Inappropriate type #003: H5Gloc.c line 241 in H5G_loc(): invalid object ID major: Invalid arguments to routine minor: Bad value October 15, 2008 HDF and HDF-EOS Workshop XII 53
  54. 54. Metadata Cache October 15, 2008 HDF and HDF-EOS Workshop XII 54
  55. 55. Metadata Cache Improvements • Why: • Improve I/O performance and memory usage when accessing many objects • What: • New metadata cache APIs • control cache size • monitor actual cache size and current hit rate • Under the hood: adaptive cache resizing • Automatically detects the current working size • Sets max cache size to the working set size October 15, 2008 HDF and HDF-EOS Workshop XII 55
  56. 56. Metadata Cache Improvements • Note: most applications do not need to worry about the cache. • See “Special topics” in the HDF5 User’s Guide for details. • And if you do see unusual memory growth or poor performance, please contact us. We want to help you. October 15, 2008 HDF and HDF-EOS Workshop XII 56
  57. 57. Dataset and Datatype Improvements October 15, 2008 HDF and HDF-EOS Workshop XII 57
  58. 58. Text-based Datatype Descriptions • Why: • Simplify data type creation • Make data type creation code more readable • Facilitate debugging by printing the text description of a data type • What: • New routines to create an HDF5 data type through the text description of the data type and get a text description from the HDF5 data type October 15, 2008 HDF and HDF-EOS Workshop XII 58
  59. 59. Text Datatype Description Example /* Create the data type from DDL text description */ dtype = H5LTtext_to_dtype( "H5T_IEEE_F32BEn”,H5LT_DDL); /* Convert the data type back to text */ H5LTtype_to_text(dtype, NULL, H5LT_DLL, str_len); dt_str = (char*)calloc(str_len, sizeof(char)); H5LTdtype_to_text(dtype, dt_str, H5LT_DDL, &str_len); October 15, 2008 HDF and HDF-EOS Workshop XII 59
  60. 60. Serialized Datatypes and Dataspaces • Why: • Allow datatype and dataspace info to be transmitted between processes • Allow datatype/dataspace to be stored in nonHDF5 files • What: • A new set of routines to serialize/deserialize HDF5 datatypes and dataspaces. October 15, 2008 HDF and HDF-EOS Workshop XII 60
  61. 61. Serialized Datatypes and Dataspaces Example /* Find the buffer length and encode a datatype into buffer */ status = H5Tencode(t_id, NULL, &cmpd_buf_size); cmpd_buf = (unsigned char*)calloc(1, cmpd_buf_size); H5Tencode(t_id, cmpd_buf, &cmpd_buf_size) /* Decode a binary description of a datatype and retune a datatype handle */ t_id = H5Tdecode(cmpd_buf); October 15, 2008 HDF and HDF-EOS Workshop XII 61
  62. 62. Integer to Float Convert During I/O • Why: • HDF5 1.6 and earlier supported conversion within the same class (16-bit integer 32-bit integer, 64bit float  32-bit float) • Conversion needed to support NetCDF-4 programming model • What: • Integer to float conversion supported during I/O October 15, 2008 HDF and HDF-EOS Workshop XII 62
  63. 63. Integer to Float Convert During I/O Example: conversion is transparent to application /* Create a dataset of 64-bit little-endian type */ dset_id = H5Dcreate(loc_id,“Mydata”, H5T_IEEE_F64LE,space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); /* Write integer data to “Mydata” */ status = H5Dwrite(dset_id, H5T_NATIVE_INT, …); October 15, 2008 HDF and HDF-EOS Workshop XII 63
  64. 64. Revised Conversion Exception Handling • Why: • Give apps greater control over exceptions (range errors, etc.) during datatype conversion • Needed to support NetCDF-4 programming model • What: • Revised conversion exception handling October 15, 2008 HDF and HDF-EOS Workshop XII 64
  65. 65. Revised Conversion Exception Handling • To handle exceptions during conversions, register handling function through H5Pset_type_conv_cb(). • Cases of exception: • • • • • • • H5T_CONV_EXCEPT_RANGE_HI H5T_CONV_EXCEPT_RANGE_LOW H5T_CONV_EXCEPT_TRUNCATE H5T_CONV_EXCEPT_PRECISION H5T_CONV_EXCEPT_PINF H5T_CONV_EXCEPT_NINF H5T_CONV_EXCEPT_NAN • Return values: H5T_CONV_ABORT, H5T_CONV_UNHANDLED, H5T_CONV_HANDLED October 15, 2008 HDF and HDF-EOS Workshop XII 65
  66. 66. Compression Filter for N-bit Data • Why: • Compact storage for user-defined datatypes • What: • When data stored on disk, padding bits chopped off and only significant bits stored • Works with compound datatypes October 15, 2008 HDF and HDF-EOS Workshop XII 66
  67. 67. N-bit Compression Example • In memory, one value of N-Bit datatype is stored like this: | byte 3 | byte 2 | byte 1 | byte 0 | |????????|????SPPP|PPPPPPPP|PPPP????| S-sign bit P-significant bit ?-padding bit • After passing through the N-Bit filter, all padding bits are chopped off, and the bits are stored on disk like this: | 1st value | 2nd value | |SPPPPPPP PPPPPPPP|SPPPPPPP PPPPPPPP|... • Opposite (decompress) when going from disk to memory • Limited to integer and floating-point data October 15, 2008 HDF and HDF-EOS Workshop XII 67
  68. 68. N-bit Compression Example Example /* Create a N-bit datatype */ dt_id = H5Tcopy(H5T_STD_I32LE); H5Tset_precision(dt_id, 16); H5Tset_offset(dt_id, 4); /* Create and write a dataset */ dcpl_id = H5Pcreate(H5P_DATASET_CREATE); H5Pset_chunk(dcpl_id, …); H5Pset_nbit(dcpl_id); dset_id = H5Dcreate(…,…,…,…,…,dcpl_id,…); H5Dwrite(dset_id,…,…,…,…,buf); October 15, 2008 HDF and HDF-EOS Workshop XII 68
  69. 69. Offset+size Storage Filter • Why: • Use less storage when less precision needed • What: • • • • Performs scale/offset operation on each value Truncates result to fewer bits before storing Currently supports integers and floats Precision may be lost October 15, 2008 HDF and HDF-EOS Workshop XII 69
  70. 70. Example with Floating-Point Type • Data: {104.561, 99.459, 100.545, 105.644} • Choose scaling factor: decimal precision to keep E.g. scale factor D = 2 1. Find minimum value (offset): 99.459 2. Subtract minimum value from each element Result: {5.102, 0, 1.086, 6.185} 3. Scale data by multiplying 10D = 100 Result: {510.2, 0, 108.6, 618.5} 4. Round the data to integer Result: {510 , 0, 109, 619} 5. Pack and store using min number of bits October 15, 2008 HDF and HDF-EOS Workshop XII 70
  71. 71. Offset+size Storage Filter Example /* Use scale+offset filter on integer data; let library figure out the number of minimum bits necessary to story the data without loss of precision */ H5Pset_scaleoffset (dcrp_id,H5Z_SO_INT,H5Z_SO_INT_MINBITS_DEFAULT); H5Pset_chunk(dcrp_id,…,…); dset_id = H5Dcreate(…,…,…,…,…,dcpl_id, …); /* Use sclae+offset filter on floating-point data; compression may be lossy */ H5Pset_scaleoffset(dcrp_id,H5Z_SO_FLOAT_DSCALE,2 ); October 15, 2008 HDF and HDF-EOS Workshop XII 71
  72. 72. “NULL” Dataspace • Why: • Allow datasets with no elements to be described • NetCDF-4 needed a “place holder” for attributes • What: • A dataset with no dimensions, no data October 15, 2008 HDF and HDF-EOS Workshop XII 72
  73. 73. NULL Dataspace Example /* Create a dataset with “NULL” dataspace*/ sp_id = H5Screate(H5S_NULL); dset_id = H5Dcreate(…,"SDS.h5”,…,sp_id,…,…,…); HDF5 "SDS.h5" { GROUP "/" { DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE NULL DATA { } } } } October 15, 2008 HDF and HDF-EOS Workshop XII 73
  74. 74. Part II How to Move an Application to HDF5 1.8 October 15, 2008 HDF and HDF-EOS Workshop XII 74
  75. 75. If you are new to HDF5 … • Start with the HDF5 Tutorial and use HDF5 1.8.1 (or later) libraries http://www.hdfgroup.org/HDF5/Tutor/ • Look at the comprehensive set of examples http://www.hdfgroup.uiuc.edu/UserSupport/examples-by- • Use HDF5 Reference Manual 1.8.1 or later http://www.hdfgroup.org/HDF5/doc to get the latest APIs • Use h5cc, h5fc, h5c++ scripts to build your applications October 15, 2008 HDF and HDF-EOS Workshop XII 75
  76. 76. If you are new to HDF5 … • To check library version • Talk to you system administrator • Try any HDF5 command line utility with –V flag ./h5dump -V h5dump: Version 1.8.1 • Look at the libhdf5.settings file on your system (under lib directory in the HDF5 installation directory); you should see the following lines HDF5 Version: 1.8.1 Default API Mapping: v18 (can be v16) • API mapping indicates API version (1.8 in this case) October 15, 2008 HDF and HDF-EOS Workshop XII 76
  77. 77. If you are new to HDF5 … • Use the latest version of API from the Reference Manual (for example, H5Gcreate or H5Gcreate2 instead of H5Gcreate1) • Be aware of forward compatibility issues • If you use HDF5 features introduced in 1.8.0 and later, you may create a file that old applications and third party tools may not be able to read http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html • All HDF5 command line utilities and tools should read HDF5 files created during the last 10 years • Report problems to help@hdfgroup.org October 15, 2008 HDF and HDF-EOS Workshop XII 77
  78. 78. If you used HDF5 before…. • And would like to switch to 1.8…. • Or you build and install HDF5 Libraries for yourself or others…. • You need to know about HDF5 API versioning. October 15, 2008 HDF and HDF-EOS Workshop XII 78
  79. 79. Introduction to HDF5 API Versioning October 15, 2008 HDF and HDF-EOS Workshop XII 79
  80. 80. API Versioning Example • Old way H5Gcreate ( loc_id, “New/My old group”, 0 ) H5Gcreate1( loc_id, “New/My old group”, 0 ) • No new features can be invoked • New way H5Gcreate ( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) H5Gcreate2( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) • New features can be invoked     Creation order Unicode names Compact storage Intermediate group creation • In 1.8.0 and later some functions have version number suffix. October 15, 2008 HDF and HDF-EOS Workshop XII 80
  81. 81. HDF5 Macros for API Compatibility Name: H5Gcreate Signature: hid_t H5Gcreate( hid_t loc_id, const char *name, size_t size_hint ) hid_t H5Gcreate( hid_t loc_id, const char *name, hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id Purpose: Creates a new empty group and links it to a location in the file. Description: H5Gcreate is a macro that is mapped to either H5Gcreate1 or H5Gcreate2, depending on the needs of the application. October 15, 2008 HDF and HDF-EOS Workshop XII 81
  82. 82. HDF5 Library Configuration • Mapping of APIs is set at configuration time • Several configuration flags are provided to enable different mappings • Macro name is mapped to old API, e.g., H5Gcreate is mapped to H5Gcreate1 • Macro name is mapped to new API, e.g., H5Gcreate is mapped to H5Gcreate2 (will have different parameters from 1.6 H5Gcreate) • Disable old APIs completely, e.g., H5Gcreate1 is not available in the library October 15, 2008 HDF and HDF-EOS Workshop XII 82
  83. 83. If you are HDF5 library maintainer… • If you do installation of the HDF5 Library, there are several choices • Install library with 1.8 APIs as default APIs (default, recommended). • Install library with 1.6 APIs. • Install library without 1.6 APIs to get smaller footprint. • Install library with strict file format checking option to detect files that do not comply with the File Format Specification. October 15, 2008 HDF and HDF-EOS Workshop XII 83
  84. 84. HDF5 Library Configuration Configure flag Public APIs has version 1.8 --with-default-api-version=v18 (default) (Example: H5Gcreate is mapped to H5Gcreate2, old H5Gcreate is H5Greate1) 1.8 --enable-strict-format-checks Library checks compliance with the file format 1.8 --disable-deprecated-symbols (Example: H5Gcreate is mapped to H5Gcreate2, H5Gcreate1 is not available) 1.6 --with-default-api-version=v16 October 15, 2008 (Example: H5Gcreate is mapped to H5Gcreate1, H5Gcreate2 is available) HDF and HDF-EOS Workshop XII 84
  85. 85. If you used HDF5 before… • And want to use new HDF5 1.8 Library with the application written for the older versions h5cc -DH5_USE_16_API my_program.c • H5Gcreate is mapped to H5Gcreate1; all three H5Gcreate1, H5Gcreate2 and H5Gcreate can be used • HDF5 compilation scripts and H5_USE_16_API flag can be used with GNU auto tools to build packages like HDF-EOS5. • The HDF Group runs daily tests for HDF-EOS5 using the latest HDF5 Libraries under development October 15, 2008 HDF and HDF-EOS Workshop XII 85
  86. 86. If you used HDF5 before … • Other available options: Assuming both deprecated and new symbols are available in the library: h5cc my_program.c • Both H5Gcreate1, H5Gcreate2 and H5Gcreate may be used • H5Gcreate will have new signature h5cc -DH5_NO_DEPRECATED_SYMBOLS my_program.c • Only new symbols are available for application; H5Gcreate is mapped to H5Gcreate2; application may use both, but cannot use H5Gcreate1 October 15, 2008 HDF and HDF-EOS Workshop XII 86
  87. 87. Example: --with-default-api-version=v18 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/A", H5P_DEAFULT, gcpt, gapt); grp2_id = H5Gcreate1(file_id,"/B",0); … grp3_id = H5Gcreate2(file_id,”New/A", H5P_DEAFULT, gcpt, gapt); October 15, 2008 HDF and HDF-EOS Workshop XII 87
  88. 88. Example: --with-default-api-version=v16 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, "/A",0); grp2_id = H5Gcreate1(file_id,"/B",0); grp3_id = H5Gcreate2(file_id,”New/C", H5P_DEAFULT, gcpt, gapt); October 15, 2008 HDF and HDF-EOS Workshop XII 88
  89. 89. Example: --disable-deprecated-symbols hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/A", H5P_DEAFULT, gcpt, gapt); /* Compilation will fail */ grp2_id = H5Gcreate1(file_id,"/B",0); grp3_id = H5Gcreate2(file_id,”New/A", H5P_DEAFULT, gcpt, gapt); October 15, 2008 HDF and HDF-EOS Workshop XII 89
  90. 90. Thank you! Questions? October 15, 2008 HDF and HDF-EOS Workshop XII 90
  91. 91. Acknowledgement • This report is based upon work supported in part by a Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA Awards NNX06AC83A and NNX08AO77A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration. October 16, 2008 HDF and HDF-EOS Workshop XII 91

×