Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

UniVerse Files

174 views

Published on

This session will take you through the File Types available for use with UniVerse. The UniVerse File Header structure will be examined. A more detailed review of UniVerse dynamic hashed, static hashed, and btree file header structures is included. And finally a review of the group and record structures associated with dynamic and static hashed files.

Published in: Software
  • Be the first to comment

UniVerse Files

  1. 1. 1 UniVerse Files Neil Morris, UniVerse Technical Support
  2. 2. 2 Credits and Acknowledgements Presenter • Neil Morris, UniVerse Technical Support Developers • Joel Yates, UniVerse Technical Support • Neil Morris, UniVerse Technical Support ©2015 Rocket Software, Inc. All Rights Reserved.
  3. 3. 3 Abstract  This session will take you through the File Types available for use with UniVerse. The UniVerse File Header structure will be examined. A more detailed review of UniVerse dynamic hashed, static hashed, and btree file header structures is included. And finally a review of the group and record structures associated with dynamic and static hashed files. An overall understanding of the UniVerse file types and structures can aid in file configuration and tuning. ©2015 Rocket Software, Inc. All Rights Reserved.
  4. 4. 4 Agenda UniVerse files general • Directories, SQL, Multilevel, Distributed, Btree, Dynamic and Static hashed UniVerse file header • Magic, Type, Modulus, Separation, and MKDBtimestamp BTree/Index files Dynamic and static hashed files UniVerse file structures • Group structure and Record structure ©2015 Rocket Software, Inc. All Rights Reserved.
  5. 5. 5 Naming Conventions - LONGNAMES  By default UniVerse limits file names to 14 characters. >CREATE.FILE FILE.WITH.A.LONG.FILENAME DYNAMIC WARNING: An operating system file will be created with a truncated name. Creating file "FILE.WITH000" as Type 30. Creating file "D_FILE.WITH000" as Type 3, Modulo 1, Separation 2. Added "@ID", the default record for RetrieVe, to "D_FILE.WITH000". >CT VOC FILE.WITH.A.LONG.FILENAME FILE.WITH.A.LONG.FILENAME 0001 F 0002 FILE.WITH000 0003 D_FILE.WITH000 > ©2015 Rocket Software, Inc. All Rights Reserved.
  6. 6. 6 Naming Conventions - LONGNAMES Changing the default behavior.  Execute 'LONGNAMES ON' in the account to enable the feature in that account.  Execute 'LONGNAMES ON NEWACC' in the main uv account to enable the feature by default for any new universe accounts  Use 'uv.load -longnames' when installing or upgrading UniVerse to enable this feature by default. ©2015 Rocket Software, Inc. All Rights Reserved.
  7. 7. 7 Naming Conventions – Special Characters Character substitutions will occur when characters with special meanings at the operating system level are used in a filename or in the key to an item in a Type 1 or 19 file. For example on UNIX: >CREATE.FILE MY/FILE DYNAMIC Creating file "MY?FILE" as Type 30. Creating file "D_MY?FILE" as Type 3, Modulo 1, Separation 2. Added "@ID", the default record for RetrieVe, to "D_MY?FILE". ©2015 Rocket Software, Inc. All Rights Reserved.
  8. 8. 8 Naming Conventions – Special Characters On Windows: >CREATE.FILE MY|FILE DYNAMIC Creating file "MY%VFILE" as Type 30. Creating file "D_MY%VFILE" as Type 3, Modulo 1, Separation 2. Added "@ID", the default record for RetrieVe, to "D_MY%VFILE". Refer to 'Naming files' in System Description manual for a complete list. ©2015 Rocket Software, Inc. All Rights Reserved.
  9. 9. 9 Naming Conventions – Special Characters Windows special files. Windows allows certain names to be used to directly access a device or feature. Attempting to use these names for UniVerse file names or record keys for Type 1 and 19 files will result in errors. ©2015 Rocket Software, Inc. All Rights Reserved.
  10. 10. 10 Naming Conventions – Special Characters Windows special files. >CREATE.FILE AUX.FILE 18 1 1 An operating system file already exists with the name "AUX.FILE". >COPYI FROM VOC TO BP AUX.PORT Failed to write record "AUX.PORT" to file "BP"! 0 records copied. Windows treats 'AUX' as a reference to COM1 and restricts these operations. ©2015 Rocket Software, Inc. All Rights Reserved.
  11. 11. 11 Naming Conventions – WINDOWS Files CON: Send data to console PRN: Send data to LPT1 (line printer) AUX: Send data to COM1 NUL: Disregard data (same as /dev/null on UNIX) LST: Send data to LPT1 (line printer) LPT1-LPT4: Send data to parallel port COM1-COM4: Send data to serial port ©2015 Rocket Software, Inc. All Rights Reserved.
  12. 12. 12 Directories Type 1: Defined by the presence of '.Type1' file in the directory Type 19: Defined by the absence of '.Type1' file in the directory On Windows this is a hidden file, use 'dir /ah' to view ©2015 Rocket Software, Inc. All Rights Reserved.
  13. 13. 13 Type 1 File Record Keys Key is broken into 14 character pieces and items stored using sub-directory structure. A key length divisible by 14 will use '?' as the file name. The key length is restricted to a maximum length of 41 characters on both Windows and UNIX. ©2015 Rocket Software, Inc. All Rights Reserved.
  14. 14. 14 Type 1 File Record Keys >CREATE.FILE TEST 1 >COPY FROM VOC TO TEST LIST,FOURTEEN.XXXXX LIST,FOURTEEN.XXXXXZ >SH -c "find TEST -print" TEST TEST/.Type1 TEST/FOURTEEN.XXXXX TEST/FOURTEEN.XXXXX/? TEST/FOURTEEN.XXXXX/Z >©2015 Rocket Software, Inc. All Rights Reserved.
  15. 15. 15 Type 1 File Record Keys >COPY FROM VOC TO TYPE1 LIST,12345678901234567890123456789012345678901234567890 File name "12345678901234567890123456789012345678901234567890" too long. Truncated to "12345678901234567890123456789012345678901". 1 record copied. >SH -c "find TYPE1 -print" TYPE1 TYPE1/.Type1 TYPE1/12345678901234 TYPE1/12345678901234/56789012345678 TYPE1/12345678901234/56789012345678/9012345678901 ©2015 Rocket Software, Inc. All Rights Reserved.
  16. 16. 16 Converting Type 1 Files Use RESIZE to convert a Type 1 file to Type 19 and vice versa. >RESIZE TEST 19 >SH -c "find TEST -print" TEST TEST/FOURTEEN.XXXXX TEST/FOURTEEN.XXXXXZ ©2015 Rocket Software, Inc. All Rights Reserved.
  17. 17. 17 Converting Type 1 Files WARNING: Items not conforming to the Type 1 format will be lost. Verify before resizing. >SELECT TEST 3 record(s) selected to SELECT list #0. >>SELECT TEST 2 record(s) selected to SELECT list #0. "FILENAMETHATEXCEEDSFOURTEENBYTES" not found. >> ©2015 Rocket Software, Inc. All Rights Reserved.
  18. 18. 18 Type 1 Files on Windows On Windows Type 1 files do not use the sub- directory structure. The primary difference between a Type 1 and Type 19 file is the 41 character record key limit on the Type 1 file. ©2015 Rocket Software, Inc. All Rights Reserved.
  19. 19. 19 Type 19 Files  On both UNIX and Windows a Type 19 file is simply a directory at the operating system level.  The maximum record key size is limited by the operating system maximum file name length. 0001 OPEN 'TEST' TO F.FILE ELSE STOP 'CANNOT OPEN TEST' 0002 KEY1 = STR('X','255');KEY2 = STR('X','256') 0003 WRITE 'TEST' ON F.FILE,KEY1 0004 WRITE 'TEST' ON F.FILE,KEY2 0005 END >RUN BP X Program "X": Line 4, Fwrite failed at DBwrite for file TEST. Program "X": Line 4, WRITE failure. ©2015 Rocket Software, Inc. All Rights Reserved.
  20. 20. 20 Type 1 and 19 files: NFS and SHARES Caution regarding using basic WRITE. Line terminator on UNIX is new-line, char(10) Line terminator on Windows is carriage-return & new line, char(13) : char(10) ©2015 Rocket Software, Inc. All Rights Reserved.
  21. 21. 21 Type 1 and 19 Files: NFS and SHARES WRITE operation will fail on UNIX to Windows share (cifs) with ALLOWNFS. Instead use sequential processing (WRITEBLK, WRITESEQ). On UNIX a WRITE to a Type 1 or 19 file is done using a temporary file to avoid loosing an existing item in the event the write fails. This is accomplished using link() and unlink() calls which are not supported on cifs mounts. ©2015 Rocket Software, Inc. All Rights Reserved.
  22. 22. 22 Machine Class: Big & Little endian Refers to the manner in which the hardware manages binary (ie., integer) data. For example integer 1 in hexidecimal Big endian : 01000000 Little endian: 00000001 ©2015 Rocket Software, Inc. All Rights Reserved.
  23. 23. 23 Machine Class: Big & Little endian Big endian (swapped) • Windows • Linux • Sun Intel Little endian (not swapped) • AIX • HPUX • Sun Sparc ©2015 Rocket Software, Inc. All Rights Reserved.
  24. 24. 24 Machine Class: Converting UniVerse files contain a combination of binary and string data File must be converted when moved between different machine classes The fnuxi or FORMAT.CONV command is used to perform the conversion ©2015 Rocket Software, Inc. All Rights Reserved.
  25. 25. 25 Machine Class >FORMAT.CONV -m1 TEST >DUMP TEST 00000000: 0C01EFAC 03000000 00000000 17000000 00000010: 04000000 00000000 00000000 00000000 >FORMAT.CONV -m0 TEST >DUMP TEST 00000000: ACEF010C 00000003 00000000 00000017 00000010: 00000004 00000000 00000000 00000000 DUMP is a basic program which uses READBLK to process the file in 'raw' mode. ©2015 Rocket Software, Inc. All Rights Reserved.
  26. 26. 26 SQL Tables and Views  Must be used within a SQL Schema  Protected by GRANTs  Tracked in uvhome/sql/catalog database SQL Tables  Use standard static and dynamic hashed files.  Will contain a SICA block with table properties  Use LIST.SICA to display SICA contents SQL Views  Virtual view of parts of an SQL Table  Can limit what parts of the table can be viewed or updated ©2015 Rocket Software, Inc. All Rights Reserved.
  27. 27. 27 Multilevel Files  Organizational method for managing files  Subfiles can be any type of file including directories (1 & 19)  Relies on application to maintain the association by accessing the appropriate subfile.  Multilevel files originated in the PICK environment Example: CUSTOMERS US CANADA MEXICO ©2015 Rocket Software, Inc. All Rights Reserved.
  28. 28. 28 Multilevel Files Creating: CREATE.FILE CUSTOMERS,US 18 3791 4 CREATE.FILE DATA CUSTOMERS,CANADA DYNAMIC Access: LIST CUSTOMERS,CANADA NAME ZIP OPEN “CUSTOMERS,US” TO F.FILE ELSE … ©2015 Rocket Software, Inc. All Rights Reserved.
  29. 29. 29 Multilevel Files >CT VOC CUSTOMERS CUSTOMERS 0001 F 0002 CUSTOMERS 0003 D_CUSTOMERS 0004 M 0005 0006 0007 CANADAýUS 0008 CANADAýUS ©2015 Rocket Software, Inc. All Rights Reserved.
  30. 30. 30 Multilevel Files >SH -c "ls -l CUSTOMERS" total 4 drwxr-xr-x 2 root root 96 Sep 1 10:55 CANADA -rw-r--r-- 1 root root 2048 Sep 1 10:55 US > ©2015 Rocket Software, Inc. All Rights Reserved.
  31. 31. 31 Distributed Files  Logical grouping of files treated as single universe file  Parts can be static hashed or dynamic hashed  Part file can be a member of more than one distributed file  Partitioning algorithm based on @ID contents  All items within a given part must adhere to the algorithm for each distributed file of which the part is a member  Distributed file must be defined local for UVNet access, parts can be remote  Cataloged in &PARTFILES& ©2015 Rocket Software, Inc. All Rights Reserved.
  32. 32. 32 Distributed Files Advantages Parts can be accessed independent of the distributed file. For example INVOICES comprised of multiple parts where each part represents a year. Can reduce the amount of time a file has to be offline for maintenance. For example BUILD.INDEX, RESIZE, fixtool, etc.. Distributed file is transparent to the application. ©2015 Rocket Software, Inc. All Rights Reserved.
  33. 33. 33 Distributed Files Disadvantages Increased demands for file units. Performance impact when opened local to subroutines and functions called frequently. Note: BSCAN only works against individual parts. ©2015 Rocket Software, Inc. All Rights Reserved.
  34. 34. 34 Distributed Files Universe commands  DEFINE.DF  VERIFY.DF  REBUILD.DF  LIST.DF Example: DEFINE.DF INVOICES ADDING INVOICE.2005 1 INTERNAL "FIELD(@ID,'*',1)" ©2015 Rocket Software, Inc. All Rights Reserved.
  35. 35. 35 UniVerse Files BTree Static hashed Dynamic hashed These files are comprised of a series of buffers Buffers are also called frames or groups In a BTree the term used is leaf ©2015 Rocket Software, Inc. All Rights Reserved.
  36. 36. 36 UniVerse Files BTree Header buffer Index buffer Data buffer Oversize item buffer Free buffer The initial buffer of a BTree is often referred to as the 'root' since all access begins at that point. ©2015 Rocket Software, Inc. All Rights Reserved.
  37. 37. 37 UniVerse Files Hashed (Dynamic/Static)  Header  Primary buffer (1 per group)  Overflow buffer to expand group beyond single primary buffer  Oversized item buffer to store large items  Free buffer, space available to be reused ©2015 Rocket Software, Inc. All Rights Reserved.
  38. 38. 38 UniVerse File Header
  39. 39. 39 UniVerse File Header Buffer  BTree, Static hashed, and Dynamic hashed files all begin with a header buffer  The header buffer is used to store information about the structure of the particular file ©2015 Rocket Software, Inc. All Rights Reserved.
  40. 40. 40 UniVerse File Header Buffer The format of the header buffer is the same for all files Some information is common across different types of files Other information in the header will vary based on the type of file  A dynamic file header will contain merge and split factors These locations would be unused in a static hashed file ©2015 Rocket Software, Inc. All Rights Reserved.
  41. 41. 41 UniVerse File Header Buffer The minimum amount of information that must be present in a file header is: MAGIC TYPE MODULUS SEPARATION MKDBTimestamp ©2015 Rocket Software, Inc. All Rights Reserved.
  42. 42. 42 UniVerse File Header Buffer The size of the header buffer depends on the separation of the file. Odd separation the header is 1024 bytes Even separation the header is separation x 512 Only the first 1024 bytes of the header are used The file header is covered in more detail later. ©2015 Rocket Software, Inc. All Rights Reserved.
  43. 43. 43 UniVerse File MAGIC 32-bit integer value that identifies the file as a UniVerse file. First 16 bits is always hexadecimal 'ACEF' Next byte is style code which indicates if the file is 32-bit (01) or 64-bit (02) Next byte is revision used to identify the file structure ©2015 Rocket Software, Inc. All Rights Reserved.
  44. 44. 44 UniVerse File MAGIC Example: ACEF010C ACEF - universe file 01 – 32-bit file 0C - created at or after release 9.5.1C ©2015 Rocket Software, Inc. All Rights Reserved.
  45. 45. 45 UniVerse File MAGIC Style code  The style code defines the type of integer values used to navigate the file  A style '01' file uses signed 32-bit integer pointers  A style '02' file uses signed 64-bit integer pointers  Style is not directly related to the physical size of the file  However a style '01' (32-bit) can not exceed 2GB  The maximum value for a signed 32-bit integer is 2GB  A style '02' (64-bit) has no specific limit other than what would be imposed by the operating system ©2015 Rocket Software, Inc. All Rights Reserved.
  46. 46. 46 UniVerse File Type UniVerse file types include: 1 Directory with special handling for key size 2-18 Static hashed files. Type controls record hashing 19 Directory 25 BTree 27 Distributed file 30 Dynamic hashed file 41 SQL View ©2015 Rocket Software, Inc. All Rights Reserved.
  47. 47. 47 UniVerse File Modulus  Identifies the number of primary groups within the file  Each group in a hashed file can be updated independent of other groups in the file ©2015 Rocket Software, Inc. All Rights Reserved.
  48. 48. 48 UniVerse File Separation  The separation determines the size of the buffers within the file  The separation is expressed in terms of 512 byte blocks  A separation of 4 would equate to a buffer size of 2048. ©2015 Rocket Software, Inc. All Rights Reserved.
  49. 49. 49 UniVerse File MKDBTimestamp The UTC timestamp when the file is created  After a resize, the MKDBTimestamp is replaced by the time of the resize operation The original timestamp is also saved and stored during the resize operation The original value is used should a file have been rotated out during the resize operation ©2015 Rocket Software, Inc. All Rights Reserved.
  50. 50. 50 UniVerse File MKDBTimestamp There are numerous tools on the internet to allow you to quickly convert a UTC timestamp to a readable time and date. For example: http://www.epochconverter.com/epoch/unix-hex-timestamp.php ©2015 Rocket Software, Inc. All Rights Reserved.
  51. 51. 51 BTree/Index Files
  52. 52. 52 BTree/Index Files Used for UniVerse Index files.  File Type 25  Separation 16 (buffer size 8192)  Items filed in sort order.  Sort order may be numeric or text  All access begins at the same point 'root leaf' ©2015 Rocket Software, Inc. All Rights Reserved.
  53. 53. 53 BTree/Index Files Buffers or leafs that makeup a BTree file. Header Index Data Oversized item Free ©2015 Rocket Software, Inc. All Rights Reserved.
  54. 54. 54 Btree/Index Files  The style of an index is based on the style of the data file when the index is created  An index created on a 32-bit file will be 32-bit  An index created on a 64-bit file will be 64-bit  If a file is resized between 32-bit and 64-bit, the style of the index will not change  An index would need to be deleted and recreated to have it match the new style ©2015 Rocket Software, Inc. All Rights Reserved.
  55. 55. 55 Btree/Index Header Required header information Magic Type (25) Modulus (1 though not used) Separation 16 (8K) Status flags (covered later) Optional Free chain pointer ©2015 Rocket Software, Inc. All Rights Reserved.
  56. 56. 56 BTree/Index Files Index buffer Used to divide the BTree into subsets based on sort order. Index leaf points to other index or data leafs May reference up to 384 index or data buffers ©2015 Rocket Software, Inc. All Rights Reserved.
  57. 57. 57 BTree/Index Files Data buffer May contain up to 128 items. ©2015 Rocket Software, Inc. All Rights Reserved.
  58. 58. 58 BTree/Index Files Oversized item buffer  Used to store data for any item over approximately 7k  Additional buffers may be linked as needed to store the item. ©2015 Rocket Software, Inc. All Rights Reserved.
  59. 59. 59 Btree/Index Files Free chain pointer As buffers become free they become part of the free chain Existing free chain pointer is written as forward pointer in free buffer The free buffer address is written to the file header as free chain pointer The same logic applies for static and dynamic hashed files ©2015 Rocket Software, Inc. All Rights Reserved.
  60. 60. 60 BTree/Index Files  The Btree structure can greatly reduce the number of read operations required to locate a specific index value in the index.  Each index read reduces potential items by a factor of 384. ©2015 Rocket Software, Inc. All Rights Reserved.
  61. 61. 61 BTree/Index Files  The record id’s within an index item are sorted and accessed sequentially.  This is done to optimize certain Retrieve operations selecting multiple indexed fields  However, poor performance can occur when updating an indexed value containing a large volume of keys.  A very large number of read/write operations may be required to insert a new record id into an index item containing thousands of keys  Caution should be used when creating an index where a large number of keys go to a single indexed value ©2015 Rocket Software, Inc. All Rights Reserved.
  62. 62. 62 Btree/Index Files  All index files have an INDEX.MAP file  The INDEX MAP file contains information regarding the specific indexes  Data contained within the INDEX.MAP includes but is not limited to: • index file (ie., INDEX.000) • index name • index type (A, C, D, or I) • (S)ingle or (M)ultivalued • Flag indicating if index need to be built • Flag indicating if null values are included • Flag indicating if index is disabled ©2015 Rocket Software, Inc. All Rights Reserved.
  63. 63. 63 BTree/Index Commands blook CREATE.INDEX BUILD.INDEX LIST.INDEX DISABLE.INDEX ENABLE.INDEX UPDATE.INDEX SET.INDEX DELETE.INDEX ENCRYPT.INDEX REENCRYPT.INDEX DECRYPT.INDEX ©2015 Rocket Software, Inc. All Rights Reserved.
  64. 64. 64 Indexes and Retrieve NO.INDEX keyword instructs Retrieve to not use an available index REQUIRE.INDEX keyword instructs Retrieve to only execute the command if an index is available EXPLAIN keyword will provide data on whether Retrieve is using an index Note, when using multiple indexes in a select that returns a significant portion of the file, not using the index may perform better. ©2015 Rocket Software, Inc. All Rights Reserved.
  65. 65. 65 Indexes and Retrieve  The use of an index is based on comparing the dictionary reference to data in the INDEX.MAP  Field name is NOT a factor in determining if an index will be used  Field, I-Type code, Correlative must match  Single/Multivalue indicator must match  Justification (L & T or R) must match  The ‘In DICT’ column of LIST.INDEX will indicate if the dictionary item matches the data within the INDEX.MAP ©2015 Rocket Software, Inc. All Rights Reserved.
  66. 66. 66 Indexes and BASIC INDICES is a function which can return information on the indexes on the file SELECTINDEX is a statement which can return a select list of items from an index BSCAN is a function which allows traversing an index in ascending or descending order BSCAN allows partial values as the beginning point of a search ©2015 Rocket Software, Inc. All Rights Reserved.
  67. 67. 67 Concurrent BUILD.INDEX The indexing properties of a file are determined at the time it is opened A process which has the file open would not be aware of new indexes created after the open BUILD.INDEX CONCURRENT should only be used after all users have reopened the file when a new index has been created ©2015 Rocket Software, Inc. All Rights Reserved.
  68. 68. 68 Dynamic and Static Hashed Files
  69. 69. 69 Dynamic and Static Hashed Files The buffers which comprise these files are the same The difference is where these buffers are located The static hashed file uses a single file to store information For a dynamic file: • The header and primary buffers (one per group) are in the DATA.30 file • All other buffers are in the OVER.30 file The OVER.30 header only holds the free chain pointer ©2015 Rocket Software, Inc. All Rights Reserved.
  70. 70. 70 Dynamic and Static Hashed Files A simple static file might be represented as: ©2015 Rocket Software, Inc. All Rights Reserved. Header Primary Primary Primary Primary Overflow Overflow Special Overflow Oversized Item Overflow
  71. 71. 71 Dynamic and Static Hashed Files A dynamic file would look as follows: DATA.30 OVER.30 ©2015 Rocket Software, Inc. All Rights Reserved. Header Primary Primary Primary Primary Header Overflow Overflow Special Overflow Oversized Item Overflow
  72. 72. 72 Dynamic and Static Hashed Files The modulus of a static hashed file is constant The modulus is loaded when the file is opened The modulus of a dynamic hashed file can change while the file is open The modulus of an open dynamic file is stored in the T30FILE shared memory structure All users having the dynamic file open use the values stored in the shared memory structure ©2015 Rocket Software, Inc. All Rights Reserved.
  73. 73. 73 Dynamic and Static Hashed Files The “smat –d” command displays the T30FILE contents  Access to the T30FILE structure is protected by the T30FILE semaphore On a system with a large number of open dynamic files, contention for access to the T30FILE structure may exist If possible, files that do not significantly change size should remain static hashed ©2015 Rocket Software, Inc. All Rights Reserved.
  74. 74. 74 Dynamic and Static Hashed Files The buffers that make up static and dynamic files include  Header  Primary buffers  Group overflow buffers  Oversized record buffers  Free buffers  Special purpose buffers ©2015 Rocket Software, Inc. All Rights Reserved.
  75. 75. 75 Dynamic and Static Hashed Files Special purpose buffers  SICA: SQL Tables and views. Triggers. Note - this buffer is encrypted  Part file: Partitioning algorithm information for each distributed file of which the part file is a member ©2015 Rocket Software, Inc. All Rights Reserved.
  76. 76. 76 Dynamic and Static Hashed File Type Record hashing is achieved by performing a calculation on the record key The file type (static) or hashing algorithm (dynamic) determines the calculation used on the key The calculation returns a value which determines the primary group location for the record ©2015 Rocket Software, Inc. All Rights Reserved.
  77. 77. 77 Dynamic and Static Hashed File Type The objective when choosing a file type or hashing algorithm is to equally distribute the records among the available primary groups UniVerse provides tools to assist with generating an even distribution • HASH.AID, FILE.STAT, HASH.HELP, etc. ©2015 Rocket Software, Inc. All Rights Reserved.
  78. 78. 78 Dynamic and Static Hashed File Type  UniVerse provides17 different file types or hashing algorithms for static hashed files The different file types are based on the characteristics of the record key Two file types or hashing algorithms are available for dynamic files ©2015 Rocket Software, Inc. All Rights Reserved.
  79. 79. 79 Static Hashed File Type 2 keys end with numbers. 3 keys end mainly with numbers. 4 keys end with letters. 5 keys end with full range of ASCII characters. 6 keys begin with numbers. 7 keys begin mainly with numbers. 8 keys begin with letters. 9 keys begin with full range of ASCII characters. ©2015 Rocket Software, Inc. All Rights Reserved.
  80. 80. 80 Static Hashed File Type 10 keys are numbers. 11 keys are mainly numbers. 12 keys are letters. 13 keys are full range of ASCII characters. 14 entire keys are numbers. 15 entire keys are mainly numbers. 16 entire keys are letters. 17 entire keys are full range of ASCII characters. 18 entire keys are of arbitrary form. ©2015 Rocket Software, Inc. All Rights Reserved.
  81. 81. 81 Dynamic Hashed File Type Dynamic files have two available file types: 20 GENERAL 21 SEQ.NUM ©2015 Rocket Software, Inc. All Rights Reserved.
  82. 82. 82 Static Hashed File Type Choosing the best file type for a static hashed file may be challenging Determining the target values for modulus and separation initially is a good approach Tools such as HASH.AID can be used to assist in selecting the most appropriate file type ©2015 Rocket Software, Inc. All Rights Reserved.
  83. 83. 83 Dynamic and Static Hashed Separation  Separation determines the size of the buffers that make up the file  For a dynamic file the separation is documented as GROUP.SIZE  Group size 1 is a separation of 4 where group size 2 is a separation of 8  The RESIZE command can be used to select a different separation for a dynamic file ©2015 Rocket Software, Inc. All Rights Reserved.
  84. 84. 84 Dynamic and Static Hashed Separation >CREATE.FILE TEST DYNAMIC >GROUP.STAT TEST Bytes Records File= TEST Modulo= 1 Sep= 4 >RESIZE TEST 30 1 16 >GROUP.STAT TEST Bytes Records File= TEST Modulo= 1 Sep= 16 ©2015 Rocket Software, Inc. All Rights Reserved.
  85. 85. 85 Dynamic and Static Hashed Separation  Choosing a separation involves balancing cpu and disk  The fewer records in a group, the less cpu required to retrieve an item The more records contained within a buffer, the fewer I/O operation required to process the file ©2015 Rocket Software, Inc. All Rights Reserved.
  86. 86. 86 Dynamic and Static Hashed Separation Given a file contains items with an average record size of 150 bytes: • Separation of 1 can fit 3 items per buffer • Separation of 2 can fit 6 items per buffer • Separation of 4 can fit 13 items per buffer ©2015 Rocket Software, Inc. All Rights Reserved.
  87. 87. 87 Dynamic and Static Hashed Separation With 3 items per buffer, the target item will be found on average in 1.5 compares With 6 items per buffer, the target item is found on average in 3 compares With 13 items per group, the target item on average is found in 7.5 compares ©2015 Rocket Software, Inc. All Rights Reserved.
  88. 88. 88 Dynamic and Static Hashed Separation Using previous example and a 1,000,000 record file: • A separation of 1 or 2 file results in a 170mb file • A separation of 4 file would be 157mb file Choose a separation that balances disk utilization with the fewest items stored per group Note - large record sizes may impact the choice of separation ©2015 Rocket Software, Inc. All Rights Reserved.
  89. 89. 89 Dynamic and Static Hashed Modulus Choosing the separation value will simplify the choice of modulus Divide total records by the estimated number of records per group to get modulus For a dynamic file this method can factor into the choice for minimum modulus ©2015 Rocket Software, Inc. All Rights Reserved.
  90. 90. 90 Dynamic and Static Hashed Modulus For static hashed files, the choice of modulus should take into account expected growth If the file is expected to grow, choosing a larger modulus would be appropriate How much larger would depend on expected growth and how often files are examined and scheduled for maintenance ©2015 Rocket Software, Inc. All Rights Reserved.
  91. 91. 91 Dynamic Hashed Files  Routine file maintenance can be minimized with the use of dynamic files A dynamic file modulus can grow and shrink based on the file contents Using dynamic files with ALLOWNFS from multiple systems is NOT recommended ©2015 Rocket Software, Inc. All Rights Reserved.
  92. 92. 92 Dynamic Hashed Files In addition to the header information already mentioned, a dynamic file header contains:  Current modulus  Base modulus  Next group to split  Data bytes (filesp)  Large record size  Hashing algorithm (GENERAL/SEQ.NUM)  Split factor  Merge factor ©2015 Rocket Software, Inc. All Rights Reserved.
  93. 93. 93 Dynamic Hashed Files Modulus The modulus value in a dynamic file is referred to as the ‘minimum modulus’ By default, the minimum modulus is set to 1 The ‘current modulus’ is used to determine the actual number of groups in the file ©2015 Rocket Software, Inc. All Rights Reserved.
  94. 94. 94 Dynamic Hashed Files Modulus The current modulus is loaded to the T30FILE memory structure when the file is first opened The current modulus stored in T30FILE will then be used for all I/O being done to the file. ©2015 Rocket Software, Inc. All Rights Reserved.
  95. 95. 95 Dynamic Hashed Files Base Modulus Base modulus is used as part of the hashing algorithm Base modulus is equal to the largest power of two of the current modulus For example, a current modulus of 216 would translate to a base modulus of 128 A current modulus of 260 would translate to a base modulus of 256 ©2015 Rocket Software, Inc. All Rights Reserved.
  96. 96. 96 Dynamic Hashed Files Base Modulus Decimal Hex Binary Curr Mod: 216 D8 0000 0000 1101 1000 Base Mod: 128 80 0000 0000 1000 0000 Curr Mod: 260 104 0000 0001 0000 0100 Base Mod: 256 100 0000 0001 0000 0000 ©2015 Rocket Software, Inc. All Rights Reserved.
  97. 97. 97 Dynamic Hashed Files Next Split The next split pointer determines the next group that will be split Next split = (Current Modulus) – Base Modulus + 1 Group split will be discussed in more detail later ©2015 Rocket Software, Inc. All Rights Reserved.
  98. 98. 98 Dynamic Hashed Files filesp  Filesp (or data bytes) is the byte count of the item data in the file  This value excludes data stored in oversized item buffers  Filesp, current modulus, and separation are used to determine file load.  File load determines if a split or merge operation is necessary ©2015 Rocket Software, Inc. All Rights Reserved.
  99. 99. 99 Dynamic Hashed Files filesp File load calculation • Filesp / (current modulus * separation * 512) * 100 • Percentage of data in file compared to physical size Given the following dynamic file values • filesp: 31636 • current mod: 20 • separation: 4 File load = (31636 / 40960) * 100 = 77% ©2015 Rocket Software, Inc. All Rights Reserved.
  100. 100. 100 Dynamic Hashed Files filesp Filesp and fixtool fixtool may report a filesp error when scanning a file that is being actively updated The filesp value is maintained in the T30FILE memory structure. Minor changes to the file will change the filesp value but the information is not immediately flushed to disk. ©2015 Rocket Software, Inc. All Rights Reserved.
  101. 101. 101 Dynamic Hashed Files Large Record  Large record size specifies the number of bytes at which a record is considered a “large record”  Large records are stored in the OVER.30 file  By default large record size is 80% (less record header) of the buffer size  A 32-bit file with separation of 4 will have large record size set to 1628  (2048 – 12) * .8 = 1628 ©2015 Rocket Software, Inc. All Rights Reserved.
  102. 102. 102 Dynamic Hashed Files Hashing Algorithm Dynamic files have two possible hashing algorithms GENERAL which is type 20 SEQ.NUM which is type 21 These file types are only valid with dynamic files ©2015 Rocket Software, Inc. All Rights Reserved.
  103. 103. 103 Dynamic Hashed Files Split Factor Split factor is expressed as a percentage, default 80 After an update the file load is calculated using the formula discussed previously If the calculated load is greater than the split factor, a split operation will occur ©2015 Rocket Software, Inc. All Rights Reserved.
  104. 104. 104 Dynamic Hashed Files Merge Factor  Merge factor is also expressed as a percentage with a default of 50  If the calculated file load is less than this value a merge operation will occur  Note - the merge operation will only occur if the current modulus is larger than the minimum modulus  The file will never decrease below minimum modulus ©2015 Rocket Software, Inc. All Rights Reserved.
  105. 105. 105 Dynamic Hashed Files T30FILE  T30FILE is a configurable shared memory structure, which is scanned when opening a dynamic file  If the file is found, the usage count is incremented by 1; otherwise, a new entry is added to the table  The usage counter is decrement by 1 when the file is closed  The table entry is cleared when usage count hits 0  The table size is based on the T30FILE uvconfig parameter ©2015 Rocket Software, Inc. All Rights Reserved.
  106. 106. 106 Dynamic Hashed Files T30FILE The dynamic file header is updated periodically from the shared memory table The memory structure gets flushed to the header on the following operations:  On a split operation  On a merge operation  When the file is closed by a UniVerse process ©2015 Rocket Software, Inc. All Rights Reserved.
  107. 107. 107 Dynamic Hashed Files SELECT  An active select counter is maintained in the T30FILE memory structure Split and merge operations are blocked when the file is being selected This prevents potentially skipping records which were merged or processing split records multiple times ©2015 Rocket Software, Inc. All Rights Reserved.
  108. 108. 108 Dynamic Hashed Files ALLOWNFS Closing a dynamic file flushes information from memory to the file header Accessing dynamic files from multiple UniVerse systems via ALLOWNFS may cause corruption If UVNet is not being used, out of date information from the remote system can be flushed to the local file header This can occur even when the remote access is simply to read the file. ©2015 Rocket Software, Inc. All Rights Reserved.
  109. 109. 109 Dynamic Hashed Files ALLOWNFS An example of file corruption User A on Sys1 opens file local to Sys1 User B on Sys2 opens the same file via ALLOWNFS User A updates the file causing split User A closes the file User B closes the file flushing memory from Sys2 which wipes out header changes made by User A ©2015 Rocket Software, Inc. All Rights Reserved.
  110. 110. 110 Dynamic Hashed Files SUSPEND.FILES and Dynamic Files  The SUSPEND.FILES command can be used to pause database updates  This can be used when making a backup copy of the database  Running fixtool on the file copy may report an error related to filesp  Minor changes to the file may change filesp but not cause a split or merge  In this situation, the updated filesp value would not be flushed to disk  The fixtool error in this case can be ignored ©2015 Rocket Software, Inc. All Rights Reserved.
  111. 111. 111 Dynamic and Static Hashed Files Header Info Additional header info • NLS • Indexes • RESIZE • Automatic Data Encryption • Transaction Logging • SQL Tables • FILE.USAGE • uvbackup • UV Replication • AUDIT ©2015 Rocket Software, Inc. All Rights Reserved.
  112. 112. 112 Dynamic and Static Hashed File Header NLS When using National Language Support (NLS). Map name - Map to use when reading and writing to the file Map checksum - Checksum used to ensure the map has not been altered Sort map name - Map to use when sorting file Sort map checksum - Checksum used to ensure the sort map has not been altered ©2015 Rocket Software, Inc. All Rights Reserved.
  113. 113. 113 Dynamic and Static Hashed File Indexes Status flags • Indexes are enabled or disabled • If updates were applied while indexes were disabled Pathname for indexing directory • Maximum of 256 bytes for revision 12 (x0C) • Maximum of 128 bytes for older revisions ©2015 Rocket Software, Inc. All Rights Reserved.
  114. 114. 114 Dynamic and Static Hashed File RESIZE Status flag to indicate the file is being resized concurrent Old MKDBTimestamp. Used to validate file when file resized while rotated out Current resize path which defines the temporary file being used to resize the file ©2015 Rocket Software, Inc. All Rights Reserved.
  115. 115. 115 Dynamic and Static Hashed File Header Encryption A 32-bit value which exists for an encrypted file • Revision (8bits) • Encryption key (24bits) The current revision level is x01 The key is used to locate the file in &ENCINFO& The &ENCINFO& file contains information on the encryption status of the file ©2015 Rocket Software, Inc. All Rights Reserved.
  116. 116. 116 Dynamic and Static Hashed File Header UVTL Contains the record id to the UV.TRANS record If UVTL Checkpoint mode is enabled • Inconsistent bit. Indicating the checkpoint process found structural problems with file • Last UVTL log where file was processed by checkpoint deamon • Last UVTL log containing a warmstart transaction for this file ©2015 Rocket Software, Inc. All Rights Reserved.
  117. 117. 117 Dynamic and Static Hashed File Header SQL Schema Column count Column map SICA Pointer (discussed previously) ©2015 Rocket Software, Inc. All Rights Reserved.
  118. 118. 118 Dynamic and Static Hashed File Header Statistics  UniVerse file usage statistics stored in the file header • FILE.USAGE • FILE.USAGE.CLEAR • FILE.USAGE.OFF  Seventeen 32-bit integer counters recording statistics related to file I/O operations  UTC Timestamp of last FILE.USAGE.CLEAR operation ©2015 Rocket Software, Inc. All Rights Reserved.
  119. 119. 119 Dynamic and Static Hashed File Header uvbackup The values maintained by uvbackup are maintained by certain other commands such as COUNT • UTC Timestamp when uvbackup or count occurred • Flag indicating type of access • Count of the number of items in the file ©2015 Rocket Software, Inc. All Rights Reserved.
  120. 120. 120 Dynamic and Static Hashed File Header UVDR UniVerse Data Replication stores information in the file header Subscriber files are marked with a read only flag Flag (1=Published, 2=Subscribed) Key to either UV_UDRPUB or UV_UDRSUB file item Note - this is NOT related to the new U2 Replication functionality ©2015 Rocket Software, Inc. All Rights Reserved.
  121. 121. 121 Dynamic and Static Hashed File Header Misc Audit log flag - Indicates the file is used by the recently added Audit Logging feature ©2015 Rocket Software, Inc. All Rights Reserved.
  122. 122. 122 UniVerse File Structures
  123. 123. 123 UniVerse Files Primary Buffer The primary buffer is the starting point for all groups The primary buffer for group 1 follows immediately after the header For a dynamic file, the primary buffers are only located in the DATA.30 file ©2015 Rocket Software, Inc. All Rights Reserved.
  124. 124. 124 UniVerse Files Primary Buffer The location of primary buffer is determined based on the physical offset from the beginning of the file The hashing algorithm determines which primary group the key hashes The RECORD command can be used to return the group in which a record hashes The primary buffer address can be calculated using the primary group number and the file separation ©2015 Rocket Software, Inc. All Rights Reserved.
  125. 125. 125 UniVerse Files Primary Buffer A record hashing to group 100 in a separation 4 file will be in the primary group starting at address • 206848 decimal • 32800 hexadecimal IF MOD(Separation,2) THEN HdrSz = 1024 ELSE HdrSz = Separation * 512 Addr = ((Group - 1) * Separation * 512) + HdrSz ©2015 Rocket Software, Inc. All Rights Reserved.
  126. 126. 126 UniVerse Files Primary and Overflow Buffers The structure of primary and overflow groups are the same Overflow groups extend a primary group which is full A buffer is a collection of items/records All space within a buffer (separation * 512) is accounted for by UniVerse ©2015 Rocket Software, Inc. All Rights Reserved.
  127. 127. 127 UniVerse Files: Item/Record Types There are two type of items: Normal • Hashed: len(item) < (separation * 512) • Dynamic: len(item) < Large record size Oversized ©2015 Rocket Software, Inc. All Rights Reserved.
  128. 128. 128 UniVerse Files Item/Record Types The illustrations that follow will use: TEST32: 32-bit hashed file, separation of 4 TEST64: 64-bit dynamic file, separation of 8 ©2015 Rocket Software, Inc. All Rights Reserved.
  129. 129. 129 UniVerse Files Normal Item The int in a 32-bit file is 4 bytes and 8 in a 64-bit file The following information exists for an item • (int) Forward pointer • (int) Blink • (int) Flags An existing item has the following • Key • Segment mark • Record data ©2015 Rocket Software, Inc. All Rights Reserved.
  130. 130. 130 UniVerse Files: Normal Item Item 1 group 1 TEST32 00000800 : 00000838 00000038 00000803 56455249 | ...8...8....VERI 00000810 : 46592E53 514CFF56 FE42502E 4F2F5645 | FY.SQL V?BP.O/VE Item 1 group 1 TEST64 0000000000001000 : 0000000000001028 0000000000000028 | .......(.......( 0000000000001010 : 00000C0300000000 4639FF44FE39FEFE | ........F9 D?9?? 0000000000001020 : FE313554FE530002 0000000000001068 | ?15T?S.........h ©2015 Rocket Software, Inc. All Rights Reserved.
  131. 131. 131 UniVerse Files: Normal Item Forward Pointer Forward pointer indicates the position of the next item in the group If the forward pointer points outside the bounds of the current buffer it must be buffer aligned That is, it must point to the beginning of another buffer A forward pointer of zero indicates the last item in the group. ©2015 Rocket Software, Inc. All Rights Reserved.
  132. 132. 132 UniVerse Files Normal Item Blink The blink or backward link value is similar to a checksum It is a bitxor of the size of the current item and a previous address It is used by UniVerse to validate that the data and pointers are not corrupted ©2015 Rocket Software, Inc. All Rights Reserved.
  133. 133. 133 UniVerse Files Normal Item Blink For the first item in a buffer, the previous address is the beginning address for the prior group buffer For other items in the group, the previous address is a pointer back the prior item within that buffer ©2015 Rocket Software, Inc. All Rights Reserved.
  134. 134. 134 UniVerse Files Normal Item Blink  Record size is calculated by subtracting the current address from the forward pointer  If the forward pointer is zero or points to another buffer • The size is calculated by subtracting the current address from the end of the current buffer ©2015 Rocket Software, Inc. All Rights Reserved.
  135. 135. 135 UniVerse Files Normal Item Blink 00000800 : 00000838 00000038 00000803 56455249 | ...8...8....VERI 00000810 : 46592E53 514CFF56 FE42502E 4F2F5645 | FY.SQL V?BP.O/VE Size = x838 – x800 (x38 or decimal 56) Previous address = x38 ^ x38 (zero, indicating primary buffer) 00000838 : 0000085C 00000824 00000C03 44495658 | ......$....DIVX 00000848 : FF56FE72 61646978 FE45FE47 56FE4449 | V?radix?E?GV?DI Size = x85c – x838 (x24 or decimal 36) Previous address = x24 ^ x824 (x800 or decimal 2048) ©2015 Rocket Software, Inc. All Rights Reserved.
  136. 136. 136 UniVerse Files Flag Field The flag field is used to define certain properties of the item  The size of the flag field is either 32-bit or 64-bit depending on the file style  However, only 16 bits are actually used ©2015 Rocket Software, Inc. All Rights Reserved.
  137. 137. 137 UniVerse Files Flag Field Big endian (treating as 16-bit int) ©2015 Rocket Software, Inc. All Rights Reserved. Bit Description Bit Description 1Uvbackup weekly 9Oversized item 2Uvbackup daily 10Oversized item buffer 3Fwd to OVER.30 11Record padded 4Blink to DATA.30 12New style padding 5Group process by resize 13Part file buffer 6not used 14Sica buffer 7Records deleted from group 15Free item 8not used 16Not used
  138. 138. 138 UniVerse Files Flag Field The bit values on a little endian machine will be reversed More detailed slides on the flag field are included at the end of the presentation ©2015 Rocket Software, Inc. All Rights Reserved.
  139. 139. 139 UniVerse Files: Normal Item For a normal item, the record key will immediately follow the flag field The record key will be terminated by a segment mark (xFF) The record data follows the segment mark ©2015 Rocket Software, Inc. All Rights Reserved.
  140. 140. 140 UniVerse Files: Large Items In a static hashed file, a large item is any item whose size exceeds the file separation In a dynamic file, a large item is any item whose size exceeds the LARGE.RECORD.SIZE parameter ©2015 Rocket Software, Inc. All Rights Reserved.
  141. 141. 141 UniVerse Files: Large Items Large items are handled differently between static and dynamic hashed files For static hashed file: • Data from the end of the record is removed and written to oversized buffers • This process continues until the remainder of the record fits into the current buffer ©2015 Rocket Software, Inc. All Rights Reserved.
  142. 142. 142 UniVerse Files: Large Items In a dynamic file all of the record data is stored in oversized buffers in OVER.30 The primary buffer contains: • Record key • Pointer to oversized buffer address • A count of the number of oversized buffers used ©2015 Rocket Software, Inc. All Rights Reserved.
  143. 143. 143 UniVerse Files: Large Items When the ‘large item’ bit is set in the flag field, the flag field will be followed by two integer values: • The first is the address of the first oversized buffer • The second is buffer count for the item This differs from a normal item, where the item key immediately follows the flags field The size of the integer values are based on whether the file is 32-bit or 64-bit ©2015 Rocket Software, Inc. All Rights Reserved.
  144. 144. 144 UniVerse Files: Large Items 00000400 : 000005A0 000001A0 00000903 00004200 | ..............B. 00000410 : 0000001F 58FF4C69 6E65206E 756D6265 | ....X Line numbe Address for first buffer Number of buffers needed to store item In a dynamic file the address is always pointing to a location in the OVER.30 file. ©2015 Rocket Software, Inc. All Rights Reserved.
  145. 145. 145 UniVerse Files: Large Items The oversized buffer contains forward pointer, blink, and flags fields in the same format as already discussed The size for calculating or verifying the blink is always equal to the buffer size based on separation 00004200 : 00004000 00000600 00000A00 37FE4C69 | ..@.........7?Li 00004210 : 6E65206E 756D6265 72203238 FE4C696E | ne number 28?Lin ©2015 Rocket Software, Inc. All Rights Reserved.
  146. 146. 146 Summary What we covered today The different types of files available for use in UniVerse The UniVerse file header structure Dynamic hashed, static hashed, and BTree files The group and record structures associated with the dynamic and static hashed files ©2015 Rocket Software, Inc. All Rights Reserved.
  147. 147. 147 Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE. ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR • ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF ROCKET SOFTWARE. ©2015 Rocket Software, Inc. All Rights Reserved.
  148. 148. 148 Trademarks and Acknowledgements The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software, Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of any such marks. Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, MvEnterprise, NetCure, Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and wIntegrate Other company, product, and service names mentioned herein may be trademarks or service marks of others. ©2015 Rocket Software, Inc. All Rights Reserved.
  149. 149. 149

×