8.1 In Depth: New 64-bit Files and File Management

1
8.1 In Depth: New 64-bit Files
and File Management
Jonathan Smith – UniData ATS
MVU Las Vegas

2
Abstract
 8.1 was the first major release of UniData in several years, so find out
about all the new features and functionality we’ve added. New features
include 64-bit files, 64-bit builds, account-based licensing, installation
improvements, improvement to ADE, IPV6 support … and the list goes on.
©2015 Rocket Software, Inc. All Rights Reserved.

3
Agenda
64-bit files
Dynamic whole file splitting
• What happens when a file splits?
• How should I set my dynamic file parameters?
New Hash Type 3
Putting all this together
• Backward compatibility, UniBasic changes
© 2015 Rocket Software, Inc. All Rights Reserved.

5
8.1 32-bit Restrictions
 A UniData static file, or a sub-file of a dynamic file, is limited to
2G in size
 Theoretically, a UniData dynamic file can grow to 500G but the
cost of the dynamic file operations is high and the internal
management of the sub-files requires complicated code that
reduces performance of the files
 The block-size of the data file is limited to 16K and as data
records get larger, this is a performance limitation of the
32-bit model
 Files break if they try to grow past this limit

6
8.1 64-bit Files
The 64-bit data model will resolve these issues so
that data files will have virtually no restrictions on file
size and block size.
• The new maximum file size is 8 Exabytes (2^63)

7
8.1 64-bit Files
The CREATE.FILE , RESIZE, MEMRESIZE and
CREATE.TABLE commands have been modified
• Accept a keyword of 32BIT or 64BIT
• The ‘modulo’ and ‘block size multiplier’ have been increased
to allow for the new larger files
• The default will be 32BIT for CREATE.FILE if no keyword is
specified. This default is not currently configurable

8
8.1 64-bit Files
Converting from 32-bit to 64-bit is done using the
RESIZE / MEMRESIZE command
• The address mode for RESIZE / MEMRESIZE will remain the
same as if no keyword is specified
 If a file is converted from one address mode to another and has one or
more indexes, the indexes will also be converted to the new address
mode

9
8.1 64-bit Files
LIST.INDEX after a file has been converted to 64-bit

10
8.1 64-bit Files
Make sure 64-bit file support is turned on in the OS and
that any size limitations are changed; for example, ulimit
If UniData detects a problem writing due to ulimit or OS
size restrictions, a message similar to the following is
shown:
• “File:DYNAMIC1,U_blkwrite():U_pwrite return 1023. at
blkoff(2147482624), errno=0, possible ulimit reached (10774820).
In BP/_ACCESSFILE at line 144 1:write_ulimit error in write_record
for file 'DYN AMIC1', key '', number=64177”

11
8.1 64-bit Files
The CREATE.INDEX command creates the index to
match the same address mode as its associated data
file
• For a 32-bit file, a 32-bit index is created
• For a 64-bit file, a 64-bit index is created
• You cannot force the address mode with the CREATE.INDEX
command
 Reminder: See the previous slide for RESIZE / MEMRESIZE

12
8.1 64-bit Files
A 64-bit dynamic file will only have one primary sub-file
and one overflow sub-file
The length of a single record will still be limited to 2G-1
The modulo will be limited to 2G-1
• The block size will still be restricted to 16K for 32-bit
• The block size will limited to 2G-1 for 64-bit
The PE (Personal Edition) of UniData will not allow the
creation and opening of a 64-bit file

13
8.1 64-bit Files
The udtconfig parameters MAX_FLENGTH and
STATIC_GROWTH_WARN_SIZE will have no effect
on 64-bit files
Sequentially hashed files (see gmekey and shfbuild)
have also been enhanced to use 64-bit files
The FILE.STAT command now adds the word “64bit”
before the word “Dynamic” or “Static” to denote a 64-
bit file

14
Dynamic Whole File
Splitting

15
8.1 Dynamic Whole File Splitting
Why change?
• Historically, UniData Dynamic files grew and shrank (changed their
modulo) based on the loading of individual group being updated
 For this type of file to perform well, records need to hash evenly and work
at their best when there is only a small standard deviation in terms of
o The average record size for the KEYDATA style
o The average key size for the KEYONLY style
 When these styles are not hashed evenly, a UniData dynamic file can grow
unnecessarily large. This contributes to wasted disk space as well as
reducing the file scan performance (SELECT, SSELECT)
 Prior to 8.1 we added UDT_SPLIT_POLICY

16
How will they work?
• A new 64-bit data structure called fileload will be stored in the
header of first overflowed part file (over001)
 This structure also will be created in KEYONLY and KEYDATA files but not
used
• Initialized to zero, the fileload is the total space that can be
occupied by the records in the primary part file(s)
 The key and its records are counted in the fileload, along with any
overhead
 For long records that must be placed into overflow, the record body itself is
not counted in the fileload calculation
 The fileload value will be used to determine split or merge operations after
a record is updated

17
Dynamic Files – Understanding Splitting
 The decision to perform a split or merge function is based upon
the ‘loading’ reaching the split or merge load
• ‘KEY DATA’ and ‘KEY ONLY’ the loading is that of the group being
updated
• ‘WHOLEFILE’ is based on the potential space that could be used in the
complete file
 The group is split based on a mathematical calculation, the
result of which is called the split pointer
• It is almost never the group being written to (once a file starts growing)
• Only one group is split during a split operation
• The calculation and process is same for all types, as it was prior to 8.1

18
Dynamic Files – Understanding Splitting
 When a file is first created, the split pointer will point to the first
group in the file (group 0). Each time a new group is added (i.e.
a split occurs) then the split pointer is moved to the next group.
This continues until the Base Modulo changes upon which the
split pointer is reset to point at group 0 again
 When a file is created, the Base Modulo will match the creation
modulo
• When a group is added to the file, the current modulo increases by 1
• When the current modulo is twice the Base Modulo, the Base Modulo is
changed to the current modulo and the split pointer reset to 0
• This step is repeated each time the current modulo reaches twice the
Base Modulo

19
Dynamic Files – Understand Splitting - Demo

20
Dynamic Files - Which Group Will Split? - Demo
 The split pointer is based on a mathematical calculation
• The code below is the code from the demonstration spread sheet
FoundBase = False
TestPwr = 0
Do Until FoundBase
TestBase = InitMod * (2 ^ TestPwr)
If TestBase <= CurrMod And (TestBase * 2) > CurrMod Then
BaseMod = TestBase
FoundBase = True
Else
TestPwr = TestPwr + 1
End If
Loop
NextSplit = CurrMod - BaseMod

21
Dynamic Files – Understanding the Effect of Split Load
So let’s see how changing the split load on a file can
affect how many records we get in a group
• We have 3 files with the same initial modulo and block size
(8K)
• The 3 files have had the same records written to them in the
same order
• The difference between the files is the split load
 One file is configured for a 65% split

22
• A higher split % loads to more overflow and slower access, but has
a small effect on reducing the overall space occupied by the file
SPLITLOAD65 SPLITLOAD80 SPLITLOAD95
Modulo At End of Test 98,712 80,204 67,540
No of Groups In Level One Overflow 23,639 30,682 34,907
% Of File in Level One Overflow 24 38 52
Size of dat001 789,704 641,640 540,328
Size of over001 192,264 256,000 296,760
Total Size 981,968 897,640 837,088

23
0
5,000
10,000
15,000
20,000
25,000
30,000
1 Record(s) in Group 2 Record(s) in Group 3 Record(s) in Group 4 Record(s) in Group 5 Record(s) in Group 6 Record(s) in Group 7 Record(s) in Group 8 Record(s) in Group 9+ Record(s) in Group
NumberofGroups
Records Per Group
Records Per Group

24
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
1K Used in Group 2K Used in Group 3K Used in Group 4K Used in Group 5K Used in Group 6K Used in Group 7K Used in Group 8K Used in Group >8K Used in Group
NumberofGroups
Space Used in Group
Space Used Per Group

25
Dynamic Files – Demo of Splitting over Time

26
fileload performance considerations
• To reduce the number of I/O operations required, the fileload value
of a dynamic file is kept in a shared memory table
 The size of this table is controlled by MAX_DS_FILES and the information
will be viewable via the ‘sms –d’ and ‘s_stat’ commands (RFS)
 An update to the file will only update the fileload in shared memory
 When a dynamic file is closed, the fileload value in shared memory is
written to the file header if the process has the file open with write
permissions and if the fileload value in the file header is different than the
shared memory value
 The update process flushes the fileload value out to file under certain
conditions to reduce the latency of the value in the file header. The RFS
checkpoint via the forcecp command (used in dbpause) will also flush and
write the fileload values of recoverable files back to the header

27
fileload after a system failure or abnormal process
termination
• In these cases the value may not have been updated in the file
header successfully and may be inaccurate
 This should only change the timing of the next split/merge slightly and we
do not consider this a performance risk at this time
 No check of the fileload is performed on opening of a file so no error will be
detected
 FILE.STAT, guide and CONFIGURE.FILE can recalculate the fileload value
and are also able to set the fileload value
 ANALYZE.FILE and fileview can be used to view the fileload in the over001
structure
o ANALYZE.FILE and guide will not report a fileload discrepancy

28
The following udtconfig parameters have been added to
support the new WHOLEFILE type
• WHOLEFILE_SPILT_LOAD is the default split load for a
WHOLEFILE dynamic file and defaults to 75
• WHOLEFILE_MERGE_LOAD is the default merge load for a
WHOLEFILE dynamic file and defaults to 40
• DEFAULT_SPLIT_TYPE is the default split style for a dynamic file
and has values of
 1 for KEYONLY
 2 for KEYDATA
 3 for WHOLEFILE (the default, as we’ve found this to perform better)

29
Modified commands
• CREATE.FILE, RESIZE, MEMRESIZE, CREATE.TABLE all now
accept a keyword of WHOLEFILE
• FILE.STAT now accepts an optional SAVING FILELOAD clause that
results in the fileload being recalculated and written back to the file
header
• CONFIGURE.FILE also accepts a keyword of WHOLEFILE with
options of ‘FILELOAD’ or ‘SAMPLE number of groups’
 When the split style is changed to WHOLEFILE, an initial value for the
fileload is needed. This can be provided via the FILELOAD option or the
system can estimate it via a sampling of files using the SAMPLE option

30
Example commands

31
This is 0x7FFF FFFF FFFF FFFF
Where is the
fileload?

32

33

34
 Recalculating and reporting fileload
• The fileload value displayed by FILE.STAT and reported in the guide statistics is
the calculated value not the actual value stored in the file
• The fileload value reported by fileview and ANALYZE.FILE is the actual value
stored in the file
• FILE.STAT has new SAVING FILELOAD keywords to update the value stored in
the file
• CONFIGURE.FILE has options to set the fileload or calculate a value based on
a sample of a number of groups and update the value the value stored in the file
• guide has a new option of -sfl to update the file with the calculated fileload value
 This new option is not used by default
• sms –d will allow you to see the value stored in the shared memory segment

35
Recalculating fileload
• The commands that calculate the fileload (FILE.STAT, guide,
and FILEINFO in Basic) will calculate the value for ALL
dynamic file types
 The fileload is a natural part of the statistical information users may
find useful even if the split style is not WHOLEFILE
 The calculated fileload value can be used in the CONFIGURE.FILE
command to set the file header if the split style is set to WHOLEFILE

36

37
• Using CONFIGURE.FILE to manually set the FILELOAD of a
DYNAMIC WHOLEFILE

38

39

41
8.1 New Hash Type 3
The new hash type has been demonstrated to provide an
even distribution when the existing types don’t, for
complex key structures, improving distribution
performance (in terms of standard deviation), and runtime
performance
• Type 0 is still the best for sequentially based keys
A new udtconfig parameter DEFAULT_HASH_TYPE
defaults the TYPE during the creation of a new file. It has
values of 0, 1, or 3
• The default value is hash type 3

42
8.1 New Hash Type 3
The CREATE.FILE , RESIZE, and MEMRESIZE
commands now accept a TYPE of 0, 1, or 3
• Old files will not be automatically resized to the value
specified in DEFAULT_HASH_TYPE. Any change in type will
need to be specified using the RESIZE / MEMRESIZE
command
 Note: TYPE 2 can be defined, but it is reserved for sequentially
hashed files (see the shfbuild command), which is why the LIMIT verb
now reports 4 hash functions although only 0, 1, and 3 can be user set

43
8.1 New Hash Type 3
Before choosing the algorithm for the new type
• 13 candidates were examined
• 6 different test patterns over millions of keys were explored,
including exact customer cases where dynamic files would overflow
extremely quickly
• Based on the AP function, which is a general hash function listed in
NIST (National Institute of Standards and Technology) Dictionary of
Algorithms and Data Structures
• Testing on a customer cases that did not perform well with Type 0
or Type 1 have shown performance improvements, some of them
quite substantial, using the new combinations

44
8.1 New Hash Type 3
CREATE.FILE and file.stat

45
8.1 New Hash Type 3 - fileview
Hash type is
now in hex

47
8.1 What Have We Changed?
Introduced 64-bit files
Introduced hash TYPE 3
Introduced a new dynamic file type (WHOLEFILE) that
changes its modulo based on the total file load rather than
on an individual groups load
• The new WHOLEFILE style will work for both 32-bit and 64-bit
dynamic files
• The existing dynamic file types KEYONLY and KEYDATA will
continue to be supported
• The default dynamic file type will be the new WHOLEFILE style

48
8.1 Backward Compatibility/Restrictions
 Compatibility
• Files created at 8.1 will be backwards compatible with earlier releases of
UniData if
 The file is 32-bit
 The hash type is 0 or 1
 If the file is a dynamic file it has a split type of KEYONLY or KEYDATA
• RFS already supports the use of 64-bit RFS log files from 7.2
• The RFS system buffer block size was made configurable from 7.3
 Restrictions at the start of 8.1
• The limitations for sequential files in Basic remains the same as of 7.2
 OPENSEQ and OSOPEN work with files over 2GB but restrictions on the use of the AT
clause still exist for values over 2GB with the OSBREAD and OSBWRITE commands

49
8.1 CREATE.FILE
CREATE.FILE (note new defaults and options)

50
8.1 FILEINFO Changes
 FILEINFO in UniBasic
• Code 4
 Static and dynamic files with KEYONLY returns 0, 1, or 3 (Hash type 0, 1, or 3)
 Dynamic files with KEYDATA returns 32, 33, or 35 (Hash type 0, 1, or 3)
 Dynamic files with WHOLEFILE returns 48, 49, or 51 (Hash type 0, 1, or 3)
• Code 11
 Returns the current fileload of a dynamic file
o For WHOLEFILE this is readily available
o For KEYDATA and KEYONLY the system needs to scan the file in order to calculate and
return the current load
• Code 31
 A return value of 0 indicates a 32-bit file
 A return value of 1 indicates a 64-bit file

51
8.1 HASH Changes
HASH in UniBasic has expanded the type to cover the
new file types
• HASH(rec.key,modulo,type)
 0, 1, or 3 for Static Hash Files with Hash Types 0, 1, and 3
 64, 65, or 67 for Dynamic Files with KEYONLY for hash types 0, 1, and 3
 96, 97, or 99 for Dynamic Files with KEYDATA for hash types 0, 1, and 3
 112, 113, or 115 for Dynamic Files with WHOLEFILE for hash types 0, 1,
and 3
• These are the file types you will see in fileview, but remember
fileview shows these values in hex

52
Next Steps – Other Presentations
Troubleshooting UniData
8.1 Technical Overview
Lab: UniData 8.1 Administration
Lab: U2 Replication
HADR Best Practices

53
Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED
IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE.
ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF
ROCKET SOFTWARE.

54
Trademarks and Acknowledgements
The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software,
Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and
Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by
Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual
property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of
any such marks.
Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, MvEnterprise, NetCure,
Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and
wIntegrate
Other company, product, and service names mentioned herein may be trademarks or service marks of
others.

8.1 In Depth: New 64-bit Files and File Management

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to 8.1 In Depth: New 64-bit Files and File Management

Similar to 8.1 In Depth: New 64-bit Files and File Management (20)

Recently uploaded

Recently uploaded (20)

8.1 In Depth: New 64-bit Files and File Management