Troubleshooting UniData

1
Troubleshooting UniData
Jonathan Smith, Principal Technical Support Engineer

2
Abstract
 When things go wrong, we all need a hand. This presentation takes you
through solving common problems and working through complex issues to
identify the cause and source of an issue. We’ll cover the common
‘gotchas’ so you can avoid them. We’ll cover the tools available to assist in
further problem determination. We’ll also discuss the online resources that
are available to you and how to determine if Rocket already knows about a
problem.
©2015 Rocket Software, Inc. All Rights Reserved.

3
Agenda
 Diagnosing an error
 Common errors
• Process details
 UniQuery problems
 Using an index
 Using guide
 Help with Replication
 Shared memory
 Working with Rocket Technical Support

5
Diagnosing an Error
What version of UniData do I have installed?
• $UDTBIN/port.note contains the version and build number
• The last line of ECL VERSION command
• SYSTEM(513) in UniBasic
Has anything changed?
• O/S patch or upgrade; more users added; application
software changed; UniData upgraded; UniData tuneables;
has other software been loaded or changed; hardware

6©2015 Rocket Software, Inc. All Rights Reserved.
Diagnosing an Error

7
It all starts with an error message
• udt.errlog
 file corruption, shared memory errors
• smm.errlog
 startud failures, license expiring messages
o see udt_lic_exp_notify in $UDTBIN to automate notification
• sbcs.errlog
 errors loading globally cataloged programs
• cleanupd.errlog
 logs cleanup of abnormal termination of udt processes
Diagnosing an Error

8
sm.log
• For RFS-enabled, this is the first place to look as it contains
messages about the RFS operations
rm.log
• This is the first place for U2 Replication as it contains messages
about all Replication operations including the errors from the
rm.errlog file
rw.errlog
• This file contains errors encountered by the U2 Replication
‘Replication Writer’ processes (Note: the writer process is also used
in object table creation so will also contain loading error messages)
Diagnosing an Error

9
More log files
• resize.log
 Logs start and finish of all file resizes
• ipclogs
 If a process has encountered a shared memory allocation error, an ipclog
will be created for each error
• minidump (Windows)
 If a process encounters an exception violation, a minidump will be created
• Windows event viewer (8.1 UniData has its own log)
• $UDTBIN/saved_logs - previous logs 20 iterations
Diagnosing an Error

10
Are all the UniData daemons/services running?
• When did they last start?
 UNIX: Look at timestamp of smm.log or cat $UDTBIN/startud.log
 Windows: Look at the Windows UniData event log (8.1+)
Are there mixed executables from multiple versions?
• Check for standard executables with an old date
• “SMM’s shared memory not reliable” message
Diagnosing an Error

11
Are there linked in C routines?
• UNIX: date/time stamp on the udt executable will be newer
• Windows: There will be an ‘E’ type entry in a VOC file with a
DLL path in attribute 2. Examine the DLL with Windows
Explorer to obtain the file “changed” date and time stamp and
the DLL “properties.”
Can the problem be reproduced with the udt.d
executable?
Diagnosing an Error

12
UNIX -showud

13
Windows – UniData Services

14
startud.log and smm.log

15
 Are concurrent UniData versions installed on the system?
• ps -ef | grep smm to check for multiple active smm’s
• Windows - Check for multiple services
 Is RFS enabled on the system?
• SB_FLAG=1 in the udtconfig
• Check to see if ARCH_FLAG=1 to determine whether archiving is enabled
 Is Replication enabled on the system?
• REP_FLAG=1 in the udtconfig
 Is the current udtconfig file in place?
• See SHOWCONF command
 See UDT-12239 in solution search
• “Collecting data for UniData: UniData Problem Profile”
Diagnosing an Error

16©2015 Rocket Software, Inc. All Rights Reserved.
Diagnosing an Error

17
Where is the Error Coming From?
Was the error produced by the application software?
• Application developers tend to have their own message scheme
• Is it a UniBasic run time error?
 In /usr/ud81/sys/CTLG/t/TEST at line 20 …
 If this preface text is missing, re-run the program after running the ECL
command HUSHBASIC OFF (SB uses HUSHBASIC OFF)
 See “UniBasic Runtime Error Logging” in latest version of the Developing
UniBasic Applications manual
 Or see UDT-3922 in the solution search “How to enable UniBasic run-time
error logging in UniData by creating the msglevelconfig file”

18

19
Is the error message a UniData error?
• Try ‘ESEARCH ERRMSG’ from ECL to validate UniData
errors based on text
• From ECL: ‘AE ERRMSG errno’ if you have an error number
 Useful for udt invocation problems where only the number displays
• Messages referencing functions named ‘U_’ … are UniData
errors, for example ‘U_read_group’

20
Customer Example: On trying to start a udt session,
the customer was getting the following error:
Error msgno = 30411. args = VOC

21
Is the error message a UNIX error?
• UNIX: Check /usr/include/sys/errno.h
• Typically a 1-3 digit number (AIX also uses nnnn-nnn)
• When compiling a UniBasic program, the error returned is:
Compiling UniBASIC: BP/TEST in mode 'p'. write
error on print_header errno=28 compilation failed
 errno 28 is defined as ENOSPC – no space left on device in UNIX
 In terms of UniBasic compilation, it means tmp is full

22
Is the error a Windows operating system error?
• Message box on console - no message in telnet window
 Try invoking a UDT shell on the console to see if different messages
appear
• Check the event viewer
 UniData has its own event log at 8.1
• Check for virus scanners interfering with UniData
 Add all UniData files to be excluded from scan

24
Common Errors – ECL Syntax Error
Check your ECLTYPE – Pick or UniData
• Paragraphs use UniData parser
• Procs use Pick parser
Check your BASICTYPE
• EXECUTE uses ECLTYPE ‘P’ for BASICTYPE ‘P’
• UDTEXECUTE forces UniData ECL parser
Lowercase verb forces UniData parser - even in Proc
&USERTYPE& in VOC sets parser for executing
commands within AE editor

25
Common Errors – ECL Syntax Error

26
Common Errors – Not a Verb
The verb must exist in the VOC or be a globally
cataloged program
Check $UDTHOME/sys/CTLG if your expected
globally cataloged program is not found
• Check to see if multiple CTLG areas exist or are expected
$UDTHOME/sys/CTLGTB is a UniData hashed file
table with information about globally cataloged
programs

27
More Common Errors
Floating point exception error
• Explore the ECL command: FLOAT.PRECISION
Review any UDT.OPTIONS that are in place
• There are currently 120
Sorting problems
• Review SORT.TYPE command
• See UDT.OPTIONS 69

29
What is the Process Doing?
Explore the PORT.STATUS command
• Review the PID CALL.STACK and FILEMAP options
• SYSTEM(49) and SYSTEM(50) in UniBasic also allow access
to the call stack and filemap information
• SYSTEM(52) returns the entire command stack

30

31

32
ENABLE.USERSTATS
• Collects udtmon-like statistics for a single process
 udtmon displays UniData statistics from a system level
• LIST.USERSTATS shows results
• DISABLE.USERSTATS to turn off collection
udstat <monitor period> <iterations> (UNIX only)
• Displays udstat statistics for a single process

33

34
pr, gr = private, global program calls rd, wt, dl = read, write, delete
po, iv = level 1, level2 overflow blocks accessed sh = shell commands
gl, gu = record lock, unlock sc = strings flushed (screen i/o)
fs = float to string conversions ph = sbcs physical reads (load new pcode)
fs = float to string conversions vr = sbcs virtual requests (access loaded
pcode)

35
Is it repeatable?
• Rocket Support will ask for the reproduction case (this is the
best answer)
• We may suggest replacing the optimized udt executable with
the non-optimized udt.d executable
• UNIX: We may suggest using tools such as truss or kill to
produce a detailed core file and stack trace
• Windows: We may suggest using process explorer

36
udtmon

37
udtmon

38
Is the Process Hung?
Check the READU locks
• ECL command LIST.READU lists locks set
 UniBasic GETREADU() function
 Note the FILENAME USERNAME and DETAIL options
• ECL command LIST.QUEUE lists users waiting for locks
 UniBasic GETQUEUE() function
• Check ‘semaphore’ or ‘file’ locks with LIST.LOCKS
• If WRITE_TO_CONSOLE=1, confirm the function is working

39

40
The glm_tool and ulc_tool allow you to examine the
UniData lock tables
• Both tools can run interactively or allow a dump file to be created
that can be read by either tool for later analysis
 The udtdiag tool (discussed later) takes advantage of this
• The glm_tool (global lock manager) displays details of the shared
memory lock table
• The ulc_tool (UniData lock control) displays internal lock tables and
structures controlling table access (toggles, latches)
 Group locks are listed as MGLM – Mini Global Lock Manager

41
glm_tool

42
glm_tool

43
ulc_tool

44
Are All the Processes Hung?
Check UNIX message queues
• $UDTBIN/ipcstat –qo
 This may hang if UNIX messaging is inoperative
• UNIX ‘ipcs –qop’ – look for non-zero CBYTES or QNUM

45
Are All the Processes Hung?
Is dbpause on?
• dbpause_status will report the status of dbpause
 We recommend that you suspend Replication (if you are using it) before a
dbpause is issued (at 8.1, dbpause –c will report this)
If SB_FLAG=1 and ARCH_FLAG=1 and all the processes
have hung…
• Review the $UDTBIN/sm.log file and ensure UniData has off-
loaded the last archive log successfully
 “The archive file /rfs_archives/archive02 cannot be off loaded.”

47
UniQuery Problems
Problems with sorting
• Refer to earlier slide on SORT.TYPE and UDT.OPTIONS
after sort precedence
Problems selecting numeric data with leading zeros
• Review UDT.OPTIONS 22
DET.SUP not displaying on break lines
• Review UDT.OPTIONS 25, 43

48
UniQuery Problems
UniQuery outputs all the multivalues but I only wanted one
• Review the multivalue association name and association phrase
• Change WITH to WHEN in UniQuery

49
UniQuery Problems

51
Using an Index
 Is your query using an index?
• Try the REQUIRE.INDEX keyword – this keyword forces the use of an index
 Does the index match the data file contents?
• Try the NO.INDEX keyword to ignore an index
• guide_ndx can check the logical integrity of an index
• Virtual field indexes that refer to other files are not updated when the other file
changes
 Is the index file corrupted?
• guide_ndx can check for physical corruption
 See Tech Note UDT-11351 “How and when does UniData use an
index in a query?”

52
Using an Index

53
Beware of Index Overflow
A UniData Dynamic Index file stores all the overflow
information for all indexes created from the data file in a
single file
• Prior to 8.1 and 64-bit files, this means the primary index file could
not exceed 2GB (or MAX_FLENGTH or ulimit)
• “The primary index file xxxxxxx has no space for expanding”
LIST.INDEX will allow you to see if an index is overflowed
The DETAIL keyword will allow you to see which keys are
overflowed

54
Beware of Index Overflow

55
Minimizing Index Overflow
Set the correct alternate key length when the index is
created
Key Size Block Size Key Size Block Size
1-13 2,048 17-177 10,240
14-54 4,096 178-218 12,288
55-95 6,144 219-259 14,336
96-136 8,192 260-300 16,384

56
Resizing Indexes
To resize an index based on the previous table
• DELETE.INDEX filename ALL
• CREATE.INDEX filename field1 field2 {options}
• Specify the new Alternate Key Length
• BUILD.INDEX filename ALL {ONLINE}
Indexes only get larger over a period of time as
index keys are not truly deleted from the file

58
How Do I Know if a File is Corrupted?
If corruption is encountered by udt or tm process at
runtime, it is recorded to:
• udt.errlog for non RFS files
• sm.log for RFS files
‘guide’
• Reports corruption in UniData hashed files
• Also reports file statistics and notes undersized files

59
A Few Tips for ‘guide’
 If you do not need the statistics or advice files:
• guide filename –ns –na
 If you need the fix and error files to go to a different file:
• guide filename -f fix file name -e error list file
 If you need to check for a specific character:
• guide filename –U0
 Checks for char(0) in the file
 Set the environment variable
SUPPRESS_ORPHAN_BLOCK_ERROR to 1 to stop orphan
block warnings being written to the guide errors file

60
■ Create a new hashed file called GSTATS
■ Copy the contents of the default dictionary
$UDTHOME/sys D_UDT_GUIDE to the dictionary of the file
■ From the command line, use the command
‘guide –r GSTATS –G –d3’
■ From ECL try ‘LIST GSTATS ALL’
■ Use UniQuery to sort by level one overflow
‘SORT GSTATS BY-DSND OVERFLOW1 OVERFLOW1
OVERFLOW2’

61

62

63

64
How to Fix the File
What tools are available
• guide
• fixfile
• dumpgroup
• fixgroup
• fileview
• memresize
White paper available from support –
Advanced File Fixing for UniData

66
Check the Status of Replication

67
Check the Status of Replication

68
Replication – Where to Start Looking
The starting point is the rm.log file
• The rm.log also contains the error reported in the rm.errlog file
 This is the exception to UniData log files. It was a design decision to have
this as it makes it easier to compare log files from two different systems
side by side.
Look at both the publisher and subscriber as they may
show different views of the error
If an error refers to another part of Replication, then check
the related errlog file
• For example, if the message refers to a udrw error or write failure,
check the rw.errlog for more details about the error

69
Replication – Determining Where the Error is
Review the Logical Sequence Number (LSN) counts
on both systems for:
• Pub Done, Sub Got, Sub Avail, and Sub Done
• Is the condition repeatable?
 Consider stopping the subscriber cleanly and resetting the log files on
the subscriber to see if the same condition occurs
• Consider using reptool to identify what Replication is trying to
update, and attempt that update from ECL

70
Replication - reptool
Using reptool or XAdmin, it is possible to work out the
next LSN to be processed
• Once the LSN has been identified, you view the update
• You could then consider turning off Replication (if the file isn’t
sub writeable) and see if you can update the record from ECL
• The LSN also contains information to denote if it is part of a
transaction

71
Replication - reptool

72
Replication – Operations Guide
Search for “UDT-22 UniData Replication Operations
Guide”
• The UniData Data Replication Operations Guide is designed
to help DBAs set up, maintain, tune, and troubleshoot
Replication

74
UniData Shared Memory - What and Why?
 Part of the UNIX Interprocess Communication facilities
• Allows memory outside of process space to be configured and accessed
by multiple processes
 In Windows, Shared Memory uses memory mapped files
 Allows processes to share specific resources (globally
cataloged programs)
 Allows processes to expand and contract memory use without
having to grow and shrink the actual OS process size
 Allows processes to quickly access system-wide tables (for
example, record locks and dynamic files current modulo)

75
UniData Shared Memory
‘ipcstat –mb’ will allow you to see the current segments
(they will be marked with a version Rn.n, for example,
R8.1)
• sbcs – Shared BASIC code server
• sm – RFS system buffer
• rm (shmbuf) – Replication buffer
• rm (ctl) – Global Replication information such as the object
update table and LSN counters

76
 smm (ctl) - Global control table, dynamic file current modulo
table, group/physical locks
 smm (glm) – Record locks (for example, READU)
 smm (shmbuf) – Storage for UniBasic variable contents & ECL
command intermediate results - known as Global Segments
(GSM)
 smm (indirect) - ‘Self-created’ or ‘private’ segments used by an
individual udt process to store a large record or string
 Note: Starting at 8.1, UniBasic variable allocation now uses
heaped memory not shared memory

77
sbcsprogs displays active shared programs
Size of each sbcs memory segment is defined via
SBCS_SHM_SIZE in udtconfig
Only one sbcs segment is needed
Cannot have more 20 segments
• If a process tries to create a 21st segment, the error message
“Numra is maxed out in installshmid” displays. Starting at 7.3.4, the
message is “Unable to create more SBCS memory segments,
consider increasing SBCS_SHM_SIZE in udtconfig.”

78
 The size of each smm (glm) segment is set by udtconfig
parameter GLM_MEM_SEGSZ
• Divided into N_GLM_GLOBAL_BUCKET buckets (hashed table)
• Lock management performance can suffer if the number of buckets you
have configured is too small
• The 4MB default is often adequate for very large user-count systems
• Record locks are stored in this segment
 Cannot have more than 16 GLM segments
• If the lock table is full, a udt process trying to create a lock will display:
“No more cells for GLM segment id” – this is also recorded in udt.errlog
 You only need one smm (glm) segment

79
The sms (ctl) Global Control Table contains:
• Information about shmbuf segments, Memory Information (MI)
table, Local Control Table (LCT), Global Control Table (GCT)
• sms can be used to display contents – various arguments
• Current modulo of active dynamic files sms –d
• udtconfig SHM_FIL_CNT is number of slots in the table (default
2048)
• File systems available for dynamic file expansion sms -F
• udtconfig N_FILESYS is the number of slots in table (default 200)
• Physical (group) locks stored in this segment (if FCNTL_ON=0)
 Can be viewed with ulc_tool

80
UniData Shared Memory – More Information
 Correcting shared memory errors would be a whole
presentation in its own right
• The manual set and knowledge base has many articles
relating to the most common shared memory errors
• Support can provide historical presentations on
understanding shared memory
• If a UniData process encounters a memory allocation
problem, an ipclog is now almost always generated
 The ipclog provides excellent information towards correcting the
problem

81
Working with Rocket
Technical Support

82
Product Availability Matrix – Notices File

83
Product Availability Matrix – Notices File

84
Customer Portal - Solution Search

85
Customer Portal - Solution Search

86
Customer Portal – Team Issues View

87
Next Steps in Diagnosis
How can I best work with Rocket Software to resolve
the problem quickly?
• A clear explanation of the problem and the context of the
problem – “The System Crashed” is not really helpful
 Ask yourself, “what would I need to know to investigate the problem?”
• Serial number, OS Version, UniData Version, and build number
• The “Problem profiles” that were mentioned earlier are a good
aid to producing this
• A reproduction case if at all available

88
Cannot find your serial number and UD version from 7.3

89
The udtdiag utility is designed to capture a complete
diagnostic view of the system
• Use the udtdiag utility while the problem is occurring
 The output can be reviewed later
 You can take a second udtdiag after any recovery steps if required
For systems running Replication, one from each
publisher and subscriber involved in Replication

90
How Do I Use udtdiag?
See Tech Note UDT-3659 “How do I use the udtdiag
utility?”
• The Tech Note always has the latest version to download
• The Tech Note covers both the UNIX and Windows versions
• The ‘-help’ option explains how to use the command
• The ‘-ftphelp’ and ‘-sftphelp’ options explain how to use non-
secure or secure ftp to get the results to Rocket Support

91
Why Do I Use udtdiag?
 Captures a snapshot of the system while the problem is
occurring
 Dumps UniData shared memory tables in a re-playable format
 Minimizes downtime
• Gather data; take immediate corrective actions; analyze later
• No waiting for support provider to connect to system and troubleshoot
 Speeds problem resolution by gathering all data in one step
• Avoids request for information; review; request for more information

92
 Specifically recommended for:
• RFS or Replication problems
• UniData daemon or application hangs
• Locking issues
• System performance (limited data available)
 All UniData logs and error logs in $UDTBIN and $UDTBIN/saved_logs
 Re-playable Shared Memory dumps
• Record locks, group locks, RFS structures, Replication structures
 UniData configuration details: active, intended, default
 UniData Shared Memory commands output

93
 UniData process and daemon listings
 RFS performance snapshot; RFS memory dump from crash
 System information:
• File systems, OS version, kernel settings, system logs, network info
• Process listing, IPCS details, env settings, sar, and vmstat output
• Windows WinMSD report
 Regularly updated
 Be sure to create a udtdiag dump before you take any
corrective action!

94
Next Steps
Other presentations:
• 8.1 in Depth: Meet the new UDT 64-Bit Files
• Lab: UniData 8.1 Administration
• Lab: U2 Replication
• HADR Best Practices
• 8.1 Technical Overview

95
Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED
IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE.
ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF
ROCKET SOFTWARE.

96
Trademarks and Acknowledgements
The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software,
Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and
Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by
Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual
property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of
any such marks.
Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, MvEnterprise, NetCure,
Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and
wIntegrate
Other company, product, and service names mentioned herein may be trademarks or service marks of
others.

Troubleshooting UniData

More Related Content

Viewers also liked

Similar to Troubleshooting UniData

Recently uploaded

Troubleshooting UniData