Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

8th Sutol Conference, November 2016
A Hitchhiker’s Guide to
Troubleshooting IBM Connections
Sharon Bellamy James
& Roberto Boccadoro

A Hitchhiker’s Guide to
Troubleshooting IBM Connections

… and no the answer is not
42

Sharon Bellamy James
• IBM WebSphere since 1999 &
Connections since 2 / 2008
• Many years experience with -
migrations, installations, administration,
integration, customization
& DOCUMENTATION!! 
• Tiny bit of a Star wars & Disney fan
• Charity fundraising Cosplayer/Costumer
• Part of the team that brings you ICON UK
5

Joined Lotus in 1994, we were selling then, Notes was V3 and was sold as
development platform. Remained in IBM since Sept. 2012 in Technical Sales. During
those years I have worked with almost all the products Lotus/IBM has created in the
collaboration space. I am now a consultant IT Specialist/IT Architect and my actual
focus is mostly IBM Connections, IBM Docs, IBM Sametime.
I have 10 IBM Technical certifications and I am an
Outside work, I am a husband and father of 4 (2 humans, 2 cats) and an amateur
astronomer.
@robboc59
blog http://rob59blog.wordpress.com/
roberto.boccadoro@eldeng.it
Roberto Boccadoro

Overview
• Different types of logs, where are they located?
• How to read system out logs
• Applying traces
• Fiddler traces
• A bunch of other useful stuff

Connecting Connections – there are many
integration points
Basic WebSphere Architecture plus:
Plugins
Search
Mobile
Portal
Cognos
FileNet
IBM Docs
Mail Integration
TAM / SiteMinder
SPNEGO / SAML

A Jigsaw of many parts

Know your Topology
• Is the installation:
Small – one server (jvms)
Medium – at least 3 servers (jvms)
Large – one server (jvm) for each Connections
application

Small Deployment
• One single node
– Deployment manager
– Nodeagent / Connections WebSphere server
deployed on the
same machine
• All applications are deployed to one JVM
(WebSphere Server)
• DB2, TDI & HTTP can also be deployed to
the same machine or another machine
• Add additional
– Mail integration, Cognos, IBM Docs,
CCM/Filenet etc….

Medium Deployment
• One or more nodes
• Applications are spread over at least 3 JVMS
(WebSphere servers)
• TDI, DB2 and HTTP usually on additional
machine(s)
• Potential for load balanced HTTP servers
• Plus additional
– IBM Docs, Cognos, CCM/Filenet,
Mail integration, SSO – Siteminder/TAM/SPNEGO,

Large Deployment
• Generally used for 100’s of thousands of users
• Multiple nodes
– Each application is deployed to its own JVM / WAS server
• Load balanced multiple HTTP servers
• Enterprise sized clustered/fail over DBs
• Clustered LDAP
• Plus additional
– IBM Docs, Cognos, CCM/Filenet, Mail integration,
– SSO – Siteminder/TAM/SPNEGO

Everything has a log
But which one do you need?
• WAS System out - First place to start
• HTTP Access / Error
– 404’s, other access issues
• HTTP Plugin
– SSL issues, HTTP to WAS connection issues
• TDI - Sync issues
• DB2 - Diag, admin logs
Associated Software
• IBM Docs – WAS logs
• CCM/Filenet – WAS logs / P8 Logs
• LDAP – LDAP logs / domino / AD etc.
• Cognos – WAS logs

Different Types of Logs, Where are They?
Because Connections connects to
everything, and because Connections is a
collection of applications that run on the
WebSphere platform, there are a variety of
logs in a variety of locations:
-WebSphere
-Connections Install
-Connections Migration
-HTTP server
-TDI
-Congnos
-FileNet
-wsadmin
-Database
-Mobile
-Fiddler

Where are the logs? WebSphere
• AppServer simply distinguishes the
WebSphere Application Server from
other WebSphere deployments that may
be under the WebSphere folder:
- The first layer of logs only pertain to the
high level install and are seldom used
- Most all of Connections log activity
pertains to the logs within the profiles
directory.

• Again, most all of our log activity
pertains to the logs within the
profiles directory.
-For example
-AppSrv01 – Profile for Connections server
-AppSrv02 – Profile for Cognos server
-Dmgr01 – Profile for Deployment Manager

- AppSrv01 – The Profile
- iccluster_server1 – The actual Connections
server JVM.
• Systemerr – System level logs that can
sometimes have key information, but not usually
• Systemout – WebSphere application server log
– this is what we use 95% of the time.
• Traces – If captured, would also be located here
• Multiply these logs by 2, 3, 4 or more for
clustered environments!

Where are the logs? Connections install and
CR updatesC:IBMConnectionslogs
While Connections runs on WebSphere, there are
additional resources that are wrapped around
WebSphere. This data is stored in the
ibmconnections folder
It includes such things as:
• Indexing
• Customization folders
• Files - attachments
• Logs:
– 5.0 - Install / CR updates
– 4.5 - Install
NOTE: As of Connections 5.0:
CRs are installed using the IBM Installation
Manager.
iFixes are still installed using the connections
update installer

Where are the logs? Connections install V5.5
• Connections 5.5 installation
log
• C:IBMConnectionsinstall.log
• /opt/IBM/Connections/install.log
• The install log contains warning,
errors and
success messages and details of the
installation

Where are the logs? iFixes
• C:IBMConnectionsversionlog
• /opt/IBM/Connections/version/log
• Each iFix has its own log with the
info, error or warning messages

Where are the logs? Current release info
• To find the current release level
you can browse these files:
• C:IBMConnectionsversion
• /opt/IBM/Connections/version

Where are the logs? TDI (TDISOL)
• TDISOL logs are in the TDISOLlogs directory
• C:IBMTDIV7.1.1tdisollogs
/opt/IBM/TDI/V7.1.1/tdisol/logs
• Or wherever you put TDISOL (same for
external users EXTTDISOL etc.)
• What is TDISOL?
• The TDI Solution directory. This holds all
of the scripts and configuration
that TDI requires to populate the profiles
DB and
keep it in sync with your LDAP directory.

Where are the logs? DB2 / Database
DB wizard logs:
In the *home* directory of the user running the
wizards – typically
Windows :
C:UsersAdministratorlcWizardlogdbWizard
Linux: /home/db2inst1/lcWizard/log/dbWizard
DB2DIAG LOG:
Windows:
C:ProgramDataIBMDB2DB2COPY1DB2DAIG
0000
Linux:
/home/db2inst1/sqllib/db2dump
Asked for by IBM for debugging issues

Where are the logs? HTTP Server
C:IBMHTTPServerlogs
/opt/IBM/HTTPServer/logs
• As all Connections application to
application interactions flow through the
HTTP server, there are times when these
logs will help to determine routing failures.
access.log – logs all access via the http
server
error.log – logs errors for the http server

Where are the logs? HTTP Server
These logs can be useful for tracking
calls to various applications.
Remember, every application to application
is actually a request to the HTTP server to
find out the location of the application.
-Connections uses LotusConnections-
Config.xml to get the URL for the
application, then contacts the HTTP
server to get to that location via the
WebSphere plugin
-The HTTP return code is second from the
end on the right

Where are the logs? Filenet / P8
C:IBMWebSphereAppServerprofilesApp
Srv01FileNetCL_connections_server
/opt/IBM/WebSphere/AppServer/profiles/Ap
pSrv01/FileNet/CL_connections_server/
• p8_server_error.log
p8_server_trace.log
pesvr_system.log
pesvr_trace.log

Where are the logs? wsadmin
The WebSphere administrative (wsadmin) scripting
program is a non-graphical command interpreter
environment enabling you to run administrative
operations in a scripting language. Connections
uses this tool to perform various maintenance and
update functions.
Wsadmin logs: (typically not logged, the
information just displays in the command window)
wsadmin>execfile("communitiesAdmin.py")
WASX7015E: Exception running command:
"execfile("communitiesAdmin.py")"; exception
information:
com.ibm.bsf.BSFException: exception from Jython:
Traceback (innermost last): File "<input>", line 1, in ?
IOError: File not found - communitiesAdmin.py (No such
file or directory)
wsadmin>
In this case, the wsadmin command was run from the
Appserver rather than from the Dmgr and therefore the file
was not present.
Wsadmin tracing:
Tracing operations using the wsadmin scripting tool
(this will help to capture wsadmin errors, not functional
errors)
http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.base.doc
%2Finfo%2Faes%2Fae%2Ftxml_traceadmincontrol.html
Right before you enter the communities.Admin
command, run the following (bolded) command to add
tracing:
wsadmin>AdminControl.trace('com.ibm.*=all')
wsadmin>execfile("communitiesAdmin.py")
wsadmin>CommunitiesService.moveCommunityToSubcommunity.............
...
This will create "wsadmin.traceout" log under the
IBMWebsphereAppserverprofilesAppSrv01logs

Switching log language to english
Open ISC Open Servers > Application Servers.
Then choose the server where you want to
modify the language for the log.
Click Java and Process Management > Process
definition in section Server Infrastructure,
Then Additional Properties > Java Virtual
Machine
In the Generic JVM Arguments box type:
-Duser.language=en -Duser.region=US *
Click OK, and Save, Logout from ISC
Restart the server to apply the changes of the
JVM.
*NOTE: depending on the machine region it
sometimes will not take the –Duser.region=US
ensure the machine LOCALE is set to US

Making sense of the SystemOut log - structure
Displays:
WebSphere release
WebSphere server name
PID – Process ID
Operating System
File location of the server
Tracing applied
Check to see if this is the startup of the server that you want!
Does not display:
Release of Connections  This is because Connections is a
collection of applications that run on WebSphere server.
We can however find this from the systemout.log that
includes the application startups. Stay tuned.

Making sense of the SystemOut log

How to Read and Analyse the Systemout Log
Searching:
- Get counts of errors
- See historical trends of errors
- “ E “ error
- “ W “ warning
Best trouble-shooting tip I learned last year, notice the spaces before and after the E
Will still need to review the log – but this can help rise to “10,000 feet” and see the context.

Making sense of the SystemOut log – stack trace
Tools:
A decent text editor – notepad++, editpad etc. for windows, gedit for linux etc.
You may need to access multiple logs in a clustered environment
Find the most recent error and work backwards
Make sure you are looking at what you think you are looking at! (Some logs have months of data, others have minutes)
Systemout is not always at the beginning of a reboot, check to see "application started“ in
systemout.
From Notepad++ you can “find all in current document” and see all applications started in a
systemout.log:

WebSphere Tracing
There is little information about reading stack traces from a support perspective and
unfortunately healthy servers may also have stack traces, so it can be hard to interpret
which ones are serious and which are not.
The first line will tell you key information about the error
The lines in-between are all pointing back to what called them
The last time is the ultimate source
Look for the “caused by” line if available

WebSphere Tracing
Log on to the WebSphere Application Server
(WAS) integrated solutions console using an
administrator ID
Go to -- Troubleshooting -> Logs and Trace -> Logging
and Tracing -> and the server you wish to add tracing to
Diagnostic Trace Server -> Change Log Detail Levels ->
Add the following string and click Apply -> OK
*=info:com.ibm.lconn.moderation.*=all: (sample trace)
This is dynamic
It will not let you save a bad trace
In a multi-node system, collect the logs from one node at a
time
• i.e. only have one node running to test / collect logs
• Makes trouble-shooting much easier
• IBM will normally request this if you have a PMR open

Fiddler Tracing
IBM support will almost always ask for a fiddler trace - http://www.fiddler2.com/fiddler2/
Ensure SSL traffic is decrypted : Tools - Fiddler Options - HTTPS tab - then select both
"Capture HTTPS CONNECTS" and "Decrypt HTTPS traffic” and accept the cert warning
Clearing the trace is easy – just use Ctrl + x
Shut down other apps/browsers that add extraneous data to the trace
Note: that Fiddler time is in your local browser’s time – Make a note of local time tests
are run and server time
Save the trace file (.saz) – send these to IBM with the systemout.logs along with time
stamps and trace info.

Fiddler Tracing

Business as usual – Useful Stuff

Useful Stuff – WAS versionInfo
From the WAS_HOMEbin (i.e
/opt/IBM/WebSphere/AppServer/bin)
run the versioninfo command
versionInfo.bat /sh
– this command will list the top level WAS
version
and major fix/feature pack
versionInfo.bat /sh
–maintenancePackages – this command
will list WAS version,
fix/feature pack and any iFixes installed

Useful Stuff – Connections Version Info
From the Connections install_homeupdateinstaller directory run the following commands
updateSilet.bat/sh –fix –installDir <connections_home>
i.e.
./updateSilent.bat/sh –fix –installDir /opt/IBM/Connections
This command will list the installed version of connections and any fixes applied

Useful Stuff - Change WAS log size
Done via the community scripts (see
http://scripting101.org) or via the ISC
Set for EACH server including nodeagent
and dmgr
Troubleshooting > Logs and trace > JVM
Logs
Default it 1mb – useless
Change to at least 10 and keep 20 historical
Don’t forget to archive them !! 

Useful Stuff - Check SIB Queue Depth
Use the following path in ISC to see if there is a backlog in the messaging bus: (keep this
handy!)
Service Integration > Buses > ConnectionsBus >Destinations > connections.events >
Publication points >connections.events@CL.000-ConnectionsBus [Runtime] >
Subscriptions
If any of the queues have a backlog it is possible to
refresh them. If they backlog up that indicates a more
serious issue. A server restart can sometimes kick
the queue into life – check SystemOut for log
messages relating to the SIBs

Useful Stuff - WAS Ports / Apps & Context Roots
Where is my app running?
Check app and manage modules
which cluster app is deployed to
Check ports via

Useful Stuff - Clear WAS Temp
Shut down all WAS processes : Connections
server(s), the DMGR, and the nodeagent.
Clear the temp/cache folders:
IBMWebSphereAppServerprofilesAppSrv01temp
IBMWebSphereAppServerprofilesAppSrv01wstemp
IBMWebSphereAppServerprofilesDmgr01temp
IBMWebSphereAppServerprofilesDmgr01wstemp
Stop and start the HTTP server – this kills off any stale
web threads /
sessions
Restart DMGR ,node agent and Connections servers
Clear the cache on the browser before you test
(EVERYTHING)
Must be done every time a fix is applied to clear
issues with UI – also good housekeeping

Useful Stuff - mtail
Mtail http://ophilipp.free.fr/op_tail.htm
When you run Connections a a Windows service, it will tell you it is started but in fact this is not
true. It takes a while between the service appears to be running and Connections is really
started.
Use Mtail to open the Connections_server SystemOut.log, when you see “server
Connections_server open for e-business” then Connections is running.
Tip – in the Mtail window filter for the word “open”
Also baretail is a free GUI tool for tailing on windows

http://rob59blog.wordpress.com
robertoboccadoro
@robboc59
Any Questions?
http://socialshazza.com
dilftechnical
@socialshazza

Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

Recommended

More Related Content

What's hot

What's hot(19)

Viewers also liked

Viewers also liked(20)

Similar to Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

Similar to Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections(20)

More from Sharon James

More from Sharon James(15)

Recently uploaded

Recently uploaded(20)

Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections