1. Toronto, June 6-7 2016
IBM Connections
Troubleshooting or
"get the cow off the ice"
Nico Meisenzahl & Christoph Stoettner
panagenda

2. PLATINUM & SPOTLIGHT SPONSORS
GOLD SPONSORS
SILVER SPONSORS
BRONZE SPONSORS

3. Christoph Stoettner
• Senior Consultant at panagenda
• IBM Notes / Domino since 1999
• IBM Connections since version 2.5 /
2009
• Many years of experience in:
• Migrations
• Administration and installation
• Performance analysis
• Joined panagenda in 2015 focusing in:
• IBM Connections deployment and optimization
• IBM Connections monitoring
• Husband of one & father of two,
Bavarian
 @stoeps
 linkedin.com/in/christophstoettner
 www.stoeps.de
 christophstoettner
 +49 173 8588719
christoph.stoettner@panagenda.com

4. Nico Meisenzahl
• Consultant at panagenda
• IBM Notes / Domino since 2008
• IBM Connections since version 3.0
/ 2010
• Many years of experience in:
• Consulting
• Migrations & Administration
• Joined panagenda in 2016
focusing in:
• IBM Connections Consulting
• ICS deployment & optimization
 @nmeisenzahl
 linkedin.com/in/nicomeisenzahl
 meisenzahl.org
 nico.meisenzahl
 +49 170 7355081
nico.meisenzahl@panagenda.com

5. Agenda
• Howto: Troubleshooting
• Troubleshooting…
• Connections itself
• Backend (DB2, TDI, WebSphere, SSO)
• Optional add-ons (Docs, FEB, Cognos, CCM)
• Tools

6. Toronto, June 6-7 2016
Howto: Troubleshooting

7. Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause

8. Be aware of the big picture
• Client-side problems
• Debug in different Browsers (IE, FF, Chrome)
• Do NOT use a server IE
• Server-side: IBM Connections is based on
many components
• debug on “high level” first
• get an overview which backend service is causing
the error

9. Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can
have big consequences!

10. Analyze log files and browser
• Analyse log files
• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• ELK stack
• Tools for client-side problems
• Firebug or Developer-Tools
• BurpSuite or Fiddler

11. Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-
config.xml“
• Error stack
Tip: In a clustered environment, start and
analyze only one Node (if possible)

12. Find support
• Knowledge Center http://goo.gl/HFzTmv
• Troubleshooting Tips https://goo.gl/mU1EQ9
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR

13. Toronto, June 6-7 2016
Troubleshooting

14. Get the cow off the ice
How to do this?

15. Agenda

16. WebSphere Application Server
(IBM Connections) logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:
• <wasroot>/profiles/<profilename>/logs/<servername>/

17. Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information

18. WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out
by an application
• R - Messages that are written directly to System.err by
an application
• Z - Place holder to indicate type was not recognized

19. WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)

20. Read trace stack
• First line displays key information
• “Caused by” displays root cause

21. Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path

22. Enable tracing
• Enable tracing using
ISC
• Runtime or configuration
only
• Define tracing based
on
• App prefix / error stack
• Must gather (PMR)

23. Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only
• Using node fqdn to access different nodes

24. Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all

25. Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps

26. CLFRW0394E: Search indexing of
services ...
• Search index not ready
• interruption at index creation
• CLFRW0283E: Search has encountered a problem while
crawling
• CLFRW0027E: Error Indexing component <app> for
search
• INDEX.READY file not present
• Recreate and enable tracing

27. com.ibm.connections.directory.services
.exception.DSOutOfServiceException
• Access Connections not possible
• DSX not working
• Check SSO settings
• Domain
• LtpaToken Cookie name
Tip: WebSphere FP sometimes resets SSO
domain

28. EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member

29. Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)
• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout

30. Database connections
• Check datasouces
• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log
• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration
enabled by default!

31. HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log

32. HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log

33. HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log

34. HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired

35. SDI / TDI logs
• <tdisol>/log/ibmdi.log
• TDI log file
• <tdisol>/employee.*
• Files include all changed users (adds, update, delete,
error, skip)
• <tdisol>/syncupdates/*
• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)
• profiles_tdi.properties
Tip: Check lock file

36. Analyze SDI logs
• Error code prefix
• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …

37. Debug SDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default

38. Cognos BI logs
• Cognos BI
• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer
• <installroot>/logs/cogserver.log
• PowerCube build
• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)

39. BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check
• Cronjobsor scheduled jobs
• trxschelog.log

40. Debug Cognos BI & Metrics
• Communication between Cognos BI and
Connections Metrics
• SonataHttpUsage=all:
SonataHttpHeader=all:
SonataHttpBody=all:
com.ibm.connections.httpClient.*=all:
com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all

41. FileNet logs
• SystemOut.log & trace.log
• FileNet logs
• <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log

42. FileNet urls
• http://<fqdn>/FileNet/Engine
• Server status, version, sonata/waltz version
• http://<fqdn>/P8CE/Health
• Health checks for authentication, stores and
database
• http://<fqdn>/dm
• FNCS version and configuration

43. Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all

44. Debug FileNet using ACCE
• http(s)://<fqdn>/acce

45. Debug FileNet using JVM Properties
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml
-DskipTLC=true
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)

46. Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version.txt

47. Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all

48. Debug Kerberos
• Configuration
• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all:
com.ibm.issw.spnegoTAI.*=all:
com.ibm.security.krb5.*=all
• Runtime
• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.websphere.wim.*=all:
com.ibm.ws.wim.*=all
• Fiddler & BurpSuite

49. Toronto, June 6-7 2016
Tools

50. Analyze logs
• Analyze logs live
• Baretail
• tail –f
• View logs
• Atom.io, Notepad++
• less, tail
• ELK Stack
• Elasticsearch, Logstash, Kibana
• Small Docker deployment

51. Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• BurpSuite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools

52. SQL Clients
• db2 command
• IBM Datastudio
• Dbeaver, DBVisualizer, squirrelSQL
Tip: Database write access is not supported!

53. LDAP Clients
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio

54. Analyze Network
• Wireshark
• tcpdump
If you ask the * admin:
There is never a problem within the *!
J
*= Network, SAN, VM, ...

55. Useful Blogs
• http://ibmconnections.com
• http://turtleblog.info
• http://portal2portal.blogspot.de
• https://www.urspringer.de
• http://socialconnections.info
• http://blog.robertfarstad.com
• http://www.curiousmitch.com
• http://www.ramsit.com/category/blog
• http://techblog.gis-ag.info
• https://milanmatejic.wordpress.com
• http://ibmdocs.com
• http://domino.elfworld.org
• https://dontforgetthe0.com
• http://dilf.me.uk/socialshazza
• http://www.stoeps.de
• http://scripting101.org
• http://meisenzahl.org
• http://martin.leyrer.priv.at
• http://kbild.ch
• http://www.notesgoddess.net
• http://www.dominodiva.com
• http://notesbusters.com
• https://rob59blog.wordpress.com
• http://connections101.info
• http://brandlrainer.blogspot.de
• https://collaborationben.com

56. Thank you very much for your attention!
panagenda GmbH – Make Your Data Work for You
Lahnstr. 17 ● 64646 Heppenheim (Germany)
Skype: christophstoettner ● Cell: +49 173 8588719
E-Mail: christoph.stoettner@panagenda.com
Christoph Stoettner
Senior Consultant
panagenda GmbH – Make Your Data Work for You
Lahnstr. 17 ● 64646 Heppenheim (Germany)
Skype: nico.meisenzahl ● Cell: +49 170 7355081
E-Mail: nico.meisenzahl@panagenda.com
Nico Meisenzahl
Consultant

57. PLATINUM & SPOTLIGHT SPONSORS
GOLD SPONSORS
SILVER SPONSORS
BRONZE SPONSORS