Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

421 views

Published on

A Hitchhiker’s Guide to troubleshooting IBM Connections as given at SUTOL16 in Prague with Roberto Boccadoro

Published in: Technology
  • Login to see the comments

Sutol - A Hitchhiker’s Guide to troubleshooting IBM Connections

  1. 1. 8th Sutol Conference, November 2016 A Hitchhiker’s Guide to Troubleshooting IBM Connections Sharon Bellamy James & Roberto Boccadoro
  2. 2. 8th Sutol Conference, November 2016 A Hitchhiker’s Guide to Troubleshooting IBM Connections
  3. 3. 8th Sutol Conference, November 2016 … and no the answer is not 42
  4. 4. Sharon Bellamy James • IBM WebSphere since 1999 & Connections since 2 / 2008 • Many years experience with - migrations, installations, administration, integration, customization & DOCUMENTATION!!  • Tiny bit of a Star wars & Disney fan • Charity fundraising Cosplayer/Costumer • Part of the team that brings you ICON UK 5
  5. 5. 8th Sutol Conference, November 2016 Joined Lotus in 1994, we were selling then, Notes was V3 and was sold as development platform. Remained in IBM since Sept. 2012 in Technical Sales. During those years I have worked with almost all the products Lotus/IBM has created in the collaboration space. I am now a consultant IT Specialist/IT Architect and my actual focus is mostly IBM Connections, IBM Docs, IBM Sametime. I have 10 IBM Technical certifications and I am an Outside work, I am a husband and father of 4 (2 humans, 2 cats) and an amateur astronomer. @robboc59 blog http://rob59blog.wordpress.com/ roberto.boccadoro@eldeng.it Roberto Boccadoro
  6. 6. 8th Sutol Conference, November 2016 Overview • Different types of logs, where are they located? • How to read system out logs • Applying traces • Fiddler traces • A bunch of other useful stuff
  7. 7. 8th Sutol Conference, November 2016 Connecting Connections – there are many integration points Basic WebSphere Architecture plus: Plugins Search Mobile Portal Cognos FileNet IBM Docs Mail Integration TAM / SiteMinder SPNEGO / SAML
  8. 8. 8th Sutol Conference, November 2016 A Jigsaw of many parts
  9. 9. 8th Sutol Conference, November 2016 Know your Topology • Is the installation: Small – one server (jvms) Medium – at least 3 servers (jvms) Large – one server (jvm) for each Connections application
  10. 10. 8th Sutol Conference, November 2016 Small Deployment • One single node – Deployment manager – Nodeagent / Connections WebSphere server deployed on the same machine • All applications are deployed to one JVM (WebSphere Server) • DB2, TDI & HTTP can also be deployed to the same machine or another machine • Add additional – Mail integration, Cognos, IBM Docs, CCM/Filenet etc….
  11. 11. 8th Sutol Conference, November 2016 Medium Deployment • One or more nodes • Applications are spread over at least 3 JVMS (WebSphere servers) • TDI, DB2 and HTTP usually on additional machine(s) • Potential for load balanced HTTP servers • Plus additional – IBM Docs, Cognos, CCM/Filenet, Mail integration, SSO – Siteminder/TAM/SPNEGO,
  12. 12. 8th Sutol Conference, November 2016 Large Deployment • Generally used for 100’s of thousands of users • Multiple nodes – Each application is deployed to its own JVM / WAS server • Load balanced multiple HTTP servers • Enterprise sized clustered/fail over DBs • Clustered LDAP • Plus additional – IBM Docs, Cognos, CCM/Filenet, Mail integration, – SSO – Siteminder/TAM/SPNEGO
  13. 13. 8th Sutol Conference, November 2016 Everything has a log But which one do you need? • WAS System out - First place to start • HTTP Access / Error – 404’s, other access issues • HTTP Plugin – SSL issues, HTTP to WAS connection issues • TDI - Sync issues • DB2 - Diag, admin logs Associated Software • IBM Docs – WAS logs • CCM/Filenet – WAS logs / P8 Logs • LDAP – LDAP logs / domino / AD etc. • Cognos – WAS logs
  14. 14. 8th Sutol Conference, November 2016 Different Types of Logs, Where are They? Because Connections connects to everything, and because Connections is a collection of applications that run on the WebSphere platform, there are a variety of logs in a variety of locations: -WebSphere -Connections Install -Connections Migration -HTTP server -TDI -Congnos -FileNet -wsadmin -Database -Mobile -Fiddler
  15. 15. 8th Sutol Conference, November 2016 Where are the logs? WebSphere • AppServer simply distinguishes the WebSphere Application Server from other WebSphere deployments that may be under the WebSphere folder: - The first layer of logs only pertain to the high level install and are seldom used - Most all of Connections log activity pertains to the logs within the profiles directory.
  16. 16. 8th Sutol Conference, November 2016 Where are the logs? WebSphere • Again, most all of our log activity pertains to the logs within the profiles directory. -For example -AppSrv01 – Profile for Connections server -AppSrv02 – Profile for Cognos server -Dmgr01 – Profile for Deployment Manager
  17. 17. 8th Sutol Conference, November 2016 Where are the logs? WebSphere - AppSrv01 – The Profile - iccluster_server1 – The actual Connections server JVM. • Systemerr – System level logs that can sometimes have key information, but not usually • Systemout – WebSphere application server log – this is what we use 95% of the time. • Traces – If captured, would also be located here • Multiply these logs by 2, 3, 4 or more for clustered environments!
  18. 18. 8th Sutol Conference, November 2016 Where are the logs? Connections install and CR updatesC:IBMConnectionslogs While Connections runs on WebSphere, there are additional resources that are wrapped around WebSphere. This data is stored in the ibmconnections folder It includes such things as: • Indexing • Customization folders • Files - attachments • Logs: – 5.0 - Install / CR updates – 4.5 - Install NOTE: As of Connections 5.0: CRs are installed using the IBM Installation Manager. iFixes are still installed using the connections update installer
  19. 19. 8th Sutol Conference, November 2016 Where are the logs? Connections install V5.5 • Connections 5.5 installation log • C:IBMConnectionsinstall.log • /opt/IBM/Connections/install.log • The install log contains warning, errors and success messages and details of the installation
  20. 20. 8th Sutol Conference, November 2016 Where are the logs? iFixes • C:IBMConnectionsversionlog • /opt/IBM/Connections/version/log • Each iFix has its own log with the info, error or warning messages
  21. 21. 8th Sutol Conference, November 2016 Where are the logs? Current release info • To find the current release level you can browse these files: • C:IBMConnectionsversion • /opt/IBM/Connections/version
  22. 22. 8th Sutol Conference, November 2016 Where are the logs? TDI (TDISOL) • TDISOL logs are in the TDISOLlogs directory • C:IBMTDIV7.1.1tdisollogs /opt/IBM/TDI/V7.1.1/tdisol/logs • Or wherever you put TDISOL (same for external users EXTTDISOL etc.) • What is TDISOL? • The TDI Solution directory. This holds all of the scripts and configuration that TDI requires to populate the profiles DB and keep it in sync with your LDAP directory.
  23. 23. 8th Sutol Conference, November 2016 Where are the logs? DB2 / Database DB wizard logs: In the *home* directory of the user running the wizards – typically Windows : C:UsersAdministratorlcWizardlogdbWizard Linux: /home/db2inst1/lcWizard/log/dbWizard DB2DIAG LOG: Windows: C:ProgramDataIBMDB2DB2COPY1DB2DAIG 0000 Linux: /home/db2inst1/sqllib/db2dump Asked for by IBM for debugging issues
  24. 24. 8th Sutol Conference, November 2016 Where are the logs? HTTP Server C:IBMHTTPServerlogs /opt/IBM/HTTPServer/logs • As all Connections application to application interactions flow through the HTTP server, there are times when these logs will help to determine routing failures. access.log – logs all access via the http server error.log – logs errors for the http server
  25. 25. 8th Sutol Conference, November 2016 Where are the logs? HTTP Server These logs can be useful for tracking calls to various applications. Remember, every application to application is actually a request to the HTTP server to find out the location of the application. -Connections uses LotusConnections- Config.xml to get the URL for the application, then contacts the HTTP server to get to that location via the WebSphere plugin -The HTTP return code is second from the end on the right
  26. 26. 8th Sutol Conference, November 2016 Where are the logs? Filenet / P8 C:IBMWebSphereAppServerprofilesApp Srv01FileNetCL_connections_server /opt/IBM/WebSphere/AppServer/profiles/Ap pSrv01/FileNet/CL_connections_server/ • p8_server_error.log p8_server_trace.log pesvr_system.log pesvr_trace.log
  27. 27. 8th Sutol Conference, November 2016 Where are the logs? wsadmin The WebSphere administrative (wsadmin) scripting program is a non-graphical command interpreter environment enabling you to run administrative operations in a scripting language. Connections uses this tool to perform various maintenance and update functions. Wsadmin logs: (typically not logged, the information just displays in the command window) wsadmin>execfile("communitiesAdmin.py") WASX7015E: Exception running command: "execfile("communitiesAdmin.py")"; exception information: com.ibm.bsf.BSFException: exception from Jython: Traceback (innermost last): File "<input>", line 1, in ? IOError: File not found - communitiesAdmin.py (No such file or directory) wsadmin> In this case, the wsadmin command was run from the Appserver rather than from the Dmgr and therefore the file was not present. Wsadmin tracing: Tracing operations using the wsadmin scripting tool (this will help to capture wsadmin errors, not functional errors) http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.base.doc %2Finfo%2Faes%2Fae%2Ftxml_traceadmincontrol.html Right before you enter the communities.Admin command, run the following (bolded) command to add tracing: wsadmin>AdminControl.trace('com.ibm.*=all') wsadmin>execfile("communitiesAdmin.py") wsadmin>CommunitiesService.moveCommunityToSubcommunity............. ... This will create "wsadmin.traceout" log under the IBMWebsphereAppserverprofilesAppSrv01logs
  28. 28. 8th Sutol Conference, November 2016 Switching log language to english Open ISC Open Servers > Application Servers. Then choose the server where you want to modify the language for the log. Click Java and Process Management > Process definition in section Server Infrastructure, Then Additional Properties > Java Virtual Machine In the Generic JVM Arguments box type: -Duser.language=en -Duser.region=US * Click OK, and Save, Logout from ISC Restart the server to apply the changes of the JVM. *NOTE: depending on the machine region it sometimes will not take the –Duser.region=US ensure the machine LOCALE is set to US
  29. 29. 8th Sutol Conference, November 2016 Making sense of the SystemOut log - structure Displays: WebSphere release WebSphere server name PID – Process ID Operating System File location of the server Tracing applied Check to see if this is the startup of the server that you want! Does not display: Release of Connections  This is because Connections is a collection of applications that run on WebSphere server. We can however find this from the systemout.log that includes the application startups. Stay tuned.
  30. 30. 8th Sutol Conference, November 2016 Making sense of the SystemOut log
  31. 31. 8th Sutol Conference, November 2016 How to Read and Analyse the Systemout Log Searching: - Get counts of errors - See historical trends of errors - “ E “ error - “ W “ warning Best trouble-shooting tip I learned last year, notice the spaces before and after the E Will still need to review the log – but this can help rise to “10,000 feet” and see the context.
  32. 32. 8th Sutol Conference, November 2016 Making sense of the SystemOut log – stack trace Tools: A decent text editor – notepad++, editpad etc. for windows, gedit for linux etc. You may need to access multiple logs in a clustered environment Find the most recent error and work backwards Make sure you are looking at what you think you are looking at! (Some logs have months of data, others have minutes) Systemout is not always at the beginning of a reboot, check to see "application started“ in systemout. From Notepad++ you can “find all in current document” and see all applications started in a systemout.log:
  33. 33. 8th Sutol Conference, November 2016 WebSphere Tracing There is little information about reading stack traces from a support perspective and unfortunately healthy servers may also have stack traces, so it can be hard to interpret which ones are serious and which are not. The first line will tell you key information about the error The lines in-between are all pointing back to what called them The last time is the ultimate source Look for the “caused by” line if available
  34. 34. 8th Sutol Conference, November 2016 WebSphere Tracing Log on to the WebSphere Application Server (WAS) integrated solutions console using an administrator ID Go to -- Troubleshooting -> Logs and Trace -> Logging and Tracing -> and the server you wish to add tracing to Diagnostic Trace Server -> Change Log Detail Levels -> Add the following string and click Apply -> OK *=info:com.ibm.lconn.moderation.*=all: (sample trace) This is dynamic It will not let you save a bad trace In a multi-node system, collect the logs from one node at a time • i.e. only have one node running to test / collect logs • Makes trouble-shooting much easier • IBM will normally request this if you have a PMR open
  35. 35. 8th Sutol Conference, November 2016 Fiddler Tracing IBM support will almost always ask for a fiddler trace - http://www.fiddler2.com/fiddler2/ Ensure SSL traffic is decrypted : Tools - Fiddler Options - HTTPS tab - then select both "Capture HTTPS CONNECTS" and "Decrypt HTTPS traffic” and accept the cert warning Clearing the trace is easy – just use Ctrl + x Shut down other apps/browsers that add extraneous data to the trace Note: that Fiddler time is in your local browser’s time – Make a note of local time tests are run and server time Save the trace file (.saz) – send these to IBM with the systemout.logs along with time stamps and trace info.
  36. 36. 8th Sutol Conference, November 2016 Fiddler Tracing
  37. 37. 8th Sutol Conference, November 2016 Business as usual – Useful Stuff
  38. 38. 8th Sutol Conference, November 2016 Useful Stuff – WAS versionInfo From the WAS_HOMEbin (i.e /opt/IBM/WebSphere/AppServer/bin) run the versioninfo command versionInfo.bat /sh – this command will list the top level WAS version and major fix/feature pack versionInfo.bat /sh –maintenancePackages – this command will list WAS version, fix/feature pack and any iFixes installed
  39. 39. 8th Sutol Conference, November 2016 Useful Stuff – Connections Version Info From the Connections install_homeupdateinstaller directory run the following commands updateSilet.bat/sh –fix –installDir <connections_home> i.e. ./updateSilent.bat/sh –fix –installDir /opt/IBM/Connections This command will list the installed version of connections and any fixes applied
  40. 40. 8th Sutol Conference, November 2016 Useful Stuff - Change WAS log size Done via the community scripts (see http://scripting101.org) or via the ISC Set for EACH server including nodeagent and dmgr Troubleshooting > Logs and trace > JVM Logs Default it 1mb – useless Change to at least 10 and keep 20 historical Don’t forget to archive them !! 
  41. 41. 8th Sutol Conference, November 2016 Useful Stuff - Check SIB Queue Depth Use the following path in ISC to see if there is a backlog in the messaging bus: (keep this handy!) Service Integration > Buses > ConnectionsBus >Destinations > connections.events > Publication points >connections.events@CL.000-ConnectionsBus [Runtime] > Subscriptions If any of the queues have a backlog it is possible to refresh them. If they backlog up that indicates a more serious issue. A server restart can sometimes kick the queue into life – check SystemOut for log messages relating to the SIBs
  42. 42. 8th Sutol Conference, November 2016 Useful Stuff - WAS Ports / Apps & Context Roots Where is my app running? Check app and manage modules which cluster app is deployed to Check ports via
  43. 43. 8th Sutol Conference, November 2016 Useful Stuff - Clear WAS Temp Shut down all WAS processes : Connections server(s), the DMGR, and the nodeagent. Clear the temp/cache folders: IBMWebSphereAppServerprofilesAppSrv01temp IBMWebSphereAppServerprofilesAppSrv01wstemp IBMWebSphereAppServerprofilesDmgr01temp IBMWebSphereAppServerprofilesDmgr01wstemp Stop and start the HTTP server – this kills off any stale web threads / sessions Restart DMGR ,node agent and Connections servers Clear the cache on the browser before you test (EVERYTHING) Must be done every time a fix is applied to clear issues with UI – also good housekeeping
  44. 44. Useful Stuff - mtail Mtail http://ophilipp.free.fr/op_tail.htm When you run Connections a a Windows service, it will tell you it is started but in fact this is not true. It takes a while between the service appears to be running and Connections is really started. Use Mtail to open the Connections_server SystemOut.log, when you see “server Connections_server open for e-business” then Connections is running. Tip – in the Mtail window filter for the word “open” Also baretail is a free GUI tool for tailing on windows
  45. 45. 8th Sutol Conference, November 2016 http://rob59blog.wordpress.com robertoboccadoro @robboc59 Any Questions? http://socialshazza.com dilftechnical @socialshazza

×