Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Soccnx11 Two wrongs don't make a right - Troubleshooting Connections

1,059 views

Published on

Curious on how to make your Connections environment run smoothly while reducing support effort? Need help debugging and getting to the core of some Connections challenges? Join Nico and Terri to find out how to resolve common issues, learn troubleshooting basics and other useful knowledge to ensure an efficient Connections on-premises environment. Level up your debugging skills while learning more about backend topics such as IBM Spectrum CfC, DB2, TDI, SSO, Directory and integrations like Docs, CCM, Cognos, FEB or Orient Me. Walk away with Connections best practice tips and tricks to help you provide steady and efficient social capabilities!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Soccnx11 Two wrongs don't make a right - Troubleshooting Connections

  1. 1. Social Connections 11 Chicago, June 1-2 2017 Two wrongs don’t make a right – Troubleshooting Connections Terri Warren, panagenda @TerriLLBean Nico Meisenzahl, panagenda @nmeisenzahl
  2. 2. PLATINUM SPONSORS GOLD SPONSORS SILVER SPONSORS
  3. 3. Social Connections 11 Chicago, June 1-2 2017 Nico Meisenzahl • Consultant at panagenda • IBM Connections since version 3.0 / 2010 • IBM Notes / Domino since 2008 • Focusing in ICS • Deployment & consulting • Optimization and migration • “panagendian” since 2016 • IBM Champion @nmeisenzahl linkedin.com/in/nicomeisenzahl meisenzahl.org nico.meisenzahl +49 170 7355081 nico.meisenzahl@panagenda.com
  4. 4. Social Connections 11 Chicago, June 1-2 2017 Terri Warren @TerriLLBean linkedin.com/in/terrillbean live:TerriLLBean terri.warren@panagenda.com • Consultant at panagenda (since 2016) • In my recent past I was a developer on IBM Connections, IBM Domino (Directory/Security) and an SRE on Watson • Frequent presenter at IBM conferences focusing in ICS • Deployment & consulting • Optimization and migration
  5. 5. Social Connections 11 Chicago, June 1-2 2017 Agenda • Howto: Troubleshooting • Troubleshooting… • Connections itself • Backend like Spectrum CfC, DB2, TDI, WebSphere, Directory, SSO • Optional add-ons like Orient Me, Docs, FEB, Cognos, CCM • Tools
  6. 6. Social Connections 11 Chicago, June 1-2 2017 Howto: Troubleshooting
  7. 7. Social Connections 11 Chicago, June 1-2 2017 Reproduce the error • Reproducible and/or periodically? • A sequence error? • Client-side or server-side problem? • Analyze the root cause
  8. 8. Social Connections 11 Chicago, June 1-2 2017 Be aware of the big picture • Client-side problems • Debug in different Browsers (IE, FF, Chrome) • Do not use a server IE • Server-side: IBM Connections is based on many components • debug on “high level” first • get an overview which backend service is causing the error
  9. 9. Social Connections 11 Chicago, June 1-2 2017 Configuration changes • Changes in… • Connections configuration • Backend (WebSphere, Database, HTTP, CfC) • Firewall or network • OS, hardware or VM Tip: Even the smallest configuration change can have big consequences!
  10. 10. Social Connections 11 Chicago, June 1-2 2017 Analyze log files and browser • Analyse log files • Atom.io, Notepad++ or less/tail • Baretail or tail –f • kubectl logs • ELK stack • Tools for client-side problems • Firebug or Developer-Tools • BurpSuite or Fiddler
  11. 11. Social Connections 11 Chicago, June 1-2 2017 Analyze root cause • Find a hint inside the log • Network timeout or DNS • SQL errors • LDAP errors • Syntax errors in configuration files „xxx-config.xml“ • Error stack Tip: In a clustered environment, start and analyze only one Node (if possible)
  12. 12. Social Connections 11 Chicago, June 1-2 2017 Find support • Knowledge Center https://goo.gl/up6cxG Troubleshooting Tips https://goo.gl/IaVinx • IBM Connections Forum http://goo.gl/CVvQCU • Community Blogs and/or Chats • Fix Central • PMR
  13. 13. Social Connections 11 Chicago, June 1-2 2017 Troubleshooting
  14. 14. Social Connections 11 Chicago, June 1-2 2017 WebSphere Application Server logs • SystemOut.log • SystemErr.log • trace.log (if tracing is enabled) • Log path: • <wasroot>/profiles/<profilename>/logs/<servername>/
  15. 15. Social Connections 11 Chicago, June 1-2 2017 Analyze WAS log files • Time stap: 24h time stamp with milli-seconds • Thread id: eight character hexadecimal value • Short name: typically java class name • Event type: one character only (E, W, I,…) • Message identifier: String based on component • Message: Some information
  16. 16. Social Connections 11 Chicago, June 1-2 2017 WAS Event types • F - Fatal message • E - Error message • W - Warning message • A - Audit message • I - Informational message • C - Configuration message • D - Detail message • O - Messages that are written directly to System.out by an application • R - Messages that are written directly to System.err by an application • Z - Place holder to indicate type was not recognized
  17. 17. Social Connections 11 Chicago, June 1-2 2017 WAS Message identifier • Prefix by Application or Server (CLFRW) • Specific application code (0042) • Event Type (I)
  18. 18. Social Connections 11 Chicago, June 1-2 2017 Read trace stack • First line displays key information • “Caused by” displays root cause
  19. 19. Social Connections 11 Chicago, June 1-2 2017 Environment information • First log lines on server startup • WebSphere version • OS version, Process id • Installation path • Java version
  20. 20. Social Connections 11 Chicago, June 1-2 2017 Enable tracing • Enable tracing using ISC • Runtime or configuration only • Define tracing based on • App prefix / error stack • Must gather (PMR)
  21. 21. Social Connections 11 Chicago, June 1-2 2017 Search issues • http(s)://<fqdn>/search/serverStatus • Display index, seedlists, log information • Data is displayed for one node only • Using node fqdn to access different nodes Tip: This will create persistent files on your hard disk!
  22. 22. Social Connections 11 Chicago, June 1-2 2017 Debug Search • Search queries (runtime) • com.ibm.connections.search.index.searching.*=all • Crawling & seedlists • com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all
  23. 23. Social Connections 11 Chicago, June 1-2 2017 Recreate Search Index • SearchService.startBackgroundIndex() • Crawls seedlists • Extracts the file content • Create index Tip: Use „all_configured“ to index all apps
  24. 24. Social Connections 11 Chicago, June 1-2 2017 CLFRW0394E: Search indexing of services ... • Search index not ready • interruption at index creation • CLFRW0283E: Search has encountered a problem while crawling • CLFRW0027E: Error Indexing component <app> for search • INDEX.READY file not present • Recreate and enable tracing
  25. 25. Social Connections 11 Chicago, June 1-2 2017 Database connections • Check datasouces • ISC – Resources – JDBC – Data sources • Check logs for more information • DB2 server log • <instanceroot>/sqllib/db2dump/ • db2diag.log • db2diag.xxx.log (log rotation, you should enable this!) Tip: Oracle users have password expiration enabled by default!
  26. 26. Social Connections 11 Chicago, June 1-2 2017 HTTP Server (IHS & Plugins) logs • IBM HTTP Server • <installroot>/logs/ • error_log • access_log • based on configuration • WebSphere AppServer Plugins • <installroot>/logs/<webserver>/http_plugin.log
  27. 27. Social Connections 11 Chicago, June 1-2 2017 HTTP 404 not found • Outdated Plugin configuration • Restart IHS • WAS Plugin configuration issue • http_plugin.log
  28. 28. Social Connections 11 Chicago, June 1-2 2017 HTTP 404 not found • AppServer or App down • Network issue • http_plugin.log
  29. 29. Social Connections 11 Chicago, June 1-2 2017 HTTP 500 Internal Server Error • Unexpected error • http_plugin.log • Configuration issue • WAS Root certificate not trusted or missing • SSL certificate expired
  30. 30. Social Connections 11 Chicago, June 1-2 2017 Debug wsadmin • Enable trace within wsadmin session • AdminControl.trace(‘com.ibm.*=all’) • <wasroot>/profiles/<profilename>/logs/wsadmin.traceout
  31. 31. Social Connections 11 Chicago, June 1-2 2017 Directory/Troubleshooting/ Tuning • Waltz and Sonata overview • Enabling Log parameters… • Cache Tuning • SSO Config Glitches • CCM Troubleshooting
  32. 32. Social Connections 11 Chicago, June 1-2 2017 Connections Troubleshooting • Waltz: "common directory services" • Directory Service eXtension (DSX) REST API. • Sonata: common HttpClient services through a RESTful service • configurable authenticators that support • SSO security tokens and cookies for secure server-to-server communication.
  33. 33. Social Connections 11 Chicago, June 1-2 2017 Troubleshooting: DSOutOfServiceExceptions • Symptoms: • Access Connections not possible • DSX not working
  34. 34. Social Connections 11 Chicago, June 1-2 2017 Troubleshooting: Enable Logging • Track all basic Waltz configuration settings and transactions. • com.ibm.connections.directory.services.*=all • Track all LDAP transactions in between Waltz & LDAP server(s). • WaltzLDAPUsage=all • Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries. • WaltzDNEntryCache=all • Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries. • WaltzExactIDMatchCache=all • Track all group membership (a list of groups) for a given user upon ID. • WaltzGroupMembershipCache=all • Track all members (a list of users) for a given group upon ID. • WaltzMemberExpansionCache=all TIP: Save runtime changes to make changes persistent
  35. 35. Social Connections 11 Chicago, June 1-2 2017 Troubleshooting: Enable Logging • WAS VMM/WIM: • com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all • WAS SPNEGO: • com.ibm.ws.security.spnego.*=all • Apache commons HttpClient: • org.apache.commons.httpclient.*=all • WAS security: • com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;
  36. 36. Social Connections 11 Chicago, June 1-2 2017 Troubleshooting: Enable Logging • Basic Sonata configuration settings and transactions: • com.ibm.connections.httpClient.*=all" • Track the amount of HTTP(S) transactions. • SonataHttpUsage=all • Track headers received for all HTTP(S) transactions. • SonataHttpHeader=all • Track all bodies received for all HTTP(S) transactions. • SonataHttpBody=all
  37. 37. Social Connections 11 Chicago, June 1-2 2017 EJPVJ9284E: Unable to get the groups from the directory for the user… • User was not able to access Connections anymore • WAS LDAP bind user had no read access to one of the groups the user was member
  38. 38. Social Connections 11 Chicago, June 1-2 2017 SSO Configuration Glitches: • Warning indicating SSO config is missing or incomplete enabling application security: • Ensure you filled in your domain Tip: Connections CRs sometimes resets SSO domain
  39. 39. Social Connections 11 Chicago, June 1-2 2017 SSO Configuration Glitches: • LtpaToken Cookies: • More secure- recommended to use version 2 LTPA cookies In other words- interoperability mode is left as disabled by default. Only enable interoperability mode if some app servers are still relying on LTPA version 1 and need to interoperate. Connections Security supports custom LTPA cookie names since 4.5 CR1 Sets com.ibm.websphere.security. customSSOCookieName custom properties (verify 3rd party products Portal/Domino can handle it)
  40. 40. Social Connections 11 Chicago, June 1-2 2017 Waltz Cache: Changing Group Membership/permissions remains in Waltz Cache • Issue: Administrator changes access or Group membership for a user. That user does not get immediate access. • Cache: Connection's Directory Services keeps a “list” of directory objects it has already searched for performance reasons ● Improves performance of connections services ● Reduces load on remote directory servers ● Cache is based on a timing mechanism ● Cache is "flushed and renewed" on a 12 hour time schedule
  41. 41. Social Connections 11 Chicago, June 1-2 2017 Tuning the Cache: Continued • How does this affect Group Membership? ● Group membership exists in the cache for 12 hours ● Operations such as renaming, deleting, or updating groups remain in the cache for that time • Configuration parameter for cache timing mechanism ● Enables Connections administrators to flush the cache in configurable time increments ● Set cache parameter via JVM arguments to flush them (we'll get to that!) ● Default remains 12 hours
  42. 42. Social Connections 11 Chicago, June 1-2 2017 Cache Configuration: Setting the cache flush parameters • Setting the cache parameters via JVM configured in WebSphere ● Google Setting generic JVM arguments in WebSphere Application Server • Time Values must be the same for all cache settings • Adjust with care! Flushing the cache has implications!
  43. 43. Social Connections 11 Chicago, June 1-2 2017 IBM Connections Content Manager (CCM): Troubleshooting • During library creation, retrieving or modifying content: • Calls are made between Communities, Profiles, and FileNet  Issues with Filenet / Connections Profile or Community configuration can adversely affect access to Libraries and CCM  Logging from Connections components (previous slides) and FileNet essential • SystemOut.log & trace.log – these are your friend • FileNet core logging: • <wasprofile>/<servername>/p8_server_error.log • <wasprofile>/<servername>/p8_server_trace.log • <wasprofile>/<servername>/pesvr_system.log • <wasprofile>/<servername>/pesvr_trace.log
  44. 44. Social Connections 11 Chicago, June 1-2 2017 CCM Troubleshooting: Filenet URLS • Status about FileNet Collaboration Services: • check the status page at http://<fqdn>/dm  Navigator version and configuration • Status about FileNet Content Engine • check the status page at http://<fqdn>/FileNet/Engine  Server status, version, sonata/waltz version • Status about FileNet Content Engine Domain and Object Store • check the status page at http://<fqdn>/P8CE/Health  Health checks for authentication, stores and database
  45. 45. Social Connections 11 Chicago, June 1-2 2017 Debug CCM Widget • Widget issues • com.ibm.quickr.communitylibrary.*=all: com.ibm.lconn.widgets.service.*=all: com.ibm.lconn.widgets.actions.*=all • Authentication issues (Remember those cool tips a few slides back? ) • com.ibm.connections.directory.services.*=all: com.ibm.connections.httpClient.*=all
  46. 46. Social Connections 11 Chicago, June 1-2 2017 Debug FileNet using ACCE • ACCE tool: check if FileNet Content Engine can authenticate using Connections and inspect other data for errors: http(s)://<fqdn>/acce
  47. 47. Social Connections 11 Chicago, June 1-2 2017 CCM: Enable logging for Connections and FileNet using JVM Properties • Download and copy the sample waltz.sonata.log4j. as log4j.xml: • Linux: /<absolute_path_for_log4j_file>/log4j.xml • Windows:<absolute_path_for_log4j_file>log4j.xml • Add generic JVM properties • -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true • Tells the Content Engine (CE) server to skip it’s tracing configuration and writes to waltz.sonata.trace.log Windows: c:Program FilesIBMWebSphereAppServerprofilesAppSrv01 Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
  48. 48. Social Connections 11 Chicago, June 1-2 2017 CCM: Enable logging for Connections and FileNet using JVM Properties • Copy & customize sample log4j.xml • <ContentEngineRoot>/config/samples • 20 subsystems (db, engine, security, search,…)
  49. 49. Social Connections 11 Chicago, June 1-2 2017 TDI logs • <tdisol>/log/ibmdi.log • TDI log file • <tdisol>/employee.* • Files include all changed users (adds, update, delete, error, skip) • <tdisol>/syncupdates/* • Temporary files within the sync • Including database dump and ldiff • sync_updates_clean_temp_files=false (default: true) • profiles_tdi.properties Tip: Check the lock file
  50. 50. Social Connections 11 Chicago, June 1-2 2017 Analyze TDI logs • Error code prefix • CLFRN: Profile & User synchronization • CTGDIS: TDI itself • Error code suffix • I, E, W, …
  51. 51. Social Connections 11 Chicago, June 1-2 2017 Debug TDI • Profile & User synchronization (<tdisol>/etc/profiles_tdi.properties) • source_ldap_debug=true • debug_update_profile=true • debug_collect=true • TDI issues (<tdisol>/etc/log4j.properties) • log4j.rootCategory=DEBUG, Default
  52. 52. Social Connections 11 Chicago, June 1-2 2017 Cognos BI logs • Cognos BI • SystemOut.log & trace.log • <installroot>/logs/cogserver.log • Cognos Transformer • <installroot>/logs/cogserver.log • PowerCube build • <installroot>/metricsmodel/trxschelog.log • <userhome>/Transformer/Logs/*.log (win only)
  53. 53. Social Connections 11 Chicago, June 1-2 2017 BMT-MD-6003 No connection to the data source … • PowerCubes not created yet • Check cronjobs or scheduled jobs • trxschelog.log
  54. 54. Social Connections 11 Chicago, June 1-2 2017 Debug Cognos BI & Metrics • Communication between Cognos BI and Connections Metrics • SonataHttpUsage=all: SonataHttpHeader=all: SonataHttpBody=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.metrics.*=all • Connections Metrics Servlet • com.ibm.connections.metrics.cognos.servlet.*=all
  55. 55. Social Connections 11 Chicago, June 1-2 2017 Docs/Viewer logs & urls • SystemOut.log & trace.log • http(s):<fqdn>/vsanity/check • http(s):<fqdn>/sanity/check?app=all&querytype=report • http(s):<fqdn>/*/version
  56. 56. Social Connections 11 Chicago, June 1-2 2017 Debug LTPA between Domino & WAS • Debug on Domino side (notes.ini) • Debug_SSO_Trace_Level=2 • Webauth_verbose_trace=1 • WebSess_verbose_trace=1 • Debug_outfile=<logfilepath> • Debug on WebSphere • com.ibm.ws.security.ltpa.*=all
  57. 57. Social Connections 11 Chicago, June 1-2 2017 Debug Kerberos • Configuration • com.ibm.ws.security.spnego.*=all: com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all: com.ibm.security.krb5.*=all • Runtime • com.ibm.connections.httpClient.*=all: com.ibm.connections.directory.services.*=all: com.ibm.websphere.wim.*=all: com.ibm.ws.wim.*=all • Fiddler & BurpSuite
  58. 58. Social Connections 11 Chicago, June 1-2 2017 IBM Spectrum CfC stats • https://fqdn:8443
  59. 59. Social Connections 11 Chicago, June 1-2 2017 Docker & Kubernetes status • kubectl get pods (--all-namespaces) • kubectl get services • kubectl get deployment • kubectl describe pods xxx • docker ps • docker info
  60. 60. Social Connections 11 Chicago, June 1-2 2017 Docker & Kubernetes status • Persistent Volume status • kubectl get pv,pvc
  61. 61. Social Connections 11 Chicago, June 1-2 2017 Orient Me logs • Display application logs • kubectl logs xxx • -f • --tail=20 • --since=1h • Special tasks for MongoDB and Solr
  62. 62. Social Connections 11 Chicago, June 1-2 2017 MongoDB Troubleshooting • Check persistent volumes • Access logs • kubectl logs xxx [mongo|mongo-sidecar] • Get replica status • kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval 'rs.status()’ • Check “health” status for all 3 members
  63. 63. Social Connections 11 Chicago, June 1-2 2017 Solr logs • kubectl exec -it xxx -- cat /var/solr/logs/solr.log • kubectl logs xxx • only display deploment logs
  64. 64. Social Connections 11 Chicago, June 1-2 2017 Pods startup issues • kubectl describe pod xxx
  65. 65. Social Connections 11 Chicago, June 1-2 2017 CfC logs • docker logs -f kubelet • docker logs -f calico
  66. 66. Social Connections 11 Chicago, June 1-2 2017 Docker tracing • service docker stop • docker daemon –debug • journalctl -u docker.service
  67. 67. Social Connections 11 Chicago, June 1-2 2017 What to get more insights on Orient Me?
  68. 68. Social Connections 11 Chicago, June 1-2 2017 Tools
  69. 69. Social Connections 11 Chicago, June 1-2 2017 Analyze logs • Analyze logs live • Baretail • tail –f • View logs • Atom.io, Notepad++ • less, tail • ELK Stack • Elasticsearch, Logstash, Kibana • Small Docker deployment • Kubectl (local client)
  70. 70. Social Connections 11 Chicago, June 1-2 2017 Analyze logs • Kubectl (local client)
  71. 71. Social Connections 11 Chicago, June 1-2 2017 Analyze Client-side • Browser • Firebug / Developer Tools • Intercepting proxies • Fiddler • Burp Suite • VMs with different IE versions • Without GPO • https://www.modern.ie/en-us/virtualization-tools
  72. 72. Social Connections 11 Chicago, June 1-2 2017 Database Clients • db2 command • Dbeaver or IBM Datastudio • Robomongo (MongoDB) Tip: Database write access is not supported!
  73. 73. Social Connections 11 Chicago, June 1-2 2017 LDAP & Search • LDAP • ldapsearch command • Softerra LDAP Browser • Apache Directory Studio • Search Index • Luke (Lucence Index Toolbox)
  74. 74. Social Connections 11 Chicago, June 1-2 2017 Analyze Network • Wireshark • tcpdump
  75. 75. Social Connections 11 Chicago, June 1-2 2017 Q&A
  76. 76. PLATINUM SPONSORS GOLD SPONSORS SILVER SPONSORS

×