Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Social Connections 13 - Troubleshooting Connections Pink

699 views

Published on

Curious on how to make your Connections PINK environment run smoothly while reducing support effort? Need help debugging and getting to the core of some Connections challenges? Join Nico to find out how to resolve common issues, learn troubleshooting basics and other useful knowledge to ensure an efficient Connections PINK on-premises environment. Level up your debugging skills while learning more about back-end topics such as IBM Cloud private, Kubernetes, Docker as well as Orient Me, Metrics and Connections Customizer. Walk away with Connections PINK best practice tips and tricks to help you provide steady and efficient social capabilities!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Social Connections 13 - Troubleshooting Connections Pink

  1. 1. Philadelphia, April 26-27 2018 13 Troubleshooting Connections PINK Nico Meisenzahl, panagenda @nmeisenzahl
  2. 2. PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR
  3. 3. • Consultant • “panagendian” since 2016 • Located in Germany • IBM Connections since 2010 – Deployment & consulting – Optimization & migration – Domino/Notes background • IBM Champion • Social Connections team member Nico Meisenzahl 3  @nmeisenzahl  linkedin.com/in/nicomeisenzahl  meisenzahl.org  nico.meisenzahl  +49 170 7355081  nico.meisenzahl@panagenda.com
  4. 4. I. Troubleshooting 101 II. Troubleshooting… – Client Request – Applications – Backend – Data migrations Agenda
  5. 5. Make Your Data Work For You Troubleshooting 101
  6. 6. Be aware of the big picture 1. Get an overview 2. Define the involved components & services 3. Start debugging on a high level 4. Track down the root cause 6
  7. 7. Track down the root cause • Reproducible and/or periodically? – Scheduler? • Sequence error? – When I do this, that occurs… • Client-side issue? – Browser, Proxy, Location • Or server-side issue? – Different behavior on different Nodes – Analyze involved components & services • Last changes? – Configuration, Frontend, Backend – OS, Hardware, Network, Firewall 7
  8. 8. Get support • Knowledge Center https://goo.gl/up6cxG – Troubleshooting Section https://goo.gl/IaVinx • IBM Connections Forum http://goo.gl/CVvQCU • IBM Cloud private Slack channel https://slack-invite-ibm-cloud- tech.mybluemix.net/ • Community Blogs and/or Chats (they have a new home!) • Fix Central • Support Case (PMR) – include logs – /opt/deployCfC/collectLogs.sh 8
  9. 9. Useful tools • Atom.io, Notepad++, Baretail or tailf • CLIs (kubectl, bx pr, helm) • kubetail (https://goo.gl/M3mrqh) • Firebug, Developer-Tools • Intercepting Proxies (Burp Suite, Fiddler) • IBM Datastudio, Dbeaver • Apache Directory Studio, ldapsearch • Wireshark, Tcpdump • ELK Stack for log management 9
  10. 10. Make Your Data Work For You Troubleshooting Client Requests
  11. 11. Client Request – as we know it 11
  12. 12. Client Request – PINK is joining 12
  13. 13. Client-side issues • Test with different Browsers & versions (Chrome, IE, FF) – Policies, Settings? – IE VMs are helpful • https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ – Do not use IE on Servers • Related only to some locations (Proxies) or languages? • Use Developer Tools (Browser Console, Network Tab) 13
  14. 14. NGINX Proxy • Part of the PINK deployment • Forwards all request to Connections Customizer • Get logs – /var/log/nginx/access.log – /var/log/nginx/error.log • Enable debugging – Customize /etc/nginx/nginx.conf – Global debugging (1) – Based on IP address (2) – systemctl reload nginx 14
  15. 15. Connections Customizer • Containerized Node.js microservice – mw-proxy • Injects customizations and forwards requests to IHS • Get logs – kubectl get pods -n connections |grep -i mw-proxy – kubectl logs -n connections mw-proxy-* • Debugging – Enabled by default – Downsize replicas or use kubetail • kubectl patch deploy -n connections mw-proxy -p '{"spec":{"replicas":1}}' 15
  16. 16. Make Your Data Work For You Troubleshooting Applications
  17. 17. Orient Me • Based on four frontend microservices – orient-web-client (1) – middleware-graphql (1) – itm-services (2) – community-suggestions (3) – and many more backend microservices • Get logs – kubectl get pods -n connections – kubectl logs -n connections * 17
  18. 18. Orient Me 18
  19. 19. PINK Sanity checks • New with 6.0.0.5 • Get port from services – kubectl get services -n connections |grep -i sanity 19
  20. 20. Metrics UI • UI and Event capturing still on WebSphere • Backend based on Elasticsearch – Cognos is not needed anymore • Migration path from Metrics DB to Elasticsearch • Enable tracing (WebSphere) – com.ibm.connections.metrics.* • UI • Event tracker • Elasticsearch 20
  21. 21. Make Your Data Work For You Troubleshooting Backend
  22. 22. IBM Cloud private • “Toolset” including Kubernetes, Registry, ELK Stack, Monitoring & Management UI and many more • Get logs – kubectl logs –n kube-system <pod> • ELK Stack for kube-system namespace (ICp 2.1+) – docker logs • kubelet, calico – journalctl –u docker.service • Enable debugging – docker daemon –debug – Install Calico CLI (calicoctl) to debug Cluster network issues • Part of the installer (6.0.0.5) 22
  23. 23. kubectl • Deploy it locally! – User menu – Configure Client – bx pr cluster-config • kubectl Cheat Sheet – https://goo.gl/pQ5ENv • Change the default namespace to skip -n connections – kubectl config set-context $(kubectl config current-context) --namespace=connections • kubectl logs – or kubetail 23
  24. 24. kubectl • kubectl logs –p – Print logs of the previous instance of the container • kubectl describe pod – Check Event section for pod creation issues 24
  25. 25. App Registry • Based on two Node.js microservices – appregistry-client – appregistry-service • Dependencies to Redis and MongoDB • Get logs – kubectl logs –n connections <pod> • Enable debugging – kubectl patch deploy <pod> -n connections -p '{"spec":{"template":{"spec":{"containers":[{"env":[{"name": "LOG_LEVEL","value":"debug"}],"name":”<pod>"}]}}}}’ 25
  26. 26. Metrics backend • Based on three Elasticsearch microservices (with 3 nodes each) – es-client – es-master – es-data • Authentication is based on a client certificate (Search Guard plugin) – Event capturing still on WebSphere! 26
  27. 27. Elasticseach debugging • kubectl exec -n connections -it <es-client-pod> -- curl --insecure -E /opt/elasticsearch-5.5.1/config/certs/elasticsearch-http.crt.pem --key /opt/elasticsearch-5.5.1/config/certs/elasticsearch-http.key -XPUT -d '{"transient" : {"logger._root" : "DEBUG"}}' https://<service- ip>:9200/_cluster/settings • Customize logger context if needed • Will ask for your private key password 27
  28. 28. MongoDB • Cluster based on three nodes • Get logs – kubectl logs –n connections mongo-X –c mongo|mongo-sidecar • Get Cluster information – rs.status().members • Enable debugging – db.adminCommand({setParameter: 1, logComponentVerbosity: {verbosity: 1,query: {verbosity: 2}}}) – db.getLogComponents() – Customize command as needed 28
  29. 29. Access MongoDB database • Authentication based on x509 certificates (one for each service) • kubectl exec -n connections -it mongo-0 -- mongo --ssl --sslPEMKeyFile /etc/mongodb/x509/user_admin.pem --sslCAFile /etc/mongodb/x509/mongo-CA-cert.crt -- authenticationMechanism=MONGODB-X509 --authenticationDatabase '$external' -u 'C=IE,ST=Ireland,L=Dublin,O=IBM,OU=Connections-Middleware- Clients,CN=admin,emailAddress=admin@mongodb' --host mongo- 0.mongo.connections.svc.cluster.local -eval ‘command‘ 29
  30. 30. Solr • Cluster based on three nodes • Get logs – kubectl exec -it -n connections solr-0 -- cat /home/solr/data/server/logs/solr.log – kubectl logs does not display runtime logs • Access Solr – kubectl exec -it solr-0 -n connections -- curl --insecure -E /home/solr/solr- 6.3.0/certs/cert.pem --key /home/solr/solr-6.3.0/certs/key.pem <url> • Get Cluster information – https://localhost:8984/solr/admin/collections?action=clusterstatus&wt=json • Enable debugging – https://localhost:8984/solr/admin/info/logging --data-binary 'set=root:FINEST&wt=json' 30
  31. 31. Redis • Cluster based on three nodes & Redis Sentinel • BLUE is forwarding events – Community creation, new user profile, … – /connections/config/highway.main.settings.tiles • c2.export.redis.host|port|pass • Subscribe Events – kubectl exec -it -n connections redis-server-0 -- redis-cli -a <password> subscribe connections.events • Use telnet to validate the connection 31
  32. 32. PINK authorization 1. User accesses Connections and will be redirected to Orient Me 2. User will be redirected to BLUE (/homepage/login) for authentication 3. User authenticates with BLUE (LDAP, SSO) and gets a LtpaToken and JSESSIONID 4. BLUE requests a PINK token (/social/auth/token) and creates a JWS token cookie afterwards 5. BLUE redirects the request back to PINK (/social) 6. PINK checks for the LtpaToken and JSESSIONID (if not present 7. à Step 2) 8. PINK authorizes the User after a last check against the Profiles API 32
  33. 33. APIs – use them, they might help you 33
  34. 34. Make Your Data Work For You Troubleshootingata migrations
  35. 35. Orient Me – Profile Migration • Microservice based on Node.js (people-migrate) – Migrates Profiles, Report Chain, Network and other information • Global configuration file /usr/src/app/migrationConfig • Talks to Profiles & Communities – Use curl to try to access – Authentication is working? • Talks to MongoDB – Up and running? • Logs (/usr/src/app/logs) – failed_users.txt – migration.log – report.html • Generate report – npm run start report mailaddress 35
  36. 36. Metrics – Event Migration • Events migration (Metrics DB to Elasticsearch) is done by wsadmin – execfile('metricsEventCapture.py’) • Get logs – <was_profile>/logs/<server>/MetricsMigration_*.log • Enable debugging – com.ibm.connections.metrics.migrate.* • Try to connect from WebSphere host – openssl pkcs12 -in elasticsearch-metrics.p12 -out cert.pem --nokeys – openssl pkcs12 -in elasticsearch-metrics.p12 -out keys.pem -nocerts --nodes – curl --insecure -E cert.pem --key keys.pem https://<service_host>:<service_port>/_cat/indices?v 36
  37. 37. Make Your Data Work For You Q&A
  38. 38. Thank you! Slides will be available soon: https://meisenzahl.org Q&A 38  @nmeisenzahl  linkedin.com/in/nicomeisenzahl  meisenzahl.org  nico.meisenzahl  +49 170 7355081  nico.meisenzahl@panagenda.com
  39. 39. Headquarters, Austria: panagenda GmbH (Ltd.) Schreyvogelgasse 3/10 AT 1010 Vienna Phone: +43 1 89 012 89 Fax: +43 1 89 012 89-15 E-Mail: info@panagenda.com Headquarters, Germany: panagenda GmbH (Ltd.) Lahnstraße 17 DE 64646 Heppenheim Phone: +49 6252 67 939-00 Fax: +49 6252 67 939-16 E-Mail: info@panagenda.com USA: panagenda Inc. 60 State Street, Suite 700 MA 02109 Boston Phone: +1 617 855 5961 Fax: +1 617 488 2292 E-Mail: info@panagenda.com Germany: panagenda Consulting GmbH (Ltd.) Donnersbergstrasse 1 DE 64646 Heppenheim Phone: +49 6252 67 939-86 Fax: +49 6252 67 939-16 E-Mail: info@panagenda.com The Netherlands: Trust Factory B.V. 11th Floor, Koningin Julianaplein 10 NL 2595 AA The Hague Phone: +31 70 80 801 96 E-Mail: info@trust-factory.com © 2007-2015 panagenda Make Your Data Work For You
  40. 40. PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR
  41. 41. Make Your Data Work For You Appendix
  42. 42. IBM HTTP Server & WAS Plugin • Get logs – /opt/IBM/HTTPServer/logs/access_log – /opt/IBM/HTTPServer/logs/error_log – /opt/IBM/WepSphere/Plugin/logs/<webserver>/http_plugin.log • Enable debugging – Customize httpd.conf (LogLevel) – Customize /opt/IBM/WebSphere/Plugins/config/<webserver>/plugin-cfg.xml – apachectl graceful • Linux only 43
  43. 43. WebSphere based Connections Apps • SystemOut.log • SystemErr.log • trace.log • Analyze them – Thread ID (1) – Event Type (2) – Message identifier (3) 44
  44. 44. Message identifier & Trace Stack • CLFRW1124I • CLFRW = Application prefix • 1124 = 4-digit code • I = Message level • List of all Application prefix: https://goo.gl/cmLPNE • Search for “Caused by” 45
  45. 45. Enable tracing • Check “Must gather” Technote – https://goo.gl/vf1aNs • Define tracing based on – Application prefix – Error stack • Enable tracing through ISC 46
  46. 46. BLUE authentication • Authentication & LDAP – com.ibm.ws.security.*=all com.ibm.websphere.security.*=all com.ibm.websphere.wim.*=all com.ibm.wsspi.wim.*=all com.ibm.ws.wim.*=all com.ibm.connections.directory.services.*=all • LTPA – com.ibm.ws.security.ltpa.*=all • Kerberos – com.ibm.ws.security.spnego.*=all com.ibm.issw.spnegoTAI.*=all com.ibm.security.krb5.*=all com.ibm.connections.httpClient.*=all 47
  47. 47. CCM & FileNet Logs • <wasprofile>/<servername>/p8_server_error.log • <wasprofile>/<servername>/p8_server_trace.log • <wasprofile>/<servername>/pesvr_system.log • <wasprofile>/<servername>/pesvr_trace.log • Health checks – https://<fqdn>/P8CE/Health 48
  48. 48. CCM & FileNet Debugging • ACCE Webinterface • Using log4j.xml – Define tracing in <filenetroot>/config/sample/log4j.xml – Customize JVM properties • -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true 49
  49. 49. Docs & Viewer • https://<fqdn>/vsanity/check • https://<fqdn>/sanity/check?app=all&querytype=report • https://<fqdn>/*/version 50
  50. 50. TDI & TDISOL • Check Error code prefix – CLFRN: Connections related – CTGDIS: TDI • TDI debugging – <tdisol>/etc/log4j.properties • Connections related debugging – <tdisol>/profiles_tdi.properties • source_ldap_debug=true • debug_*=true • trace_profiles_tdi_javascript=debug|fine|finer|all 51
  51. 51. DB2 • Get logs – <instance_root>/sqllib/db2dump/ – db2diag command • Enable debugging – db2 update dbm cfg using DIAGLEVEL 4 • Default is 3 • No restart required 52

×