Social Connections 13 - Troubleshooting Connections Pink
•
0 likes•1,206 views
Curious on how to make your Connections PINK environment run smoothly while reducing support effort? Need help debugging and getting to the core of some Connections challenges? Join Nico to find out how to resolve common issues, learn troubleshooting basics and other useful knowledge to ensure an efficient Connections PINK on-premises environment. Level up your debugging skills while learning more about back-end topics such as IBM Cloud private, Kubernetes, Docker as well as Orient Me, Metrics and Connections Customizer. Walk away with Connections PINK best practice tips and tricks to help you provide steady and efficient social capabilities!
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Social Connections 13 - Troubleshooting Connections Pink
Similar to Social Connections 13 - Troubleshooting Connections Pink (20)
More from Nico Meisenzahl
More from Nico Meisenzahl (20)
Recently uploaded
Recently uploaded (20)
Social Connections 13 - Troubleshooting Connections Pink
- 1. Philadelphia, April 26-27 2018 13 Troubleshooting Connections PINK Nico Meisenzahl, panagenda @nmeisenzahl
- 2. PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR
- 3. • Consultant • “panagendian” since 2016 • Located in Germany • IBM Connections since 2010 – Deployment & consulting – Optimization & migration – Domino/Notes background • IBM Champion • Social Connections team member Nico Meisenzahl 3 @nmeisenzahl linkedin.com/in/nicomeisenzahl meisenzahl.org nico.meisenzahl +49 170 7355081 nico.meisenzahl@panagenda.com
- 4. I. Troubleshooting 101 II. Troubleshooting… – Client Request – Applications – Backend – Data migrations Agenda
- 5. Make Your Data Work For You Troubleshooting 101
- 6. Be aware of the big picture 1. Get an overview 2. Define the involved components & services 3. Start debugging on a high level 4. Track down the root cause 6
- 7. Track down the root cause • Reproducible and/or periodically? – Scheduler? • Sequence error? – When I do this, that occurs… • Client-side issue? – Browser, Proxy, Location • Or server-side issue? – Different behavior on different Nodes – Analyze involved components & services • Last changes? – Configuration, Frontend, Backend – OS, Hardware, Network, Firewall 7
- 8. Get support • Knowledge Center https://goo.gl/up6cxG – Troubleshooting Section https://goo.gl/IaVinx • IBM Connections Forum http://goo.gl/CVvQCU • IBM Cloud private Slack channel https://slack-invite-ibm-cloud- tech.mybluemix.net/ • Community Blogs and/or Chats (they have a new home!) • Fix Central • Support Case (PMR) – include logs – /opt/deployCfC/collectLogs.sh 8
- 9. Useful tools • Atom.io, Notepad++, Baretail or tailf • CLIs (kubectl, bx pr, helm) • kubetail (https://goo.gl/M3mrqh) • Firebug, Developer-Tools • Intercepting Proxies (Burp Suite, Fiddler) • IBM Datastudio, Dbeaver • Apache Directory Studio, ldapsearch • Wireshark, Tcpdump • ELK Stack for log management 9
- 10. Make Your Data Work For You Troubleshooting Client Requests
- 11. Client Request – as we know it 11
- 12. Client Request – PINK is joining 12
- 13. Client-side issues • Test with different Browsers & versions (Chrome, IE, FF) – Policies, Settings? – IE VMs are helpful • https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ – Do not use IE on Servers • Related only to some locations (Proxies) or languages? • Use Developer Tools (Browser Console, Network Tab) 13
- 14. NGINX Proxy • Part of the PINK deployment • Forwards all request to Connections Customizer • Get logs – /var/log/nginx/access.log – /var/log/nginx/error.log • Enable debugging – Customize /etc/nginx/nginx.conf – Global debugging (1) – Based on IP address (2) – systemctl reload nginx 14
- 15. Connections Customizer • Containerized Node.js microservice – mw-proxy • Injects customizations and forwards requests to IHS • Get logs – kubectl get pods -n connections |grep -i mw-proxy – kubectl logs -n connections mw-proxy-* • Debugging – Enabled by default – Downsize replicas or use kubetail • kubectl patch deploy -n connections mw-proxy -p '{"spec":{"replicas":1}}' 15
- 16. Make Your Data Work For You Troubleshooting Applications
- 17. Orient Me • Based on four frontend microservices – orient-web-client (1) – middleware-graphql (1) – itm-services (2) – community-suggestions (3) – and many more backend microservices • Get logs – kubectl get pods -n connections – kubectl logs -n connections * 17
- 18. Orient Me 18
- 19. PINK Sanity checks • New with 6.0.0.5 • Get port from services – kubectl get services -n connections |grep -i sanity 19
- 20. Metrics UI • UI and Event capturing still on WebSphere • Backend based on Elasticsearch – Cognos is not needed anymore • Migration path from Metrics DB to Elasticsearch • Enable tracing (WebSphere) – com.ibm.connections.metrics.* • UI • Event tracker • Elasticsearch 20
- 21. Make Your Data Work For You Troubleshooting Backend
- 22. IBM Cloud private • “Toolset” including Kubernetes, Registry, ELK Stack, Monitoring & Management UI and many more • Get logs – kubectl logs –n kube-system <pod> • ELK Stack for kube-system namespace (ICp 2.1+) – docker logs • kubelet, calico – journalctl –u docker.service • Enable debugging – docker daemon –debug – Install Calico CLI (calicoctl) to debug Cluster network issues • Part of the installer (6.0.0.5) 22
- 23. kubectl • Deploy it locally! – User menu – Configure Client – bx pr cluster-config • kubectl Cheat Sheet – https://goo.gl/pQ5ENv • Change the default namespace to skip -n connections – kubectl config set-context $(kubectl config current-context) --namespace=connections • kubectl logs – or kubetail 23
- 24. kubectl • kubectl logs –p – Print logs of the previous instance of the container • kubectl describe pod – Check Event section for pod creation issues 24
- 25. App Registry • Based on two Node.js microservices – appregistry-client – appregistry-service • Dependencies to Redis and MongoDB • Get logs – kubectl logs –n connections <pod> • Enable debugging – kubectl patch deploy <pod> -n connections -p '{"spec":{"template":{"spec":{"containers":[{"env":[{"name": "LOG_LEVEL","value":"debug"}],"name":”<pod>"}]}}}}’ 25
- 26. Metrics backend • Based on three Elasticsearch microservices (with 3 nodes each) – es-client – es-master – es-data • Authentication is based on a client certificate (Search Guard plugin) – Event capturing still on WebSphere! 26
- 27. Elasticseach debugging • kubectl exec -n connections -it <es-client-pod> -- curl --insecure -E /opt/elasticsearch-5.5.1/config/certs/elasticsearch-http.crt.pem --key /opt/elasticsearch-5.5.1/config/certs/elasticsearch-http.key -XPUT -d '{"transient" : {"logger._root" : "DEBUG"}}' https://<service- ip>:9200/_cluster/settings • Customize logger context if needed • Will ask for your private key password 27
- 28. MongoDB • Cluster based on three nodes • Get logs – kubectl logs –n connections mongo-X –c mongo|mongo-sidecar • Get Cluster information – rs.status().members • Enable debugging – db.adminCommand({setParameter: 1, logComponentVerbosity: {verbosity: 1,query: {verbosity: 2}}}) – db.getLogComponents() – Customize command as needed 28
- 29. Access MongoDB database • Authentication based on x509 certificates (one for each service) • kubectl exec -n connections -it mongo-0 -- mongo --ssl --sslPEMKeyFile /etc/mongodb/x509/user_admin.pem --sslCAFile /etc/mongodb/x509/mongo-CA-cert.crt -- authenticationMechanism=MONGODB-X509 --authenticationDatabase '$external' -u 'C=IE,ST=Ireland,L=Dublin,O=IBM,OU=Connections-Middleware- Clients,CN=admin,emailAddress=admin@mongodb' --host mongo- 0.mongo.connections.svc.cluster.local -eval ‘command‘ 29
- 30. Solr • Cluster based on three nodes • Get logs – kubectl exec -it -n connections solr-0 -- cat /home/solr/data/server/logs/solr.log – kubectl logs does not display runtime logs • Access Solr – kubectl exec -it solr-0 -n connections -- curl --insecure -E /home/solr/solr- 6.3.0/certs/cert.pem --key /home/solr/solr-6.3.0/certs/key.pem <url> • Get Cluster information – https://localhost:8984/solr/admin/collections?action=clusterstatus&wt=json • Enable debugging – https://localhost:8984/solr/admin/info/logging --data-binary 'set=root:FINEST&wt=json' 30
- 31. Redis • Cluster based on three nodes & Redis Sentinel • BLUE is forwarding events – Community creation, new user profile, … – /connections/config/highway.main.settings.tiles • c2.export.redis.host|port|pass • Subscribe Events – kubectl exec -it -n connections redis-server-0 -- redis-cli -a <password> subscribe connections.events • Use telnet to validate the connection 31
- 32. PINK authorization 1. User accesses Connections and will be redirected to Orient Me 2. User will be redirected to BLUE (/homepage/login) for authentication 3. User authenticates with BLUE (LDAP, SSO) and gets a LtpaToken and JSESSIONID 4. BLUE requests a PINK token (/social/auth/token) and creates a JWS token cookie afterwards 5. BLUE redirects the request back to PINK (/social) 6. PINK checks for the LtpaToken and JSESSIONID (if not present 7. à Step 2) 8. PINK authorizes the User after a last check against the Profiles API 32
- 33. APIs – use them, they might help you 33
- 34. Make Your Data Work For You Troubleshootingata migrations
- 35. Orient Me – Profile Migration • Microservice based on Node.js (people-migrate) – Migrates Profiles, Report Chain, Network and other information • Global configuration file /usr/src/app/migrationConfig • Talks to Profiles & Communities – Use curl to try to access – Authentication is working? • Talks to MongoDB – Up and running? • Logs (/usr/src/app/logs) – failed_users.txt – migration.log – report.html • Generate report – npm run start report mailaddress 35
- 36. Metrics – Event Migration • Events migration (Metrics DB to Elasticsearch) is done by wsadmin – execfile('metricsEventCapture.py’) • Get logs – <was_profile>/logs/<server>/MetricsMigration_*.log • Enable debugging – com.ibm.connections.metrics.migrate.* • Try to connect from WebSphere host – openssl pkcs12 -in elasticsearch-metrics.p12 -out cert.pem --nokeys – openssl pkcs12 -in elasticsearch-metrics.p12 -out keys.pem -nocerts --nodes – curl --insecure -E cert.pem --key keys.pem https://<service_host>:<service_port>/_cat/indices?v 36
- 37. Make Your Data Work For You Q&A
- 38. Thank you! Slides will be available soon: https://meisenzahl.org Q&A 38 @nmeisenzahl linkedin.com/in/nicomeisenzahl meisenzahl.org nico.meisenzahl +49 170 7355081 nico.meisenzahl@panagenda.com
- 39. Headquarters, Austria: panagenda GmbH (Ltd.) Schreyvogelgasse 3/10 AT 1010 Vienna Phone: +43 1 89 012 89 Fax: +43 1 89 012 89-15 E-Mail: info@panagenda.com Headquarters, Germany: panagenda GmbH (Ltd.) Lahnstraße 17 DE 64646 Heppenheim Phone: +49 6252 67 939-00 Fax: +49 6252 67 939-16 E-Mail: info@panagenda.com USA: panagenda Inc. 60 State Street, Suite 700 MA 02109 Boston Phone: +1 617 855 5961 Fax: +1 617 488 2292 E-Mail: info@panagenda.com Germany: panagenda Consulting GmbH (Ltd.) Donnersbergstrasse 1 DE 64646 Heppenheim Phone: +49 6252 67 939-86 Fax: +49 6252 67 939-16 E-Mail: info@panagenda.com The Netherlands: Trust Factory B.V. 11th Floor, Koningin Julianaplein 10 NL 2595 AA The Hague Phone: +31 70 80 801 96 E-Mail: info@trust-factory.com © 2007-2015 panagenda Make Your Data Work For You
- 40. PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR
- 42. Make Your Data Work For You Appendix
- 43. IBM HTTP Server & WAS Plugin • Get logs – /opt/IBM/HTTPServer/logs/access_log – /opt/IBM/HTTPServer/logs/error_log – /opt/IBM/WepSphere/Plugin/logs/<webserver>/http_plugin.log • Enable debugging – Customize httpd.conf (LogLevel) – Customize /opt/IBM/WebSphere/Plugins/config/<webserver>/plugin-cfg.xml – apachectl graceful • Linux only 43
- 44. WebSphere based Connections Apps • SystemOut.log • SystemErr.log • trace.log • Analyze them – Thread ID (1) – Event Type (2) – Message identifier (3) 44
- 45. Message identifier & Trace Stack • CLFRW1124I • CLFRW = Application prefix • 1124 = 4-digit code • I = Message level • List of all Application prefix: https://goo.gl/cmLPNE • Search for “Caused by” 45
- 46. Enable tracing • Check “Must gather” Technote – https://goo.gl/vf1aNs • Define tracing based on – Application prefix – Error stack • Enable tracing through ISC 46
- 47. BLUE authentication • Authentication & LDAP – com.ibm.ws.security.*=all com.ibm.websphere.security.*=all com.ibm.websphere.wim.*=all com.ibm.wsspi.wim.*=all com.ibm.ws.wim.*=all com.ibm.connections.directory.services.*=all • LTPA – com.ibm.ws.security.ltpa.*=all • Kerberos – com.ibm.ws.security.spnego.*=all com.ibm.issw.spnegoTAI.*=all com.ibm.security.krb5.*=all com.ibm.connections.httpClient.*=all 47
- 48. CCM & FileNet Logs • <wasprofile>/<servername>/p8_server_error.log • <wasprofile>/<servername>/p8_server_trace.log • <wasprofile>/<servername>/pesvr_system.log • <wasprofile>/<servername>/pesvr_trace.log • Health checks – https://<fqdn>/P8CE/Health 48
- 49. CCM & FileNet Debugging • ACCE Webinterface • Using log4j.xml – Define tracing in <filenetroot>/config/sample/log4j.xml – Customize JVM properties • -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true 49
- 50. Docs & Viewer • https://<fqdn>/vsanity/check • https://<fqdn>/sanity/check?app=all&querytype=report • https://<fqdn>/*/version 50
- 51. TDI & TDISOL • Check Error code prefix – CLFRN: Connections related – CTGDIS: TDI • TDI debugging – <tdisol>/etc/log4j.properties • Connections related debugging – <tdisol>/profiles_tdi.properties • source_ldap_debug=true • debug_*=true • trace_profiles_tdi_javascript=debug|fine|finer|all 51
- 52. DB2 • Get logs – <instance_root>/sqllib/db2dump/ – db2diag command • Enable debugging – db2 update dbm cfg using DIAGLEVEL 4 • Default is 3 • No restart required 52