Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing the Jahia Log Analyzer

In this presentation I introduce the Jahia Log Analyzer tool, that can perform advanced parsing of logs (Jahia specific or not) and then generate output to CSV, JSON or ElasticSearch for use with Kibana

  • Login to see the comments

Introducing the Jahia Log Analyzer

  1. 1. #jahiaXDC 1 Digital CSI : The Jahia Log Analyzer Serge Huber Jahia CTO & Co-Founder shuber@jahia.com
  2. 2. Title of Presentation I Date of Presentation 2 Introduction
  3. 3. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 3 Have you ever… Needed to analyze (very quickly) large amount of log file data ? Been trying to make some sense of large amount of thread dumps ? Been trying to understand exception frequency and seriousness ? Needed to understand performance issues without impacting a live production system ? Wanted to generate reports in multiple formats such as CSV, JSON or push into ElasticSearch ? Wanted to customize/extend the log analysis tool in Java ? Want a solution that won’t require sending your logs to an external company in the cloud ?
  4. 4. Title of Presentation I Date of Presentation 4 How it works
  5. 5. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 5 How it works Log Files CSV JSON ElasticSearch … Jahia Log Analyzer Line Analyzers Exceptions Garbage Collection Log line filter … Progressive Stack Trace Analysis GeoIP
  6. 6. Title of Presentation I Date of Presentation 6 Features
  7. 7. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 7 Features Input processing - Can process log files or thread dump files - No impact on production servers ! - “Live” or post-mortem analysis - Single node log processing - Cluster nodes log processing - Pattern-based log line matching - Progressive analysis (BETA) - Can easily handle large files (over 800MB), they are not fully loaded in memory
  8. 8. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 8 Features Line Analyzers • Generic • Exceptions • Log lines with level filter (>= WARN) • Thread dumps (BETA) • Garbage collection activity • Jahia-specific • Jahia DX server-side page processing time with GeoIP resolution • Jahia DX Request Load Average • Jackrabbit Bundle Cache logs
  9. 9. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 9 Features Output Writers: - CSV - JSON - ElasticSearch (embedded or remote) Stack trace analysis: (beta) - Pattern-based stack element matching - Tagging based on recognized patterns Progressive Analysis (beta) User interfaces: - Swing UI - Command line UI Kibana predefined dashboard and visualizations
  10. 10. Title of Presentation I Date of Presentation 10 User Interfaces
  11. 11. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 11 Swing UI
  12. 12. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 12 Command line interface java -DremoteServers=localhost:9300 -jar loganalyzer/target/loganalyzer-0.5-SNAPSHOT.jar cluster-logs [INFO] LogParserConfiguration - Using default parser-config.xml file for configuration... [INFO] JahiaLogAnalyzer - Analyzing logs in cluster-logs... [INFO] ElasticSearchService - Connecting to remote ElasticSearch at addresses: [localhost:9300]... avr. 19, 2016 3:23:11 PM org.elasticsearch.plugins.PluginsService <init> INFO: [Katherine Reynolds] modules [], plugins [], sites [] [INFO] JahiaLogAnalyzer - File cluster-logs/.DS_Store already parsed previously, skipping [INFO] JahiaLogAnalyzer - File cluster-logs/jvm-app1/.DS_Store already parsed previously, skipping [INFO] JahiaLogAnalyzer - Parsing file cluster-logs/jvm-app1/catalina.out-20160301_app1... [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 132517=tcher.include(ApplicationDispatcher.java:523) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 614108=lect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 631819=ava:206) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 691268=a:206) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 754226=ilter.TemplateNodeFilter.prepare(TemplateNodeFilter.java:142)
  13. 13. Title of Presentation I Date of Presentation 13 Live demo
  14. 14. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 14 Thread Dump Line Analyzer Processes Thread Dumps and generates - A detail entry for each thread in the dump containing - Timestamp - Thread ID - Thread State - Stack trace, length, dashcode and stack analysis tags (BETA) - Locks information - A global entry for each thread dump containing - Timestamp - Number of threads in dump - Potential “Stuck” threads (in same state since last dump) (BETA) - New/dead threads since last dump
  15. 15. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 15 Stack Trace Analysis { "id": "jackrabbit-bundle-load-from-database", "level": "NORMAL", "threadState": "RUNNING", "ignore": false, "linesToMatch": [ "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper.execute(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper.reallyExec(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper$3.call(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper$3.call(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper$RetryManager.doTry(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.util.db.ConnectionHelper.exec(ConnectionHelper.java:Ed+Q)E", "Qat org.apache.jackrabbit.core.persistence.pool.BundleDbPersistenceManager.loadBundle(BundleDbPersistenceManager.java:Ed+ Q)E" ] }
  16. 16. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 16 Stack Trace Analysis, continued
  17. 17. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 17 Kibana integration • All data is timestamped • Common fields • Predefined dashboards • Predefined visualizations • Cluster aware
  18. 18. Title of Presentation I Date of Presentation 18 Roadmap
  19. 19. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 19 Project Status Ready for usage Lots of improvements possible (modules, new line analyzers) Some features still under development: - Thread dump analyzer - Stack trace analysis - Progressive analysis Configuration is not yet as flexible as desired, needs improvements (configurable patterns for example)
  20. 20. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 20 Roadmap Develop proper plugin system (OSGi?) Make it possible to quickly define new filters using regexps (no code) Improve configuration system Make it possible to use scripts (Groovy, Javascript) to develop line analyzers Migrate to Apache Karaf ?
  21. 21. Title of Presentation I Date of Presentation 21 Q & A
  22. 22. Title of Presentation I Date of Presentation 22 Thank you !
  23. 23. Digital CSI : The Jahia Log Analyzer I #jahiaXDC 23 Resources We need your help ! Don’t hesitate to get involved ! Get/fork it now here : https://github.com/Jahia/jahia-loganalyzer Reach me at : shuber@jahia.com, @sergehuber

×