A Mind Map Based Framework for Automated Software Log File Analysis<br />Dileepa  Jayathilake<br />Department of Electrica...
Conclusion<br />Implementation<br />AGENDA<br />Solution Design<br />Solution Overview<br />Problem Identification<br />Ba...
Functional Conformance<br />BACKGROUND<br />Quality Verification<br />Troubleshooting<br />System Administrators<br />Doma...
BACKGROUND<br />Labor Intensive<br />Require Expertise<br />Error-prone<br />Advantage of Recurrence not used<br />PITFALL...
Different log formats & structure<br />Lack of a common platform<br />Making rules human & machine readable<br />PROBLEM I...
EXISTING SUPPORT<br />PROBLEM IDENTIFICATION<br />XML<br /><ul><li> Universal format
Ubiquitous use
Many tools available
 Costly meta data
Less human readable
Associated languages are complex
Not every log is xml</li></ul>Log File Grammars<br /><ul><li> Formal definitions
 Regular expression based
Assume line logs
Fail with complex log file structures
Unable to handle difficult syntax
Distant from XML </li></li></ul><li>Handle arbitrary formats and structures of log files<br />SOLUTION OVERVIEW<br />Resil...
Log Files<br />SOLUTION OVERVIEW<br />SOLUTION OVERVIEW<br />Interpretation<br />Processing<br />Presentation<br />Unified...
Upcoming SlideShare
Loading in …5
×

A Mind Map Based Framework for Automated Software Log File Analysis

1,307 views

Published on

Software log file analysis is involved heavily in both Software development and maintenance phases. It serves for various purposes such as verifying the conformance of the software functionality to the specification, software quality check and troubleshooting. Application log files or the logs generated by other monitoring tools are subjected to analysis for extracting information that can be vital in an investigation. These tasks demand expertise to a great deal and are labor intensive when performed manually. The lack of a commonly used technique to record expert knowledge stands as an impediment to automate the analysis tasks. The need for correlating information extracted from different locations in the same log file or multiple log files further ads to this complexity. This paper describes a framework based on mind maps which formulates a homogeneous platform for recording expert knowledge as well as for performing other tasks such as extracting information from log files, drawing inferences and creating reports. The framework includes a scripting language, a parallel application programming interface and a set of tools. Usage is illustrated by a proof of concept system built using the framework that creates a useful report after analyzing a log file generated by a widely used software monitoring tool.

My speech in ICSCA 2011 - http://dileepaj.blogspot.com/2011/07/speech-in-icsca-2011.html

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,307
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Mind Map Based Framework for Automated Software Log File Analysis

  1. 1. A Mind Map Based Framework for Automated Software Log File Analysis<br />Dileepa Jayathilake<br />Department of Electrical Engineering<br /> University of Moratuwa <br /> Sri Lanka<br />ICSCA 2011<br />
  2. 2. Conclusion<br />Implementation<br />AGENDA<br />Solution Design<br />Solution Overview<br />Problem Identification<br />Background<br />
  3. 3. Functional Conformance<br />BACKGROUND<br />Quality Verification<br />Troubleshooting<br />System Administrators<br />Domain Experts<br />Application Logs<br />Developers<br />Monitoring Tool Logs<br />Testers<br />LOG FILE ANALYSIS<br />
  4. 4. BACKGROUND<br />Labor Intensive<br />Require Expertise<br />Error-prone<br />Advantage of Recurrence not used<br />PITFALLS IN MANUAL APPROACH <br />
  5. 5. Different log formats & structure<br />Lack of a common platform<br />Making rules human & machine readable<br />PROBLEM IDENTIFICATION<br />Challenges<br />Result<br />Proprietary Implementation<br />Automation abandoned<br />Reports not customizable<br />Costly<br />Rules not human readable<br />Less resilient to format changes<br />Difficult to add new rules<br />CHALLENGES<br />
  6. 6. EXISTING SUPPORT<br />PROBLEM IDENTIFICATION<br />XML<br /><ul><li> Universal format
  7. 7. Ubiquitous use
  8. 8. Many tools available
  9. 9. Costly meta data
  10. 10. Less human readable
  11. 11. Associated languages are complex
  12. 12. Not every log is xml</li></ul>Log File Grammars<br /><ul><li> Formal definitions
  13. 13. Regular expression based
  14. 14. Assume line logs
  15. 15. Fail with complex log file structures
  16. 16. Unable to handle difficult syntax
  17. 17. Distant from XML </li></li></ul><li>Handle arbitrary formats and structures of log files<br />SOLUTION OVERVIEW<br />Resilient to log file format and structure changes<br />A knowledge representation which is both human and machine readable<br />EXPECTATIONS<br />In lined with XML<br />Friendly for non-developers<br />+<br />Ability to generate custom reports<br />A GENERIC LOG ANALYSIS FRAMEWORK<br />
  18. 18. Log Files<br />SOLUTION OVERVIEW<br />SOLUTION OVERVIEW<br />Interpretation<br />Processing<br />Presentation<br />Unified mechanism for extracting information of interest from both text and binary log files with arbitrary structure and format<br />Easy mechanism to build and maintain a rule base for inferences<br />Flexible means for generating custom reports from inferences<br />Knowledge Representation Schema<br />
  19. 19. Easy to add content<br />SOLUTION DESIGN<br />Easy to visualize<br />Resembles human knowledge organization better<br />Easy to combine<br />MIND MAPS<br />Easily convertible to XML<br />Easy access to computers<br />Tree<br />Can utilize existing tree algorithms<br />Can utilize existing tools<br />MIND MAP AS KNOWLEDGE UNIT<br />
  20. 20. SYSTEM ARCHITECTURE<br />SOLUTION DESIGN<br />
  21. 21. NEW SCRIPTING LANGUAGE<br />SOLUTION IMPLEMENTATION<br />Mind map is the basic processing unit<br />Configurable syntax<br />Advanced filtering<br />Multiple executions in a single statement<br />Supports basic and compound data types<br />Built-in and user defined functions<br />
  22. 22. $Map1.TypeIs(#ERROR)::$MY.LeftSibling.IsNotNull, Level < 2.LeftSibling->Category.Unique.Count = $ERROR_CATEGORIES_COUNT<br />SOLUTION IMPLEMENTATION<br /><ul><li> Follows the flow of a text in natural language
  23. 23. Uses statement chaining
  24. 24. No distant memory calls
  25. 25. More suitable for expressing rules
  26. 26. Independent small chunks of execution </li></ul>$Found = FALSE<br />$Map1.TypeIs(#ERROR) = $Set1<br />$Set1.Unique = $Errors<br />$Map1.TypeIs(#WARN) = $Set2<br />$Set2.Unique = $Warnings<br />Foreach $Error in $Errors<br /> $Error->Category = $Cat<br /> $Warnings::Category==$Cat = $X<br /> If ( $X.Count > 0 )<br /> $Found = TRUE<br /> Break<br />EndIf<br />EndFor<br /><ul><li> Suits Advanced Programming
  27. 27. Difficult for non-developers
  28. 28. Memory intensive </li></ul>PROGRAMMING MODELS<br />
  29. 29. Log Files<br />SOLUTION SUMMARY<br />SOLUTION IMPLEMENTATION<br />Interpretation<br />Processing<br />Presentation<br /><ul><li> Special support for splitting text and binary data
  30. 30. Support for structural data extraction
  31. 31. Rich platform to add and edit rules
  32. 32. Support for combining mind maps
  33. 33. Turing complete
  34. 34. Custom reports generated by scripts</li></ul>Mind Maps<br />
  35. 35. SOLUTION IMPLEMENTATION<br />USAGE SCENARIO<br />
  36. 36. CONCLUSION<br />The new framework<br />provides a unified platform for generic log analysis. It enables users to perform different tasks in a homogeneous fashion. In addition it formulates infrastructure for a shared rule base. <br />
  37. 37. FUTURE WORK<br />Interpretation<br />Processing<br />Presentation<br /><ul><li> Script library for common tool logs
  38. 38. Declarative language
  39. 39. Support for fuzzy rules
  40. 40. Design driven reports</li></li></ul><li>REFERENCES<br />J. Valdman. Log file analysis. Technical Report DCSE/TR-2001-04, Department of Computer Science and Engineering (FAV UWB), 2001.<br />Tony Buzan. The Mind Map Book. Penguin Books, 1996, ch. 2<br />John E. Hopcroft, Jeffery D. Ullman. Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979, pp. 13-137 <br />J. H. Andrews. Theory and practice of log file analysis. Technical Report 524, Department of Computer Science, University of Western Ontario, May 1998. <br />S. G. Eick, M. C. Nelson, J. D. Schmidt. Graphical Analysis of Computer Log Files. Communications of the ACM, Vol. 37, No. 12, pp. 50-56, 1994. <br />H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Mining for relevant terms from log files. In: KDIR’09. Proc. of International Conference on Knowledge Discovery and Information Retrieval. Madeira, Portugal. 2009. <br />H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Terminology extraction from log files. In: KDIR’09. Proc. Of 20th International Conference on Database and Expert Systems Applications. pp. 769-776. Lecture Notes in Computer Science, Springer 2009. <br />
  41. 41. QUESTIONS<br />

×