A Mind Map Based Framework for Automated  Software Log File Analysis
Upcoming SlideShare
Loading in...5
×
 

A Mind Map Based Framework for Automated Software Log File Analysis

on

  • 1,072 views

Software log file analysis is involved heavily in both Software development and maintenance phases. It serves for various purposes such as verifying the conformance of the software functionality to ...

Software log file analysis is involved heavily in both Software development and maintenance phases. It serves for various purposes such as verifying the conformance of the software functionality to the specification, software quality check and troubleshooting. Application log files or the logs generated by other monitoring tools are subjected to analysis for extracting information that can be vital in an investigation. These tasks demand expertise to a great deal and are labor intensive when performed manually. The lack of a commonly used technique to record expert knowledge stands as an impediment to automate the analysis tasks. The need for correlating information extracted from different locations in the same log file or multiple log files further ads to this complexity. This paper describes a framework based on mind maps which formulates a homogeneous platform for recording expert knowledge as well as for performing other tasks such as extracting information from log files, drawing inferences and creating reports. The framework includes a scripting language, a parallel application programming interface and a set of tools. Usage is illustrated by a proof of concept system built using the framework that creates a useful report after analyzing a log file generated by a widely used software monitoring tool.

My speech in ICSCA 2011 - http://dileepaj.blogspot.com/2011/07/speech-in-icsca-2011.html

Statistics

Views

Total Views
1,072
Views on SlideShare
1,066
Embed Views
6

Actions

Likes
0
Downloads
6
Comments
0

2 Embeds 6

http://www.linkedin.com 5
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

A Mind Map Based Framework for Automated  Software Log File Analysis A Mind Map Based Framework for Automated Software Log File Analysis Presentation Transcript

  • A Mind Map Based Framework for Automated Software Log File Analysis
    Dileepa Jayathilake
    Department of Electrical Engineering
    University of Moratuwa
    Sri Lanka
    ICSCA 2011
  • Conclusion
    Implementation
    AGENDA
    Solution Design
    Solution Overview
    Problem Identification
    Background
  • Functional Conformance
    BACKGROUND
    Quality Verification
    Troubleshooting
    System Administrators
    Domain Experts
    Application Logs
    Developers
    Monitoring Tool Logs
    Testers
    LOG FILE ANALYSIS
    View slide
  • BACKGROUND
    Labor Intensive
    Require Expertise
    Error-prone
    Advantage of Recurrence not used
    PITFALLS IN MANUAL APPROACH
    View slide
  • Different log formats & structure
    Lack of a common platform
    Making rules human & machine readable
    PROBLEM IDENTIFICATION
    Challenges
    Result
    Proprietary Implementation
    Automation abandoned
    Reports not customizable
    Costly
    Rules not human readable
    Less resilient to format changes
    Difficult to add new rules
    CHALLENGES
  • EXISTING SUPPORT
    PROBLEM IDENTIFICATION
    XML
    • Universal format
    • Ubiquitous use
    • Many tools available
    • Costly meta data
    • Less human readable
    • Associated languages are complex
    • Not every log is xml
    Log File Grammars
    • Formal definitions
    • Regular expression based
    • Assume line logs
    • Fail with complex log file structures
    • Unable to handle difficult syntax
    • Distant from XML
  • Handle arbitrary formats and structures of log files
    SOLUTION OVERVIEW
    Resilient to log file format and structure changes
    A knowledge representation which is both human and machine readable
    EXPECTATIONS
    In lined with XML
    Friendly for non-developers
    +
    Ability to generate custom reports
    A GENERIC LOG ANALYSIS FRAMEWORK
  • Log Files
    SOLUTION OVERVIEW
    SOLUTION OVERVIEW
    Interpretation
    Processing
    Presentation
    Unified mechanism for extracting information of interest from both text and binary log files with arbitrary structure and format
    Easy mechanism to build and maintain a rule base for inferences
    Flexible means for generating custom reports from inferences
    Knowledge Representation Schema
  • Easy to add content
    SOLUTION DESIGN
    Easy to visualize
    Resembles human knowledge organization better
    Easy to combine
    MIND MAPS
    Easily convertible to XML
    Easy access to computers
    Tree
    Can utilize existing tree algorithms
    Can utilize existing tools
    MIND MAP AS KNOWLEDGE UNIT
  • SYSTEM ARCHITECTURE
    SOLUTION DESIGN
  • NEW SCRIPTING LANGUAGE
    SOLUTION IMPLEMENTATION
    Mind map is the basic processing unit
    Configurable syntax
    Advanced filtering
    Multiple executions in a single statement
    Supports basic and compound data types
    Built-in and user defined functions
  • $Map1.TypeIs(#ERROR)::$MY.LeftSibling.IsNotNull, Level < 2.LeftSibling->Category.Unique.Count = $ERROR_CATEGORIES_COUNT
    SOLUTION IMPLEMENTATION
    • Follows the flow of a text in natural language
    • Uses statement chaining
    • No distant memory calls
    • More suitable for expressing rules
    • Independent small chunks of execution
    $Found = FALSE
    $Map1.TypeIs(#ERROR) = $Set1
    $Set1.Unique = $Errors
    $Map1.TypeIs(#WARN) = $Set2
    $Set2.Unique = $Warnings
    Foreach $Error in $Errors
    $Error->Category = $Cat
    $Warnings::Category==$Cat = $X
    If ( $X.Count > 0 )
    $Found = TRUE
    Break
    EndIf
    EndFor
    • Suits Advanced Programming
    • Difficult for non-developers
    • Memory intensive
    PROGRAMMING MODELS
  • Log Files
    SOLUTION SUMMARY
    SOLUTION IMPLEMENTATION
    Interpretation
    Processing
    Presentation
    • Special support for splitting text and binary data
    • Support for structural data extraction
    • Rich platform to add and edit rules
    • Support for combining mind maps
    • Turing complete
    • Custom reports generated by scripts
    Mind Maps
  • SOLUTION IMPLEMENTATION
    USAGE SCENARIO
  • CONCLUSION
    The new framework
    provides a unified platform for generic log analysis. It enables users to perform different tasks in a homogeneous fashion. In addition it formulates infrastructure for a shared rule base.
  • FUTURE WORK
    Interpretation
    Processing
    Presentation
    • Script library for common tool logs
    • Declarative language
    • Support for fuzzy rules
    • Design driven reports
  • REFERENCES
    J. Valdman. Log file analysis. Technical Report DCSE/TR-2001-04, Department of Computer Science and Engineering (FAV UWB), 2001.
    Tony Buzan. The Mind Map Book. Penguin Books, 1996, ch. 2
    John E. Hopcroft, Jeffery D. Ullman. Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979, pp. 13-137
    J. H. Andrews. Theory and practice of log file analysis. Technical Report 524, Department of Computer Science, University of Western Ontario, May 1998.
    S. G. Eick, M. C. Nelson, J. D. Schmidt. Graphical Analysis of Computer Log Files. Communications of the ACM, Vol. 37, No. 12, pp. 50-56, 1994.
    H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Mining for relevant terms from log files. In: KDIR’09. Proc. of International Conference on Knowledge Discovery and Information Retrieval. Madeira, Portugal. 2009.
    H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Terminology extraction from log files. In: KDIR’09. Proc. Of 20th International Conference on Database and Expert Systems Applications. pp. 769-776. Lecture Notes in Computer Science, Springer 2009.
  • QUESTIONS