Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
JCSSE 2012DileepaJayathilake
Functional               Troubleshooting                                   ConformanceLog Analysis   in Use               ...
Log Analysis                      DomainsWeb server logsNetwork logsSecurity logsSystem logsApplication logs
Even with              Manual analysis                                      expertise,                  needs             ...
Log Analysis Automation                              ChallengesLack of a standard• “Universal Format for Logger Messages” ...
Existing Log Management                   Tools
Identifying common constructsLog indexingHandling different log sourcesDealing with different log typesRich user interface...
Why Structured Log                                    Analysis?Many log filesmanifest a structure  Analysis needs  context...
Structured Log Analysis            Framework
ConclusionsExisting tools solve a subset of automated log analysis requirements,but ignore the importance of structureNew ...
Future WorkAdd more log management capabilitiesReal time analysisBuilt-in format declarations for common log formatsOptimi...
Upcoming SlideShare
Loading in …5
×

Towards structured log analysis

Value of software log file analysis has been constantly increasing with the value of information to organizations. Log management tools still have a lot to deliver in order to empower their customers with the true strength of log information. In addition to the traditional uses such as testing software functional conformance, troubleshooting and performance benchmarking, log analysis has proven its capabilities in fields like intrusion detection and compliance evaluation. This is verified by the emphasis on log analysis in regulations like PCI DSS, FISMA, HIPAA and frameworks such as ISO 27001 and COBIT.
In this paper we present an in depth analysis into current log analysis domains and common problems. A practical guide to the use of few popular log analysis tools is also included. Lack of proper support for structured analysis is identified as one major flaw in existing tools. After that, we describe a framework we developed for structured log analysis with the view of providing a solution to open problems in the domain. The core strength of the framework is its ability to handle many log file formats that are not well served by existing tools and providing sophisticated infrastructure for automating recurring log analysis procedures. We prove the usefulness of the framework with a simple experiment.

  • Login to see the comments

Towards structured log analysis

  1. 1. JCSSE 2012DileepaJayathilake
  2. 2. Functional Troubleshooting ConformanceLog Analysis in Use Monitoring Statistical Insight System Health
  3. 3. Log Analysis DomainsWeb server logsNetwork logsSecurity logsSystem logsApplication logs
  4. 4. Even with Manual analysis expertise, needs manual log acquaintance analysis is with format laborious Manually Manual analysis dealing with hinders reusingvast amount of recurringlog information analysis is difficult Automation patterns will save lot of costs
  5. 5. Log Analysis Automation ChallengesLack of a standard• “Universal Format for Logger Messages” - Expired without a successor• “Syslog” – Serves only a limited range of system logsLog file corruptions• Erasing parts of a log file, mixing up multiple log entries, presence of log entries in wrong order and garbage in the middle of log filesInappropriate log content• Problem stems from incorrect judgments of developers regarding the importance of log entriesVarying log semantics• Format and the content logged can continue to evolveHuge sizes of log files• Log files can easily grow into gigabyte sizes in a commercial environment
  6. 6. Existing Log Management Tools
  7. 7. Identifying common constructsLog indexingHandling different log sourcesDealing with different log typesRich user interfacesAlertsIntrusion detectionCompliance validationAutomate recurring analysisproceduresStructured LogAnalysis
  8. 8. Why Structured Log Analysis?Many log filesmanifest a structure Analysis needs contextual correctness Automation requires a structure-aware Example tool
  9. 9. Structured Log Analysis Framework
  10. 10. ConclusionsExisting tools solve a subset of automated log analysis requirements,but ignore the importance of structureNew declarative language is capable of expressing any log file formatand is resilient to corruptionsThe scripting language provides solid infrastructure for rule basedautomationData management scheme offers flexibilityCurrent UI generation method is not appropriate
  11. 11. Future WorkAdd more log management capabilitiesReal time analysisBuilt-in format declarations for common log formatsOptimize data management module to handle heterogeneous dataefficientlyUI generation based on HTML5

×