SMART LOG ANALYSIS
A General Framework
and SMB Prototype
Windows Serviceability
Tim Burke, Kishore Chintalapati (manager)
...
TALK OVERVIEW
 Problem Space
 Current Approaches
 Design Objectives
 My Project: Smart Log Analysis and SMB Prototype
...
PROBLEM SPACE
 Multiple Data Sources
 Multiple Tools (Netmon, Perfmon, Notepad, …)
 Difficulty in correlating different...
CURRENT APPROACHES
 Open Notepad
 Open NetMon
 Repeat
 The Nuclear Option
 Perl
 Grep
Credit: Eric Roode
b(25[0-5]|2...
THE RADIANT FUTURE
Network Captures ETW Traces Custom Logs
Smart Analysis
Framework
Viewer Automatic Analysis
DESIGN OBJECTIVES
 A unified way of viewing, searching, and
analyzing data
 Easily track and highlight relationships
amo...
DESIGN CONSIDERATIONS
 Data is data, independent of the source
 Data consists of sets of named values
 Modular
 Easy r...
MY PROJECT
 Framework
 Viewer Prototype
 Text Rule Editor
 From Logs
 From Source
 Extensible
 Component Agnostic
...
THE FRAMEWORK
Storage Plugins
Provider RulesFile Format Plugins
Log Viewer
Query Engine
SQL Server
Parsed
Data
Log Parser
...
LOG VIEWER
 Boolean expression filters
Filter based on any tag or value
Similar to Netmon filters
Procedural queries
...
TEXT LOG RULE EDITOR
 Easy creation of parsing rules
 From text logs
 From source code
 Preview rule effects
BENEFITS
 Allows quicker, easier debugging
 Automates common analysis tasks
 Merges data sources to allow cross-source
...
FUTURE PLANS
 Complete the prototypes
 Implement more log parsers (Netmon, …)
 Have component experts create rule sets
...
DEMO
QUESTIONS?
Upcoming SlideShare
Loading in …5
×

Smart Log Analysis

686 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Smart Log Analysis

  1. 1. SMART LOG ANALYSIS A General Framework and SMB Prototype Windows Serviceability Tim Burke, Kishore Chintalapati (manager) Mike Tiberio (coach), Apurva Sharma, Samarth Shetty Badilaguthu
  2. 2. TALK OVERVIEW  Problem Space  Current Approaches  Design Objectives  My Project: Smart Log Analysis and SMB Prototype  Benefits  Future Plans  Demo
  3. 3. PROBLEM SPACE  Multiple Data Sources  Multiple Tools (Netmon, Perfmon, Notepad, …)  Difficulty in correlating different source  Information Overload  Manual Analysis  Knowledge Loss
  4. 4. CURRENT APPROACHES  Open Notepad  Open NetMon  Repeat  The Nuclear Option  Perl  Grep Credit: Eric Roode b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?). (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)b http://www.regular-expressions.info/examples.html
  5. 5. THE RADIANT FUTURE Network Captures ETW Traces Custom Logs Smart Analysis Framework Viewer Automatic Analysis
  6. 6. DESIGN OBJECTIVES  A unified way of viewing, searching, and analyzing data  Easily track and highlight relationships among data.  Group data into high-level operations  Extensibility and Flexibility
  7. 7. DESIGN CONSIDERATIONS  Data is data, independent of the source  Data consists of sets of named values  Modular  Easy rule creation  Performance and Scalability  Developer focused
  8. 8. MY PROJECT  Framework  Viewer Prototype  Text Rule Editor  From Logs  From Source  Extensible  Component Agnostic  Scalable  Embeddable
  9. 9. THE FRAMEWORK Storage Plugins Provider RulesFile Format Plugins Log Viewer Query Engine SQL Server Parsed Data Log Parser ETW Parser Windows Events Etc. RDR SRV Log FIles Config Files Custom Storage Parsed Data Storage Manager Format Engine CLR Adapter Formatting Rules Saved Queries
  10. 10. LOG VIEWER  Boolean expression filters Filter based on any tag or value Similar to Netmon filters Procedural queries Data correlation Complex scenarios Custom formatting
  11. 11. TEXT LOG RULE EDITOR  Easy creation of parsing rules  From text logs  From source code  Preview rule effects
  12. 12. BENEFITS  Allows quicker, easier debugging  Automates common analysis tasks  Merges data sources to allow cross-source analysis.
  13. 13. FUTURE PLANS  Complete the prototypes  Implement more log parsers (Netmon, …)  Have component experts create rule sets  Implement automatic analyses on top of the framework  Integrate with other tools for capturing data like MSDT
  14. 14. DEMO
  15. 15. QUESTIONS?

×