This document describes a framework and prototype for analyzing log files from multiple sources. It aims to address the problems of information overload, difficulty correlating different data sources, and manual analysis when debugging issues. The framework allows storage and parsing of different log file types and formats data as sets of named values. A log viewer prototype allows filtering, querying, and custom formatting of parsed log data. Future plans include completing the prototypes, adding more log parsers, automatic analysis capabilities, and integrating with other tools.

1. SMART LOG ANALYSIS
A General Framework
and SMB Prototype
Windows Serviceability
Tim Burke, Kishore Chintalapati (manager)
Mike Tiberio (coach), Apurva Sharma,
Samarth Shetty Badilaguthu

2. TALK OVERVIEW
 Problem Space
 Current Approaches
 Design Objectives
 My Project: Smart Log Analysis and SMB Prototype
 Benefits
 Future Plans
 Demo

3. PROBLEM SPACE
 Multiple Data Sources
 Multiple Tools (Netmon, Perfmon, Notepad, …)
 Difficulty in correlating different source
 Information Overload
 Manual Analysis
 Knowledge Loss

4. CURRENT APPROACHES
 Open Notepad
 Open NetMon
 Repeat
 The Nuclear Option
 Perl
 Grep
Credit: Eric Roode
b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)b
http://www.regular-expressions.info/examples.html

5. THE RADIANT FUTURE
Network Captures ETW Traces Custom Logs
Smart Analysis
Framework
Viewer Automatic Analysis

6. DESIGN OBJECTIVES
 A unified way of viewing, searching, and
analyzing data
 Easily track and highlight relationships
among data.
 Group data into high-level operations
 Extensibility and Flexibility

7. DESIGN CONSIDERATIONS
 Data is data, independent of the source
 Data consists of sets of named values
 Modular
 Easy rule creation
 Performance and Scalability
 Developer focused

8. MY PROJECT
 Framework
 Viewer Prototype
 Text Rule Editor
 From Logs
 From Source
 Extensible
 Component Agnostic
 Scalable
 Embeddable

9. THE FRAMEWORK
Storage Plugins
Provider RulesFile Format Plugins
Log Viewer
Query Engine
SQL Server
Parsed
Data
Log Parser
ETW Parser
Windows Events
Etc.
RDR
SRV
Log FIles
Config Files
Custom Storage
Parsed
Data
Storage Manager
Format Engine
CLR Adapter
Formatting
Rules
Saved
Queries

10. LOG VIEWER
 Boolean expression filters
Filter based on any tag or value
Similar to Netmon filters
Procedural queries
Data correlation
Complex scenarios
Custom formatting

11. TEXT LOG RULE EDITOR
 Easy creation of parsing rules
 From text logs
 From source code
 Preview rule effects

12. BENEFITS
 Allows quicker, easier debugging
 Automates common analysis tasks
 Merges data sources to allow cross-source
analysis.

13. FUTURE PLANS
 Complete the prototypes
 Implement more log parsers (Netmon, …)
 Have component experts create rule sets
 Implement automatic analyses on top of the
framework
 Integrate with other tools for capturing data
like MSDT

14. DEMO

15. QUESTIONS?