The document summarizes the development of a new parser tool called the FlexConnector Wizard that improves upon existing logfile parsing methods. It provides an easy to use interface for users to parse logfiles into a Common Event Format readable by ArcSight software. The tool allows users to save and reuse parsing rules, offers a more intuitive layout than existing regex-based tools, and includes features like token filtering testing and import/export functionality. The document also outlines the developer's contributions, including token filtering logic and base regex tokenization, and explains some key terminology used in logfile parsing.

1. HPE Restricted
Cody Zeng
UC Berkeley, 2018
Electrical Engineering &
Computer Science Software Developer Intern
Next Generation
Parser
Import Logfile
Create Base
Regex
Create Tokens
& Token Filters
Test Token
Filters
Create
Properties File
Use in
FlexConnector
Present vs Future
• Use once and finish, no
reusability
• Not user friendly
• Everything crammed onto one page
• Only for those experienced with
regex
Flex Connector ParserFlexConnector Wizard
• Ability to save/reuse tokens and
token filters
• Pick & choose, no need to type every time
• Comfortable layout
• Up-to-date UI
• Multiple pages, greater control
• Instant response, easy to use
Overview / Problem
• Need to bridge gap between the logfiles customers need to analyze
and ArcSight’s software (ESM, Logger) that analyzes them
• ArcSight software requires input in CEF (Common Event Format)
• Need to parse logfile into CEF
• Done by ArcSight Connectors (400+)
• So many different logfile formats, can’t support all
• Implement parser for unsupported formats
• Customers quickly, easily convert logfile into data readable by FlexConnector
My Contributions
• Token Filter testing logic
• Extensive use of regex matching
• Allows users to view success rate
of filters created
• Base regex tokenization
• Extract capture groups from regex
and wrap each group into a token
• Executable
Front endBack end
• Context-sensitive help
• Help window whose content changes
depending on which page is open
• Field validation
• Make sure user enters valid info
• Ensure no overriding
• Import/Export windows
• Allow customers to reuse tokens and
token filters
Terminology
• Schema: An event schema used in Connectors and
ESM (Enterprise Security Manager)
• Operations: Connector supported operations for parsers
e.g. _oneOf(arg1,arg2,arg3)
• Token: Represents a unit of information in log file
e.g. %TimeStamp% or %TimeZone%
• Mapping: Represents an assignment of a token value to
a schema field. It can use operations as well
• Token Filters: Combination of tokens and constants.
Should match at least one log file line.
FlexConnector
Wizard
Flex
Connector
Enterprise
Security
Management
Logfile
Next Gen
Parser
Current Sequence
Technologies Used
Replace