Centralizing Centralization v. 0.2 October 2003 Anton Chuvakin, Ph.D., GCIA, GCIH Senior Security Analyst

Outline Security data centralization overview Value of centralization Single device type and cross-device centralization Normalization, pros and cons Categorization, security event types and standards Correlation, types and methods Why do you have to do it?

Security data centralization overview

Value of centralization

Single device type and cross-device centralization

Normalization, pros and cons

Categorization, security event types and standards

Correlation, types and methods

Why do you have to do it?

Terms Message – some system indication that the event has occurred Log or audit record – recorded message related to the event Log file – collection of the above records Alert – a message usually sent to notify an operator Device – a source of security-relevant logs

Message – some system indication that the event has occurred

Log or audit record – recorded message related to the event

Log file – collection of the above records

Alert – a message usually sent to notify an operator

Device – a source of security-relevant logs

Centralization Centralized security controls: Cheaper to manage Easier to audit Save money on staff Reduce training costs

Centralized security controls:

Cheaper to manage

Easier to audit

Save money on staff

Reduce training costs

Security Data Overview What data? Audit logs Transaction logs Intrusion logs Connection logs System performance records User activity logs Various alerts From where? Firewalls/intrusion prevention Routers/switches Intrusion detection Hosts Business applications (databases, servers) Anti-virus VPNs

What data?

Audit logs

Transaction logs

Intrusion logs

Connection logs

System performance records

User activity logs

Various alerts

From where?

Firewalls/intrusion prevention

Routers/switches

Intrusion detection

Hosts

Business applications (databases, servers)

Anti-virus

VPNs

Centralized Data Why centralize security data? Accessibility All audit records in one place Cross-device searchability and analysis Categorization Correlation De-duplication / volume reduction Reduced response time Increase in the efficiency of existing security point solutions

Why centralize security data?

Accessibility

All audit records in one place

Cross-device searchability and analysis

Categorization

Correlation

De-duplication / volume reduction

Reduced response time

Increase in the efficiency of existing security point solutions

Requirements What do you need to start? Collect the data Convert to common format Reduce in size, if possible Transport securely to a central location Process in real-time Alert on threats Store securely Report on trends

What do you need to start?

Collect the data

Convert to common format

Reduce in size, if possible

Transport securely to a central location

Process in real-time

Alert on threats

Store securely

Report on trends

Challenges Need to overcome these: Too much data Not enough data Diverse records False alarms Duplicate data Hard to get data Chain of custody concerns

Need to overcome these:

Too much data

Not enough data

Diverse records

False alarms

Duplicate data

Hard to get data

Chain of custody concerns

Case Made! So... ...everybody build a “central console” for their stuff. Results : central consoles for ... Each firewall vendor Multiple firewalls and routers Each IDS type IDS and vulnerability scanners Routers and network management data Ad infinitum... 

...everybody build a “central console” for their stuff. Results : central consoles for ...

Each firewall vendor

Multiple firewalls and routers

Each IDS type

IDS and vulnerability scanners

Routers and network management data

Ad infinitum... 

Case I: NIDS Console Relatively small number of devices False alarms False positives Large volume (needs tuning) Many of the recorded events require response The data sometimes need to be viewed (and responded to) in near real-time

Relatively small number of devices

False alarms

False positives

Large volume (needs tuning)

Many of the recorded events require response

The data sometimes need to be viewed (and responded to) in near real-time

Example: ACID Console Collects events from multiple Snort NIDS sensors Uses relational DBMS to store data Retains (and can show) full packet payload Web front-end Advanced search queries Search across sensors Search by any packet field/combination Data graphing No real-time tools

Collects events from multiple Snort NIDS sensors

Uses relational DBMS to store data

Retains (and can show) full packet payload

Web front-end

Advanced search queries

Search across sensors

Search by any packet field/combination

Data graphing

No real-time tools

Case II: Desktop Protection Such as personal firewall/IDS, anti-virus, host IPS Characteristics: Huge number of devices Low volume from each device Needs status monitoring (disabled by the user?) Data might be transmitted over the slow WAN link Rarely looked at Requires cross-device correlation for meaningful analysis

Such as personal firewall/IDS, anti-virus, host IPS

Characteristics:

Huge number of devices

Low volume from each device

Needs status monitoring (disabled by the user?)

Data might be transmitted over the slow WAN link

Rarely looked at

Requires cross-device correlation for meaningful analysis

Diverse devices Moving from a single type of data source to heterogeneous sources Volume is getting even higher Data diversity problem arises Binary and text logs Undocumented formats Free form logs Same events described differently Different level of detail in collected data How to analyze?

Moving from a single type of data source to heterogeneous sources

Volume is getting even higher

Data diversity problem arises

Binary and text logs

Undocumented formats

Free form logs

Same events described differently

Different level of detail in collected data

How to analyze?

Normalization Defined Solution: normalization i.e. converting recorded events to a common format or schema (often XML) How to normalize? Look at common fields in security event records Source IP, port, protocol Event type Device instance Severity Create a data model to cover all these and more Map the original event fields to the new general schema

Solution: normalization i.e. converting recorded events to a common format or schema (often XML)

How to normalize?

Look at common fields in security event records

Source IP, port, protocol

Event type

Device instance

Severity

Create a data model to cover all these and more

Map the original event fields to the new general schema

Normalization Example

Normalization Advantages: Store – known storage requirements Analyze (c orrelate, c ategorize) and search – same attributes Prioritize – uniform severity Present/visualize – common reports Challenges: Data loss What if something does not fit the model? Overhead Too much of a good thing? Over-normalization What if its not really the same ? Mapping incompatibilities Is this more of a source or a destination after all?

Advantages:

Store – known storage requirements

Analyze (c orrelate, c ategorize) and search – same attributes

Prioritize – uniform severity

Present/visualize – common reports

Challenges:

Data loss

What if something does not fit the model?

Overhead

Too much of a good thing?

Over-normalization

What if its not really the same ?

Mapping incompatibilities

Is this more of a source or a destination after all?

Categorization Data format is the same, but what about the content ? MSBlaster, Nimda, CodeRed -> Malware Statd Attack, SSH Exploit -> Unix Exploits UDP Bomb, Boink, Smurf -> Legacy DoS Select Categories Malware Attacks and Exploits Vulnerable Software System Failures AAA Change Management

Data format is the same, but what about the content ?

MSBlaster, Nimda, CodeRed -> Malware

Statd Attack, SSH Exploit -> Unix Exploits

UDP Bomb, Boink, Smurf -> Legacy DoS

Select Categories

Malware

Attacks and Exploits

Vulnerable Software

System Failures

AAA

Change Management

Value of Categorization Value of Categorization Adds intelligence to event data collection Enhances high-level reporting Provides understanding of the detected threat types and supplies the context for their interpretation Challenges with Categorization No universal standard (but work in progress!) Too much variety in data makes every categorization effort incomplete Every security vendor is trying to create its own scheme (yak!)

Value of Categorization

Adds intelligence to event data collection

Enhances high-level reporting

Provides understanding of the detected threat types and supplies the context for their interpretation

Challenges with Categorization

No universal standard

(but work in progress!)

Too much variety in data makes every categorization effort incomplete

Every security vendor is trying to create its own scheme (yak!)

Correlation Defined : General: “establishing or finding relationships between entities” Security: “improving threat identification and assessment by looking not only at individual events, but at their sets , bound by some common parameter ('related')” Correlation is enabled by centralization, normalization and also enhanced by categorization.

Defined :

General: “establishing or finding relationships between entities”

Security: “improving threat identification and assessment by looking not only at individual events, but at their sets , bound by some common parameter ('related')”

Correlation is enabled by centralization, normalization and also enhanced by categorization.

Correlation Types Rule-based Uses pre-existing knowledge of the attack (the rule) and is able to define what has been detected in precise terms Statistical Relies upon the knowledge of normal activities, which has been accumulated over time to detect the deviations

Rule-based

Uses pre-existing knowledge of the attack (the rule) and is able to define what has been detected in precise terms

Statistical

Relies upon the knowledge of normal activities, which has been accumulated over time to detect the deviations

Policy, Vulnerability and Incident Management Other data (information, knowledge) Knowledge Security policies and procedures Industry and organization security guidelines Vulnerability and asset data Scans Asset attributes Security Incidents Centralized incident handling and reporting

Other data (information, knowledge)

Knowledge

Security policies and procedures

Industry and organization security guidelines

Vulnerability and asset data

Scans

Asset attributes

Security Incidents

Centralized incident handling and reporting

Conclusion Centralization of security data is crucial for... Large organization Can't do security without it! Small/medium companies Needed to succeed with little security staff

Centralization of security data is crucial for...

Large organization

Can't do security without it!

Small/medium companies

Needed to succeed with little security staff

Thanks for Viewing the Presentation Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA http://www.chuvakin.org Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org Read my blog at http:// chuvakin.blogspot.com Book on logs is coming soon! See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs

Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA

http://www.chuvakin.org

Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org

Read my blog at http:// chuvakin.blogspot.com

Book on logs is coming soon!

See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs