Centralizing security data provides several benefits such as easier management, auditing, and analysis across devices. It allows access to all audit records in one place, cross-device search and correlation, categorization of event types, and reduced response time. To centralize data, organizations need to collect, normalize, transport securely, process, store, and report on data from various sources like firewalls, intrusion detection systems, and applications. Challenges include dealing with large volumes of diverse data formats and ensuring chain of custody of the data. Normalization converts data to a common format to facilitate storage, analysis, search, and reporting. Correlation establishes relationships between events to improve threat identification by looking at sets of related events.

1. Centralizing Centralization v. 0.2 October 2003 Anton Chuvakin, Ph.D., GCIA, GCIH Senior Security Analyst

2. Outline Security data centralization overview Value of centralization Single device type and cross-device centralization Normalization, pros and cons Categorization, security event types and standards Correlation, types and methods Why do you have to do it?

3. Terms Message – some system indication that the event has occurred Log or audit record – recorded message related to the event Log file – collection of the above records Alert – a message usually sent to notify an operator Device – a source of security-relevant logs

4. Centralization Centralized security controls: Cheaper to manage Easier to audit Save money on staff Reduce training costs

5. Security Data Overview What data? Audit logs Transaction logs Intrusion logs Connection logs System performance records User activity logs Various alerts From where? Firewalls/intrusion prevention Routers/switches Intrusion detection Hosts Business applications (databases, servers) Anti-virus VPNs

6. Centralized Data Why centralize security data? Accessibility All audit records in one place Cross-device searchability and analysis Categorization Correlation De-duplication / volume reduction Reduced response time Increase in the efficiency of existing security point solutions

7. Requirements What do you need to start? Collect the data Convert to common format Reduce in size, if possible Transport securely to a central location Process in real-time Alert on threats Store securely Report on trends

8. Challenges Need to overcome these: Too much data Not enough data Diverse records False alarms Duplicate data Hard to get data Chain of custody concerns

9. Case Made! So... ...everybody build a “central console” for their stuff. Results : central consoles for ... Each firewall vendor Multiple firewalls and routers Each IDS type IDS and vulnerability scanners Routers and network management data Ad infinitum... 

10. Case I: NIDS Console Relatively small number of devices False alarms False positives Large volume (needs tuning) Many of the recorded events require response The data sometimes need to be viewed (and responded to) in near real-time

11. Example: ACID Console Collects events from multiple Snort NIDS sensors Uses relational DBMS to store data Retains (and can show) full packet payload Web front-end Advanced search queries Search across sensors Search by any packet field/combination Data graphing No real-time tools

12. Case II: Desktop Protection Such as personal firewall/IDS, anti-virus, host IPS Characteristics: Huge number of devices Low volume from each device Needs status monitoring (disabled by the user?) Data might be transmitted over the slow WAN link Rarely looked at Requires cross-device correlation for meaningful analysis

13. Diverse devices Moving from a single type of data source to heterogeneous sources Volume is getting even higher Data diversity problem arises Binary and text logs Undocumented formats Free form logs Same events described differently Different level of detail in collected data How to analyze?

14. Normalization Defined Solution: normalization i.e. converting recorded events to a common format or schema (often XML) How to normalize? Look at common fields in security event records Source IP, port, protocol Event type Device instance Severity Create a data model to cover all these and more Map the original event fields to the new general schema

15. Normalization Example

16. Normalization Advantages: Store – known storage requirements Analyze (c orrelate, c ategorize) and search – same attributes Prioritize – uniform severity Present/visualize – common reports Challenges: Data loss What if something does not fit the model? Overhead Too much of a good thing? Over-normalization What if its not really the same ? Mapping incompatibilities Is this more of a source or a destination after all?

17. Categorization Data format is the same, but what about the content ? MSBlaster, Nimda, CodeRed -> Malware Statd Attack, SSH Exploit -> Unix Exploits UDP Bomb, Boink, Smurf -> Legacy DoS Select Categories Malware Attacks and Exploits Vulnerable Software System Failures AAA Change Management

18. Value of Categorization Value of Categorization Adds intelligence to event data collection Enhances high-level reporting Provides understanding of the detected threat types and supplies the context for their interpretation Challenges with Categorization No universal standard (but work in progress!) Too much variety in data makes every categorization effort incomplete Every security vendor is trying to create its own scheme (yak!)

19. Correlation Defined : General: “establishing or finding relationships between entities” Security: “improving threat identification and assessment by looking not only at individual events, but at their sets , bound by some common parameter ('related')” Correlation is enabled by centralization, normalization and also enhanced by categorization.

20. Correlation Types Rule-based Uses pre-existing knowledge of the attack (the rule) and is able to define what has been detected in precise terms Statistical Relies upon the knowledge of normal activities, which has been accumulated over time to detect the deviations

21. Policy, Vulnerability and Incident Management Other data (information, knowledge) Knowledge Security policies and procedures Industry and organization security guidelines Vulnerability and asset data Scans Asset attributes Security Incidents Centralized incident handling and reporting

22. Conclusion Centralization of security data is crucial for... Large organization Can't do security without it! Small/medium companies Needed to succeed with little security staff

23. Thanks for Viewing the Presentation Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA http://www.chuvakin.org Author of “Security Warrior” (O’Reilly) – http://www.securitywarrior.org Read my blog at http:// chuvakin.blogspot.com Book on logs is coming soon! See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs