SlideShare a Scribd company logo
1 of 25
LogLogic Confidential Thursday, March 19, 20151
The Top Five Log
Analysis Mistakes
Dr Anton Chuvakin
Chief Logging Evangelist
LogLogic, Inc
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
2Confidential |
Summary
1. System, Network and Security Logs
2. Why Look at Logs?
3. Brief Log Analysis Overview
4. Log Analysis Mistakes
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
3Confidential |
Log Data Overview
 Audit logs
 Transaction logs
 Intrusion logs
 Connection logs
 System performance records
 User activity logs
 Various alerts and other
messages
 Firewalls/intrusion prevention
 Routers/switches
 Intrusion detection
 Servers, desktops, mainframes
 Business applications
 Databases
 Anti-virus
 VPNs
What logs? From Where?
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
4Confidential |
Login? Logon? Log in?
<122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from
::ffff:192.168.138.35 port 2895 ssh2
<13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success Audit
ENTERPRISE Account Logon
Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon acco
unt: POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC000006A
4574
<57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:Login Success
[user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28
2006
<18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-
warning-00515: Admin User netscreen has logged on via Telnet from
10.14.98.55:39073 (2002-12-17 15:50:53)
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
5Confidential |
“Arrgh! Why
Don’t We Just
Ignore’Em?”
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
6Confidential |
Log Management Mandate and Regulations
Regulations
Require LMI
 SOX
 GLBA
 FISMA
 JPA
 NIST 800-53
 Capture audit records
 Regularly review audit records
for unusual activity and
violations
 Automatically process audit
records
 Protect audit information from
unauthorized deletion
 Retain audit logs
 PCI
 HIPAA
 SLAs
Mandates
Demand It
 PCI : Requirement 10
and beyond
 Logging and user activities
tracking are critical
 Automate and secure audit trails
for event reconstruction
 Review logs daily
 Retain audit trail history for
at least one year
 COBIT
 ISO
 ITIL
 COBIT 4
 Provide audit trail
for root-cause analysis
 Use logging to detect unusual or
abnormal activities
 Regularly review access, privileges,
changes
 Verify backup completion
 ISO17799
 Maintain audit logs for system
access and use, changes, faults,
corrections, capacity demands
 Review the results of monitoring
activities regularly and ensure the
accuracy of logs
Controls
Require it
“Get fined, Get
Sanctioned”
“Lose Customers,
Reputation, Revenue or Job”
“Get fined, Go To Jail”
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
7Confidential |
So, How Do People Do It?
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
8Confidential |
Log Analysis Basics
 Manual
– ‘Tail’, ‘more’, ‘grep’, ‘notepad’, etc
 Filtering
– Positive and negative (“Artificial ignorance”)
 Summarization and reports
– “Top X of Y”
 Visualization
 Log indexing and searching
 Correlation
– Rule-based and other
 Log data mining
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
9Confidential |
From Log Analysis to Log Management
 Threat protection and discovery
 Incident response
 Forensics, “e-discovery” and litigation support
 Regulatory compliance
 Internal policies and procedure compliance
 Internal and external audit support
 IT system and network troubleshooting
 IT performance management
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
10Confidential |
Looks Complicated?! No
Wonder People Make
Mistakes …
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
11Confidential |
Six Mistakes of Log Analysis and Log Management
0. Not logging at all.
1. Not looking at the logs
2. Storing logs for too short a time
3. Prioritizing the log records before collection
4. Ignoring the logs from applications
5. Only looking for “known bad” stuff
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
12Confidential |
Mistake 0: Not Logging AT ALL …
… and its aggravated version: “… and not
knowing that you don’t”
 No logging? -> well, no logs for incident
response, audits, compliance
Got logs?
If your answer is ‘NO”, don’t listen further: run
and enable logging right now!
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
13Confidential |
Example: Oracle
 Defaults:
– minimum system logging
– minimum database server access
– no data access logging
 So, where is …
– data access audit
– schema and data change audit
– configuration change audit
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
14Confidential |
Mistake 1: Not looking at logs
 Collection of logs has value!
 But review boosts the value 10-fold (numbers are estimates
)
 More in-depth analysis boosts it a lot more!
 Two choices here …
– Review after an incident
– Ongoing review
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
15Confidential |
Example Log Review Priorities
1. DMZ NIDS
2. DMZ firewall
3. DMZ servers with applications
4. Critical internal servers
5. Other servers
6. Select critical application
7. Other applications
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
16Confidential |
Mistake 2: Storing Logs For Too Short A Time
 You are saying you HAD logs? And how is it
useful?
 Retention question is a hard one. Truly,
nobody has the answer!
– Seven years? A year? 90 days? A week? Until the
disk runs out?
 Common: 90 days online and up to 1-3 years
“nearline” or offline
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
17Confidential |
Also A Mistake: Storing Logs for TOO LONG?!
 Retention = storage + destruction
 Why DESTROY LOGS?
– Privacy regulations (mostly EU)
– Litigation risk management
– System resource utilization
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
18Confidential |
Example Retention Strategy
Type + network + storage tier
 IDS + DMZ + online = 90 days
 Firewall + DMZ + online = 30 days
 Servers + internal + online = 90 days
 ALL + DMZ + archive = 3 years
 Critical + internal + archive = 5 years
 OTHER + internal + archive = 1 year
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
19Confidential |
Mistake 3: Deciding What’s Relevant Before
Collection
 How would you know what is …
– … Security-relevant
– … Compliance-relevant
– … or will solve the problem you’d have
TOMORROW!?
 Also affects “forensic quality” of logs
 Prioritization Challenge – Got ESP? 
 “Simple” – just grab everything!
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
20Confidential |
Example Common Logging Order
Log everything
Retain most everything
Analyze enough
Summarize and report on a subset
Look at some
Act in real-time on a few
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
21Confidential |
Mistake 4: Ignoring Logs from Applications
 Firewall – Yes, Linux – Yes, Windows –
Yes, NIDS and NIPS – Yes
but …
 Oracle - ?
 SAP - ?
 Your Application X – No
Log standards are coming: MITRE CEE!
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
22Confidential |
Example: Jumbled Mess of Application Logs
|22:01:40|BTC| 7|000|DDIC | |LC2|Systemerror when
executing external command DB6_DATA_COLLECTOR on
gneisenau ()
|22:02:32|BTC| 7|000|DDIC | |R49|Communication error,
CPIC return code 020, SAP return code 456
|22:02:32|BTC| 7|000|DDIC | |R5A|> Conversation ID:
38910614
|22:02:32|BTC| 7|000|DDIC | |R64|> CPI-C function:
CMSEND(SAP)
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
23Confidential |
Mistake 5: Looking for only the bad stuff
 Correlation, filters, regex matching – oh, no! 
 Why such approaches?
– You have to know what you are looking for!
 Can we somehow just “see what we need to
see”?
– Data mining technology can help
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
24Confidential |
Conclusions: Mistakes Summary
0. Not logging at all.
1. Not looking at the logs
2. Storing logs for too short a time
3. Prioritizing the log records before collection
4. Ignoring the logs from applications
5. Only looking for “known bad” stuff
Thursday, March 19, 2015
Mitigating Risk. Automating Compliance.
25Confidential |
Thanks for Attending the Presentation
Dr Anton Chuvakin, GCIH, GCFA
Chief Logging Evangelist
http://www.chuvakin.org
Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI
Compliance” (Syngress, 2007)
See http://www.info-secure.org for my papers, books, reviews
and other security resources related to logs. Book on logs is
coming soon! Also see http://chuvakin.blogspot.com

More Related Content

Similar to O'Reilly Webinar Five Mistakes Log Analysis

Log management and compliance: What's the real story? by Dr. Anton Chuvakin
Log management and compliance: What's the real story? by Dr. Anton ChuvakinLog management and compliance: What's the real story? by Dr. Anton Chuvakin
Log management and compliance: What's the real story? by Dr. Anton ChuvakinAnton Chuvakin
 
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...How to Gain Visibility and Control: Compliance Mandates, Security Threats and...
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...Anton Chuvakin
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big DataRaffael Marty
 
5 Things Your Security Administrator Should Tell You
5 Things Your Security Administrator Should Tell You5 Things Your Security Administrator Should Tell You
5 Things Your Security Administrator Should Tell YouHelpSystems
 
Qualys user group presentation - vulnerability management - November 2009 v1 3
Qualys user group presentation - vulnerability management - November 2009 v1 3Qualys user group presentation - vulnerability management - November 2009 v1 3
Qualys user group presentation - vulnerability management - November 2009 v1 3Tom King
 
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergIbm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergdawnrk
 
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergIbm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergdawnrk
 
IBM i Security SIEM Integration
IBM i Security SIEM IntegrationIBM i Security SIEM Integration
IBM i Security SIEM IntegrationPrecisely
 
Automation: Embracing the Future of SecOps
Automation: Embracing the Future of SecOpsAutomation: Embracing the Future of SecOps
Automation: Embracing the Future of SecOpsIBM Security
 
The impact of consumerization
The impact of consumerizationThe impact of consumerization
The impact of consumerizationMichel de Goede
 
SACON - Incident Response Automation & Orchestration (Amit Modi)
SACON - Incident Response Automation & Orchestration (Amit Modi)SACON - Incident Response Automation & Orchestration (Amit Modi)
SACON - Incident Response Automation & Orchestration (Amit Modi)Priyanka Aash
 
Finding attacks with these 6 events
Finding attacks with these 6 eventsFinding attacks with these 6 events
Finding attacks with these 6 eventsMichael Gough
 
Maximize your IT Data and Analytics
Maximize your IT Data and AnalyticsMaximize your IT Data and Analytics
Maximize your IT Data and AnalyticsIvanti
 
Tizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptTizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptwebhostingguy
 
Tizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptTizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptwebhostingguy
 
Log Analytics for Distributed Microservices
Log Analytics for Distributed MicroservicesLog Analytics for Distributed Microservices
Log Analytics for Distributed MicroservicesKai Wähner
 
IT Audit For Non-IT Auditors
IT Audit For Non-IT AuditorsIT Audit For Non-IT Auditors
IT Audit For Non-IT AuditorsEd Tobias
 
How to Perform Continuous Vulnerability Management
How to Perform Continuous Vulnerability ManagementHow to Perform Continuous Vulnerability Management
How to Perform Continuous Vulnerability ManagementIvanti
 

Similar to O'Reilly Webinar Five Mistakes Log Analysis (20)

Log management and compliance: What's the real story? by Dr. Anton Chuvakin
Log management and compliance: What's the real story? by Dr. Anton ChuvakinLog management and compliance: What's the real story? by Dr. Anton Chuvakin
Log management and compliance: What's the real story? by Dr. Anton Chuvakin
 
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...How to Gain Visibility and Control: Compliance Mandates, Security Threats and...
How to Gain Visibility and Control: Compliance Mandates, Security Threats and...
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big Data
 
5 Things Your Security Administrator Should Tell You
5 Things Your Security Administrator Should Tell You5 Things Your Security Administrator Should Tell You
5 Things Your Security Administrator Should Tell You
 
Qualys user group presentation - vulnerability management - November 2009 v1 3
Qualys user group presentation - vulnerability management - November 2009 v1 3Qualys user group presentation - vulnerability management - November 2009 v1 3
Qualys user group presentation - vulnerability management - November 2009 v1 3
 
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergIbm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
 
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenbergIbm ofa ottawa_ how_secure_is_your_data_eric_offenberg
Ibm ofa ottawa_ how_secure_is_your_data_eric_offenberg
 
IBM i Security SIEM Integration
IBM i Security SIEM IntegrationIBM i Security SIEM Integration
IBM i Security SIEM Integration
 
Automation: Embracing the Future of SecOps
Automation: Embracing the Future of SecOpsAutomation: Embracing the Future of SecOps
Automation: Embracing the Future of SecOps
 
The impact of consumerization
The impact of consumerizationThe impact of consumerization
The impact of consumerization
 
SACON - Incident Response Automation & Orchestration (Amit Modi)
SACON - Incident Response Automation & Orchestration (Amit Modi)SACON - Incident Response Automation & Orchestration (Amit Modi)
SACON - Incident Response Automation & Orchestration (Amit Modi)
 
B3948
B3948B3948
B3948
 
Finding attacks with these 6 events
Finding attacks with these 6 eventsFinding attacks with these 6 events
Finding attacks with these 6 events
 
Maximize your IT Data and Analytics
Maximize your IT Data and AnalyticsMaximize your IT Data and Analytics
Maximize your IT Data and Analytics
 
Tizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptTizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.ppt
 
Tizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.pptTizor_Data-Best-Practices.ppt
Tizor_Data-Best-Practices.ppt
 
Log Analytics for Distributed Microservices
Log Analytics for Distributed MicroservicesLog Analytics for Distributed Microservices
Log Analytics for Distributed Microservices
 
IT Audit For Non-IT Auditors
IT Audit For Non-IT AuditorsIT Audit For Non-IT Auditors
IT Audit For Non-IT Auditors
 
How to Perform Continuous Vulnerability Management
How to Perform Continuous Vulnerability ManagementHow to Perform Continuous Vulnerability Management
How to Perform Continuous Vulnerability Management
 
CISA (1).pdf
CISA (1).pdfCISA (1).pdf
CISA (1).pdf
 

More from Anton Chuvakin

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinAnton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...Anton Chuvakin
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinAnton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothAnton Chuvakin
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022Anton Chuvakin
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton ChuvakinAnton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020  Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020  Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsAnton Chuvakin
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC Anton Chuvakin
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020Anton Chuvakin
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton Chuvakin
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)Anton Chuvakin
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinAnton Chuvakin
 

More from Anton Chuvakin (20)

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less Operations
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020  Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020  Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in Brief
 
Generic siem how_2017
Generic siem how_2017Generic siem how_2017
Generic siem how_2017
 
Tips on SIEM Ops 2015
Tips on SIEM Ops 2015Tips on SIEM Ops 2015
Tips on SIEM Ops 2015
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics Presentation
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
 
SIEM Primer:
SIEM Primer:SIEM Primer:
SIEM Primer:
 

Recently uploaded

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxMasterG
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxjbellis
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistandanishmna97
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard37
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهMohamed Sweelam
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 

Recently uploaded (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 

O'Reilly Webinar Five Mistakes Log Analysis

  • 1. LogLogic Confidential Thursday, March 19, 20151 The Top Five Log Analysis Mistakes Dr Anton Chuvakin Chief Logging Evangelist LogLogic, Inc
  • 2. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 2Confidential | Summary 1. System, Network and Security Logs 2. Why Look at Logs? 3. Brief Log Analysis Overview 4. Log Analysis Mistakes
  • 3. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 3Confidential | Log Data Overview  Audit logs  Transaction logs  Intrusion logs  Connection logs  System performance records  User activity logs  Various alerts and other messages  Firewalls/intrusion prevention  Routers/switches  Intrusion detection  Servers, desktops, mainframes  Business applications  Databases  Anti-virus  VPNs What logs? From Where?
  • 4. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 4Confidential | Login? Logon? Log in? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon acco unt: POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC000006A 4574 <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system- warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
  • 5. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 5Confidential | “Arrgh! Why Don’t We Just Ignore’Em?”
  • 6. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 6Confidential | Log Management Mandate and Regulations Regulations Require LMI  SOX  GLBA  FISMA  JPA  NIST 800-53  Capture audit records  Regularly review audit records for unusual activity and violations  Automatically process audit records  Protect audit information from unauthorized deletion  Retain audit logs  PCI  HIPAA  SLAs Mandates Demand It  PCI : Requirement 10 and beyond  Logging and user activities tracking are critical  Automate and secure audit trails for event reconstruction  Review logs daily  Retain audit trail history for at least one year  COBIT  ISO  ITIL  COBIT 4  Provide audit trail for root-cause analysis  Use logging to detect unusual or abnormal activities  Regularly review access, privileges, changes  Verify backup completion  ISO17799  Maintain audit logs for system access and use, changes, faults, corrections, capacity demands  Review the results of monitoring activities regularly and ensure the accuracy of logs Controls Require it “Get fined, Get Sanctioned” “Lose Customers, Reputation, Revenue or Job” “Get fined, Go To Jail”
  • 7. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 7Confidential | So, How Do People Do It?
  • 8. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 8Confidential | Log Analysis Basics  Manual – ‘Tail’, ‘more’, ‘grep’, ‘notepad’, etc  Filtering – Positive and negative (“Artificial ignorance”)  Summarization and reports – “Top X of Y”  Visualization  Log indexing and searching  Correlation – Rule-based and other  Log data mining
  • 9. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 9Confidential | From Log Analysis to Log Management  Threat protection and discovery  Incident response  Forensics, “e-discovery” and litigation support  Regulatory compliance  Internal policies and procedure compliance  Internal and external audit support  IT system and network troubleshooting  IT performance management
  • 10. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 10Confidential | Looks Complicated?! No Wonder People Make Mistakes …
  • 11. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 11Confidential | Six Mistakes of Log Analysis and Log Management 0. Not logging at all. 1. Not looking at the logs 2. Storing logs for too short a time 3. Prioritizing the log records before collection 4. Ignoring the logs from applications 5. Only looking for “known bad” stuff
  • 12. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 12Confidential | Mistake 0: Not Logging AT ALL … … and its aggravated version: “… and not knowing that you don’t”  No logging? -> well, no logs for incident response, audits, compliance Got logs? If your answer is ‘NO”, don’t listen further: run and enable logging right now!
  • 13. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 13Confidential | Example: Oracle  Defaults: – minimum system logging – minimum database server access – no data access logging  So, where is … – data access audit – schema and data change audit – configuration change audit
  • 14. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 14Confidential | Mistake 1: Not looking at logs  Collection of logs has value!  But review boosts the value 10-fold (numbers are estimates )  More in-depth analysis boosts it a lot more!  Two choices here … – Review after an incident – Ongoing review
  • 15. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 15Confidential | Example Log Review Priorities 1. DMZ NIDS 2. DMZ firewall 3. DMZ servers with applications 4. Critical internal servers 5. Other servers 6. Select critical application 7. Other applications
  • 16. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 16Confidential | Mistake 2: Storing Logs For Too Short A Time  You are saying you HAD logs? And how is it useful?  Retention question is a hard one. Truly, nobody has the answer! – Seven years? A year? 90 days? A week? Until the disk runs out?  Common: 90 days online and up to 1-3 years “nearline” or offline
  • 17. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 17Confidential | Also A Mistake: Storing Logs for TOO LONG?!  Retention = storage + destruction  Why DESTROY LOGS? – Privacy regulations (mostly EU) – Litigation risk management – System resource utilization
  • 18. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 18Confidential | Example Retention Strategy Type + network + storage tier  IDS + DMZ + online = 90 days  Firewall + DMZ + online = 30 days  Servers + internal + online = 90 days  ALL + DMZ + archive = 3 years  Critical + internal + archive = 5 years  OTHER + internal + archive = 1 year
  • 19. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 19Confidential | Mistake 3: Deciding What’s Relevant Before Collection  How would you know what is … – … Security-relevant – … Compliance-relevant – … or will solve the problem you’d have TOMORROW!?  Also affects “forensic quality” of logs  Prioritization Challenge – Got ESP?   “Simple” – just grab everything!
  • 20. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 20Confidential | Example Common Logging Order Log everything Retain most everything Analyze enough Summarize and report on a subset Look at some Act in real-time on a few
  • 21. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 21Confidential | Mistake 4: Ignoring Logs from Applications  Firewall – Yes, Linux – Yes, Windows – Yes, NIDS and NIPS – Yes but …  Oracle - ?  SAP - ?  Your Application X – No Log standards are coming: MITRE CEE!
  • 22. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 22Confidential | Example: Jumbled Mess of Application Logs |22:01:40|BTC| 7|000|DDIC | |LC2|Systemerror when executing external command DB6_DATA_COLLECTOR on gneisenau () |22:02:32|BTC| 7|000|DDIC | |R49|Communication error, CPIC return code 020, SAP return code 456 |22:02:32|BTC| 7|000|DDIC | |R5A|> Conversation ID: 38910614 |22:02:32|BTC| 7|000|DDIC | |R64|> CPI-C function: CMSEND(SAP)
  • 23. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 23Confidential | Mistake 5: Looking for only the bad stuff  Correlation, filters, regex matching – oh, no!   Why such approaches? – You have to know what you are looking for!  Can we somehow just “see what we need to see”? – Data mining technology can help
  • 24. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 24Confidential | Conclusions: Mistakes Summary 0. Not logging at all. 1. Not looking at the logs 2. Storing logs for too short a time 3. Prioritizing the log records before collection 4. Ignoring the logs from applications 5. Only looking for “known bad” stuff
  • 25. Thursday, March 19, 2015 Mitigating Risk. Automating Compliance. 25Confidential | Thanks for Attending the Presentation Dr Anton Chuvakin, GCIH, GCFA Chief Logging Evangelist http://www.chuvakin.org Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Book on logs is coming soon! Also see http://chuvakin.blogspot.com

Editor's Notes

  1. “Six Mistakes of Log Management” Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Director of Product Management @ LogLogic, Inc Top Log Mistakes #1 not logging at all. #2 not looking at the logs #3 storing logs for too short a time #4 prioritizing the log records before collection #5 ignoring the logs from applications #6 only looking at what they know is bad Since I wrote my log mistakes paper a few years ago, the domain of log analysis changed a lot. Many factors affected it; among those are new regulatory compliance requirements, wider adoption of “best practice” and governance frameworks such as ISO, COBIT and ITIL as well as new technologies with their log files. New standards, such as NIST 800-92 Guide, have been created. Thus, I am updating this article with newly committed mistakes as well as new prospective on the old ones. Thus, this article, just like its predecessor, again covers the typical mistakes organizations make while approaching management of computer logs and other records produced by IT infrastructure components. As digital technology continues to spread (“A web-enabled fridge, anybody? Its just 8 (!) grand today, you know” ) and computers start playing even more important role in our lives (I do have a penchant for the obvious, don’t I? ), the records that they produce, a.k.a. logs, start to play bigger and bigger role. From firewalls and intrusion prevention systems to databases and enterprise applications to wireless access points and VOIP gateways, logs are being spewed forth at an every increasing pace. Both security and other IT components not only increase in numbers, but often come with more logging enabled out of the box. Example of that trend include Linux systems as well as web servers that now ship with increased level of logging. All those systems, both legacy and novel, are known to generate copious amounts of logs, audit trails, records and alerts, that beg for constant attention. Thus, many companies and government agencies are trying to set up repeatable log collection, centralization and analysis processes and tools. However, when planning and implementing log collection and analysis infrastructure, the organizations often discover that they are not realizing the full promise of such a system and, in fact, sometimes notice that the efficiency is not gained but lost as a result. This often happens due to the following common log analysis mistakes. We will start from the obvious, but unfortunately all too common even in this age of Sarbanes-Oxley and PCI. This mistake destroys all possible chances of benefiting from logs. It is the mistake #1: not logging at all. The more exciting flavor of this mistake is: “not logging and not even knowing it until it is too late.” How can it be “too late”, some say? “Its just logs!” Welcome to 2006! Not having “just logs” can lead to losing your income (PCI that contain logging requirements implies that violations might lead to your credit card processing privileges being cancelled by Visa or Mastercard and thus putting you out of business), reputation (somebody stole a few credit card number from your database, but the media reported that all of the 40 million credit card have been stolen since you were unable to prove otherwise) or even your freedom (see various Sarbanes-Oxley horror stories in the media) Even better-prepared organizations fall for this one. Here is a recent example. Does your web server have logging enabled? Sure, it is a default option on both of the popular web servers: Apache and Microsoft IIS. Does your server operating system log messages? Sure, nobody cancelled /var/log/messages. But does your database? Oops! Default option in Oracle is to not do any audit logging. Maybe MS SQL fares better? Nope, same thing, you need to dig deep in the system to even start a moderate level of audit trail generation. Thus, to avoid this mistake one needs to sometimes go beyond the defaults and make sure that the software and hardware deployed does have some level of logging enabled. In case of Oracle, for example, it might boil down to making sure that the ‘audit_trail’ variable is set to ‘db’; for other systems it might be more complicated. #2 Not looking at the logs is the second mistake. While making sure that logs do exist and then collecting and storing them is important, it is only a means to an end – knowing what is going on in your environment and being able to respond to it as well as possibly predict what will happen later. Thus, once the technology is in place and logs are collected, there needs to be a process of ongoing monitoring and review that hooks into actions and possible escalations, if needed. In addition, personnel reviewing or monitoring logs should have enough information to be able to determine what they really mean and what – if any – action is required. It is worthwhile to note that some organizations take a half-step in the right direction: they only review logs (provided they didn’t commit the first mistake and they actually have something to review) after a major incident (be it a compromise, information leak or a mysterious server crash) and avoid ongoing monitoring and log review, often by quoting “the lack of resources”. This gives them the reactive benefit of log analysis, which is important, but fails to realize the proactive one – knowing when bad stuff is about to happen or become worse. For example, if you review logs, you might learn that the failover was activated on a firewall, and, even though the connection stayed on, the incident is certainly worth looking into. If you don’t and your network connectivity goes away, you’d have to rely on your ever-helpful logs in investigation why *both* failover devices went down … In fact, looking at logs proactively helps organizations to better realize the value of their existing network, security and system infrastructure. It is also critical to stress that some types of organizations have to look at log files and audit tracks due to regulatory pressure of some kind. For example, US HIPAA regulation compels medical organizations to establish audit record and analysis program (even though the enforcement action is notorious lacking). In the even more extreme case, PCI (Payment Card Industry) security standard has provisions for both log collection and log monitoring and periodic review, highlighting the fact that collection of logs does not stand on its own. #3 The third common mistake is storing logs for too short a time. This makes the security or IT operations team think they have all the logs needed for monitoring and investigation or troubleshooting and then leading to the horrible realization after the incident that all logs are gone due to their shortsighted retention policy. It often happens (especially in the case of insider attacks) that the incident is discovered a long time – sometimes many months - after the crime or abuse has been committed. One might save some money on storage hardware, but lose the tenfold due to regulatory fines. If low cost is critical, the solution is sometimes in splitting the retention in two parts: shorter-term online storage (that costs more) and long-term offline storage (that is much cheaper). A better three-tier approach is also common and resolves some of the limitations of the previous one. In this case, shorter-term online storage is complemented by a near-line storage where logs are still accessible and searchable. The oldest and the least relevant log records are offloaded to the third tier, such as tape or DVDs, where they can be stored inexpensively, but without any way to selectively access the needed logs. More specifically, one financial institution was storing logs online for 90 days, then in the near-line searchable storage for 2 years and then on tape for up to 7 years or even more. #4 The fourth mistake is related to log record prioritization. While people need a sense of priority to better organize their log analysis efforts, the common mistake nowadays is in prioritizing the log records before collection. In fact, even some “best practice” documents recommend only collecting “the important stuff.” But what is important? This is where the above guidance documents fall short by not specifying it in any useful form. While there are some approaches to the problem, all that I am aware of can lead to glaring holes in security posture or even undermine the regulatory compliance efforts. For example, many people would claim that network intrusion detection and prevention logs are inherently more important than, say, VPN concentrator logs. Well, it might be true in the world where external threats completely dominate the insider abuse (i.e. not in this one). VPN logs, together with server and workstation logs, is what you would most likely need to conduct an internal investigation about the information leak or even a malware infection. Thus, similar claims about the elevated importance of whatever other log type can be similarly disputed, which would lead us to a painful realization that you do need to collect everything. But can you? Before you answer this, try to answer whether you can make the right call on which log is more important even before seeing it and this problem will stop looking unsolvable. In fact, there are cost-effective solutions to achieve just that. The mistake #5 is in ignoring the logs from applications, by only focusing on the perimeter and internal network devices and possibly also servers, but not going “higher up the stack” to look at the application logging. The realm of enterprise applications ranges from SAPs and PeopleSofts of the worlds to small homegrown applications, which nevertheless handle mission-critical processes for many enterprises. Legacy applications, running on mainframes and midrange systems, are out there as well, often running the core business processes as well. The availability and quality of logs differs wildly across the application, ranging from missing (the case for many home-grown applications) to extremely detailed and voluminous (the case for many mainframe applications). Lack of common logging standards and even of logging guidance for software developers lead to many challenges with application logs. Despite the challenges, one needs to make sure that the application logs are collected and made available for analysis as well as for longer term-retention. This can be accomplished by configuring your log management software to collect them and by establishing a log review policy, both for the on-incident review and periodic proactive log review. #6 Even the most advanced and mature organizations fall into the pitfall of the sixth error. It is sneaky and insidious, and can severely reduce the value of a log analysis project. It occurs when organization is only looking at what they know is bad in the logs. Indeed, a vast majority of open source and some commercial tools are set up to filter and look for bad log lines, attack signatures, critical events, etc. For example, “swatch” is a classic free log analysis tool that is powerful, but only at one thing: looking for defined bad things in log files. Moreover, when people talk about log analysis they usually mean sifting through logs looking for things of note. However, to fully realize the value of log data one has to take it to the next level to log mining: actually discovering things of interest in log files without having any preconceived notion of ‘what we need to find’. It sounds obvious - how can we be sure that we know of all the possible malicious behavior in advance – but it disregarded so often. Sometimes, it is suggested that it is simpler to just list all the known good things and then look for the rest. It sounds like a solution, but such task is not only onerous, but also thankless: it is usually even harder to list all the good things than it is to list all the bad things that might happen on a system or network. So many different things occur, malfunction or misbehave, that weeding out attack traces just by listing all the possibilities is not effective. A more intelligent approach is needed! Some of the data mining (also called “knowledge discovery in databases” or KDD) and visualization methods actually work on log data with great success. They allow organizations to look for real anomalies in log data, beyond ‘known bad’ and ‘known good’. To conclude, avoiding the above six mistakes we covered will take your log analysis program to a next level and enhance the value of the existing security and logging infrastructures. Dr Anton Chuvakin, GCIA, GCIH, GCFA (http://www.chuvakin.org) is a recognized security expert and book author. He currently works at LogLogic, where he is involved with defining and executing on a product vision and strategy, driving the product roadmap, conducting research as well as assisting key customers with their LogLogic implementations. He was previously a Security Strategist with a security information management company. He is an author of a book &amp;quot;Security Warrior&amp;quot; and a contributor to &amp;quot;Know Your Enemy II&amp;quot;, &amp;quot;Information Security Management Handbook&amp;quot;, &amp;quot;Hacker&amp;apos;s Challenge 3&amp;quot; and the upcoming book on PCI. Anton also published numerous papers on a broad range of security subjects. In his spare time he maintains his security portal http://www.info-secure.org and several blogs.
  2. Abstract: This presentation will cover operational security challenges that organizations face while deploying log and alert collection and analysis infrastructure, highlighting the top five most common mistakes organizations make in this process, including: not storing logs long enough to comply with government regulation and mandates, not preserving the forensic quality of the logs, only looking for known &amp;quot;bad records.&amp;quot; From there, the session will dive into how to avoid these, and other, mistakes. Additionally, it will provide tips and tricks for how government users can get the most value out of various log files generated by systems, applications and security devices.
  3. I did mention security data, events, etc on the previous slides. But what am I really talking about? In other words, what do we LOG and MONITOR? What is called “security data” in this presentation consists of various audit records (left), generated by various devices and softwares (right). It should be noted that business applications also generate security data, such as by recording access decisions or generating messages indicative of exploitation attempts.
  4. Notice even though each of these examples are from different sources, all have the fabulous five data: Time Sending machine Sending process or program Severity Message
  5. This slide summarizes the methods and techniques for making sense of logs that we will cover on the next few slides. They range from trivial perusing of logs all the way into machine intelligence and data mining. Manual log analysis includes using such tools as ‘tail’, ‘more’, “notepad”, ‘vi’, etc to look at log files and try to understand them. Filtering logs is the next techniques with two variations: Positive and negative (“artificial ignorance”). Positive filtering is trying to focus on the “bad” things that one need to to see, investigate and then act on: attacks, failures, etc. Negative filtering, which Marcus Ranum called “artificial ignorance” (as an opposite of artificial intelligence) is about looking for all the stuff you know is normal (looking for “good”) then throwing it away and focusing on what is left. The latter is often much more effective since it is hard to know the “bad” logs up front. Summarization and reports is the mainstay of log analysis: “Top Connection by Bandwidth”, “Top Attacks by Country”, “Top Users with Authentication Failures” and other reports based on logs is what most people think about when they think about “log analysis.” Simple visualization: is this picture “…worth a thousand words?” The answer is “sometimes” since many visualization methods actually confuse, not clarify the data. Visualization techniques range from very simple (such as pie charts and bar charts) to maps, trees and other advanced visual methods. Log searching is pretty much “googing” the logs: most people who collected vast amounts of log data and now have to resort to ‘grep’ to analyze the logs knows this method and the frustrations it brings. Correlation is typically understood to be “rule-based” correlation (even though there are other “correlation” types). The best way to understand is to note that there is “correlation” (small ‘c’ – which typically means using rules to tie log messages together) and “Correlation” (big ‘C’ – which means any method for looking at logs in relationship to each other) Using Log Data mining methods for looking at logs are still on the drawing boards. DM methods will – hopefully AND probably! – allow future log analysts to generate conclusion automatically from raw data and move even more of the log analysis tasks to the systems from human brains … This slide summarizes the methods and techniques for making sense of logs that we will cover on the next few slides. They range from trivial perusing of logs all the way into machine intelligence and data mining. Manual log analysis includes using such tools as ‘tail’, ‘more’, “notepad”, ‘vi’, etc to look at log files and try to understand them. Filtering logs is the next techniques with two variations: Positive and negative (“artificial ignorance”). Positive filtering is trying to focus on the “bad” things that one need to to see, investigate and then act on: attacks, failures, etc. Negative filtering, which Marcus Ranum called “artificial ignorance” (as an opposite of artificial intelligence) is about looking for all the stuff you know is normal (looking for “good”) then throwing it away and focusing on what is left. The latter is often much more effective since it is hard to know the “bad” logs up front. Summarization and reports is the mainstay of log analysis: “Top Connection by Bandwidth”, “Top Attacks by Country”, “Top Users with Authentication Failures” and other reports based on logs is what most people think about when they think about “log analysis.” Simple visualization: is this picture “…worth a thousand words?” The answer is “sometimes” since many visualization methods actually confuse, not clarify the data. Visualization techniques range from very simple (such as pie charts and bar charts) to maps, trees and other advanced visual methods. Log searching is pretty much “googing” the logs: most people who collected vast amounts of log data and now have to resort to ‘grep’ to analyze the logs knows this method and the frustrations it brings. Correlation is typically understood to be “rule-based” correlation (even though there are other “correlation” types). The best way to understand is to note that there is “correlation” (small ‘c’ – which typically means using rules to tie log messages together) and “Correlation” (big ‘C’ – which means any method for looking at logs in relationship to each other) Using Log Data mining methods for looking at logs are still on the drawing boards. DM methods will – hopefully AND probably! – allow future log analysts to generate conclusion automatically from raw data and move even more of the log analysis tasks to the systems from human brains …
  6. All those require quick and intelligent access to logs! SECURITY + OPS + COMPLIANCE
  7. It is the mistake #1: not logging at all. The more exciting flavor of this mistake is: “not logging and not even knowing it until it is too late.” How can it be “too late”, some say? “Its just logs!” Welcome to 2006! Not having “just logs” can lead to losing your income (PCI that contain logging requirements implies that violations might lead to your credit card processing privileges being cancelled by Visa or Mastercard and thus putting you out of business), reputation (somebody stole a few credit card number from your database, but the media reported that all of the 40 million credit card have been stolen since you were unable to prove otherwise) or even your freedom (see various Sarbanes-Oxley horror stories in the media) Even better-prepared organizations fall for this one. Here is a recent example. Does your web server have logging enabled? Sure, it is a default option on both of the popular web servers: Apache and Microsoft IIS. Does your server operating system log messages? Sure, nobody cancelled /var/log/messages. But does your database? Oops! Default option in Oracle is to not do any audit logging. Maybe MS SQL fares better? Nope, same thing, you need to dig deep in the system to even start a moderate level of audit trail generation. Thus, to avoid this mistake one needs to sometimes go beyond the defaults and make sure that the software and hardware deployed does have some level of logging enabled. In case of Oracle, for example, it might boil down to making sure that the ‘audit_trail’ variable is set to ‘db’; for other systems it might be more complicated.
  8. #2 Not looking at the logs is the second mistake. While making sure that logs do exist and then collecting and storing them is important, it is only a means to an end – knowing what is going on in your environment and being able to respond to it as well as possibly predict what will happen later. Thus, once the technology is in place and logs are collected, there needs to be a process of ongoing monitoring and review that hooks into actions and possible escalations, if needed. In addition, personnel reviewing or monitoring logs should have enough information to be able to determine what they really mean and what – if any – action is required. It is worthwhile to note that some organizations take a half-step in the right direction: they only review logs (provided they didn’t commit the first mistake and they actually have something to review) after a major incident (be it a compromise, information leak or a mysterious server crash) and avoid ongoing monitoring and log review, often by quoting “the lack of resources”. This gives them the reactive benefit of log analysis, which is important, but fails to realize the proactive one – knowing when bad stuff is about to happen or become worse. For example, if you review logs, you might learn that the failover was activated on a firewall, and, even though the connection stayed on, the incident is certainly worth looking into. If you don’t and your network connectivity goes away, you’d have to rely on your ever-helpful logs in investigation why *both* failover devices went down … In fact, looking at logs proactively helps organizations to better realize the value of their existing network, security and system infrastructure. It is also critical to stress that some types of organizations have to look at log files and audit tracks due to regulatory pressure of some kind. For example, US HIPAA regulation compels medical organizations to establish audit record and analysis program (even though the enforcement action is notorious lacking). In the even more extreme case, PCI (Payment Card Industry) security standard has provisions for both log collection and log monitoring and periodic review, highlighting the fact that collection of logs does not stand on its own.
  9. They went from “high external risk” to inside and from servers to desktops
  10. Mandatory 7 years is a myth! #3 The third common mistake is storing logs for too short a time. This makes the security or IT operations team think they have all the logs needed for monitoring and investigation or troubleshooting and then leading to the horrible realization after the incident that all logs are gone due to their shortsighted retention policy. It often happens (especially in the case of insider attacks) that the incident is discovered a long time – sometimes many months - after the crime or abuse has been committed. One might save some money on storage hardware, but lose the tenfold due to regulatory fines. If low cost is critical, the solution is sometimes in splitting the retention in two parts: shorter-term online storage (that costs more) and long-term offline storage (that is much cheaper). A better three-tier approach is also common and resolves some of the limitations of the previous one. In this case, shorter-term online storage is complemented by a near-line storage where logs are still accessible and searchable. The oldest and the least relevant log records are offloaded to the third tier, such as tape or DVDs, where they can be stored inexpensively, but without any way to selectively access the needed logs. More specifically, one financial institution was storing logs online for 90 days, then in the near-line searchable storage for 2 years and then on tape for up to 7 years or even more.
  11. NIST 800-92 guide is confusing on that! #4 The fourth mistake is related to log record prioritization. While people need a sense of priority to better organize their log analysis efforts, the common mistake nowadays is in prioritizing the log records before collection. In fact, even some “best practice” documents recommend only collecting “the important stuff.” But what is important? This is where the above guidance documents fall short by not specifying it in any useful form. While there are some approaches to the problem, all that I am aware of can lead to glaring holes in security posture or even undermine the regulatory compliance efforts. For example, many people would claim that network intrusion detection and prevention logs are inherently more important than, say, VPN concentrator logs. Well, it might be true in the world where external threats completely dominate the insider abuse (i.e. not in this one). VPN logs, together with server and workstation logs, is what you would most likely need to conduct an internal investigation about the information leak or even a malware infection. Thus, similar claims about the elevated importance of whatever other log type can be similarly disputed, which would lead us to a painful realization that you do need to collect everything. But can you? Before you answer this, try to answer whether you can make the right call on which log is more important even before seeing it and this problem will stop looking unsolvable. In fact, there are cost-effective solutions to achieve just that.
  12. The mistake #5 is in ignoring the logs from applications, by only focusing on the perimeter and internal network devices and possibly also servers, but not going “higher up the stack” to look at the application logging. The realm of enterprise applications ranges from SAPs and PeopleSofts of the worlds to small homegrown applications, which nevertheless handle mission-critical processes for many enterprises. Legacy applications, running on mainframes and midrange systems, are out there as well, often running the core business processes as well. The availability and quality of logs differs wildly across the application, ranging from missing (the case for many home-grown applications) to extremely detailed and voluminous (the case for many mainframe applications). Lack of common logging standards and even of logging guidance for software developers lead to many challenges with application logs. Despite the challenges, one needs to make sure that the application logs are collected and made available for analysis as well as for longer term-retention. This can be accomplished by configuring your log management software to collect them and by establishing a log review policy, both for the on-incident review and periodic proactive log review.
  13. #6 Even the most advanced and mature organizations fall into the pitfall of the sixth error. It is sneaky and insidious, and can severely reduce the value of a log analysis project. It occurs when organization is only looking at what they know is bad in the logs. Indeed, a vast majority of open source and some commercial tools are set up to filter and look for bad log lines, attack signatures, critical events, etc. For example, “swatch” is a classic free log analysis tool that is powerful, but only at one thing: looking for defined bad things in log files. Moreover, when people talk about log analysis they usually mean sifting through logs looking for things of note. However, to fully realize the value of log data one has to take it to the next level to log mining: actually discovering things of interest in log files without having any preconceived notion of ‘what we need to find’. It sounds obvious - how can we be sure that we know of all the possible malicious behavior in advance – but it disregarded so often. Sometimes, it is suggested that it is simpler to just list all the known good things and then look for the rest. It sounds like a solution, but such task is not only onerous, but also thankless: it is usually even harder to list all the good things than it is to list all the bad things that might happen on a system or network. So many different things occur, malfunction or misbehave, that weeding out attack traces just by listing all the possibilities is not effective. A more intelligent approach is needed! Some of the data mining (also called “knowledge discovery in databases” or KDD) and visualization methods actually work on log data with great success. They allow organizations to look for real anomalies in log data, beyond ‘known bad’ and ‘known good’.