SlideShare a Scribd company logo

Making Logs Sexy Again: Can We Finally Lose The Regexes

•
•
Anton Chuvakin
Anton Chuvakin

Making Logs Sexy Again: Can We Finally Lose The Regexes?, presented at DeepSec 2008 in Vienna, Austria

Technology
1 of 41
Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],C O N F I D E N T I A L
Who is Anton? ,[object Object],[object Object]
First: WTH is Logs? ,[object Object],C O M P A N Y C O N F I D E N T I A L II. A huge messy pile of undocumented, unstructured or poorly structured “ stuff ” that might or might not mean anything – at all?
WTF?!! ,[object Object],[object Object],[object Object],[object Object]
Second: What Do We Do With Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
[object Object],C O M P A N Y C O N F I D E N T I A L
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Manual ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Filtering ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Correlation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Management “Grand Challenges?” ,[object Object],[object Object],[object Object],[object Object]
GC1 – Log Chaos ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
GC2 – Secure and Reliable Log Collection ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC3 - Log Parsing and Regexs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC4 – Automated Meaning Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC5 – “Fuzzy” Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],C O M P A N Y C O N F I D E N T I A L
Handling The Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preliminary DATA Requirements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What Do We “Mine” for? ,[object Object],[object Object],[object Object],[object Object]
Simple Example ,[object Object],[object Object],[object Object],[object Object]
Deeper into interesting - I ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Can You Guess What Happened?!
Deeper into interesting - II ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where Is The DATA? ,[object Object],[object Object],[object Object],[object Object]
“ Search+” ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Text Mining Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Keywords to Meaning? ,[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Visualization ,[object Object],[object Object],[object Object],[object Object]
Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
Finally, Can We Make Better Logs? CEE = Syntax + Vocabulary + Transport + Log Recommendations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions and CALL TO ACTION! ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Thank You! ,[object Object],[object Object],[object Object],[object Object],[object Object]
Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
“ Top 11 Reasons People Hate Logs” ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining LogsAnton Chuvakin
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = AccountabilityAnton Chuvakin
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network securityInformation Technology Society Nepal
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs InsidersAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 
Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 

More Related Content

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes (20)

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining Logs
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = Accountability
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logs
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network security
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs Insiders
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data Logistics
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log Standardization
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMA
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real World
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application logging
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 

More from Anton Chuvakin

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinAnton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...Anton Chuvakin
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinAnton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothAnton Chuvakin
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022Anton Chuvakin
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton ChuvakinAnton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsAnton Chuvakin
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC Anton Chuvakin
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020Anton Chuvakin
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton Chuvakin
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)Anton Chuvakin
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinAnton Chuvakin
 

More from Anton Chuvakin (20)

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less Operations
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in Brief
 
Generic siem how_2017
Generic siem how_2017Generic siem how_2017
Generic siem how_2017
 
Tips on SIEM Ops 2015
Tips on SIEM Ops 2015Tips on SIEM Ops 2015
Tips on SIEM Ops 2015
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics Presentation
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
 
SIEM Primer:
SIEM Primer:SIEM Primer:
SIEM Primer:
 

Recently uploaded

Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfjimielynbastida
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Key Features Of Token Development (1).pptx
Key Features Of Token Development (1).pptxKey Features Of Token Development (1).pptx
Key Features Of Token Development (1).pptxLBM Solutions
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfngoud9212
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Neo4j
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGAPIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGMarianaLemus7
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 

Recently uploaded (20)

Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdf
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Key Features Of Token Development (1).pptx
Key Features Of Token Development (1).pptxKey Features Of Token Development (1).pptx
Key Features Of Token Development (1).pptx
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdf
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDGAPIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April Automation LPDG
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptxVulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
 

Making Logs Sexy Again: Can We Finally Lose The Regexes

  • 1. Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29. Can You Guess What Happened?!
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36. Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
  • 37.
  • 38.
  • 39.
  • 40. Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
  • 41.

Editor's Notes

  1. “ Will be high-level, but the subject is definitely pretty DEEP… so DeepSec” Making Logs Sexy Again: Can We Finally Lose The Regexes?   As I was talking to one conference organizer a few years ago and he complained that there hasn’t been anything new in the field of automated log analysis for many years and that we have not progressed much since regular expressions (regexes) were used for extracting bits of useful information from logs. This talk is intended to change that! It will go through a set of approaches to “log problem”: some completely new (text mining of logs), some borrowed from other fields (Bayesian methods for log analysis) and show how people can use them to solve practical problems using these new methods. Only real-world-tested analysis methods and only those that can be used by the audience will be shown (and, of course, only the “0technically cool” and novel ones!) Example questions that will be answered: are logs data or text? Which data mining methods works really well on logs (which is still simple enough to be used by the non-Ph.Ds)? Can you make sense of all the keywords and key phrases in logs? Does Bayes works for logs? Can you predict when the system will crash based on logs?