Six Mistakes of Log Management 2008

Slide 1: Dr Anton Chuvakin, GCIA, GCIH, GCFA Chief Logging Evangelist LogLogic, Inc Six Mistakes of Security Log Management LogLogic Confidential 1 Wednesday, April 23, 2008

Slide 2: Summary  System, Network and Security Logs  Why Look at Logs?  Brief Log Analysis Overview  From Log Analysis to Log Management  Log Mistakes: from 0 to 6 Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 2

Slide 3: Log Data Overview What logs? From Where?  Audit logs  Firewalls/intrusion prevention  Transaction logs  Routers/switches  Intrusion logs  Intrusion detection  Connection logs  Servers, desktops, mainframes  System performance records  Business applications  User activity logs  Databases  Various alerts and other  Anti-virus messages  VPNs Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 3

Slide 4: Login? Logon? Log in? <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system- warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53) <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon acc ount: POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC000006 A 4574 Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 4

Slide 5: “Arrgh! Why Don’t We Just Ignore’Em?” Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 5

Slide 6: Log Management Mandate and Regulations Regulations Mandates Controls Require LMI Demand It Require it  SOX  FIS MA  PCI  S LAs  COBIT  ITIL  GLBA  JPA  HIPAA  ISO  NIST 800-53  PCI : Requirement 10  COBIT 4  Capture audit records and beyond  Provide audit trail for root-cause analysis  Regularly review audit records  Logging and user activities for unusual activity and tracking are critical  Use logging to detect unusual or violations abnormal activities  Automate and secure audit trails  Automatically process audit for event reconstruction  Regularly review access, privileges, records changes  Review logs daily  Protect audit information from  Verify backup completion  Retain audit trail history for unauthorized deletion at least one year  Retain audit logs  ISO17799  Maintain audit logs for system access and use, changes, faults, corrections, capacity demands  Review the results of monitoring activities regularly and ensure the accuracy of logs “Get fined, Get “Lose Customers, “Get fined, Go To Jail” Sanctioned” Reputation, Revenue or Job” Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 6

Slide 7: Also: NIST 800-92 “This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “ Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 7

Slide 8: So, How Do People Do It? Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 8

Slide 9: Log Analysis Basics  Manual – ‘Tail’, ‘more’, ‘grep’, ‘notepad’, etc  Filtering – Positive and negative (“Artificial ignorance”)  Summarization and reports – “Top X of Y”  Visualization  Log indexing and searching  Correlation – Rule-based and other  Log data mining Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 9

Slide 10: From Log Analysis to Log Management  Threat protection and discovery  Incident response  Forensics, “e-discovery” and litigation support  Regulatory compliance  Internal policies and procedure compliance  Internal and external audit support  IT system and network troubleshooting  IT performance management Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 10

Slide 11: Log Management Lifecycle Files, syslog, other Share Collect Secure “As needed “ basis Make Alert SNMP, Email, etc Conclusions Search Report Store Search, Report and Analytics Immutable Logs Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 11

Slide 12: Looks Complicated?! No Wonder People Make Mistakes … Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 12

Slide 13: Seven Mistakes of Log Analysis and Management 0. Not logging at all. 1. Not looking at the logs 2. Storing logs for too short a time 3. Prioritizing the log records before collection 4. Ignoring the logs from applications 5. Approaching logs in a siloed fashion Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 13

Slide 14: Mistake 0: Not Logging AT ALL … … and its aggravated version: “… and not knowing that you don’t”  No logging? -> well, no logs for incident response, audits, compliance Got logs? If your answer is ‘NO”, don’t listen further: run and enable logging right now! Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 14

Slide 15: Example: Oracle  Defaults: – minimum system logging – minimum database server access – no data access logging  So, where is … – data access audit – schema and data change audit – configuration change audit Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 15

Slide 16: Mistake 1: Not looking at logs  Collection of logs has value!  But review boosts the value 10-fold ( numbers are estimates )  More in-depth analysis boosts it a lot more!  Two choices here … – Review after an incident – Ongoing review Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 16

Slide 17: Example Log Review Priorities 1. DMZ NIDS 2. DMZ firewall 3. DMZ servers with applications 4. Critical internal servers 5. Other servers 6. Select critical application 7. Desktops 8. Other applications Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 17

Slide 18: Mistake 2: Storing logs for too short a time  You are saying you HAD logs? And how is it useful?  Retention question is a hard one. Truly, nobody has the answer! – Seven years? A year? 90 days? A week? Until the disk runs out?  Common: 90 days online and up to 1-3 years “nearline” or offline Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 18

Slide 19: Also A Mistake: Storing Logs for TOO LONG?!  Retention = storage + destruction  Why DESTROY LOGS? – Privacy regulations – Litigation risk management – Due diligence and security policy – System resource utilization Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 19

Slide 20: Example Retention Strategy Type + network + storage tier  IDS + DMZ + online = 90 days  Firewall + DMZ + online = 30 days  Servers + internal + online = 90 days  ALL + DMZ + archive = 3 years  Critical + internal + archive = 5 years  OTHER + internal + archive = 1 year Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 20

Slide 21: Quiz: Name Which Are Security Relevant? 1. System or software startup, shutdown, restart, and abnormal termination (crash) 2. Various thresholds being exceeded or reaching dangerous levels such as disk space full, memory exhausted, or processor load too high 3. Hardware health messages that the system can troubleshoot or at least detect and log 4. User access to the system such as remote (telnet, ssh, etc.) and local login, network access (FTP) initiated to and from the system, failed and successful 5. User access privilege changes such as the su command—both failed and successful 6. User credentials and access right changes, such as account updates, creation, and deletion—both failed and successful 7. System configuration changes and software updates—both failed and successful Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 21

Slide 22: Mistake 3: Deciding What’s Relevant Before Collection  How would you know what is … – … Security-relevant – … Compliance-relevant – … or will solve the problem you’d have TOMORROW!?  Also affects “forensic quality” of logs  Prioritization Challenge – Got ESP?   Simple – just grab everything! Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 22

Slide 23: Example Common Logging Order Log everything Retain most everything Analyze enough Summarize and report on a subset Look at some Act in real-time on a few Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 23

Slide 24: Mistake 4: Ignoring Logs from Applications  Firewall – Yes, Linux – Yes, Windows – Yes. NIDS – Yes but …  Oracle - ?  SAP - ?  Your Application X – No? Log standards are coming: MITRE CEE! Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 24

Slide 25: Example: Jumbled Mess of SAP Logs |22:01:40|BTC| 7|000|DDIC | |LC2|Systemerror when executing external command DB6_DATA_COLLECTOR on gneisenau () |22:02:32|BTC| 7|000|DDIC | |R49|Communication error, CPIC return code 020, SAP return code 456 |22:02:32|BTC| 7|000|DDIC | |R5A|> Conversation ID: 38910614 |22:02:32|BTC| 7|000|DDIC | |R64|> CPI-C function: CMSEND(SAP) |22:02:32|BTC| 7|000|DDIC Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 | |LC2|Systemerror when 25

Slide 26: Mistake 5: Siloed Approach to Log Management  Imagine… – Database logs -> database monitoring system – Syslog -> syslog server – Windows log -> stay where they are – Firewall logs -> PIX logger – Application logs -> don’t exist   What about forensics, incident response, audit?  How do you analyze the activities across systems? Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 26

Slide 27: Example Platform vs Siloes Operational IT & Network Identity Governance & Operational IT & Network Identity Governance & Security Operations Management Compliance Security Operations Management Compliance Log Tool Log S ilo ? ? ? ? ???? ? ? ? ?? ? ? ? ???? ? ? ? Log Jam ? ? ?? ? Log platform ? ? ? ?? ? ? ???? ? ? ? ??? ? ? ? LOGS ?? ?? ? Network Servers Databases Homegrown Network Servers Databases Homegrown Applications Applications Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 27

Slide 28: Conclusions  Now you know: – What are the logs? – Where they come from? – Why look at them? – How people do it? – What are some of the relevant regulations? – How to deal with them?  And how to AVOID MISTAKES in dealing with logs! Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 28

Slide 29: Seven Mistakes of Log Analysis and Management 0. Not logging at all. 1. Not looking at the logs 2. Storing logs for too short a time 3. Prioritizing the log records before collection 4. Ignoring the logs from applications 5. Only looking at what you know is bad 6. Approaching logs in a siloed fashion Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 29

Slide 30: Thanks for Attending the Presentation Dr Anton Chuvakin, GCIA, GCIH, GCFA Chief Logging Evangelist http://www.chuvakin.org Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Book on logs is coming soon! Also see http://chuvakin.blogspot.com Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 30

Slide 31: Further Reading  Check out my longer paper “Mistakes of Log Management”, published at http://www. infosecwriters.org  Other fun reading – section on log management on my blog http://chuvakin.blogspot.com/search/label/log%20manag  My chapter on logging for PCI from “PCI Compliance” book (posted on Syngress web site) Mitigating Risk. Automating Compliance. Confidential | Wednesday, April 23, 2008 31