Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008


Published on

Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Title : "Logging for Incident Response and Forensics: Key Issues" Outline : "The presentation will cover the use of various system, network and security logs and audit trails in the incident response process , from methodology to practical case studies and tools. It will touch upon incident response practices and the role of logs in them , using logs for forensics and e-discovery as well as for incident detection. Recommended logging settings that organizations use to increase the productivity and effectiveness of their incident response process will be discussed.  In addition, a few key questions on using logs as forensic evidence will be discussed"
  • Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008

    1. 1. Logging for Incident Response and Forensics Key Issues Dr Anton Chuvakin Chief Logging Evangelist GOVCERT 2008 Symposium Sep 2008
    2. 2. Logs for Cybercrime Investigations <ul><li>A few thoughts to start us off … </li></ul><ul><li>All attackers leave traces. Period!  </li></ul><ul><li>It is just that you don’t always know what and where </li></ul><ul><li>And almost never know why… </li></ul><ul><li>Logs are the place to look, first </li></ul>
    3. 3. Outline <ul><li>Log and logging overview </li></ul><ul><li>A brief on Incident response and forensics </li></ul><ul><li>Logs in incident investigations </li></ul><ul><li>Just what is log forensics? </li></ul><ul><li>Conclusions and call to action! </li></ul>
    4. 4. Definitions <ul><li>Log = timed record of whatever activities occurring on an information system </li></ul><ul><li>Also: alert, “event”, alarm, message, audit record, etc </li></ul><ul><li>BTW: s tandard definitions are coming soon: CEE standard by MITRE </li></ul>
    5. 5. Log Data Overview <ul><li>Audit logs </li></ul><ul><li>Transaction logs </li></ul><ul><li>Intrusion logs </li></ul><ul><li>Connection logs </li></ul><ul><li>System performance records </li></ul><ul><li>User activity logs </li></ul><ul><li>Various alerts and other messages </li></ul><ul><li>Firewalls/intrusion prevention </li></ul><ul><li>Routers/switches </li></ul><ul><li>Intrusion detection </li></ul><ul><li>Servers, desktops, mainframes </li></ul><ul><li>Business applications </li></ul><ul><li>Databases </li></ul><ul><li>Anti-virus </li></ul><ul><li>VPNs </li></ul>What logs? From Where?
    6. 6. Login? Logon? Log in? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff: port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0    Logon account:  POWERUSER    Source Workstation: ENTERPRISE    Error Code: 0xC000006A     4574 <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from (2002-12-17 15:50:53)
    7. 7. Why Log Management? <ul><li>Threat protection and discovery </li></ul><ul><li>Incident response </li></ul><ul><li>Forensics , “e-discovery” and litigation support </li></ul><ul><li>Regulatory compliance </li></ul><ul><li>Internal policies and procedure compliance </li></ul><ul><li>Internal and external audit support </li></ul><ul><li>IT system and network troubleshooting </li></ul><ul><li>IT performance management </li></ul>
    8. 8. Select Open Source Logging Tools <ul><li>Log collection </li></ul><ul><ul><li>Syslog-ng, kiwi, ntsyslog, LASSO, DAD, Apache2syslog, etc </li></ul></ul><ul><li>Secure centralization </li></ul><ul><ul><li>Stunnel, ssh, free IPSec VPNs </li></ul></ul><ul><li>Pre-processing </li></ul><ul><ul><li>LogPP </li></ul></ul><ul><li>Storage </li></ul><ul><ul><li>MySQL or design your own </li></ul></ul><ul><li>Analysis </li></ul><ul><ul><li>SEC for correlation </li></ul></ul><ul><ul><li>OSSEC, OSSIM for [ some ] intelligence </li></ul></ul><ul><ul><li>Swatch, logwatch, logsentry, other match-n-bug  scripts </li></ul></ul>
    9. 9. Logs at Various Stage of Incident Response <ul><li>Preparation : verify controls, collect normal usage data, baseline, etc </li></ul><ul><li>Identification : detect an incident , confirm incident, etc <- really? </li></ul><ul><li>Containment : scope the damage, learn what else is “lost”, what else the attacker visited/tried, etc </li></ul><ul><li>Eradication : preserving logs for the future, etc </li></ul><ul><li>Recovery : confirming the restoration, etc </li></ul><ul><li>Follow-Up : logs for “peaceful” purposes (training, etc) as well as preventing the recurrence </li></ul>
    10. 10. <ul><li>Various Logs for Incident Response </li></ul>
    11. 11. Firewall Logs in Incident Response <ul><li>Proof of Connectivity </li></ul><ul><li>Proof of NO Connectivity </li></ul><ul><li>Scans </li></ul><ul><li>Malware: Worms, Spyware </li></ul><ul><li>Compromised Systems </li></ul><ul><li>Misconfigured Systems </li></ul><ul><li>Unauthorized Access and Access Attempts </li></ul>
    12. 12. Example: Firewall Logs in Place of Netflow <ul><li>Why Look at Firewall Logs During Incident Investigation? </li></ul><ul><li>1990-2001 – to see what external threats got blocked (in, failure) </li></ul><ul><li>2002-2008 – to see what internal system got connected (out, success) </li></ul><ul><li>Thus, firewall logs is “poor man’s” netflow … </li></ul>
    13. 13. Recommendations <ul><li>Log denied connections </li></ul><ul><li>Log allowed connection </li></ul><ul><li>If cannot log all allowed connections, then log outbound allowed connections </li></ul><ul><li>Watch for rule changes </li></ul>
    14. 14. NIDS Logs in Incident Response <ul><li>Attack, Intrusion and Compromise Detection </li></ul><ul><li>Malware Detection: Worms, Viruses, Spyware, etc </li></ul><ul><li>Network Abuses and Policy Violations </li></ul><ul><li>Unauthorized Access and Access Attempts </li></ul><ul><li>Recon Activity </li></ul><ul><li>[NIPS] Blocked Attacks </li></ul>
    15. 15. Just A Thought… <ul><li>Q: How many serious incidents were discovered by an IDS/IPS? </li></ul><ul><li>A: Answers I’ve heard.. </li></ul><ul><li>0, 0, 0, 0 with an average of 0  </li></ul>
    16. 16. Server Logs in Incident Response <ul><li>Confirmed Access by an Intruder </li></ul><ul><li>Service Crashes and Restarts </li></ul><ul><li>Reboots </li></ul><ul><li>Password, Trust and Other Account Changes </li></ul><ul><li>System Configuration Changes </li></ul><ul><li>A World of Other Things  </li></ul>
    17. 17. Example: “Irrelevant, You Say” <ul><li>Using disk failures for IDS  </li></ul><ul><li>What is really there? </li></ul><ul><li>Is this OUR server? Well … </li></ul><ul><li>“Detection by catastrophe” </li></ul><ul><li>Is CNN you IDS ? </li></ul>
    18. 18. Recommendations <ul><li>On a typical Unix system log: </li></ul><ul><ul><li>Syslog = usually defaults are sensible </li></ul></ul><ul><ul><li>Select file access = via kernel logging </li></ul></ul><ul><ul><li>Select process execution = via kernel logging </li></ul></ul><ul><li>On a typical Windows server: </li></ul><ul><ul><li>Authentication = failure/success </li></ul></ul><ul><ul><li>Account changes = failure/success </li></ul></ul><ul><ul><li>Privilege use = failure/success </li></ul></ul><ul><ul><li>Others as needed (see MS Recommended Audit Policy ) </li></ul></ul>
    19. 19. Database Logs in Incident Response <ul><li>Database and Schema Modifications </li></ul><ul><li>Data and Object Modifications </li></ul><ul><li>User and Privileged User Access </li></ul><ul><li>Failed User Access </li></ul><ul><li>Failures, Crashes and Restarts </li></ul>LOOK AT LOGS! 
    20. 20. Example: And What is NOT Stolen? <ul><li>Supposedly , all of ChoicePoint 40 mil cards were not stolen… They just couldn’t prove they weren’t. </li></ul><ul><li>Database logs as a way of non-intrusion detection (or, rather, confirmation) </li></ul><ul><li>On the other hand, TJMaxx 47 mil cards were stolen … oops  </li></ul>
    21. 21. Recommendations <ul><li>Default: almost nothing! </li></ul><ul><li>Changes/deletions of SOME data </li></ul><ul><li>Access to SMALLER SET of data </li></ul><ul><li>Backups </li></ul><ul><li>Schema changes and other structural changes </li></ul><ul><li>Authentication success/failure (over all channels) </li></ul><ul><li>Privilege changes </li></ul>
    22. 22. Proxy Logs in Incident Response <ul><li>Internet access patterns </li></ul><ul><li>Policy violations </li></ul><ul><li>Accidental/malicious information disclosure </li></ul><ul><li>Malware: spyware, trojans, etc </li></ul><ul><li>Outbound web-hacking </li></ul>
    23. 23. Example: Proxy Logs vs Uploads <ul><li>How to look for data uploads? </li></ul><ul><li>HTTP method (logged as &quot;cs-method&quot; by BlueCoat) = POST   </li></ul><ul><li>For information uploads: content type (logged as &quot;RS(content-type)&quot; by BlueCoat) = anything but &quot;html/text&quot; (which is the type used for uploading  web form contents) - especially try content types  &quot; application/octet-stream &quot;, &quot; application/msword &quot;, &quot; application/powerpoint &quot;, &quot; application/vnd.ms-excel &quot;, </li></ul><ul><li>User-agent set to anything but the common ones (i.e. not Mozilla , iTunes , LiveUpdate , etc) or even to &quot;unknown“ (also check agent names matching messaging applications) </li></ul>
    24. 24. Antivirus Logs in Incident Response <ul><li>Virus Detection and Clean-up (or lack thereof!) </li></ul><ul><li>Failed and Successful Antivirus Signature Updates </li></ul><ul><li>Other Protection Failures and Issues </li></ul><ul><li>Antivirus Software Crashes and Terminations </li></ul>
    25. 25. Example: Sorry, I AM A Failure  <ul><li>System is compromised </li></ul><ul><li>Analysis reveals major vendor AV was running </li></ul><ul><li>AV logs shows that that there was a “detection-only” event with “left alone” as an action </li></ul><ul><li>Malware is tracked through other logs as well, including those on and off the 0wned box </li></ul>
    26. 26. <ul><li>Log Forensics </li></ul>
    27. 27. Logs and Forensics <ul><li>What Makes Your Incident Investigation a “Forensic” Investigation? </li></ul><ul><li>Incident Response vs Forensics </li></ul><ul><li>… and is the ‘vs’ really appropriate? </li></ul>
    28. 28. So, What is “Log Forensics” <ul><li>Log analysis is trying to make sense of system and network logs </li></ul><ul><li>“ Computer forensics is application of the scientific method to digital media in order to establish factual information for judicial review.” </li></ul><ul><li>So…. </li></ul><ul><li>Log Forensics = trying to make sense of system and network logs + in order to establish factual information for judicial review </li></ul><ul><li>Overall, a bizarro mix of science, technology and law </li></ul>
    29. 29. How Logs Help… Sometimes <ul><li>If logs are there , we can try to </li></ul><ul><li>… figure out who , where , what , when , how, etc </li></ul><ul><li>but </li></ul><ul><li>Who as a person or a system? </li></ul><ul><li>Is where spoofed? </li></ul><ul><li>When ? In what time zone? </li></ul><ul><li>How ? More like ‘how’d you think’… </li></ul><ul><li>What happened or what got recorded? </li></ul>
    30. 30. Who? <ul><li>Man vs machine: identity fight </li></ul><ul><li>Just who is ? Do you know him ? </li></ul><ul><li>Is jsmith.example.com a who ? </li></ul><ul><li>Is JSMITH at SMITHEXAMPLE ? </li></ul><ul><li>Is JSMITH authenticated by an RSA token at SMITHEXAMPLE and also logged to another system as “jsmith” a who ? </li></ul><ul><li>Is JSMITH authenticated by a fingerprint reader at SMITHEXAMPLE a who? </li></ul>
    31. 31. When? <ul><li>Got timestamp ?  - challenges to log timing! </li></ul><ul><li>Completely false timestamp in logs (BTW, today is Jan 1, 1969) </li></ul><ul><li>It’s always 5PM somewhere: which timezone are your logs in? </li></ul><ul><li>Are you in drift? Your clock might be ( those pesky seconds turn into minutes… ) </li></ul><ul><li>Syslog forwarder mysteries: “his” time vs “my” time </li></ul><ul><li>So, which one is right? Systems with two timestamps! </li></ul><ul><li>It got logged at 5:17AM. When did it happen ? Log lag! </li></ul>
    32. 32. Where? <ul><li>The attack came from which belongs to Guanjou Internet Alliance, Beijng China </li></ul><ul><li>The stolen data then went to which belongs to PakNet ISP in Karachi, Pakistan </li></ul><ul><li>Result : Romanian hackers attack!  </li></ul><ul><li>or </li></ul><ul><li>Result : it was a guy from the office on the 3 rd floor? </li></ul><ul><li>or … </li></ul>
    33. 33. What? <ul><li>“ False positives” – need I say more?  </li></ul><ul><li>Other “lying logs” include: </li></ul><ul><ul><li>System errors that are corrected automatically </li></ul></ul><ul><ul><li>Misinterpreted log messages ( Oh my, error 4632 strikes again! ) </li></ul></ul><ul><ul><li>Artificially inserted logs (ever used /usr/bin/logger ?) </li></ul></ul><ul><ul><li>Finally, logs that someone edited </li></ul></ul><ul><ul><ul><li>Added (log flooding) </li></ul></ul></ul><ul><ul><ul><li>Removed (log “cleaning”) </li></ul></ul></ul><ul><ul><ul><li>Changed (log corruption) </li></ul></ul></ul>
    34. 34. Logs Forensics Challenges: From US DoJ <ul><li>“ Computer Records and the Federal Rules of Evidence “ </li></ul><ul><li>“ First , parties may challenge the authenticity of both computer-generated and computer-stored records by questioning whether the records were altered, manipulated, or damaged after they were created. </li></ul><ul><li>Second , parties may question the authenticity of computer-generated records by challenging the reliability of the computer program that generated the records. </li></ul><ul><li>Third , parties may challenge the authenticity of computer-stored records by questioning the identity of their author.” </li></ul>
    35. 35. <ul><li>Case Study: Horror 1016 </li></ul>
    36. 36. Case Study: Horror of 1016 IMAGE CANNOT BE SHARED!
    37. 37. <ul><li>But I didn’t do it!!!!! </li></ul>
    38. 38. The Answer!
    39. 39. “Case 1016” Lessons <ul><li>Log reading IS NOT log analysis </li></ul><ul><li>Another angle to logs in court: log misinterpretation </li></ul><ul><li>A question that will never be answered: were they stupid or were they evil!? </li></ul>
    40. 40. Conclusions <ul><li>Turn ON Logging!!! </li></ul><ul><li>Make sure logs are there when you need them </li></ul><ul><li>When going into the incident-induced panic think ‘its all logged somewhere – we just need to dig it out’ </li></ul><ul><li>Review logs to know your environment! </li></ul><ul><li>Logs for Forensics </li></ul><ul><ul><li>Logs can tell you things, but are they “good evidence”? </li></ul></ul><ul><ul><li>Logs become evidence only if precautions are taken </li></ul></ul>
    41. 41. Thanks for Attending the Presentation! <ul><li>Dr Anton Chuvakin </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li>Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) </li></ul><ul><li>See http://www.info-secure.org for my papers, books, reviews and other security resources. Also see my blog http://www.SecurityWarrior.org </li></ul>