Search

Six Mistakes of Log Management 2008

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    “ Six Mistakes of Log Management” Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Director of Product Management @ LogLogic, Inc Top Log Mistakes #1 not logging at all. #2 not looking at the logs #3 storing logs for too short a time #4 prioritizing the log records before collection #5 ignoring the logs from applications #6 only looking at what they know is bad Since I wrote my log mistakes paper a few years ago, the domain of log analysis changed a lot. Many factors affected it; among those are new regulatory compliance requirements, wider adoption of “best practice” and governance frameworks such as ISO, COBIT and ITIL as well as new technologies with their log files. New standards, such as NIST 800-92 Guide, have been created. Thus, I am updating this article with newly committed mistakes as well as new prospective on the old ones. Thus, this article, just like its predecessor , again covers the typical mistakes organizations make while approaching management of computer logs and other records produced by IT infrastructure components. As digital technology continues to spread (“A web-enabled fridge , anybody? Its just 8 (!) grand today, you know”  ) and computers start playing even more important role in our lives (I do have a penchant for the obvious , don’t I?  ), the records that they produce, a.k.a. logs, start to play bigger and bigger role. From firewalls and intrusion prevention systems to databases and enterprise applications to wireless access points and VOIP gateways, logs are being spewed forth at an every increasing pace. Both security and other IT components not only increase in numbers, but often come with more logging enabled out of the box. Example of that trend include Linux systems as well as web servers that now ship with increased level of logging. All those systems, both legacy and novel, are known to generate copious amounts of logs, audit trails, records and alerts, that beg for constant attention. Thus, many companies and government agencies are trying to set up repeatable log collection, centralization and analysis processes and tools. However, when planning and implementing log collection and analysis infrastructure, the organizations often discover that they are not realizing the full promise of such a system and, in fact, sometimes notice that the efficiency is not gained but lost as a result. This often happens due to the following common log analysis mistakes. We will start from the obvious, but unfortunately all too common even in this age of Sarbanes-Oxley and PCI. This mistake destroys all possible chances of benefiting from logs. It is the mistake #1 : not logging at all. The more exciting flavor of this mistake is: “not logging and not even knowing it until it is too late.” How can it be “too late”, some say? “Its just logs!” Welcome to 2006! Not having “just logs” can lead to losing your income (PCI that contain logging requirements implies that violations might lead to your credit card processing privileges being cancelled by Visa or Mastercard and thus putting you out of business), reputation (somebody stole a few credit card number from your database, but the media reported that all of the 40 million credit card have been stolen since you were unable to prove otherwise) or even your freedom (see various Sarbanes-Oxley horror stories in the media) Even better-prepared organizations fall for this one. Here is a recent example. Does your web server have logging enabled? Sure, it is a default option on both of the popular web servers: Apache and Microsoft IIS. Does your server operating system log messages? Sure, nobody cancelled /var/log/messages . But does your database ? Oops! Default option in Oracle is to not do any audit logging. Maybe MS SQL fares better? Nope, same thing, you need to dig deep in the system to even start a moderate level of audit trail generation. Thus, to avoid this mistake one needs to sometimes go beyond the defaults and make sure that the software and hardware deployed does have some level of logging enabled. In case of Oracle, for example, it might boil down to making sure that the ‘ audit_trail’ variable is set to ‘db’; for other systems it might be more complicated. #2 Not looking at the logs is the second mistake. While making sure that logs do exist and then collecting and storing them is important, it is only a means to an end – knowing what is going on in your environment and being able to respond to it as well as possibly predict what will happen later. Thus, once the technology is in place and logs are collected, there needs to be a process of ongoing monitoring and review that hooks into actions and possible escalations, if needed. In addition, personnel reviewing or monitoring logs should have enough information to be able to determine what they really mean and what – if any – action is required. It is worthwhile to note that some organizations take a half-step in the right direction: they only review logs (provided they didn’t commit the first mistake and they actually have something to review) after a major incident (be it a compromise, information leak or a mysterious server crash) and avoid ongoing monitoring and log review, often by quoting “the lack of resources”. This gives them the reactive benefit of log analysis, which is important, but fails to realize the proactive one – knowing when bad stuff is about to happen or become worse. For example, if you review logs, you might learn that the failover was activated on a firewall, and, even though the connection stayed on, the incident is certainly worth looking into. If you don’t and your network connectivity goes away, you’d have to rely on your ever-helpful logs in investigation why *both* failover devices went down … In fact, looking at logs proactively helps organizations to better realize the value of their existing network, security and system infrastructure. It is also critical to stress that some types of organizations have to look at log files and audit tracks due to regulatory pressure of some kind. For example, US HIPAA regulation compels medical organizations to establish audit record and analysis program (even though the enforcement action is notorious lacking). In the even more extreme case, PCI (Payment Card Industry) security standard has provisions for both log collection and log monitoring and periodic review, highlighting the fact that collection of logs does not stand on its own. #3 The third common mistake is storing logs for too short a time . This makes the security or IT operations team think they have all the logs needed for monitoring and investigation or troubleshooting and then leading to the horrible realization after the incident that all logs are gone due to their shortsighted retention policy. It often happens (especially in the case of insider attacks) that the incident is discovered a long time – sometimes many months - after the crime or abuse has been committed. One might save some money on storage hardware, but lose the tenfold due to regulatory fines. If low cost is critical, the solution is sometimes in splitting the retention in two parts: shorter-term online storage (that costs more) and long-term offline storage (that is much cheaper). A better three-tier approach is also common and resolves some of the limitations of the previous one. In this case, shorter-term online storage is complemented by a near-line storage where logs are still accessible and searchable. The oldest and the least relevant log records are offloaded to the third tier, such as tape or DVDs, where they can be stored inexpensively, but without any way to selectively access the needed logs. More specifically, one financial institution was storing logs online for 90 days, then in the near-line searchable storage for 2 years and then on tape for up to 7 years or even more. #4 The fourth mistake is related to log record prioritization. While people need a sense of priority to better organize their log analysis efforts, the common mistake nowadays is in prioritizing the log records before collection. In fact, even some “best practice” documents recommend only collecting “the important stuff.” But what is important? This is where the above guidance documents fall short by not specifying it in any useful form. While there are some approaches to the problem, all that I am aware of can lead to glaring holes in security posture or even undermine the regulatory compliance efforts. For example, many people would claim that network intrusion detection and prevention logs are inherently more important than, say, VPN concentrator logs. Well, it might be true in the world where external threats completely dominate the insider abuse (i.e. not in this one). VPN logs, together with server and workstation logs, is what you would most likely need to conduct an internal investigation about the information leak or even a malware infection. Thus, similar claims about the elevated importance of whatever other log type can be similarly disputed, which would lead us to a painful realization that you do need to collect everything. But can you? Before you answer this, try to answer whether you can make the right call on which log is more important even before seeing it and this problem will stop looking unsolvable. In fact, there are cost-effective solutions to achieve just that. The mistake #5 is in ignoring the logs from applications , by only focusing on the perimeter and internal network devices and possibly also servers, but not going “higher up the stack” to look at the application logging. The realm of enterprise applications ranges from SAPs and PeopleSofts of the worlds to small homegrown applications, which nevertheless handle mission-critical processes for many enterprises. Legacy applications, running on mainframes and midrange systems, are out there as well, often running the core business processes as well. The availability and quality of logs differs wildly across the application, ranging from missing (the case for many home-grown applications) to extremely detailed and voluminous (the case for many mainframe applications). Lack of common logging standards and even of logging guidance for software developers lead to many challenges with application logs. Despite the challenges, one needs to make sure that the application logs are collected and made available for analysis as well as for longer term-retention. This can be accomplished by configuring your log management software to collect them and by establishing a log review policy, both for the on-incident review and periodic proactive log review. #6 Even the most advanced and mature organizations fall into the pitfall of the sixth error. It is sneaky and insidious, and can severely reduce the value of a log analysis project. It occurs when organization is only looking at what they know is bad in the logs . Indeed, a vast majority of open source and some commercial tools are set up to filter and look for bad log lines, attack signatures, critical events, etc. For example, “swatch” is a classic free log analysis tool that is powerful, but only at one thing: looking for defined bad things in log files. Moreover, when people talk about log analysis they usually mean sifting through logs looking for things of note. However, to fully realize the value of log data one has to take it to the next level to log mining: actually discovering things of interest in log files without having any preconceived notion of ‘what we need to find’. It sounds obvious - how can we be sure that we know of all the possible malicious behavior in advance – but it disregarded so often. Sometimes, it is suggested that it is simpler to just list all the known good things and then look for the rest. It sounds like a solution, but such task is not only onerous, but also thankless: it is usually even harder to list all the good things than it is to list all the bad things that might happen on a system or network. So many different things occur, malfunction or misbehave, that weeding out attack traces just by listing all the possibilities is not effective. A more intelligent approach is needed! Some of the data mining (also called “knowledge discovery in databases” or KDD) and visualization methods actually work on log data with great success. They allow organizations to look for real anomalies in log data, beyond ‘known bad’ and ‘known good’. To conclude, avoiding the above six mistakes we covered will take your log analysis program to a next level and enhance the value of the existing security and logging infrastructures. Dr Anton Chuvakin, GCIA, GCIH, GCFA (http://www.chuvakin.org) is a recognized security expert and book author. He currently works at LogLogic, where he is involved with defining and executing on a product vision and strategy, driving the product roadmap, conducting research as well as assisting key customers with their LogLogic implementations. He was previously a Security Strategist with a security information management company. He is an author of a book "Security Warrior" and a contributor to "Know Your Enemy II", "Information Security Management Handbook", "Hacker's Challenge 3" and the upcoming book on PCI. Anton also published numerous papers on a broad range of security subjects. In his spare time he maintains his security portal http://www.info-secure.org and several blogs.

    4 Favorites

    Six Mistakes of Log Management 2008 - Presentation Transcript

    1. Six Mistakes of Security Log Management Dr Anton Chuvakin, GCIA, GCIH, GCFA Chief Logging Evangelist LogLogic, Inc
    2. Summary
      • System, Network and Security Logs
      • Why Look at Logs?
      • Brief Log Analysis Overview
      • From Log Analysis to Log Management
      • Log Mistakes: from 0 to 6
    3. Log Data Overview
      • Audit logs
      • Transaction logs
      • Intrusion logs
      • Connection logs
      • System performance records
      • User activity logs
      • Various alerts and other messages
      • Firewalls/intrusion prevention
      • Routers/switches
      • Intrusion detection
      • Servers, desktops, mainframes
      • Business applications
      • Databases
      • Anti-virus
      • VPNs
      What logs? From Where?
    4. Login? Logon? Log in? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0    Logon account:  POWERUSER    Source Workstation: ENTERPRISE    Error Code: 0xC000006A     4574 <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
      • “ Arrgh! Why Don’t We Just Ignore’Em?”
    6. Log Management Mandate and Regulations Regulations Require LMI
      • SOX
      • GLBA
      • FISMA
      • JPA
      • NIST 800-53
        • Capture audit records
        • Regularly review audit records for unusual activity and violations
        • Automatically process audit records
        • Protect audit information from unauthorized deletion
        • Retain audit logs
      • PCI
      • HIPAA
      • SLAs
      Mandates Demand It
      • PCI : Requirement 10 and beyond
        • Logging and user activities tracking are critical
        • Automate and secure audit trails for event reconstruction
        • Review logs daily
        • Retain audit trail history for at least one year
      • COBIT
      • ISO
      • ITIL
      • COBIT 4
        • Provide audit trail for root-cause analysis
        • Use logging to detect unusual or abnormal activities
        • Regularly review access, privileges, changes
        • Verify backup completion
      • ISO17799
        • Maintain audit logs for system access and use, changes, faults, corrections, capacity demands
        • Review the results of monitoring activities regularly and ensure the accuracy of logs
      Controls Require it “ Get fined, Get Sanctioned” “ Lose Customers, Reputation, Revenue or Job” “ Get fined, Go To Jail”
    7. Also: NIST 800-92
      • “This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “
      • So, How Do People Do It?
    9. Log Analysis Basics
      • Manual
        • ‘ Tail’, ‘more’, ‘grep’, ‘notepad’, etc
      • Filtering
        • Positive and negative (“Artificial ignorance”)
      • Summarization and reports
        • “ Top X of Y”
      • Visualization
      • Log indexing and searching
      • Correlation
        • Rule-based and other
      • Log data mining
    10. From Log Analysis to Log Management
      • Threat protection and discovery
      • Incident response
      • Forensics , “e-discovery” and litigation support
      • Regulatory compliance
      • Internal policies and procedure compliance
      • Internal and external audit support
      • IT system and network troubleshooting
      • IT performance management
    11. Log Management Lifecycle Files, syslog, other Immutable Logs Secure Share Collect SNMP, Email, etc Alert Search, Report and Analytics Store Search Report Make Conclusions “ As needed “ basis
      • Looks Complicated?! No Wonder People Make Mistakes …
    13. Seven Mistakes of Log Analysis and Management
      • 0. Not logging at all.
      • 1. Not looking at the logs
      • 2. Storing logs for too short a time
      • 3. Prioritizing the log records before collection
      • 4. Ignoring the logs from applications
      • 5. Approaching logs in a siloed fashion
    14. Mistake 0: Not Logging AT ALL …
      • … and its aggravated version : “… and not knowing that you don’t”
      • No logging? -> well , no logs for incident response, audits, compliance
      • Got logs?
      • If your answer is ‘NO”, don’t listen further: run and enable logging right now !
    15. Example: Oracle
      • Defaults :
        • minimum system logging
        • minimum database server access
        • no data access logging
      • So, where is …
        • data access audit
        • schema and data change audit
        • configuration change audit
    16. Mistake 1: Not looking at logs
      • Collection of logs has value!
      • But review boosts the value 10-fold ( numbers are estimates  )
      • More in-depth analysis boosts it a lot more!
      • Two choices here …
        • Review after an incident
        • Ongoing review
    17. Example Log Review Priorities
      • DMZ NIDS
      • DMZ firewall
      • DMZ servers with applications
      • Critical internal servers
      • Other servers
      • Select critical application
      • Desktops
      • Other applications
    18. Mistake 2: Storing logs for too short a time
      • You are saying you HAD logs? And how is it useful?
      • Retention question is a hard one. Truly, nobody has the answer!
        • Seven years? A year? 90 days? A week? Until the disk runs out?
      • Common : 90 days online and up to 1-3 years “nearline” or offline
    19. Also A Mistake: Storing Logs for TOO LONG?!
      • Retention = storage + destruction
      • Why DESTROY LOGS ?
        • Privacy regulations
        • Litigation risk management
        • Due diligence and security policy
        • System resource utilization
    20. Example Retention Strategy
      • Type + network + storage tier
      • IDS + DMZ + online = 90 days
      • Firewall + DMZ + online = 30 days
      • Servers + internal + online = 90 days
      • ALL + DMZ + archive = 3 years
      • Critical + internal + archive = 5 years
      • OTHER + internal + archive = 1 year
    21. Quiz: Name Which Are Security Relevant ?
      • System or software startup, shutdown, restart, and abnormal termination (crash)
      • Various thresholds being exceeded or reaching dangerous levels such as disk space full, memory exhausted, or processor load too high
      • Hardware health messages that the system can troubleshoot or at least detect and log
      • User access to the system such as remote (telnet, ssh, etc.) and local login, network access (FTP) initiated to and from the system, failed and successful
      • User access privilege changes such as the su command—both failed and successful
      • User credentials and access right changes , such as account updates, creation, and deletion—both failed and successful
      • System configuration changes and software updates—both failed and successful
    22. Mistake 3: Deciding What’s Relevant Before Collection
      • How would you know what is …
        • … Security-relevant
        • … Compliance-relevant
        • … or will solve the problem you’d have TOMORROW !?
      • Also affects “ forensic quality” of logs
      • Prioritization Challenge – Got ESP? 
      • Simple – just grab everything!
    23. Example Common Logging Order
      • Log everything
      • Retain most everything
      • Analyze enough
      • Summarize and report on a subset
      • Look at some
      • Act in real-time on a few
    24. Mistake 4: Ignoring Logs from Applications
      • Firewall – Yes , Linux – Yes , Windows – Yes . NIDS – Yes
      • but …
      • Oracle - ?
      • SAP - ?
      • Your Application X – No?
      • Log standards are coming: MITRE CEE!
    25. Example: Jumbled Mess of SAP Logs
      • |22:01:40|BTC| 7|000|DDIC | |LC2|Systemerror when executing external command DB6_DATA_COLLECTOR on gneisenau ()
      • |22:02:32|BTC| 7|000|DDIC | |R49|Communication error, CPIC return code 020, SAP return code 456
      • |22:02:32|BTC| 7|000|DDIC | |R5A|> Conversation ID: 38910614
      • |22:02:32|BTC| 7|000|DDIC | |R64|> CPI-C function: CMSEND(SAP)
      • |22:02:32|BTC| 7|000|DDIC | |LC2|Systemerror when executing external command DB6_DATA_COLLECTOR on gneisenau ()
    26. Mistake 5: Siloed Approach to Log Management
      • Imagine…
        • Database logs -> database monitoring system
        • Syslog -> syslog server
        • Windows log -> stay where they are
        • Firewall logs -> PIX logger
        • Application logs -> don’t exist 
      • What about forensics, incident response, audit?
      • How do you analyze the activities across systems?
    27. Network Servers Databases Homegrown Applications Example Platform vs Siloes Log Silo ?????? ????? ???? ??? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? LOGS ? ? ? ? ? ? ? ? ? ? ? ? Identity Management IT & Network Operations Operational Security Governance & Compliance Log Tool Log Jam Log platform Network Servers Databases Homegrown Applications Identity Management IT & Network Operations Operational Security Governance & Compliance
    28. Conclusions
      • Now you know:
        • What are the logs?
        • Where they come from?
        • Why look at them?
        • How people do it?
        • What are some of the relevant regulations?
        • How to deal with them?
      • And how to AVOID MISTAKES in dealing with logs!
    29. Seven Mistakes of Log Analysis and Management
      • 0. Not logging at all.
      • 1. Not looking at the logs
      • 2. Storing logs for too short a time
      • 3. Prioritizing the log records before collection
      • 4. Ignoring the logs from applications
      • 5. Only looking at what you know is bad
      • 6. Approaching logs in a siloed fashion
    30. Thanks for Attending the Presentation
      • Dr Anton Chuvakin, GCIA, GCIH, GCFA
      • Chief Logging Evangelist
      • http://www.chuvakin.org
      • Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007)
      • See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Book on logs is coming soon! Also see http://chuvakin.blogspot.com
    31. Further Reading
      • Check out my longer paper “ Mistakes of Log Management ”, published at http://www. infosecwriters.org
      • Other fun reading – section on log management on my blog http://chuvakin.blogspot.com/search/label/log%20management/
      • My chapter on logging for PCI from “PCI Compliance” book (posted on Syngress web site)

    + Anton ChuvakinAnton Chuvakin, 2 years ago

    custom

    2608 views, 4 favs, 4 embeds more stats

    This is a full Six Mistakes of Log Management prese more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 2608
      • 2476 on SlideShare
      • 132 from embeds
    • Comments 0
    • Favorites 4
    • Downloads 0
    Most viewed embeds
    • 74 views on http://open.loglogic.com
    • 56 views on http://chuvakin.blogspot.com
    • 1 views on http://alpha
    • 1 views on https://open.loglogic.com

    more

    All embeds
    • 74 views on http://open.loglogic.com
    • 56 views on http://chuvakin.blogspot.com
    • 1 views on http://alpha
    • 1 views on https://open.loglogic.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved