Anton's Log Management 'Worst Practices'


Published on

Log Management 'Worst Practices' - log management tool from planning to deployment to operation. All the mistakes to avoid! All the pitfalls to skip! This was given at SANS Lunch and Learn a few times.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Not LOGGING worst practices
  • Anton's Log Management 'Worst Practices'

    1. 1. Log Management “ Worst Practices” Anton Chuvakin, Ph.D., GCIH, GCFA Chief Logging Evangelist LogLogic, Inc Mitigating Risk. Automating Compliance.
    2. 2. Outline <ul><li>Are you convinced: why log management? </li></ul><ul><ul><li>Hey, why not just ignore the logs, as usual !  </li></ul></ul><ul><li>How to do log management WRONG – an idiot’s guide  </li></ul><ul><ul><li>Planning </li></ul></ul><ul><ul><li>Purchasing </li></ul></ul><ul><ul><li>Deploying </li></ul></ul><ul><ul><li>Running </li></ul></ul><ul><li>Conclusions </li></ul>
    3. 3. Log Data Overview <ul><li>Audit logs </li></ul><ul><li>Transaction logs </li></ul><ul><li>Intrusion logs </li></ul><ul><li>Connection logs </li></ul><ul><li>System performance records </li></ul><ul><li>User activity logs </li></ul><ul><li>Various alerts and other messages </li></ul><ul><li>Firewalls/intrusion prevention </li></ul><ul><li>Routers/switches </li></ul><ul><li>Intrusion detection </li></ul><ul><li>Servers, desktops, mainframes </li></ul><ul><li>Business applications </li></ul><ul><li>Databases </li></ul><ul><li>Anti-virus </li></ul><ul><li>VPNs </li></ul>What logs? From Where?
    4. 4. Why Log Management? <ul><li>Threat protection and discovery </li></ul><ul><li>Incident response </li></ul><ul><li>Forensics , “e-discovery” and litigation support </li></ul><ul><li>Regulatory compliance </li></ul><ul><li>Internal policies and procedure compliance </li></ul><ul><li>Internal and external audit support </li></ul><ul><li>IT system and network troubleshooting </li></ul><ul><li>IT performance management </li></ul>
    5. 5. Log Management Mandate and Regulations Regulations Require LMI <ul><li>SOX </li></ul><ul><li>GLBA </li></ul><ul><li>FISMA </li></ul><ul><li>JPA </li></ul><ul><li>NIST 800-53 </li></ul><ul><ul><li>Capture audit records </li></ul></ul><ul><ul><li>Regularly review audit records for unusual activity and violations </li></ul></ul><ul><ul><li>Automatically process audit records </li></ul></ul><ul><ul><li>Protect audit information from unauthorized deletion </li></ul></ul><ul><ul><li>Retain audit logs </li></ul></ul><ul><li>PCI </li></ul><ul><li>HIPAA </li></ul><ul><li>SLAs </li></ul>Mandates Demand It <ul><li>PCI : Requirement 10 and beyond </li></ul><ul><ul><li>Logging and user activities tracking are critical </li></ul></ul><ul><ul><li>Automate and secure audit trails for event reconstruction </li></ul></ul><ul><ul><li>Review logs daily </li></ul></ul><ul><ul><li>Retain audit trail history for at least one year </li></ul></ul><ul><li>COBIT </li></ul><ul><li>ISO </li></ul><ul><li>ITIL </li></ul><ul><li>COBIT 4 </li></ul><ul><ul><li>Provide audit trail for root-cause analysis </li></ul></ul><ul><ul><li>Use logging to detect unusual or abnormal activities </li></ul></ul><ul><ul><li>Regularly review access, privileges, changes </li></ul></ul><ul><ul><li>Verify backup completion </li></ul></ul><ul><li>ISO17799 </li></ul><ul><ul><li>Maintain audit logs for system access and use, changes, faults, corrections, capacity demands </li></ul></ul><ul><ul><li>Review the results of monitoring activities regularly and ensure the accuracy of logs </li></ul></ul>Controls Require it “ Get fined, Get Sanctioned” “ Lose Customers, Reputation, Revenue or Job” “ Get fined, Go To Jail”
    6. 6. Also: NIST 800-92 <ul><li>“This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “ </li></ul>
    7. 7. Log Management Process Files, syslog, other Immutable Logs Secure Share Collect SNMP, Email, etc Alert Search, Report and Analytics Store Search Report Make Conclusions “ As needed “ basis
    8. 8. So, You Decided to Acquire a LM Tool … <ul><li>What’s next? </li></ul><ul><li>What do you want, specifically? </li></ul><ul><li>How to choose a product? </li></ul><ul><li>How not to screw it up? </li></ul><ul><li>How to make sure that it goes smoothly, now and later? </li></ul><ul><li>Overall, how to be wildly happy </li></ul><ul><li>… with your log management purchase? </li></ul>
    9. 9. What is a “Worst Practice”? <ul><li>As opposed to the “ best practice ” it is … </li></ul><ul><ul><li>What the losers in the field are doing today </li></ul></ul><ul><ul><li>A practice that generally leads to disastrous results , despite its popularity </li></ul></ul>
    10. 10. Log Management Project Lifecycle <ul><li>Determine the need </li></ul><ul><li>Define scope of log management </li></ul><ul><li>Select and evaluate the vendor </li></ul><ul><li>Run proof of Concept – POC </li></ul><ul><li>Deploy (in phases) </li></ul><ul><li>Run the tool </li></ul><ul><li>Expand deployment </li></ul>
    11. 11. 1. Determine the Need <ul><li>WP1: Skip this step altogether – just buy something </li></ul><ul><ul><li>“ John said that we need a correlation engine” </li></ul></ul><ul><ul><li>“ I know this guy who sells log management tools …” </li></ul></ul><ul><li>WP2: Define the need in general </li></ul><ul><ul><li>“ We need, you know, manage logs and stuff”  </li></ul></ul><ul><li>Questions : Real-time? Platform? Appliance? Service? Correlation? Indexing? RDBMS vs files? Volume of logs? Agents? Collectors? Connectors? Users? Your use cases? </li></ul>
    12. 12. Case Study A – Just Buy a SIEM! <ul><li>Medium-sized financial company </li></ul><ul><li>New CSO comes in from a much larger organization </li></ul><ul><li>“We need a SIEM! ASAP!” </li></ul><ul><li>Can you spell “boondoggle?  </li></ul><ul><li>Lessons learned: which problem did we solve? Huh!? None? </li></ul>
    13. 13. 2. Define scope <ul><li>WP3: Postpone scope until after the purchase </li></ul><ul><ul><li>“ The vendor says ‘it scales’ so we will just feed ALL our logs” </li></ul></ul><ul><ul><li>Windows, Linux, i5/OS, OS/390, Cisco – send’em in! </li></ul></ul><ul><li>WP4: Assume you will be the only user of the tool </li></ul><ul><ul><li>“ Steak holders”? What’s that?  </li></ul></ul><ul><ul><li>Common consequence: two or more similar tools are bought </li></ul></ul><ul><ul><li>Forgetting that logs are useful to many people for many reasons … </li></ul></ul>
    14. 14. Case Study B: “We Use’em All” <ul><li>SANS Log Management Summit 2006 </li></ul><ul><li>Vendors X, Y and Z claim “Big Finance” as a customer </li></ul><ul><li>How can that be? </li></ul><ul><li>Well, different teams purchased different products … </li></ul><ul><li>About $2.3m wasted on tools that do the same! </li></ul>
    15. 15. 3. Initial vendor selection <ul><li>WP5: Choose by price alone </li></ul><ul><ul><li>Ignore hardware, extra modules, </li></ul></ul><ul><ul><li>training, service, support, etc costs </li></ul></ul><ul><ul><li>“OMG, this tool is 30% cheaper. And it is only twice as bad. ”  </li></ul></ul><ul><ul><li>Advanced version : be suckered by the vendor’s TCO and ROI “formulas” </li></ul></ul><ul><li>WP6: Choose by relationship or “PowerPoint power” </li></ul><ul><ul><li>“We got it with the latest router purchase…” </li></ul></ul>
    16. 16. 4. Vendor evaluation and POC <ul><li>WP7: Don’t ask for and don’t check references </li></ul><ul><ul><li>“Our environment is unique” </li></ul></ul><ul><li>WP8: Don’t do a POC </li></ul><ul><ul><li>“We can save time!” </li></ul></ul><ul><ul><li>“We can just choose the best product, right?” </li></ul></ul><ul><ul><li>“The vendor said it works just peachy ”  </li></ul></ul><ul><li>WP9: If doing a POC, let vendor dictate how OR ignore what the vendor says </li></ul><ul><ul><li>“Windows? Sure, we will test on Windows!” </li></ul></ul><ul><ul><li>“ Proof of concept!? Why prove what we already know! ” </li></ul></ul>
    17. 17. Case Study C: Performance-Shmerformance  <ul><li>Retail organization deciding between two log management products, A and B </li></ul><ul><li>Vendor A: “We scale like there is no tomorrow”  </li></ul><ul><li>Vendor B: “We scale like we invented scaling”  </li></ul><ul><li>“Can you prove it?!” </li></ul><ul><li>Results : </li></ul><ul><ul><li>Vendor A claims 75,000 MPS, dies at 2300 (!) </li></ul></ul><ul><ul><li>Vendor B claims 75,000 MPS, runs at 85000 (!!) <- LogLogic </li></ul></ul>
    18. 18. 5. Deployment <ul><li>WP10: Expect The Vendor To Write Your Logging Policy OR Ignore Vendor Recommendations </li></ul><ul><ul><li>“ Tell us what we need – tell us what you have” forever… </li></ul></ul><ul><li>WP11: Unpack the boxes and go! </li></ul><ul><ul><li>“ Coordinating with network and system folks is for cowards!” </li></ul></ul><ul><ul><li>Do you know why LM projects take months sometimes? </li></ul></ul><ul><li>WP12: Don’t prepare the infrastructure </li></ul><ul><ul><li>“ Time synchronization? Pah, who needs it” </li></ul></ul><ul><li>WP13: Ignore legal team </li></ul><ul><ul><li>Pain … </li></ul></ul>
    19. 19. Case Study D: Shelfware Forever! <ul><li>Financial company gets a SIEM tool after many months of “evaluations” </li></ul><ul><li>Vendor SEs deploy it </li></ul><ul><li>One year passes by </li></ul><ul><li>A new CSO comes in; looks for what is deployed </li></ul><ul><li>Finds a SIEM tool – which database contains exactly 53 log records (!) </li></ul><ul><ul><li>It was never connected to a production network… </li></ul></ul>
    20. 20. 6. Running the Tool <ul><li>WP14: Deploy Everywhere At Once </li></ul><ul><ul><li>“ We need log management everywhere!” </li></ul></ul><ul><li>WP15: “Save Money” on Vendor Support Contract </li></ul><ul><ul><li>“ We Have to Pay 18% for What? ”  </li></ul></ul><ul><li>WP16: Ignore Upgrades </li></ul><ul><ul><li>“ It works just fine – why touch it?” </li></ul></ul><ul><li>WP17: Training? They said it is ‘ intuitive ’! </li></ul><ul><ul><li>“’ A chance to “save” more money here? Suuure.” </li></ul></ul>
    21. 21. Case Study E: Intuitive? To Me It Isn’t! <ul><li>A major retailer procures a log management tool from an integrator </li></ul><ul><li>A classic “high-level” sales, golf and all  </li></ul><ul><li>“Intuitive UI” is high on the list of criteria </li></ul><ul><li>The tool is deployed in production </li></ul><ul><li>Security engineers hate it – and don’t touch it </li></ul><ul><li>Simple: UI workflow doesn’t match what they do every day </li></ul>
    22. 22. 7. Expanding Deployment <ul><li>WP18: Don’t Bother With A Product Owner </li></ul><ul><ul><li>“We all use it – we all run it (=nobody does)” </li></ul></ul><ul><li>WP19: Don’t Check For Changed Needs – Just Buy More of the Same </li></ul><ul><ul><li>“We made the decision – why fuss over it?” </li></ul></ul><ul><li>WP20: If it works for 10, it will be OK for 10,000 </li></ul><ul><ul><li>“1,10,100, …, 1 trillion – they are just numbers” </li></ul></ul>
    23. 23. Case Study F: Today - Datacenter, Tomorrow … Oops! <ul><li>Log management tool is tested and deployed at two datacenters – with great success! </li></ul><ul><li>PCI DSS comes in; scope is expanded to wireless systems and POS branch servers </li></ul><ul><li>The tool is prepared to be deployed in 410 (!) more locations </li></ul><ul><li>“ Do you think it will work?” - “Suuuuure!”, says the vendor </li></ul><ul><li>Security director resigns … </li></ul>
    24. 24. Conclusions – Serious ! <ul><li>Turn ON logging! </li></ul><ul><li>Learn about logging and log management </li></ul><ul><ul><li>Read NIST 800-92 and other guides; do the research! </li></ul></ul><ul><li>Match what you need with what they have </li></ul><ul><ul><li>Not doing it as a key source of PAIN </li></ul></ul><ul><li>Plan carefully – and plan your planning too  </li></ul><ul><li>Work WITH the vendor – not ‘against’, not ‘without’, not ‘for’ </li></ul><ul><li>Final word : do big IT projects have “shortcuts” to easy and effortless success – what are they? </li></ul>
    25. 25. Thank You for Attending! <ul><li>Dr Anton Chuvakin, GCIH, GCFA </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li> </li></ul><ul><li>See for my papers, books, reviews, etc </li></ul><ul><li>and other security and logging resources; check my blog at </li></ul>