Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Anton's Log Management 'Worst Practices'


Published on

Log Management 'Worst Practices' - log management tool from planning to deployment to operation. All the mistakes to avoid! All the pitfalls to skip! This was given at SANS Lunch and Learn a few times.

Published in: Technology, Business
  • Be the first to comment

Anton's Log Management 'Worst Practices'

  1. 1. Log Management “ Worst Practices” Anton Chuvakin, Ph.D., GCIH, GCFA Chief Logging Evangelist LogLogic, Inc Mitigating Risk. Automating Compliance.
  2. 2. Outline <ul><li>Are you convinced: why log management? </li></ul><ul><ul><li>Hey, why not just ignore the logs, as usual !  </li></ul></ul><ul><li>How to do log management WRONG – an idiot’s guide  </li></ul><ul><ul><li>Planning </li></ul></ul><ul><ul><li>Purchasing </li></ul></ul><ul><ul><li>Deploying </li></ul></ul><ul><ul><li>Running </li></ul></ul><ul><li>Conclusions </li></ul>
  3. 3. Log Data Overview <ul><li>Audit logs </li></ul><ul><li>Transaction logs </li></ul><ul><li>Intrusion logs </li></ul><ul><li>Connection logs </li></ul><ul><li>System performance records </li></ul><ul><li>User activity logs </li></ul><ul><li>Various alerts and other messages </li></ul><ul><li>Firewalls/intrusion prevention </li></ul><ul><li>Routers/switches </li></ul><ul><li>Intrusion detection </li></ul><ul><li>Servers, desktops, mainframes </li></ul><ul><li>Business applications </li></ul><ul><li>Databases </li></ul><ul><li>Anti-virus </li></ul><ul><li>VPNs </li></ul>What logs? From Where?
  4. 4. Why Log Management? <ul><li>Threat protection and discovery </li></ul><ul><li>Incident response </li></ul><ul><li>Forensics , “e-discovery” and litigation support </li></ul><ul><li>Regulatory compliance </li></ul><ul><li>Internal policies and procedure compliance </li></ul><ul><li>Internal and external audit support </li></ul><ul><li>IT system and network troubleshooting </li></ul><ul><li>IT performance management </li></ul>
  5. 5. Log Management Mandate and Regulations Regulations Require LMI <ul><li>SOX </li></ul><ul><li>GLBA </li></ul><ul><li>FISMA </li></ul><ul><li>JPA </li></ul><ul><li>NIST 800-53 </li></ul><ul><ul><li>Capture audit records </li></ul></ul><ul><ul><li>Regularly review audit records for unusual activity and violations </li></ul></ul><ul><ul><li>Automatically process audit records </li></ul></ul><ul><ul><li>Protect audit information from unauthorized deletion </li></ul></ul><ul><ul><li>Retain audit logs </li></ul></ul><ul><li>PCI </li></ul><ul><li>HIPAA </li></ul><ul><li>SLAs </li></ul>Mandates Demand It <ul><li>PCI : Requirement 10 and beyond </li></ul><ul><ul><li>Logging and user activities tracking are critical </li></ul></ul><ul><ul><li>Automate and secure audit trails for event reconstruction </li></ul></ul><ul><ul><li>Review logs daily </li></ul></ul><ul><ul><li>Retain audit trail history for at least one year </li></ul></ul><ul><li>COBIT </li></ul><ul><li>ISO </li></ul><ul><li>ITIL </li></ul><ul><li>COBIT 4 </li></ul><ul><ul><li>Provide audit trail for root-cause analysis </li></ul></ul><ul><ul><li>Use logging to detect unusual or abnormal activities </li></ul></ul><ul><ul><li>Regularly review access, privileges, changes </li></ul></ul><ul><ul><li>Verify backup completion </li></ul></ul><ul><li>ISO17799 </li></ul><ul><ul><li>Maintain audit logs for system access and use, changes, faults, corrections, capacity demands </li></ul></ul><ul><ul><li>Review the results of monitoring activities regularly and ensure the accuracy of logs </li></ul></ul>Controls Require it “ Get fined, Get Sanctioned” “ Lose Customers, Reputation, Revenue or Job” “ Get fined, Go To Jail”
  6. 6. Also: NIST 800-92 <ul><li>“This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “ </li></ul>
  7. 7. Log Management Process Files, syslog, other Immutable Logs Secure Share Collect SNMP, Email, etc Alert Search, Report and Analytics Store Search Report Make Conclusions “ As needed “ basis
  8. 8. So, You Decided to Acquire a LM Tool … <ul><li>What’s next? </li></ul><ul><li>What do you want, specifically? </li></ul><ul><li>How to choose a product? </li></ul><ul><li>How not to screw it up? </li></ul><ul><li>How to make sure that it goes smoothly, now and later? </li></ul><ul><li>Overall, how to be wildly happy </li></ul><ul><li>… with your log management purchase? </li></ul>
  9. 9. What is a “Worst Practice”? <ul><li>As opposed to the “ best practice ” it is … </li></ul><ul><ul><li>What the losers in the field are doing today </li></ul></ul><ul><ul><li>A practice that generally leads to disastrous results , despite its popularity </li></ul></ul>
  10. 10. Log Management Project Lifecycle <ul><li>Determine the need </li></ul><ul><li>Define scope of log management </li></ul><ul><li>Select and evaluate the vendor </li></ul><ul><li>Run proof of Concept – POC </li></ul><ul><li>Deploy (in phases) </li></ul><ul><li>Run the tool </li></ul><ul><li>Expand deployment </li></ul>
  11. 11. 1. Determine the Need <ul><li>WP1: Skip this step altogether – just buy something </li></ul><ul><ul><li>“ John said that we need a correlation engine” </li></ul></ul><ul><ul><li>“ I know this guy who sells log management tools …” </li></ul></ul><ul><li>WP2: Define the need in general </li></ul><ul><ul><li>“ We need, you know, manage logs and stuff”  </li></ul></ul><ul><li>Questions : Real-time? Platform? Appliance? Service? Correlation? Indexing? RDBMS vs files? Volume of logs? Agents? Collectors? Connectors? Users? Your use cases? </li></ul>
  12. 12. Case Study A – Just Buy a SIEM! <ul><li>Medium-sized financial company </li></ul><ul><li>New CSO comes in from a much larger organization </li></ul><ul><li>“We need a SIEM! ASAP!” </li></ul><ul><li>Can you spell “boondoggle?  </li></ul><ul><li>Lessons learned: which problem did we solve? Huh!? None? </li></ul>
  13. 13. 2. Define scope <ul><li>WP3: Postpone scope until after the purchase </li></ul><ul><ul><li>“ The vendor says ‘it scales’ so we will just feed ALL our logs” </li></ul></ul><ul><ul><li>Windows, Linux, i5/OS, OS/390, Cisco – send’em in! </li></ul></ul><ul><li>WP4: Assume you will be the only user of the tool </li></ul><ul><ul><li>“ Steak holders”? What’s that?  </li></ul></ul><ul><ul><li>Common consequence: two or more similar tools are bought </li></ul></ul><ul><ul><li>Forgetting that logs are useful to many people for many reasons … </li></ul></ul>
  14. 14. Case Study B: “We Use’em All” <ul><li>SANS Log Management Summit 2006 </li></ul><ul><li>Vendors X, Y and Z claim “Big Finance” as a customer </li></ul><ul><li>How can that be? </li></ul><ul><li>Well, different teams purchased different products … </li></ul><ul><li>About $2.3m wasted on tools that do the same! </li></ul>
  15. 15. 3. Initial vendor selection <ul><li>WP5: Choose by price alone </li></ul><ul><ul><li>Ignore hardware, extra modules, </li></ul></ul><ul><ul><li>training, service, support, etc costs </li></ul></ul><ul><ul><li>“OMG, this tool is 30% cheaper. And it is only twice as bad. ”  </li></ul></ul><ul><ul><li>Advanced version : be suckered by the vendor’s TCO and ROI “formulas” </li></ul></ul><ul><li>WP6: Choose by relationship or “PowerPoint power” </li></ul><ul><ul><li>“We got it with the latest router purchase…” </li></ul></ul>
  16. 16. 4. Vendor evaluation and POC <ul><li>WP7: Don’t ask for and don’t check references </li></ul><ul><ul><li>“Our environment is unique” </li></ul></ul><ul><li>WP8: Don’t do a POC </li></ul><ul><ul><li>“We can save time!” </li></ul></ul><ul><ul><li>“We can just choose the best product, right?” </li></ul></ul><ul><ul><li>“The vendor said it works just peachy ”  </li></ul></ul><ul><li>WP9: If doing a POC, let vendor dictate how OR ignore what the vendor says </li></ul><ul><ul><li>“Windows? Sure, we will test on Windows!” </li></ul></ul><ul><ul><li>“ Proof of concept!? Why prove what we already know! ” </li></ul></ul>
  17. 17. Case Study C: Performance-Shmerformance  <ul><li>Retail organization deciding between two log management products, A and B </li></ul><ul><li>Vendor A: “We scale like there is no tomorrow”  </li></ul><ul><li>Vendor B: “We scale like we invented scaling”  </li></ul><ul><li>“Can you prove it?!” </li></ul><ul><li>Results : </li></ul><ul><ul><li>Vendor A claims 75,000 MPS, dies at 2300 (!) </li></ul></ul><ul><ul><li>Vendor B claims 75,000 MPS, runs at 85000 (!!) <- LogLogic </li></ul></ul>
  18. 18. 5. Deployment <ul><li>WP10: Expect The Vendor To Write Your Logging Policy OR Ignore Vendor Recommendations </li></ul><ul><ul><li>“ Tell us what we need – tell us what you have” forever… </li></ul></ul><ul><li>WP11: Unpack the boxes and go! </li></ul><ul><ul><li>“ Coordinating with network and system folks is for cowards!” </li></ul></ul><ul><ul><li>Do you know why LM projects take months sometimes? </li></ul></ul><ul><li>WP12: Don’t prepare the infrastructure </li></ul><ul><ul><li>“ Time synchronization? Pah, who needs it” </li></ul></ul><ul><li>WP13: Ignore legal team </li></ul><ul><ul><li>Pain … </li></ul></ul>
  19. 19. Case Study D: Shelfware Forever! <ul><li>Financial company gets a SIEM tool after many months of “evaluations” </li></ul><ul><li>Vendor SEs deploy it </li></ul><ul><li>One year passes by </li></ul><ul><li>A new CSO comes in; looks for what is deployed </li></ul><ul><li>Finds a SIEM tool – which database contains exactly 53 log records (!) </li></ul><ul><ul><li>It was never connected to a production network… </li></ul></ul>
  20. 20. 6. Running the Tool <ul><li>WP14: Deploy Everywhere At Once </li></ul><ul><ul><li>“ We need log management everywhere!” </li></ul></ul><ul><li>WP15: “Save Money” on Vendor Support Contract </li></ul><ul><ul><li>“ We Have to Pay 18% for What? ”  </li></ul></ul><ul><li>WP16: Ignore Upgrades </li></ul><ul><ul><li>“ It works just fine – why touch it?” </li></ul></ul><ul><li>WP17: Training? They said it is ‘ intuitive ’! </li></ul><ul><ul><li>“’ A chance to “save” more money here? Suuure.” </li></ul></ul>
  21. 21. Case Study E: Intuitive? To Me It Isn’t! <ul><li>A major retailer procures a log management tool from an integrator </li></ul><ul><li>A classic “high-level” sales, golf and all  </li></ul><ul><li>“Intuitive UI” is high on the list of criteria </li></ul><ul><li>The tool is deployed in production </li></ul><ul><li>Security engineers hate it – and don’t touch it </li></ul><ul><li>Simple: UI workflow doesn’t match what they do every day </li></ul>
  22. 22. 7. Expanding Deployment <ul><li>WP18: Don’t Bother With A Product Owner </li></ul><ul><ul><li>“We all use it – we all run it (=nobody does)” </li></ul></ul><ul><li>WP19: Don’t Check For Changed Needs – Just Buy More of the Same </li></ul><ul><ul><li>“We made the decision – why fuss over it?” </li></ul></ul><ul><li>WP20: If it works for 10, it will be OK for 10,000 </li></ul><ul><ul><li>“1,10,100, …, 1 trillion – they are just numbers” </li></ul></ul>
  23. 23. Case Study F: Today - Datacenter, Tomorrow … Oops! <ul><li>Log management tool is tested and deployed at two datacenters – with great success! </li></ul><ul><li>PCI DSS comes in; scope is expanded to wireless systems and POS branch servers </li></ul><ul><li>The tool is prepared to be deployed in 410 (!) more locations </li></ul><ul><li>“ Do you think it will work?” - “Suuuuure!”, says the vendor </li></ul><ul><li>Security director resigns … </li></ul>
  24. 24. Conclusions – Serious ! <ul><li>Turn ON logging! </li></ul><ul><li>Learn about logging and log management </li></ul><ul><ul><li>Read NIST 800-92 and other guides; do the research! </li></ul></ul><ul><li>Match what you need with what they have </li></ul><ul><ul><li>Not doing it as a key source of PAIN </li></ul></ul><ul><li>Plan carefully – and plan your planning too  </li></ul><ul><li>Work WITH the vendor – not ‘against’, not ‘without’, not ‘for’ </li></ul><ul><li>Final word : do big IT projects have “shortcuts” to easy and effortless success – what are they? </li></ul>
  25. 25. Thank You for Attending! <ul><li>Dr Anton Chuvakin, GCIH, GCFA </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li> </li></ul><ul><li>See for my papers, books, reviews, etc </li></ul><ul><li>and other security and logging resources; check my blog at </li></ul>