Log Standards & Future Trends by Dr. Anton Chuvakin


Published on

The presentation will discuss how to bring order (in the form of standards!) to the chaotic world of logging.
It will give a brief introduction to logs and logging and explain how and why logs grew so chaotic and disorganized.
Next it will cover why log standards are sorely needed.
It will offer a walk-through that highlights the critical areas of log standardization. Current standard efforts will be discussion.

Finally, the presentation will cover a few of the emerging and yet-to-emerge trends related to logging and log management.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • “DATA” is not really DATA – more like crap. BIG CRAP!
  • Applies to logs perfectly!Cloud might enable better log analysis though
  • http://chuvakin.blogspot.com/
  • Log Standards & Future Trends by Dr. Anton Chuvakin

    1. 1. Log Standards and Future Trends<br />Dr. Anton Chuvakin<br />Security Warrior Consulting<br />www.securitywarriorconsulting.com<br />August 2010 <br />
    2. 2. Outline<br />World of logs today<br />Log chaos? Order is sorely needed!<br />Past attempts to bring order to chaos!<br />Why all failed?<br />What does the future hold?<br />Logging trends of the next few years<br />
    3. 3. Log Data Overview<br />From Where?<br />What Logs?<br /><ul><li>Firewalls/intrusion prevention
    4. 4. Routers/switches
    5. 5. Intrusion detection
    6. 6. Servers, desktops, mainframes
    7. 7. Business applications
    8. 8. Databases
    9. 9. Anti-virus
    10. 10. VPNs
    11. 11. Audit logs
    12. 12. Transaction logs
    13. 13. Intrusion logs
    14. 14. Connection logs
    15. 15. System performance records
    16. 16. User activity logs
    17. 17. Various alerts and other messages</li></li></ul><li>From Log Analysis to Log Management<br />Threatprotection and discovery<br />Incidentresponse and forensics<br />Regulatory compliance and audit<br />Internal policies and procedure compliance<br />IT system and network troubleshooting<br />System performancemanagement<br />
    18. 18. Log Chaos: Login<br /><18> Dec 17 15:45:57 ns5xp: NetScreendevice_id=ns5xp system-warning-00515: Admin User chuvakinhas logged on via Telnet from (2002-12-17 15:50:53) <br /><57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:LoginSuccess [user:antonc] [Source:] [localport:23] at 20:55:40 UTC Fri Feb 28 2006<br /><122> Mar 4 09:23:15 localhostsshd[27577]: Accepted password for anton from ::ffff: port 2895 ssh2<br /><13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0    Logon account:  ACHUVAKIN   <br />
    19. 19. Log Chaos Everywhere!<br />No standard format<br />No standard schema <br />No standard meaning<br />No taxonomy<br />No standard transport<br />No shared knowledge on what to log and how<br />No logging guidance for developers<br />No standard API / libraries for log production<br />
    20. 20. Result?<br />%PIX|ASA-3-713185 Error: Username too long - connection aborted<br />Aug 11 09:11:19 xx null pif ? exit! 0 <br />ERROR: transport error 202: send failed: Success<br />userenv[error] 1030 RCI-CORPwsupx No description available<br />
    21. 21. So, what can we expect?<br />
    22. 22. If We Don’t Stop It …<br />MORE logs (learn what’s a “petabyte”!)<br />Distributed logging -> WIDELY distributed logging across applications, systems, etc<br />More REALLY bad logs from custom applications<br />We work harder – and still MISS important things in logs (see VzBIR 2010!)<br />BIG log DATA comes and kills us! <br />
    23. 23. Cloud to the Rescue?<br />Question: do you think “cloud” will make logging better due to APIs, XML, structured data, etc?<br />Answer: <br /> "If your security and trust models suck now, you'll be pleasantly surprised by the lack of change when you move to cloud“<br />Chris Hoff @ Cisco<br />
    24. 24. Any solutions, Anton?<br />
    25. 25. Standards: The ONLY Way Out!<br />FIRST: make it easier to know what logs tell us!<br />Easier to report on logs and explain the reports<br />Deeper insight into future problems <br />Easier system interoperability<br />Common logging practices<br />Easier to explain what is in the logs to management and non-IT people<br />
    26. 26. What Becomes Possible?<br />All those super-smart people at SIEM vendors can stop parsing and start analyzing<br />What the events mean? Consequences? Actions? Maybe even prediction?<br />Different systems can mitigate consequences of each others’ failures<br />We can finally tell the developers “what to log?” and have them “get it!”<br />
    27. 27. Various Logging Standards by Type<br />Log format<br />Example: Syslog, a non-standard standard <br />Example: IDMEF, a failed standard <br />Log contents<br />No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!)<br />Log transport<br />Example: Syslog (TCP/UDP port 514)<br />Logging practices / recommendations<br />Example: NIST 800-92 (for security only)<br />
    28. 28. Log Standard Cemetery and Market <br />Vendor “standard” efforts:<br /><ul><li>CBE - IBM
    29. 29. WELF - Webtrends
    30. 30. CEF - ArcSight
    31. 31. OLF – eIQnetworks
    32. 32. SDEE – Cisco+</li></ul>Old, mostly dead standards: <br />CIDF – DARPA (became IDMEF)<br />IDMEF – IETF (never adopted by anybody)<br />CIEL – MITRE (cancelled early)<br />Alive: <br />OpenXDAS and syslog RFC5224<br />
    33. 33. What Killed’em ALL?<br />Lack of adoption – BIG one!<br />“Solution in search of a problem”<br />“Overthinking” designers <br />Standard complexity<br />Emphasis on XML<br />Vendors and their tactical focus (or “marketing standards”)<br />Narrow approach (e.g. just IDS)<br />
    34. 34. What Worked? NIST 800-92 Guide to LM<br />“This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “<br />
    35. 35. Pause …<br />How we want the future world of logging to look like?<br />
    36. 36. What is a Common Event Expression – CEE?<br />CEE = Taxonomy + Syntax + Transport + Log Recommendations <br /><ul><li>Taxonomy and Dictionary
    37. 37. To specify the event in a common representation
    38. 38. Common Log Syntax
    39. 39. For parsing out relevant data from received log messages
    40. 40. Common Log Transport
    41. 41. For exchanging log messages
    42. 42. Log Recommendations
    43. 43. For guiding events and details needed to be logged by devices (OS, IDS, FWs, etc)</li></ul>Common Event Expression Impacts<br /><ul><li>Log management capabilities
    44. 44. Log correlation (SIEM) capabilities
    45. 45. Device intercommunication enabling autonomic computing
    46. 46. Enterprise-level situational awareness
    47. 47. Infosec attacker modeling and other security analysis capability</li></li></ul><li>Key Area of CEE<br />Common Event Expression (CEE) by MITRE (http://cee.mitre.org) <br />Key standard areas:<br /><ul><li>Create an event expression taxonomyand field dictionary for uniform and precise log definitions that lead to a common event representation</li></ul>Create log syntaxes utilizing a single data dictionary to provide consistent event specific details.<br />Standardize flexible event transport mechanisms to support multiple environments.<br />Propose log recommendations for the events and attributes devices generate.<br />
    48. 48. Conclusions: Future of Logging<br />Log standard is sorely needed<br />About 30 years of IT has passed without it<br />If we don’t do it, BIG log DATA will eat us alive!<br />Current analysis methods are failing fast<br />A log standard will be created<br />CEE team has learned the lessons of others<br />Let’s get to work!<br />LogChaos must die! <br />
    49. 49. … but wait!<br />What about the growing volumes?<br />
    50. 50. Log Math<br />100,000 log messages / second x 300 bytes / log message ~ 28.6 MB<br />x 3600 seconds  ~ 100.6 GB / hour<br />x 24 hours ~ 2.35 TB / day<br />x 365 days ~ 860.5 TB / year<br />x 3 years ~ 2.52 PB<br />Now you know what is a petabyte and a trillion!<br />
    51. 51. Well…<br />Well, that’s the battle we still have to fight<br />
    52. 52. Questions?<br />Dr. Anton Chuvakin <br />Security Warrior Consulting<br />Log management , SIEM, PCI DSS<br />Email:anton@chuvakin.org<br />Site:http://www.chuvakin.org<br />Blog:http://www.securitywarrior.org<br />Twitter:@anton_chuvakin<br />Consulting:http://www.securitywarriorconsulting.com<br />
    53. 53. More on Anton<br />Consultant: http://www.securitywarriorconsulting.com<br />Book author: “Security Warrior”, “PCI Compliance”, “Information Security Management Handbook”, “Know Your Enemy II”, “Hacker’s Challenge 3”, etc<br />Conference speaker: SANS, FIRST, GFIRST, ISSA, CSI, RSA, Interop, many, many others worldwide<br />Standard developer: CEE, CVSS, OVAL, etc<br />Community role: SANS, Honeynet Project, WASC, CSI, ISSA, OSSTMM, InfraGard, ISSA, others<br />Past roles: Researcher, Security Analyst, Strategist, Evangelist, Product Manager<br />
    54. 54. Security Warrior Consulting Services<br />Logging and log management strategy, procedures and practices<br />Develop logging policies and processes, log review procedures, workflows and periodic tasks as well as help architect those to solve organization problems <br />Plan and implement log management architecture to support your business cases; develop specific components such as log data collection, filtering, aggregation, retention, log source configuration as well as reporting, review and validation<br />Customize industry “best practices” related to logging and log review to fit your environment, help link these practices to business services and regulations<br />Help integrate logging tools and processes into IT and business operations<br />SIEM and log management content development<br />Develop correlation rules, reports and other content to make your SIEM and log management product more useful to you and more applicable to your risk profile and compliance needs<br />Create and refine policies, procedures and operational practices for logging and log management to satisfy requirements of PCI DSS, HIPAA, NERC, FISMA and other regulations<br />More at www.SecurityWarriorConsulting.com<br />