This document discusses log standards and future trends in logging. It outlines the current state of log chaos, with no standard formats, schemas, taxonomies, or transport mechanisms. This results in logs being difficult to analyze and make sense of. The document then discusses past attempts at log standards that failed and proposes that a common event expression standard could help bring order to the chaos by establishing common log syntax, taxonomy, transport, and recommendations. This would allow for improved log management, correlation, and security analysis capabilities. It also notes that growing log volumes will continue to pose challenges and will need to be addressed.

1. Log Standards and Future Trends Dr. Anton Chuvakin Security Warrior Consulting www.securitywarriorconsulting.com August 2010

2. Outline World of logs today Log chaos? Order is sorely needed! Past attempts to bring order to chaos! Why all failed? What does the future hold? Logging trends of the next few years

4. Routers/switches

5. Intrusion detection

6. Servers, desktops, mainframes

7. Business applications

8. Databases

9. Anti-virus

10. VPNs

11. Audit logs

12. Transaction logs

13. Intrusion logs

14. Connection logs

15. System performance records

16. User activity logs

18. Log Chaos: Login <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreendevice_id=ns5xp system-warning-00515: Admin User chuvakinhas logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53) <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:LoginSuccess [user:antonc] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <122> Mar 4 09:23:15 localhostsshd[27577]: Accepted password for anton from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: ACHUVAKIN

19. Log Chaos Everywhere! No standard format No standard schema No standard meaning No taxonomy No standard transport No shared knowledge on what to log and how No logging guidance for developers No standard API / libraries for log production

20. Result? %PIX|ASA-3-713185 Error: Username too long - connection aborted Aug 11 09:11:19 xx null pif ? exit! 0 ERROR: transport error 202: send failed: Success userenv[error] 1030 RCI-CORPsupx No description available

21. So, what can we expect?

22. If We Don’t Stop It … MORE logs (learn what’s a “petabyte”!) Distributed logging -> WIDELY distributed logging across applications, systems, etc More REALLY bad logs from custom applications We work harder – and still MISS important things in logs (see VzBIR 2010!) BIG log DATA comes and kills us! 

23. Cloud to the Rescue? Question: do you think “cloud” will make logging better due to APIs, XML, structured data, etc? Answer: "If your security and trust models suck now, you'll be pleasantly surprised by the lack of change when you move to cloud“ Chris Hoff @ Cisco

24. Any solutions, Anton?

25. Standards: The ONLY Way Out! FIRST: make it easier to know what logs tell us! Easier to report on logs and explain the reports Deeper insight into future problems Easier system interoperability Common logging practices Easier to explain what is in the logs to management and non-IT people

26. What Becomes Possible? All those super-smart people at SIEM vendors can stop parsing and start analyzing What the events mean? Consequences? Actions? Maybe even prediction? Different systems can mitigate consequences of each others’ failures We can finally tell the developers “what to log?” and have them “get it!”

27. Various Logging Standards by Type Log format Example: Syslog, a non-standard standard Example: IDMEF, a failed standard Log contents No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!) Log transport Example: Syslog (TCP/UDP port 514) Logging practices / recommendations Example: NIST 800-92 (for security only)

29. WELF - Webtrends

30. CEF - ArcSight

31. OLF – eIQnetworks

32. SDEE – Cisco+Old, mostly dead standards: CIDF – DARPA (became IDMEF) IDMEF – IETF (never adopted by anybody) CIEL – MITRE (cancelled early) Alive: OpenXDAS and syslog RFC5224

33. What Killed’em ALL? Lack of adoption – BIG one! “Solution in search of a problem” “Overthinking” designers Standard complexity Emphasis on XML Vendors and their tactical focus (or “marketing standards”) Narrow approach (e.g. just IDS)

34. What Worked? NIST 800-92 Guide to LM “This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “

35. Pause … How we want the future world of logging to look like?

37. To specify the event in a common representation

38. Common Log Syntax

39. For parsing out relevant data from received log messages

40. Common Log Transport

41. For exchanging log messages

42. Log Recommendations

44. Log correlation (SIEM) capabilities

45. Device intercommunication enabling autonomic computing

46. Enterprise-level situational awareness

48. Conclusions: Future of Logging Log standard is sorely needed About 30 years of IT has passed without it If we don’t do it, BIG log DATA will eat us alive! Current analysis methods are failing fast A log standard will be created CEE team has learned the lessons of others Let’s get to work! LogChaos must die! 

49. … but wait! What about the growing volumes?

50. Log Math 100,000 log messages / second x 300 bytes / log message ~ 28.6 MB x 3600 seconds ~ 100.6 GB / hour x 24 hours ~ 2.35 TB / day x 365 days ~ 860.5 TB / year x 3 years ~ 2.52 PB Now you know what is a petabyte and a trillion!

51. Well… Well, that’s the battle we still have to fight

52. Questions? Dr. Anton Chuvakin Security Warrior Consulting Log management , SIEM, PCI DSS Email:anton@chuvakin.org Site:http://www.chuvakin.org Blog:http://www.securitywarrior.org Twitter:@anton_chuvakin Consulting:http://www.securitywarriorconsulting.com

53. More on Anton Consultant: http://www.securitywarriorconsulting.com Book author: “Security Warrior”, “PCI Compliance”, “Information Security Management Handbook”, “Know Your Enemy II”, “Hacker’s Challenge 3”, etc Conference speaker: SANS, FIRST, GFIRST, ISSA, CSI, RSA, Interop, many, many others worldwide Standard developer: CEE, CVSS, OVAL, etc Community role: SANS, Honeynet Project, WASC, CSI, ISSA, OSSTMM, InfraGard, ISSA, others Past roles: Researcher, Security Analyst, Strategist, Evangelist, Product Manager

54. Security Warrior Consulting Services Logging and log management strategy, procedures and practices Develop logging policies and processes, log review procedures, workflows and periodic tasks as well as help architect those to solve organization problems Plan and implement log management architecture to support your business cases; develop specific components such as log data collection, filtering, aggregation, retention, log source configuration as well as reporting, review and validation Customize industry “best practices” related to logging and log review to fit your environment, help link these practices to business services and regulations Help integrate logging tools and processes into IT and business operations SIEM and log management content development Develop correlation rules, reports and other content to make your SIEM and log management product more useful to you and more applicable to your risk profile and compliance needs Create and refine policies, procedures and operational practices for logging and log management to satisfy requirements of PCI DSS, HIPAA, NERC, FISMA and other regulations More at www.SecurityWarriorConsulting.com