CEE Logging Standard: Today and Tomorrow


Published on

CEE Logging Standard: Today and Tomorrow

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Emerging Log Standards: Challenges and Opportunities The presentation will discuss how to bring order (in the form of standards!) to the chaotic world of logging. It will offer a walkthrough that highlights the critical areas of log standardization. Past failed standards will be looked at and their lessons learned. Finally, CEE logging standard effort will be presented and described. Key takeaways: Why log standards are sorely needed Why none succeeded so far? Why CEE will succeed?
  • CEE Logging Standard: Today and Tomorrow

    1. 1. CEE Logging Standard: Today and Tomorrow Dr Anton Chuvakin Chief Logging Evangelist LogLogic, Inc
    2. 2. Outline <ul><li>World of logs today </li></ul><ul><ul><li>Where is chaos? Everywhere! Why? Why order is needed? </li></ul></ul><ul><li>Past attempts to bring order to log chaos! </li></ul><ul><ul><li>Why ALL failed? </li></ul></ul><ul><li>CEE Approach </li></ul><ul><ul><li>Brief history </li></ul></ul><ul><ul><li>4 pillars of CEE and their today’s status </li></ul></ul><ul><li>Future possibilities </li></ul>
    3. 3. Log Chaos I - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff: port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    Source Workstation: ENTERPRISE    Error Code: 0xC000006A     4574 <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from (2002-12-17 15:50:53)
    4. 4. Log Chaos II - Accept? messages:Dec 16 17:28:49 ns5xp: NetScreen device_id=ns5xp system-notification-00257(traffic): start_time=&quot;2002-12-16 17:33:36&quot; duration=5 policy_id=0 service=telnet proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1170 rcvd=1500 src= dst= src_port=1384 dst_port=23 translated ip= port=1206 Mar 6 06:06:02 winonasu-pix %PIX-6-302013: Built outbound TCP connection 315210 596 for outside: ( to inside: ( Apr 6 06:06:02 Checkpoint NGX SRC=Any,DEST=ANY,Accept=nosubstitute,Do Not Log,Installspyware,lieonyourtaxes,orbetteryet,dontpaythem
    5. 5. Definitions <ul><li>Log = message generated by an IT system to record whatever event happening </li></ul><ul><li>Log format = layout of log messages in the form of fields, separators, delimiters, tags, etc </li></ul><ul><li>Log syntax = fields and values that are present in logs </li></ul><ul><li>Log taxonomy = a taxonomy of log messages that categorizes log messages and codifies their meaning </li></ul><ul><li>Log transport = a method of moving logs from one system to another; typically a network protocol </li></ul>
    6. 6. Log Chaos <ul><li>No standard format </li></ul><ul><ul><li>No standard schema, no level of details </li></ul></ul><ul><li>No standard meaning </li></ul><ul><ul><li>No taxonomy </li></ul></ul><ul><li>No standard transport </li></ul><ul><li>No shared knowledge on what to log and how </li></ul><ul><li>No logging guidance for developers </li></ul><ul><li>No standard API / libraries for log production </li></ul>
    7. 7. Chaos2order: Why Logging Standards? <ul><li>Common language so that people and systems understand what is in the logs </li></ul><ul><li>Easier to report on logs and explain the reports </li></ul><ul><li>Deeper insight into future problems as indicated by the log data </li></ul><ul><li>Easier system interoperability leading to reduced cost and complexity </li></ul><ul><li>Common logging practices simplify audits and compliance </li></ul><ul><li>Easier to explain what is in the logs to management and non-IT people </li></ul>
    8. 8. Various Logging Standards by Type <ul><li>Log format </li></ul><ul><ul><li>Example: Syslog, a non-standard standard  </li></ul></ul><ul><ul><li>Example: IDMEF, a failed standard </li></ul></ul><ul><li>Log contents </li></ul><ul><ul><li>No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!) </li></ul></ul><ul><li>Log transport </li></ul><ul><ul><li>Example: Syslog (TCP/UDP port 514) </li></ul></ul><ul><li>Logging practices </li></ul><ul><ul><li>Example: NIST 800-92 (for security only) </li></ul></ul>
    9. 9. Old, Dead and Vendor Log Standards <ul><li>Old, mostly dead standards : </li></ul><ul><ul><li>CIDF – DARPA (became IDMEF) </li></ul></ul><ul><ul><li>IDMEF – IETF (never adopted by anybody ) </li></ul></ul><ul><ul><li>CIEL – MITRE (cancelled early) </li></ul></ul><ul><li>Vendor “standard” efforts: </li></ul><ul><ul><li>CBE - IBM </li></ul></ul><ul><ul><li>WELF - Webtrends </li></ul></ul><ul><ul><li>CEF - ArcSight </li></ul></ul><ul><ul><li>OLF – eIQnetworks </li></ul></ul><ul><ul><li>SDEE – Cisco+ (also mostly dead as a standard) </li></ul></ul>
    10. 10. What Killed’em ALL?  <ul><li>Lack of adoption – BIG one! </li></ul><ul><li>“ Solution in search of a problem” </li></ul><ul><li>IETF “overthinkers”  </li></ul><ul><li>Overly complex standards </li></ul><ul><li>Vendors and their tactical focus (or “marketing standards”) </li></ul><ul><li>Narrow approach (e.g. just security) </li></ul>
    11. 11. What Worked? NIST 800-92 Guide to LM <ul><li>“ This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “ </li></ul>
    12. 12. What is a Common Event Expression – CEE? CEE = Syntax + Vocabulary + Transport + Log Recommendations <ul><li>Common Event Expression Impacts </li></ul><ul><ul><li>Log management capabilities </li></ul></ul><ul><ul><li>Log correlation (SIEM) capabilities </li></ul></ul><ul><ul><li>Device intercommunication enabling autonomic computing </li></ul></ul><ul><ul><li>Enterprise-level situational awareness </li></ul></ul><ul><ul><li>Infosec attacker modeling and other security analysis capability </li></ul></ul><ul><li>Common Event Expression Taxonomy </li></ul><ul><ul><li>To specify the event in a common representation </li></ul></ul><ul><li>Common Log Syntax </li></ul><ul><ul><li>For parsing out relevant data from received log messages </li></ul></ul><ul><li>Common Log Transport </li></ul><ul><ul><li>For exchanging log messages </li></ul></ul><ul><li>Log Recommendations </li></ul><ul><ul><li>For guiding events and details needed to be logged by devices (OS, IDS, FWs, etc) </li></ul></ul>
    13. 13. CEE Introduction <ul><li>Brief history </li></ul><ul><ul><li>Used to be called: Common Event eXpression  </li></ul></ul><ul><ul><li>Now called: Common Event Expression  </li></ul></ul><ul><li>First conceived : discussions over 2004 about CIEL </li></ul><ul><li>Started unofficially: email conversations 8/2005 </li></ul><ul><li>Started officially : meeting 01/2007 </li></ul>
    14. 14. CEE Current Status <ul><li>“ Alive and well!” </li></ul><ul><li>CEE board creative and active </li></ul><ul><li>Public Working Group created and active (join via [email_address] ) </li></ul><ul><li>Positioning note released </li></ul><ul><li>Website : http:// cee.mitre.org </li></ul><ul><li>Longer white paper under review (TBA) </li></ul><ul><li>Project FAQ released (update TBA) </li></ul><ul><li>Public WG list archives (soon TBA) </li></ul>
    15. 15. Key Area of CEE <ul><li>Common Event Expression (CEE) by MITRE </li></ul><ul><li>Key standard areas : </li></ul><ul><li>“ Create an event expression taxonomy for uniform and precise log definitions that lead to a common event representation. </li></ul><ul><li>Create log syntaxes utilizing a single data dictionary to provide consistent event specific details. </li></ul><ul><li>Standardize flexible event transport mechanisms to support multiple environments. </li></ul><ul><li>Propose log recommendations for the events and attributes devices generate.” </li></ul>
    16. 16. CEE in One Glance
    17. 17. CEE Components <ul><li>Log transport </li></ul><ul><ul><li>Plan : “bless” existing transports ( ongoing !) </li></ul></ul><ul><li>Log format/syntax </li></ul><ul><ul><li>Plan: analyze existing and create ( ongoing !) </li></ul></ul><ul><li>Log taxonomy </li></ul><ul><ul><li>Plan: create one ( ongoing !) </li></ul></ul><ul><li>Logging recommendations </li></ul><ul><ul><li>Plan: collect, organize and unify ( ongoing !) </li></ul></ul>
    18. 18. 1. CEE on Log Transport <ul><li>How logs move around? </li></ul><ul><ul><li>Syslog via UDP/TCP 514 </li></ul></ul><ul><ul><li>XML over SOAP over HTTP (and same over SSL) </li></ul></ul><ul><ul><li>XDAS event transport service </li></ul></ul><ul><ul><li>SNMP traps </li></ul></ul><ul><ul><li>Windows logs over WMI or RPC </li></ul></ul><ul><li>CEE will adopt or “bless” those that are common and/or “clean” </li></ul><ul><li>Questions : encryption? Filtering? </li></ul>
    19. 19. 2. CEE on Log Format/Syntax <ul><li>Need a generic syntax to map into these below: </li></ul><ul><li>Key=value pair format </li></ul><ul><ul><li>Need a canonical list of keys! </li></ul></ul><ul><li>CSV/delimited or database format </li></ul><ul><ul><li>Need a list of column names </li></ul></ul><ul><li>XML format </li></ul><ul><ul><li>Needs a list of XML tags </li></ul></ul><ul><li>Binary format </li></ul><ul><ul><li>Need field names </li></ul></ul>
    20. 20. 3. CEE on Log Taxonomy <ul><li>The main idea – every log MUST have: </li></ul><ul><li>OBJECT </li></ul><ul><li>ACTION </li></ul><ul><li>STATUS </li></ul><ul><li>Example : </li></ul><ul><li>User jsmith login successful </li></ul><ul><li>OBJECT ACTION STATUS </li></ul>
    21. 21. 4. CEE on Logging Recommendations <ul><li>Based on existing guidance (e.g. regulatory, etc) </li></ul><ul><li>What to log? </li></ul><ul><li>What details? </li></ul><ul><li>What fields? </li></ul><ul><li>What format? </li></ul><ul><li>What scenario, industry, regulation, etc? </li></ul>
    22. 22. Adoption, ADOPTION, A-D-O-P-T-I-O-N! <ul><li>Why CEE WILL be adopted? </li></ul><ul><li>MITRE can drive government procurement </li></ul><ul><li>MITRE sponsor organizations will use it immediately </li></ul><ul><li>Open process with everybody welcome (ALL stakeholders’ needs incorporated) </li></ul><ul><li>If you have comments, ideas, share them at [email_address] list! </li></ul>
    23. 23. Why CEE WILL Succeed!! <ul><li>Time has come – logs are more important today than ever </li></ul><ul><li>MITRE as a host – many successful standards </li></ul><ul><li>Approach – attack key “pain” areas of “log chaos” </li></ul><ul><li>Vendors on board – can adopt and use </li></ul>
    24. 24. Conclusions: Future of CEE <ul><li>Develop 4 pillars of CEE standard </li></ul><ul><li>Publish and collect feedback from the community </li></ul><ul><li>Drive adoption from the software vendor side </li></ul><ul><li>Drive adoption from the government side </li></ul><ul><li>Drive adoption from the log analysis vendor side </li></ul>
    25. 25. CEE + XDAS Together (Suggested!) <ul><li>XDAS audit transport service = to be “blessed” as one of the CEE log transport mechanisms </li></ul><ul><li>XDAS event fields = to be included in CEE syntax </li></ul><ul><li>XDAS events = to follow CEE taxonomy (i.e. have OBJECT, ACTION, STATUS or fields mapped to them) </li></ul><ul><li>Suggestion on what XDAS audit options to enable = to include in CEE logging recommendations </li></ul><ul><li>Common registry for changes </li></ul>
    26. 26. Thanks for Attending the Presentation <ul><li>Dr Anton Chuvakin, GCIH, GCFA www.chuvakin.org </li></ul><ul><li>Chief Logging Evangelist </li></ul><ul><li>LogLogic, Inc </li></ul><ul><li>Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) books </li></ul><ul><li>See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Also see http://chuvakin.blogspot.com </li></ul>
    27. 27. Backup and Example Slides from MITRE
    28. 28. CEE Example <ul><li>Sep 26 12:00:00 myhost-- root[808]: ROOT LOGIN ON tty1 </li></ul><ul><li>Apr 10 12:30:34 hostname sshd[16682]: error: PAM: Authentication failure for user1 from host.domain.com </li></ul><ul><li>Sep 19 08:26:10 zuric CEF:0|security|threatmanager|1.0|100|worm successfully stopped|10|src= dst= spt=1232 </li></ul>Example Log Messages Transport - successful log transmission Syslog (each log message is transmitted in a single UDP packet usually over port 514/UDP) Syntax - details specific to event being logged Format (3) CEF message CEF:Version|Device Vendor|DeviceProduct|DeviceVersion|Signature ID|Name|Severity|Extension Extension is a meta item – it imposes a sytax extension to specify addition details - a data dictionary can enumerate these details Syntax - details specific to event being logged Format (1 and 2) month day time host program[pid]: message In CEE, each of these would be a possible syntax element, whose value and definition would be well defined by a Data Dictionary Data Dictionary needed to enumerate details associated with the event It needs to provide flexible syntax options by defining the elements and formats Ex: dst in the CEF event – defined by a dotted quad IPv4 address
    29. 29. CEE Example (Cont.) <ul><li>Scenario: An attacker has breached our network </li></ul><ul><ul><li>1. Determine if any successful logins </li></ul></ul><ul><ul><ul><li>What do we search for – ‘log in’, ‘log in’, ‘logged on’ etc. </li></ul></ul></ul>Taxonomy – a reduced language set for consistent log messages Right now they are semantically similar – ok for humans – but not for computers