CEE Logging Standard: Today and Tomorrow

Slide 1: Dr Anton Chuvakin Chief Logging Evangelist LogLogic, Inc CEE Logging Standard: Today and Tomorrow LogLogic Confidential 1 Friday, February 1, 2008

Slide 2: Outline  World of logs today – Where is chaos? Everywhere! Why? Why order is needed?  Past attempts to bring order to log chaos! – Why ALL failed?  CEE Approach – Brief history – 4 pillars of CEE and their today’s status  Future possibilities LogLogic, Inc Friday, February 1, 2008 2

Slide 3: Log Chaos I - Login? <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system- warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53) <57> Dec 25 00:04:32:%SEC_LOGIN-5- LOGIN_SUCCESS:Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon a ccount: POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC00 0006A 4574 LogLogic, Inc Friday, February 1, 2008 3

Slide 4: Log Chaos II - Accept? messages:Dec 16 17:28:49 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-notification-00257(traffic): start_time="2002-12-16 17:33:36" duration=5 policy_id=0 service=telnet proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1170 rcvd=1500 src=10.14.94.221 dst=10.14.98.107 src_port=1384 dst_port=23 translated ip=10.14.93.7 port=1206 Apr 6 06:06:02 Checkpoint NGX SRC=Any,DEST=ANY,Accept=nosubstitute,Do Not Log,Installspyware,lieonyourtaxes,orbetteryet,dontpaythem Mar 6 06:06:02 winonasu-pix %PIX-6-302013: Built outbound TCP connection 315210 596 for outside:172.196.9.206/1214 (172.196.9.206/1214) to inside:199.17.151.103/1438 (199.17.151.103/1438) LogLogic, Inc Friday, February 1, 2008 4

Slide 5: Definitions  Log = message generated by an IT system to record whatever event happening  Log format = layout of log messages in the form of fields, separators, delimiters, tags, etc  Log syntax = fields and values that are present in logs  Log taxonomy = a taxonomy of log messages that categorizes log messages and codifies their meaning  Log transport = a method of moving logs from one system to another; typically a network protocol LogLogic, Inc Friday, February 1, 2008 5

Slide 6: Log Chaos  No standard format – No standard schema, no level of details  No standard meaning – No taxonomy  No standard transport  No shared knowledge on what to log and how  No logging guidance for developers  No standard API / libraries for log production LogLogic, Inc Friday, February 1, 2008 6

Slide 7: Chaos2order: Why Logging Standards?  Common language so that people and systems understand what is in the logs  Easier to report on logs and explain the reports  Deeper insight into future problems as indicated by the log data  Easier system interoperability leading to reduced cost and complexity  Common logging practices simplify audits and compliance  Easier to explain what is in the logs to management and non- IT people LogLogic, Inc Friday, February 1, 2008 7

Slide 8: Various Logging Standards by Type  Log format – Example: Syslog, a non-standard standard  – Example: IDMEF, a failed standard  Log contents – No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!)  Log transport – Example: Syslog (TCP/UDP port 514)  Logging practices – Example: NIST 800-92 (for security only) LogLogic, Inc Friday, February 1, 2008 8

Slide 9: Old, Dead and Vendor Log Standards  Old, mostly dead standards: – CIDF – DARPA (became IDMEF) – IDMEF – IETF (never adopted by anybody) – CIEL – MITRE (cancelled early)  Vendor “standard” efforts: – CBE - IBM – WELF - Webtrends – CEF - ArcSight – OLF – eIQnetworks – SDEE – Cisco+ (also mostly dead as a standard) LogLogic, Inc Friday, February 1, 2008 9

Slide 10: What Killed’em ALL?  1. Lack of adoption – BIG one! 2. “Solution in search of a problem” 3. IETF “overthinkers”  4. Overly complex standards 5. Vendors and their tactical focus (or “marketing standards”) 6. Narrow approach (e.g. just security) LogLogic, Inc Friday, February 1, 2008 10

Slide 11: What Worked? NIST 800-92 Guide to LM “This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “ LogLogic, Inc Friday, February 1, 2008 11

Slide 12: What is a Common Event Expression – CEE? CEE = Syntax + Vocabulary + Transport + Log Recommendations  Common Event Expression Taxonomy – To specify the event in a common representation  Common Log Syntax – For parsing out relevant data from received log messages  Common Log Transport – For exchanging log messages Common Event ExpressionImpactsRecommendations Log – Log management capabilities – For guiding events and details needed to – Log correlation (SIEM) capabilities logged by devices (OS, IDS, FWs, be etc) – Device intercommunication enabling autonomic computing – Enterprise-level situational awareness LogLogic, Inc Friday, FebruaryInfosec attacker modeling and other security analysis capability – 1, 2008 12

Slide 13: CEE Introduction  Brief history – Used to be called: Common Event eXpression  – Now called: Common Event Expression   First conceived: discussions over 2004 about CIEL  Started unofficially: email conversations 8/2005  Started officially: meeting 01/2007 LogLogic, Inc Friday, February 1, 2008 13

Slide 14: CEE Current Status “Alive and well!”  CEE board creative and active  Public Working Group created and active (join via cee@mitre.org)  Positioning note released  Website: http://cee.mitre.org  Longer white paper under review (TBA)  Project FAQ released (update TBA)  Public WG list archives (soon TBA) LogLogic, Inc Friday, February 1, 2008 14

Slide 15: Key Area of CEE Common Event Expression (CEE) by MITRE Key standard areas: • “Create an event expression taxonomy for uniform and precise log definitions that lead to a common event representation.  Create log syntaxes utilizing a single data dictionary to provide consistent event specific details.  Standardize flexible event transport mechanisms to support multiple environments.  Propose log recommendations for the events and attributes devices generate.” LogLogic, Inc Friday, February 1, 2008 15

Slide 16: CEE in One Glance LogLogic, Inc Friday, February 1, 2008 16

Slide 17: CEE Components 1. Log transport  Plan: “bless” existing transports (ongoing!) 2. Log format/syntax  Plan: analyze existing and create (ongoing!) 3. Log taxonomy  Plan: create one (ongoing!)  Logging recommendations  Plan: collect, organize and unify (ongoing!) LogLogic, Inc Friday, February 1, 2008 17

Slide 18: 1. CEE on Log Transport  How logs move around? – Syslog via UDP/TCP 514 – XML over SOAP over HTTP (and same over SSL) – XDAS event transport service – SNMP traps – Windows logs over WMI or RPC  CEE will adopt or “bless” those that are common and/or “clean”  Questions: encryption? Filtering? LogLogic, Inc Friday, February 1, 2008 18

Slide 19: 2. CEE on Log Format/Syntax Need a generic syntax to map into these below:  Key=value pair format – Need a canonical list of keys!  CSV/delimited or database format – Need a list of column names  XML format – Needs a list of XML tags  Binary format – Need field names LogLogic, Inc Friday, February 1, 2008 19

Slide 20: 3. CEE on Log Taxonomy The main idea – every log MUST have:  OBJECT  ACTION  STATUS Example: User jsmith login successful OBJECT ACTION STATUS LogLogic, Inc Friday, February 1, 2008 20

Slide 21: 4. CEE on Logging Recommendations Based on existing guidance (e.g. regulatory, etc)  What to log?  What details?  What fields?  What format?  What scenario, industry, regulation, etc? LogLogic, Inc Friday, February 1, 2008 21

Slide 22: Adoption, ADOPTION, A-D-O-P-T-I-O-N! Why CEE WILL be adopted?  MITRE can drive government procurement  MITRE sponsor organizations will use it immediately  Open process with everybody welcome (ALL stakeholders’ needs incorporated)  If you have comments, ideas, share them at cee@mitre.org list! LogLogic, Inc Friday, February 1, 2008 22

Slide 23: Why CEE WILL Succeed!!  Time has come – logs are more important today than ever  MITRE as a host – many successful standards  Approach – attack key “pain” areas of “log chaos”  Vendors on board – can adopt and use LogLogic, Inc Friday, February 1, 2008 23

Slide 24: Conclusions: Future of CEE  Develop 4 pillars of CEE standard  Publish and collect feedback from the community  Drive adoption from the software vendor side  Drive adoption from the government side  Drive adoption from the log analysis vendor side LogLogic, Inc Friday, February 1, 2008 24

Slide 25: CEE + XDAS Together (Suggested!) 1. XDAS audit transport service = to be “blessed” as one of the CEE log transport mechanisms 2. XDAS event fields = to be included in CEE syntax 3. XDAS events = to follow CEE taxonomy (i.e. have OBJECT, ACTION, STATUS or fields mapped to them) 4. Suggestion on what XDAS audit options to enable = to include in CEE logging recommendations 5. Common registry for changes LogLogic, Inc Friday, February 1, 2008 25

Slide 26: Thanks for Attending the Presentation Dr Anton Chuvakin, GCIH, GCFA www.chuvakin.org Chief Logging Evangelist LogLogic, Inc Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) books See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Also see http://chuvakin.blogspot.com LogLogic, Inc Friday, February 1, 2008 26

Slide 27: Backup and Example Slides from MITRE LogLogic, Inc Friday, February 1, 2008 27

Slide 28: CEE Example Syntax - details specific to event being logged Format (1 and 2) Example Log Messages month day time host program[pid]: message In CEE, each of these would be a possible syntax element, whose value and definition would be well defined by a Data Dictionary 1.Sep 26 12:00:00 myhost-- root[808]: ROOT LOGIN ON tty1 2. Apr 10 12:30:34 hostname sshd[16682]: error: PAM: Authentication failure for user1 from host.domain.com 3. Sep 19 08:26:10 details specific to event being Syntax - zuric CEF:0|security| Data Dictionary needed to logged threatmanager|1.0|100|worm successfully stopped| associated Format (3) CEF message enumerate details Transport - successful log with the event transmission 10|src=10.0.0.1 dst=2.1.2.2 spt=1232 It needs to provide flexible CEF:Version|Device Vendor| syntax options by defining the Syslog (each log message is DeviceProduct|DeviceVersion| elements and formats transmitted in a single UDP Signature ID|Name|Severity| packet usually over port Extension Ex: dst in the CEF event – 514/UDP) Extension is a meta item – it imposes a defined by a dotted quad IPv4 sytax extension to specify addition details address - a data dictionary can enumerate these LogLogic, Inc details Friday, February 1, 2008 28

Slide 29: CEE Example (Cont.) Taxonomy – a reduced language set for consistent log messages Right now they are semantically similar – ok for humans – but not for computers  Scenario: An attacker has breached our network – 1. Determine if any successful logins – What do we search for – ‘log in’, ‘log in’, ‘logged on’ etc. LogLogic, Inc Friday, February 1, 2008 29