CEE Logging Standard: Today and Tomorrow Dr Anton Chuvakin Chief Logging Evangelist LogLogic, Inc

Outline World of logs today Where is chaos? Everywhere! Why? Why order is needed? Past attempts to bring order to log chaos! Why ALL failed? CEE Approach Brief history 4 pillars of CEE and their today’s status Future possibilities

World of logs today

Where is chaos? Everywhere! Why? Why order is needed?

Past attempts to bring order to log chaos!

Why ALL failed?

CEE Approach

Brief history

4 pillars of CEE and their today’s status

Future possibilities

Log Chaos I - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account : POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC000006A 4574 <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)

Log Chaos II - Accept? messages:Dec 16 17:28:49 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-notification-00257(traffic): start_time=&quot;2002-12-16 17:33:36&quot; duration=5 policy_id=0 service=telnet proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1170 rcvd=1500 src=10.14.94.221 dst=10.14.98.107 src_port=1384 dst_port=23 translated ip=10.14.93.7 port=1206 Mar 6 06:06:02 winonasu-pix %PIX-6-302013: Built outbound TCP connection 315210 596 for outside:172.196.9.206/1214 (172.196.9.206/1214) to inside:199.17.151.103/1438 (199.17.151.103/1438) Apr 6 06:06:02 Checkpoint NGX SRC=Any,DEST=ANY,Accept=nosubstitute,Do Not Log,Installspyware,lieonyourtaxes,orbetteryet,dontpaythem

Definitions Log = message generated by an IT system to record whatever event happening Log format = layout of log messages in the form of fields, separators, delimiters, tags, etc Log syntax = fields and values that are present in logs Log taxonomy = a taxonomy of log messages that categorizes log messages and codifies their meaning Log transport = a method of moving logs from one system to another; typically a network protocol

Log = message generated by an IT system to record whatever event happening

Log format = layout of log messages in the form of fields, separators, delimiters, tags, etc

Log syntax = fields and values that are present in logs

Log taxonomy = a taxonomy of log messages that categorizes log messages and codifies their meaning

Log transport = a method of moving logs from one system to another; typically a network protocol

Log Chaos No standard format No standard schema, no level of details No standard meaning No taxonomy No standard transport No shared knowledge on what to log and how No logging guidance for developers No standard API / libraries for log production

No standard format

No standard schema, no level of details

No standard meaning

No taxonomy

No standard transport

No shared knowledge on what to log and how

No logging guidance for developers

No standard API / libraries for log production

Chaos2order: Why Logging Standards? Common language so that people and systems understand what is in the logs Easier to report on logs and explain the reports Deeper insight into future problems as indicated by the log data Easier system interoperability leading to reduced cost and complexity Common logging practices simplify audits and compliance Easier to explain what is in the logs to management and non-IT people

Common language so that people and systems understand what is in the logs

Easier to report on logs and explain the reports

Deeper insight into future problems as indicated by the log data

Easier system interoperability leading to reduced cost and complexity

Common logging practices simplify audits and compliance

Easier to explain what is in the logs to management and non-IT people

Various Logging Standards by Type Log format Example: Syslog, a non-standard standard  Example: IDMEF, a failed standard Log contents No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!) Log transport Example: Syslog (TCP/UDP port 514) Logging practices Example: NIST 800-92 (for security only)

Log format

Example: Syslog, a non-standard standard 

Example: IDMEF, a failed standard

Log contents

No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!)

Log transport

Example: Syslog (TCP/UDP port 514)

Logging practices

Example: NIST 800-92 (for security only)

Old, Dead and Vendor Log Standards Old, mostly dead standards : CIDF – DARPA (became IDMEF) IDMEF – IETF (never adopted by anybody ) CIEL – MITRE (cancelled early) Vendor “standard” efforts: CBE - IBM WELF - Webtrends CEF - ArcSight OLF – eIQnetworks SDEE – Cisco+ (also mostly dead as a standard)

Old, mostly dead standards :

CIDF – DARPA (became IDMEF)

IDMEF – IETF (never adopted by anybody )

CIEL – MITRE (cancelled early)

Vendor “standard” efforts:

CBE - IBM

WELF - Webtrends

CEF - ArcSight

OLF – eIQnetworks

SDEE – Cisco+ (also mostly dead as a standard)

What Killed’em ALL?  Lack of adoption – BIG one! “ Solution in search of a problem” IETF “overthinkers”  Overly complex standards Vendors and their tactical focus (or “marketing standards”) Narrow approach (e.g. just security)

Lack of adoption – BIG one!

“ Solution in search of a problem”

IETF “overthinkers” 

Overly complex standards

Vendors and their tactical focus (or “marketing standards”)

Narrow approach (e.g. just security)

What Worked? NIST 800-92 Guide to LM “ This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “

“ This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. “

What is a Common Event Expression – CEE? CEE = Syntax + Vocabulary + Transport + Log Recommendations Common Event Expression Impacts Log management capabilities Log correlation (SIEM) capabilities Device intercommunication enabling autonomic computing Enterprise-level situational awareness Infosec attacker modeling and other security analysis capability Common Event Expression Taxonomy To specify the event in a common representation Common Log Syntax For parsing out relevant data from received log messages Common Log Transport For exchanging log messages Log Recommendations For guiding events and details needed to be logged by devices (OS, IDS, FWs, etc)

Common Event Expression Impacts

Log management capabilities

Log correlation (SIEM) capabilities

Device intercommunication enabling autonomic computing

Enterprise-level situational awareness

Infosec attacker modeling and other security analysis capability

Common Event Expression Taxonomy

To specify the event in a common representation

Common Log Syntax

For parsing out relevant data from received log messages

Common Log Transport

For exchanging log messages

Log Recommendations

For guiding events and details needed to be logged by devices (OS, IDS, FWs, etc)

CEE Introduction Brief history Used to be called: Common Event eXpression  Now called: Common Event Expression  First conceived : discussions over 2004 about CIEL Started unofficially: email conversations 8/2005 Started officially : meeting 01/2007

Brief history

Used to be called: Common Event eXpression 

Now called: Common Event Expression 

First conceived : discussions over 2004 about CIEL

Started unofficially: email conversations 8/2005

Started officially : meeting 01/2007

CEE Current Status “ Alive and well!” CEE board creative and active Public Working Group created and active (join via [email_address] ) Positioning note released Website : http:// cee.mitre.org Longer white paper under review (TBA) Project FAQ released (update TBA) Public WG list archives (soon TBA)

“ Alive and well!”

CEE board creative and active

Public Working Group created and active (join via [email_address] )

Positioning note released

Website : http:// cee.mitre.org

Longer white paper under review (TBA)

Project FAQ released (update TBA)

Public WG list archives (soon TBA)

Key Area of CEE Common Event Expression (CEE) by MITRE Key standard areas : “ Create an event expression taxonomy for uniform and precise log definitions that lead to a common event representation. Create log syntaxes utilizing a single data dictionary to provide consistent event specific details. Standardize flexible event transport mechanisms to support multiple environments. Propose log recommendations for the events and attributes devices generate.”

Common Event Expression (CEE) by MITRE

Key standard areas :

“ Create an event expression taxonomy for uniform and precise log definitions that lead to a common event representation.

Create log syntaxes utilizing a single data dictionary to provide consistent event specific details.

Standardize flexible event transport mechanisms to support multiple environments.

Propose log recommendations for the events and attributes devices generate.”

CEE in One Glance

CEE Components Log transport Plan : “bless” existing transports ( ongoing !) Log format/syntax Plan: analyze existing and create ( ongoing !) Log taxonomy Plan: create one ( ongoing !) Logging recommendations Plan: collect, organize and unify ( ongoing !)

Log transport

Plan : “bless” existing transports ( ongoing !)

Log format/syntax

Plan: analyze existing and create ( ongoing !)

Log taxonomy

Plan: create one ( ongoing !)

Logging recommendations

Plan: collect, organize and unify ( ongoing !)

1. CEE on Log Transport How logs move around? Syslog via UDP/TCP 514 XML over SOAP over HTTP (and same over SSL) XDAS event transport service SNMP traps Windows logs over WMI or RPC CEE will adopt or “bless” those that are common and/or “clean” Questions : encryption? Filtering?

How logs move around?

Syslog via UDP/TCP 514

XML over SOAP over HTTP (and same over SSL)

XDAS event transport service

SNMP traps

Windows logs over WMI or RPC

CEE will adopt or “bless” those that are common and/or “clean”

Questions : encryption? Filtering?

2. CEE on Log Format/Syntax Need a generic syntax to map into these below: Key=value pair format Need a canonical list of keys! CSV/delimited or database format Need a list of column names XML format Needs a list of XML tags Binary format Need field names

Need a generic syntax to map into these below:

Key=value pair format

Need a canonical list of keys!

CSV/delimited or database format

Need a list of column names

XML format

Needs a list of XML tags

Binary format

Need field names

3. CEE on Log Taxonomy The main idea – every log MUST have: OBJECT ACTION STATUS Example : User jsmith login successful OBJECT ACTION STATUS

The main idea – every log MUST have:

OBJECT

ACTION

STATUS

Example :

User jsmith login successful

OBJECT ACTION STATUS

4. CEE on Logging Recommendations Based on existing guidance (e.g. regulatory, etc) What to log? What details? What fields? What format? What scenario, industry, regulation, etc?

Based on existing guidance (e.g. regulatory, etc)

What to log?

What details?

What fields?

What format?

What scenario, industry, regulation, etc?

Adoption, ADOPTION, A-D-O-P-T-I-O-N! Why CEE WILL be adopted? MITRE can drive government procurement MITRE sponsor organizations will use it immediately Open process with everybody welcome (ALL stakeholders’ needs incorporated) If you have comments, ideas, share them at [email_address] list!

Why CEE WILL be adopted?

MITRE can drive government procurement

MITRE sponsor organizations will use it immediately

Open process with everybody welcome (ALL stakeholders’ needs incorporated)

If you have comments, ideas, share them at [email_address] list!

Why CEE WILL Succeed!! Time has come – logs are more important today than ever MITRE as a host – many successful standards Approach – attack key “pain” areas of “log chaos” Vendors on board – can adopt and use

Time has come – logs are more important today than ever

MITRE as a host – many successful standards

Approach – attack key “pain” areas of “log chaos”

Vendors on board – can adopt and use

Conclusions: Future of CEE Develop 4 pillars of CEE standard Publish and collect feedback from the community Drive adoption from the software vendor side Drive adoption from the government side Drive adoption from the log analysis vendor side

Develop 4 pillars of CEE standard

Publish and collect feedback from the community

Drive adoption from the software vendor side

Drive adoption from the government side

Drive adoption from the log analysis vendor side

CEE + XDAS Together (Suggested!) XDAS audit transport service = to be “blessed” as one of the CEE log transport mechanisms XDAS event fields = to be included in CEE syntax XDAS events = to follow CEE taxonomy (i.e. have OBJECT, ACTION, STATUS or fields mapped to them) Suggestion on what XDAS audit options to enable = to include in CEE logging recommendations Common registry for changes

XDAS audit transport service = to be “blessed” as one of the CEE log transport mechanisms

XDAS event fields = to be included in CEE syntax

XDAS events = to follow CEE taxonomy (i.e. have OBJECT, ACTION, STATUS or fields mapped to them)

Suggestion on what XDAS audit options to enable = to include in CEE logging recommendations

Common registry for changes

Thanks for Attending the Presentation Dr Anton Chuvakin, GCIH, GCFA www.chuvakin.org Chief Logging Evangelist LogLogic, Inc Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) books See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Also see http://chuvakin.blogspot.com

Dr Anton Chuvakin, GCIH, GCFA www.chuvakin.org

Chief Logging Evangelist

LogLogic, Inc

Coauthor of “Security Warrior” (O’Reilly, 2004) and “PCI Compliance” (Syngress, 2007) books

See http://www.info-secure.org for my papers, books, reviews and other security resources related to logs. Also see http://chuvakin.blogspot.com

Backup and Example Slides from MITRE

CEE Example Sep 26 12:00:00 myhost-- root[808]: ROOT LOGIN ON tty1 Apr 10 12:30:34 hostname sshd[16682]: error: PAM: Authentication failure for user1 from host.domain.com Sep 19 08:26:10 zuric CEF:0|security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232 Example Log Messages Transport - successful log transmission Syslog (each log message is transmitted in a single UDP packet usually over port 514/UDP) Syntax - details specific to event being logged Format (3) CEF message CEF:Version|Device Vendor|DeviceProduct|DeviceVersion|Signature ID|Name|Severity|Extension Extension is a meta item – it imposes a sytax extension to specify addition details - a data dictionary can enumerate these details Syntax - details specific to event being logged Format (1 and 2) month day time host program[pid]: message In CEE, each of these would be a possible syntax element, whose value and definition would be well defined by a Data Dictionary Data Dictionary needed to enumerate details associated with the event It needs to provide flexible syntax options by defining the elements and formats Ex: dst in the CEF event – defined by a dotted quad IPv4 address

Sep 26 12:00:00 myhost-- root[808]: ROOT LOGIN ON tty1

Apr 10 12:30:34 hostname sshd[16682]: error: PAM: Authentication failure for user1 from host.domain.com

Sep 19 08:26:10 zuric CEF:0|security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232

CEE Example (Cont.) Scenario: An attacker has breached our network 1. Determine if any successful logins What do we search for – ‘log in’, ‘log in’, ‘logged on’ etc. Taxonomy – a reduced language set for consistent log messages Right now they are semantically similar – ok for humans – but not for computers

Scenario: An attacker has breached our network

1. Determine if any successful logins

What do we search for – ‘log in’, ‘log in’, ‘logged on’ etc.