LogChaos: Challenges and Opportunities of Security Log Standardization
Report
Share
•15 likes•2,152 views
![LogChaos: Challenges and Opportunities of Security Log Standardization,[object Object],Dr. Anton Chuvakin,[object Object],Security Warrior Consulting,[object Object],Oct 2009,[object Object],5th Annual IT Security Automation Conference,[object Object],Baltimore, MD,[object Object],October 26-29, 2009,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-1-2048.jpg?cb=1669234207)
![Outline,[object Object],World of logs today,[object Object],Log chaos? Why? Why order is sorely needed!,[object Object],Past attempts to bring order chaos!,[object Object],Why ALL failed?,[object Object],What does the future hold?,[object Object],You already know about CEE ,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-2-2048.jpg?cb=1669234207)
![Log Data Overview,[object Object],From Where?,[object Object],What Logs?,[object Object],[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-4-2048.jpg?cb=1669234207)
![Various alerts and other messages,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-18-2048.jpg?cb=1669234207)
![Log Chaos I - Login?,[object Object],<18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53) ,[object Object],<57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS:LoginSuccess [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006,[object Object],<122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2,[object Object],<13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: POWERUSER ,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-19-2048.jpg?cb=1669234207)
![Log Chaos II - Accept?,[object Object],messages:Dec 16 17:28:49 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-notification-00257(traffic): start_time="2002-12-16 17:33:36" duration=5 policy_id=0 service=telnet proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1170 rcvd=1500 src=10.14.94.221 dst=10.14.98.107 src_port=1384 dst_port=23 translated ip=10.14.93.7 port=1206,[object Object],Apr 6 06:06:02 Checkpoint NGX SRC=Any,DEST=ANY,Accept=nosubstitute,Do Not Log,Installspyware,lieonyourtaxes,orbetteryet,dontpaythem,[object Object],Mar 6 06:06:02 winonasu-pix %PIX-6-302013: Built outbound TCP connection 315210 596 for outside:172.196.9.206/1214 (172.196.9.206/1214) to inside:199.17.151.103/1438 (199.17.151.103/1438),[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-20-2048.jpg?cb=1669234207)
![Log Format Consistency,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-21-2048.jpg?cb=1669234207)
![Log Chaos Everywhere!,[object Object],No standard format,[object Object],No standard schema, no level of details,[object Object],No standard meaning,[object Object],No taxonomy,[object Object],No standard transport,[object Object],No shared knowledge on what to log and how,[object Object],No logging guidance for developers,[object Object],No standard API / libraries for log production,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-22-2048.jpg?cb=1669234207)
![Result?,[object Object],%PIX|ASA-3-713185 Error: Username too long - connection aborted,[object Object],%PIX|ASA-5-501101 User transitioning priv level,[object Object],ERROR: transport error 202: send failed: Success,[object Object],sles10sp1oes oesaudit: type=CWD msg=audit(09/27/07 22:09:45.683:318) : cwd=/home/user1 ,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-23-2048.jpg?cb=1669234207)
![More results?,[object Object],userenv[error] 1030 RCI-CORPsupx No description available,[object Object],Aug 11 09:11:19 xx null pif ? exit! 0 ,[object Object],Apr 23 23:03:08 support last message repeated 3 times,[object Object],Apr 23 23:04:23 support last message repeated 5 times,[object Object],Apr 23 23:05:38 support last message repeated 5 times,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-24-2048.jpg?cb=1669234207)
![But This … This Here Takes The Cake…,[object Object],Logging username AND passwords to “debug” authentication (niiiice! ),[object Object],Logging numeric error codes – and not having documentation ANYWHERE (please read my mind!),[object Object],Logging chunks of source code to syslog (care to see a 67kB syslog message? ),[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-25-2048.jpg?cb=1669234207)
![Chaos2order: Why Logging Standards?,[object Object],Common language,[object Object],Easier to report on logs and explain the reports,[object Object],Deeper insight into future problems ,[object Object],Easier system interoperability,[object Object],Common logging practices,[object Object],Easier to explain what is in the logs to management and non-IT people,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-26-2048.jpg?cb=1669234207)
![What Becomes Possible?,[object Object],All those super-smart people at SIEM vendors can stop parsing and start analyzing,[object Object],What the events mean? Consequences? Actions? Maybe even prediction?,[object Object],Different systems can mitigate consequences of each others’ failures,[object Object],We can finally tell the developers “what to log?” and have them “get it!”,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-27-2048.jpg?cb=1669234207)
![Example Logging for Developers,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-28-2048.jpg?cb=1669234207)
![Definitions,[object Object],Log = message generated by an IT system to record whatever event happening ,[object Object],Log format = layout of log messages in the form of fields, separators, delimiters, tags, etc,[object Object],Log syntax = fields and values that are present in logs,[object Object],Log taxonomy = a taxonomy of log messages that categorizes log messages and codifies their meaning,[object Object],Log transport = a method of moving logs from one system to another; typically a network protocol,[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-29-2048.jpg?cb=1669234207)
![Various Logging Standards by Type,[object Object],Log format,[object Object],Example: Syslog, a non-standard standard ,[object Object],Example: IDMEF, a failed standard ,[object Object],Log contents,[object Object],No standard to speak of: logs = trash can because application developers dump what they want there (and how they want!),[object Object],Log transport,[object Object],Example: Syslog (TCP/UDP port 514),[object Object],Logging practices / recommendations,[object Object],Example: NIST 800-92 (for security only),[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-30-2048.jpg?cb=1669234207)
![Old, Dead and Vendor Log Standards,[object Object],Vendor “standard” efforts:,[object Object],[object Object]](https://image.slidesharecdn.com/nistlogstandard-challengesrel-091027164614-phpapp02/75/logchaos-challenges-and-opportunities-of-security-log-standardization-31-2048.jpg?cb=1669234207)
LogChaos: Challenges and Opportunities of Security Log Standardization
•15 likes•2,152 views
Report
Share
LogChaos: Challenges and Opportunities of Security Log Standardization Abstract: The presentation will discuss how to bring order (in the form of standards!) to the chaotic world of logging. It will give a brief introduction to logs and logging and explain how and why logs grew so chaotic and disorganized. Next it will cover why log standards are sorely needed. It will offer a walkthrough that highlights the critical areas of log standardization. Past failed standards will be looked at and their lessons learned. Finally, current logging standard efforts will be presented briefly.
More Related Content
What's hot
What's hot(20)
Similar to LogChaos: Challenges and Opportunities of Security Log Standardization
Similar to LogChaos: Challenges and Opportunities of Security Log Standardization(20)
![First Responders Course - Session 6 - Detection Systems [2004]](https://cdn.slidesharecdn.com/ss_thumbnails/publiccopy-6-detectionsystems-130419071428-phpapp01-thumbnail.jpg?width=768&fit=bounds)
More from Anton Chuvakin
More from Anton Chuvakin(20)
Recently uploaded
Recently uploaded(20)
LogChaos: Challenges and Opportunities of Security Log Standardization
- 9. Databases
- 10. Anti-virus
- 11. VPNs
- 12. Audit logs
- 13. Transaction logs
- 14. Intrusion logs
- 15. Connection logs
- 32. WELF - Webtrends
- 33. CEF - ArcSight
- 46. To specify the event in a common representation
- 48. For parsing out relevant data from received log messages
- 50. For exchanging log messages
- 53. Log correlation (SIEM) capabilities
- 54. Device intercommunication enabling autonomic computing
Editor's Notes
- Some Logging RulesI seem to be building logging infrastructure today. I keep recalling one or another of the rules for playing this game. Might as well try to put them down.Who? – The speaker’s unique ID and type should be in each log line.Transcript – The speaker’s utterances should have a serial number, so you can notice gaps.Checksum – A running check sum is a big help in proving things.When – The utterances should have a time stamp (daemontoolsmultilog t is good)Synchronize our watches – NTP is a must everywhere.Breadcrumbs – Jobs/tasks/work-items/requests should have a unique ID that is threaded these across to the logs and across process/module/machinesHealth – All processes (machines, threads …) should emit a heart beat; heart beats should include some health indicators so other parties can notice when they expire or get sickReplay – Logs that enable a rebuild from last snapshot will save your butt. Often your close and only some minor optimization (truncating output, discarding binary info, say) is preventing it. I once rebuilt an entire source repository from years of mail to prove an intrusion had not touched the sources.Syntax – It’s good if the logs are well tokenized, i.e. embedded strings are escaped; and character encodings are worked out.Standardized – It’s good, but it’s hopeless. This is the worst case of the 2nd part of “”Be strict in what you send, but generous in what you receive”Innummerable – You can lay an ontology over the space of exceptions. Accept that, and then proceed as usual.Now – The sooner the log analysis takes place the better. Don’t wait until your patient is in intensive care. Analogy: test driven development.Email – The accumulated headers in modern email are full of lessons learnedFSEvents – the asynchronous file system journaling/notifications of (BeOS, et. al.) are worth looking at closely.Fast – I tend to embrace that writing the log is not transactional or even particularly reliable so I can have volume instead.
- <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE IDMEF-Message PUBLIC "-//IETF//DTD RFC XXXX IDMEFv1.0//EN“ "idmef-message.dtd"><IDMEF-Message version="1.0"><Alert ident="abc123456789"><Analyzer analyzerid="hq-dmz-analyzer62"><Node category="dns"><location>Headquarters Web Server</location><name>analyzer62.example.com</name></Node></Analyzer><CreateTimentpstamp="0xbc72b2b4.0x00000000"> 2000-03-09T15:31:00-08:00</CreateTime><Source ident="abc01"><Node ident="abc01-01"><Address ident="abc01-02" category="ipv4-addr"><address>192.0.2.200</address></Address></Node></Source><Target ident="def01"><Node ident="def01-01" category="dns"><name>www.example.com</name><Address ident="def01-02" category="ipv4-addr"><address>192.0.2.50</address></Address></Node><Service ident="def01-03"><portlist>5-25,37,42,43,53,69-119,123-514</portlist></Service></Target><Classification origin="vendor-specific"><name>portscan</name><url>http://www.vendor.com/portscan</url></Classification></Alert></IDMEF-Message>
- WTsyslog[1998-08-01 00:04:11 ip=10.0.0.1 pri=6] id=firewall time="1998-08-01 00:08:52" fw=WebTrendsSamplepri=6 proto=http src=10.0.0.2 dst=10.0.0.3 dstname=1.example.com arg=/selfupd/x86/en/WULPROTO.CAB op=GET result=304 sent=898 WTsyslog[1998-08-01 00:04:12 ip=10.0.0.1 pri=6] id=firewall time="1998-08-01 00:08:52" fw=WebTrendsSamplepri=6 proto=http src=10.0.0.2 dst=10.0.0.3 dstname=1.example.com arg=/selfupd/x86/en/CUNPROT2.CAB op=GET result=304 sent=853 WTsyslog[1998-08-01 00:04:23 ip=10.0.0.1 pri=6] id=firewall time="1998-08-01 00:09:03" fw=WebTrendsSamplepri=6 proto=http src=10.0.0.2 dst=10.0.0.3 dstname=1.example.com arg=/R510/v31content/90820/0x00000409.gng op=GET result=304 sent=2983 WTsyslog[1998-08-01 03:02:03 ip=10.0.0.1 pri=6] id=firewall time="1998-08-01 03:06:43" fw=WebTrendsSamplepri=6 proto=http src=10.0.0.2 dst=10.0.0.4 dstname=2.example.com arg=/ op=POST result=200 sent=2195 WTsyslog[1998-08-01 16:25:33 ip=10.0.0.1 pri=6] id=firewall time="1998-08-01 06:30:09" fw=WebTrendsSamplepri=6 proto=http src=10.0.0.5 dst=10.0.0.6 dstname=3.example.com arg=/portal/brand/images/logo_pimg.gif op=GET result=304 rcvd=1036
- Example of converting a DB2 Content Manager event to a CBE-formatted eventA DB2® Content Manager event is converted to a Common Base Event (CBE) formatted event. A CBE-formatted event contains additional information about the host name, product name, and the environment.The following example shows how a DB2 Content Manager event is converted to a CBE-formatted event. Information in the CBE-formatted event is entered by the event monitor:<CommonbaseEventcreationTime="2008-03-19T01:03:11:256Z" extensionName="CMEvent" globalInstanceId="AA1234567890123456789012345678901234567890123001" priority="100" version="1.0.1"> <contextDataElements name="cmevent" type="string"> <contextValue>placeholder for DB2 Content Manager event</contextValue> </contextDataElements> <sourceComponentId application="Content Manager Event Monitor" component="Content Manager V8.4.01.000" componentIdType="Productname" executionEnvironment="Windows XP[x86]" instanceId="1" location="myhost/9.30.44.123" locationType="Hostname" subComponent="Event Monitor" componentType="Content Manager"/> <situation categoryName="OtherSituation"> <situationTypexmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="OtherSituation" reasoningScope="EXTERNAL"> Application Event </situationType> </situation> </CommonBaseEvent> Parent topic: CBE XML element format