Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin

Logs: Can’t Hate Them, Won’t Love Them!,[object Object],Dr. Anton Chuvakin,[object Object],Security Warrior Consulting,[object Object],www.securitywarriorconsulting.com,[object Object],April 2010,[object Object]

What Is It?,[object Object],This is a short log analysis and log management class given by Dr. Anton Chuvakin of Security Warrior Consulting at Project HoneynetAnnual Event 2010 in Mexico City, Mexico,[object Object],www.chuvakin.org,[object Object],www.SecurityWarriorConsulting.com,[object Object]

Outline,[object Object],Logs, WTH?,[object Object],Logs and Log Analysis,[object Object],Log Analysis Methods,[object Object],Log Analysis -> Log Management,[object Object],Log Management Mistakes,[object Object],Future Ideas,[object Object],Conclusions,[object Object]

Hilarity!!!,[object Object],“Logs Are Data??! ,[object Object],Bua-ha-ha-ha-ha-haaa!”,[object Object],Aug 11 09:11:19 xx null pif ? exit! 0 ,[object Object]

Log Data Overview,[object Object],From Where?,[object Object],What Logs?,[object Object],[object Object]

Various alerts and other messages,[object Object]

Log Chaos II - Accept?,[object Object],messages:Dec 16 17:28:49 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-notification-00257(traffic): start_time="2002-12-16 17:33:36" duration=5 policy_id=0 service=telnet proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1170 rcvd=1500 src=10.14.94.221 dst=10.14.98.107 src_port=1384 dst_port=23 translated ip=10.14.93.7 port=1206,[object Object],Apr 6 06:06:02 Checkpoint NGX SRC=Any,DEST=ANY,Accept=nosubstitute,Do Not Log,Installspyware,lieonyourtaxes,orbetteryet,dontpaythem,[object Object],Mar 6 06:06:02 winonasu-pix %PIX-6-302013: Built outbound TCP connection 315210 596 for outside:172.196.9.206/1214 (172.196.9.206/1214) to inside:199.17.151.103/1438 (199.17.151.103/1438),[object Object]

SHOCK!!!,[object Object],… and that is BEFORE we even mention application logs!,[object Object]

Log Chaos Everywhere!,[object Object],No standard format,[object Object],No standard schema, no level of details,[object Object],No standard meaning,[object Object],No taxonomy,[object Object],No standard transport,[object Object],No shared knowledge on what to log and how,[object Object],No logging guidance for developers,[object Object],No standard API / libraries for log production,[object Object]

Result?,[object Object],%PIX|ASA-3-713185 Error: Username too long - connection aborted,[object Object],%PIX|ASA-5-501101 User transitioning priv level,[object Object],ERROR: transport error 202: send failed: Success,[object Object],sles10sp1oes oesaudit: type=CWD msg=audit(09/27/07 22:09:45.683:318) : cwd=/home/user1 ,[object Object]

More results?,[object Object],userenv[error] 1030 RCI-CORPsupx No description available,[object Object],Aug 11 09:11:19 xx null pif ? exit! 0 ,[object Object],Apr 23 23:03:08 support last message repeated 3 times,[object Object],Apr 23 23:04:23 support last message repeated 5 times,[object Object],Apr 23 23:05:38 support last message repeated 5 times,[object Object]

It DOES Suck!,[object Object],Well, it does…,[object Object],… but we need to analyze logs every time an incident occurs and in many other cases!,[object Object]

LOG ANALYSIS,[object Object],We will discuss,[object Object],[object Object]

Some log analysis tools,[object Object]

Log Analysis Basics: Summary,[object Object],Manual,[object Object],Filtering,[object Object],Summarization and reports,[object Object],Simple visualization,[object Object],Log searching,[object Object],Correlation,[object Object],Log Data mining,[object Object]

Log Analysis Basics: Manual,[object Object],Manual log review,[object Object],Just fire your trusty tail, more, notepad, vi, Event Viewer, etc and hop to it! ,[object Object],Pros:,[object Object],Easy, no tools required (neither build nor buy),[object Object],Cons:,[object Object],Try it with 10GB log file one day ,[object Object],Boring as Hell! ,[object Object]

See!?,[object Object],Log for VMware Server, pid=2364, version=e.x.p, build=build-63231, option=BETA, section=2[2007-12-03 14:57:00.931 'App' 4516 info] Current working directory: C:ocuments and Settingsll Userspplication DataMwareMware Server,[object Object],[2007-12-03 14:57:00.946 'BaseLibs' 4516 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 2 numThreadsPerCore 1.,[object Object],[2007-12-03 14:57:00.946 'BaseLibs' 4516 info] HOSTINFO: This machine has 1 physical CPUS, 2 total cores, and 2 logical CPUs.,[object Object],[2007-12-03 14:57:00.946 'App' 4516 info] Trying blklistsvc,[object Object],[2007-12-03 14:57:00.946 'App' 4516 info] Trying cimsvc,[object Object],[2007-12-03 14:57:00.946 'App' 4516 info] Trying directorysvc,[object Object],[2007-12-03 14:57:00.946 'App' 4516 info] Trying hostsvc,[object Object],[2007-12-03 14:57:01.571 'NetworkProvider' 4516 info] Using netmap configuration file C:ocuments and Settingsll Userspplication DataMwareMware Serveretmap.conf,[object Object],[2007-12-03 14:57:01.587 'NetworkProvider' 4516 error] VNL_GetBriggeState call failed with status 1.Refreshing network information failed,[object Object],[2007-12-03 14:57:03.165 'NetworkProvider' 4516 info] Active ftp is 1,[object Object],[2007-12-03 14:57:03.165 'NetworkProvider' 4516 info] Allowanyoui is 0,[object Object],[2007-12-03 14:57:03.165 'NetworkProvider' 4516 info] udptimeout is 30,[object Object],[2007-12-03 14:57:03.337 'HostsvcPlugin' 4516 warning] No advanced options found,[object Object],[2007-12-03 14:57:03.368 'Hostsvc::AutoStartManager' 4516 info] VM autostart configuration: C:ocuments and Settingsll Userspplication DataMwareMware ServerostdmAutoStart.xml,[object Object],[2007-12-03 14:57:04.212 'Locale' 4516 info] Locale subsystem initialized from C:rogram FilesMwareMware Serverocale/ with default locale en.,[object Object],[2007-12-03 14:57:04.212 'ResourcePool ha-root-pool' 4516 info] Resource pool instantiated,[object Object],[2007-12-03 14:57:04.212 'ResourcePool ha-root-pool' 4516 info] Refresh interval: 60 seconds,[object Object],[2007-12-03 14:57:04.212 'HostsvcPlugin' 4516 info] Plugin initialized,[object Object],[2007-12-03 14:57:04.212 'App' 4516 info] Trying internalsvc,[object Object],[2007-12-03 14:57:04.259 'App' 4516 info] Trying nfcsvc,[object Object],[2007-12-03 14:57:04.305 'Nfc' 4516 info] Breakpoints disabled,[object Object],[2007-12-03 14:57:04.321 'BaseLibs' 4516 info] Using system libcrypto, version 9070AF,[object Object],[2007-12-03 14:57:06.399 'BaseLibs' 4516 info] [NFC DEBUG] Successfully loaded the diskLib library,[object Object],[2007-12-03 14:57:06.415 'Nfc' 4516 info] Plugin initialized,[object Object],[2007-12-03 14:57:06.415 'App' 4516 info] Trying partitionsvc,[object Object],[2007-12-03 14:57:06.415 'App' 4516 info] Trying proxysvc,[object Object]

Log Analysis Basics: Filtering,[object Object],Log Filtering,[object Object],Just show me the bad stuff; here is the list (positive),[object Object],Just ignore the good stuff; here is the list (negative or Artificial Ignorance),[object Object],Pros:,[object Object],Easy result interpretation: see->act,[object Object],Many tools or write your own,[object Object],Cons:,[object Object],Patterns beyond single messages?,[object Object],Neither good nor bad, but interesting?,[object Object]

Example: How to grep Logs?,[object Object],The easiest log analysis method (Linux/Unix):,[object Object],# grepailure /var/log/messages,[object Object],Filter interesting failure message in messages log,[object Object],# grep –v uccess /var/log/messages,[object Object],Filter messages other than success in messages log,[object Object],# grep –vf LIST /var/log/messages,[object Object],Filter messages other than those listed in FILE,[object Object]

Log Analysis Basics: Summary,[object Object],Summarization and reports,[object Object],Top X Users, Connections by IP, etc ,[object Object],Pros:,[object Object],Dramatically reduces the size of data ,[object Object],Suitable for high-level reporting ,[object Object],Cons:,[object Object],Loss of information by summarizing,[object Object],Which report to pick for a task?,[object Object]

Make A Summary,[object Object],SELECT source, destination, proto, user, COUNT(*) FROMlog_tableWHERE user LIKE ‘an%’ GROUP BY source, destination, proto, user ORDER BY source DESC,[object Object],P.S. Pray tell me, how those nasty logs ended up in a nice database like that? ,[object Object]

Log Analysis Basics: Search,[object Object],Googling Logs,[object Object],User specifies a time period, a log source or all, and an expression; gets back logs that match (regexvs Boolean),[object Object],Pro,[object Object],Easy to understand,[object Object],Quick to do,[object Object],Con,[object Object],What do you search for?,[object Object],A LOT of data back, sometimes,[object Object]

How to Do It: Splunk Search,[object Object]

Log Analysis Basics: Correlation,[object Object],Correlation,[object Object],Rule-based and other 'correlation' and 'Correlation' algorithms,[object Object],Pro,[object Object],Highly automated,[object Object],Con,[object Object],Needs rules written by experts,[object Object],Needs tuning for each site,[object Object]

Example Rule,[object Object],<rule id="40112" level="12" timeframe="240">,[object Object],<if_group>authentication_success</if_group>,[object Object],<if_matched_group>authentication_failures,[object Object], </if_matched_group> ,[object Object],<same_source_ip /> ,[object Object],<description>Multiple authentication failures followed a success.</description> ,[object Object],</rule>,[object Object],OSSEC rule shown; see OSSEC.net for details,[object Object]

Log Analysis Basics: Data Mining,[object Object],Log mining,[object Object],Algorithms that extract meaning from raw data,[object Object],Pro,[object Object],Promises fully-automated analysis ,[object Object],Con,[object Object],Still research-grade technology,[object Object]

Example Ranum NBS,[object Object],Ranum’s “nbs” (never before seen) – the simplest log mining tool.,[object Object],No knowledge about “bad” goes in -> insight comes out!,[object Object],Look Ma, NO RULES!,[object Object],Use the tool to pick up anomalous messages from your log pool,[object Object],See for more: http://www.slideshare.net/anton_chuvakin/log-mining-beyond-log-analysis,[object Object]

Log Analysis Basics: Visualization,[object Object],Visualization, from simple to 4D,[object Object],A pie chart worth a thousand words?,[object Object],Pro,[object Object],You just look at it and know what it means and what to do,[object Object],Con,[object Object],You just look at it, and hmmm…. ,[object Object]

How to Do It: afterglow Tool,[object Object]

Log Analysis Basics: When,[object Object],Real time vs. historical analysis,[object Object],Do you always need real-time?,[object Object],What data cannot be analyzed in real-time?,[object Object],A day later vs. never question,[object Object],Historical analysis for deep insight,[object Object]

How To Start Using The Tools?,[object Object],1. Collect logs,[object Object], Tools: Syslog-ng, standard syslog, etc,[object Object],2. Store logs,[object Object],Tools: MySQL, etc,[object Object],3. Search logs,[object Object], Tools: grep, splunk, etc,[object Object],4. Correlate and alert,[object Object], Tools: OSSEC, OSSIM, sec, nbs, logwatch, etc,[object Object]

Key Points to Remember,[object Object],Techniques review,[object Object],Tools review,[object Object],Any other tool suggestions?,[object Object],Start thinking buy vs. build,[object Object]

From Log Analysis to Log Management,[object Object],We will discuss,[object Object],[object Object]

Log management motivations,[object Object]

Log Analysis to Log Management,[object Object],Files, syslog, other,[object Object],Act,[object Object],Collect,[object Object],Secure,[object Object],Humans still needed!,[object Object],Make ,[object Object],Conclusions,[object Object],SNMP, E-mail, etc,[object Object],Alert,[object Object],Search,[object Object],Report,[object Object],Store,[object Object],Search, Report and Analytics,[object Object],Immutable Logs,[object Object]

Log Management Challenges,[object Object],Not enough data,[object Object],Too much data,[object Object],Diverse records,[object Object],Time out of sync,[object Object],False records,[object Object],Duplicate data,[object Object],Hard to get data,[object Object]

LOG RETENTION – A TRIVIAL MATTER?,[object Object],We will discuss,[object Object],[object Object]

Issues with various log retention technologies,[object Object]

What is NOT Retention?,[object Object],A database that stores a few fields from each log ,[object Object],A tape closet with log data,[object Object],tapes that were never verified,[object Object], – and lurking rats,[object Object],A syslog server that just spools logs into files,[object Object]

Retention Time Question,[object Object],I have the answer!  No, not really.,[object Object],Regulations?,[object Object],Unambiguous: PCI – keep’em for 1 year,[object Object],Tiered retention strategy,[object Object],Online,[object Object],Near line,[object Object],Offline/tape,[object Object]

Example: Retention Strategy,[object Object],Type + network + storage tier,[object Object],IDS + DMZ + online = 90 days,[object Object],Firewall + DMZ + online = 30 days,[object Object],Servers + internal + online = 90 days,[object Object],ALL + DMZ + archive = 3 years,[object Object],Critical + internal + archive = 5 years,[object Object],OTHER + internal + archive = 1 year,[object Object]

How to Create A Log Retention Strategy,[object Object],Assess applicable compliance requirements ,[object Object],Look at risk posture and other needs,[object Object],Look at various log sources and their log volumes,[object Object],Review available storage options,[object Object],Decide on tiers,[object Object]

Log Storage Tiers: Options,[object Object],RDBMS ,[object Object],[object Object],Flat files,[object Object],[object Object],Hybrid,[object Object],[object Object],Proprietary datastore,[object Object],[object Object],Tape,[object Object]

Example: How to Deal with A Trillion Log Messages,[object Object],How to manage a trillion (~1000 billions) log messages?,[object Object],Hundreds of terabytes (1/2 of a petabyte …) of data,[object Object],Which tool to pick?,[object Object],"Sorry, buddy, you are writing your own code here!”,[object Object]

Key Points to Remember,[object Object],What is really log retention?,[object Object],Review log storage option to use (or to buy in a vendor tool),[object Object],Learn about storage challenges,[object Object]

LOGGING MISTAKES,[object Object],We will discuss,[object Object],[object Object],[object Object]

Mistake 1: Not Logging AT ALL …,[object Object],… and its aggravated version: “… and not knowing that you don’t”,[object Object],No logging? -> well, no logs for incident investigation and response, audits, C&A, control validation, compliance,[object Object],Got logs?,[object Object],If your answer is ‘NO' don’t listen further: run and enable logging right now!,[object Object]

Example: Oracle,[object Object],Defaults: ,[object Object],minimum system logging,[object Object],minimum database server access,[object Object],no data access logging,[object Object],So, where is …,[object Object],data access audit,[object Object],schema and data change audit,[object Object],configuration change audit,[object Object]

Mistake 2: Not looking at logs,[object Object],Collection of logs has value!,[object Object],But review boosts the value 10-fold(numbersare estimates ),[object Object],More in-depth analysis boosts it a lot more!,[object Object],Two choices here …,[object Object],Review after an incident ,[object Object],Ongoing review,[object Object]

Example Log Review Priorities,[object Object],DMZ NIDS,[object Object],DMZ firewall,[object Object],DMZ servers with applications,[object Object],Critical internal servers,[object Object],Other servers,[object Object],Select critical application,[object Object],Other applications,[object Object]

Mistake 3: Storing logs for too short a time,[object Object],You are saying you HAD logs? And how is it useful?,[object Object],Retention question is a hard one. Truly, nobody has the answer!,[object Object],Seven years? A year? 90 days? A week? Until the disk runs out?,[object Object],Common: 90 days online and up to 1-3 years near line or offline,[object Object]

Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin

Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(16)

Similar to Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin

Similar to Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin(20)

More from Anton Chuvakin

More from Anton Chuvakin(20)

Recently uploaded

Recently uploaded(20)

Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton Chuvakin

Editor's Notes