Your SlideShare is downloading. ×
Log Visualization - Bellua BCS 2006
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Log Visualization - Bellua BCS 2006

770
views

Published on

Log Visualization from Bellua BCS in Jakarta, 2006

Log Visualization from Bellua BCS in Jakarta, 2006

Published in: Technology, Education

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
770
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Focus on the little circles (especially on the bottom of the graph). These circles indicate sources (red nodes) that are connecting to many machines (green nodes) on the same port (white node). The zoom on the right side shows that there is one machine (the left red node) which connects to about a dozen machines on the same port. Depending on the source machine, this is normal or possibly anomalous behavior! Certainly worth investigating. For graphs like this it might make sense to apply a filter which prevents servers (especially Windows Domain Controllers) from being drawn. Those usually show very different behavior than all the other machines.
  • The graph shows a configuration that uses the destination address (green nodes) and target ports (white nodes). The contiguous port numbers either represent a part of a portscan or, what is more likely, a device which reports source ports as destination ports for some of the events.
  • In this graph we are looking at a zoom of the graph from the previous slide again. Because we chose to show the destination ports only once in the graph (configure the graph to be show nodes “once per distinct source node”), we can quickly identify all the machines that are using a specific service on the network (red nodes connecting to to the same white node) and also what machines are making use of those services (green nodes connecting to the white nodes). Filter out all the services (i.e., ports) that you know are running on your network and you will be able to spot servers that you did not know of and should not exist on the network!
  • Transcript

    • 1. Logfile Visualization– The Beauty of Graphs BCS 2006, Jakarta Raffael Marty, GCIA, CISSP Manager Solutions @ ArcSight August 30th, 2006 *
    • 2. Raffael Marty, GCIA, CISSP  Enterprise Security Management (ESM) specialist  Strategic Application Solutions @ ArcSight, Inc.  Intrusion Detection Research @ IBM Research  See http://thor.cryptojail.net  IT Security Consultant @ PriceWaterhouse Coopers  Open Vulnerability and Assessment Language (OVAL) board member  Passion for Visual Security Event AnalysisRaffael Marty BCS 2006 Jakarta 2
    • 3. Table Of Contents ► Introduction ► Graphing Basics ► Graph Use Cases ► Visual Analysis Process ► AfterGlow ► Firewall Log VisualizationRaffael Marty BCS 2006 Jakarta 3
    • 4. IntroductionRaffael Marty BCS 2006 Las Vegas 4
    • 5. Disclaimer IP addresses and host names showing up in event graphs and descriptions were obfuscated/changed. The addresses are completely random and any resemblance with well-known addresses or host names are purely coincidental.Raffael Marty BCS 2006 Jakarta 5
    • 6. A Picture is Worth a Thousand Log Entries Detect the Expected Detect the Expected & Discover the Unexpected & Discover the Unexpected Reduce Analysis and Response Times Reduce Analysis and Response Times Make Better Decisions Make Better DecisionsRaffael Marty BCS 2006 Jakarta 6
    • 7. Text or Visuals? ►What would you rather look at? Jun 17 09:42:30 rmarty ifup: Determining IP information for eth0... Jun 17 09:42:35 rmarty ifup: failed; no link present. Check cable? Jun 17 09:42:35 rmarty network: Bringing up interface eth0: failed Jun 17 09:42:38 rmarty sendmail: sendmail shutdown succeeded Jun 17 09:42:38 rmarty sendmail: sm-client shutdown succeeded Jun 17 09:42:39 rmarty sendmail: sendmail startup succeeded Jun 17 09:42:39 rmarty sendmail: sm-client startup succeeded Jun 17 09:43:39 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 09:45:42 rmarty last message repeated 2 times Jun 17 09:45:47 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 09:56:02 rmarty vmnet-dhcpd: DHCPDISCOVER from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 09:56:03 rmarty vmnet-dhcpd: DHCPOFFER on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 Jun 17 09:56:03 rmarty vmnet-dhcpd: DHCPREQUEST for 172.16.48.128 from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 09:56:03 rmarty vmnet-dhcpd: DHCPACK on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:00:03 rmarty crond(pam_unix)[30534]: session opened for user root by (uid=0) Jun 17 10:00:10 rmarty crond(pam_unix)[30534]: session closed for user root Jun 17 10:01:02 rmarty crond(pam_unix)[30551]: session opened for user root by (uid=0) Jun 17 10:01:07 rmarty crond(pam_unix)[30551]: session closed for user root Jun 17 10:05:02 rmarty crond(pam_unix)[30567]: session opened for user idabench by (uid=0) Jun 17 10:05:05 rmarty crond(pam_unix)[30567]: session closed for user idabench Jun 17 10:13:05 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.19/192.168.80.19 to UDP port: 192 Jun 17 10:13:05 rmarty portsentry[4797]: attackalert: Host: 192.168.80.19/192.168.80.19 is already blocked Ignoring Jun 17 10:14:09 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.8/192.168.80.8 to UDP port: 68 Jun 17 10:14:09 rmarty portsentry[4797]: attackalert: Host: 192.168.80.8/192.168.80.8 is already blocked Ignoring Jun 17 10:14:09 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.8/192.168.80.8 to UDP port: 68 Jun 17 10:14:09 rmarty portsentry[4797]: attackalert: Host: 192.168.80.8/192.168.80.8 is already blocked Ignoring Jun 17 10:21:30 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.8/192.168.80.8 to UDP port: 68 Jun 17 10:21:30 rmarty portsentry[4797]: attackalert: Host: 192.168.80.8/192.168.80.8 is already blocked Ignoring Jun 17 10:28:40 rmarty vmnet-dhcpd: DHCPDISCOVER from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:28:41 rmarty vmnet-dhcpd: DHCPOFFER on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:28:41 rmarty vmnet-dhcpd: DHCPREQUEST for 172.16.48.128 from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:28:45 rmarty vmnet-dhcpd: DHCPACK on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:30:47 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.8/192.168.80.8 to UDP port: 68 Jun 17 10:30:47 rmarty portsentry[4797]: attackalert: Host: 192.168.80.8/192.168.80.8 is already blocked Ignoring Jun 17 10:30:47 rmarty portsentry[4797]: attackalert: UDP scan from host: 192.168.80.8/192.168.80.8 to UDP port: 68 Jun 17 10:30:47 rmarty portsentry[4797]: attackalert: Host: 192.168.80.8/192.168.80.8 is already blocked Ignoring Jun 17 10:35:28 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 10:35:31 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 10:38:51 rmarty vmnet-dhcpd: DHCPREQUEST for 172.16.48.128 from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:38:52 rmarty vmnet-dhcpd: DHCPACK on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 Jun 17 10:42:35 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 10:42:38 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128Raffael Marty BCS 2006 Jakarta 7
    • 8. Graphing BasicsRaffael Marty BCS 2006 Las Vegas 8
    • 9. How To Generate A Graph ... | Normalization | ... Device Parser Event Visualizer Jun 17 09:42:30 rmarty ifup: Determining IP information for eth0... Jun 17 09:42:35 rmarty ifup: failed; no link present. Check cable? Jun 17 09:42:35 rmarty network: Bringing up interface eth0: failed Jun 17 09:42:38 rmarty sendmail: sendmail shutdown succeeded Jun 17 09:42:38 rmarty sendmail: sm-client shutdown succeeded Jun 17 09:42:39 rmarty sendmail: sendmail startup succeeded Jun 17 09:42:39 rmarty sendmail: sm-client startup succeeded Jun 17 09:43:39 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Jun 17 09:45:42 rmarty last message repeated 2 times Jun 17 09:45:47 rmarty vmnet-dhcpd: DHCPINFORM from 172.16.48.128 Visual Jun 17 09:56:02 rmarty vmnet-dhcpd: DHCPDISCOVER from 00:0c:29:b7:b2:47 via vmnet8 Jun 17 09:56:03 rmarty vmnet-dhcpd: DHCPOFFER on 172.16.48.128 to 00:0c:29:b7:b2:47 via vmnet8 NH Log FileRaffael Marty BCS 2006 Jakarta 9
    • 10. Visual Types Link Graphs TreeMaps AfterGlow 1.x - Perl AfterGlow 2.0 - JAVARaffael Marty BCS 2006 Jakarta 10
    • 11. Link Graph Configurations Raw Event: [**] [1:1923:2] RPC portmap UDP proxy attempt [**] [Classification: Decode of an RPC Query] [Priority: 2] 06/04-15:56:28.219753 192.168.10.90:32859 -> 192.168.10.255:111 UDP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:148 DF Len: 120 Different node configurations: SIP Name DIP SIP DIP DPort 192.168.10.90 RPC portmap 192.168.10.255 192.168.10.90 192.168.10.255 111 SIP SPort DPort Name SIP DIP 192.168.10.90 32859 111 RPC portmap 192.168.10.90 192.168.10.255Raffael Marty BCS 2006 Jakarta 11
    • 12. Tree Maps All Network TrafficRaffael Marty BCS 2006 Jakarta 12
    • 13. Tree Maps 20% 80% UDP TCP Configuration (Hierarchy): ProtocolRaffael Marty BCS 2006 Jakarta 13
    • 14. Tree Maps UDP TCP HTTP DNS UDP TCP SSH SNMP FTP Configuration (Hierarchy): Protocol -> ServiceRaffael Marty BCS 2006 Jakarta 14
    • 15. Graph Use CasesRaffael Marty BCS 2006 Las Vegas 15
    • 16. Graph Use-Cases Situational Awareness DashboardRaffael Marty BCS 2006 Jakarta 16
    • 17. Graph Use-Cases Suspicious Activity?Raffael Marty BCS 2006 Jakarta 17
    • 18. Graph Use-Cases Network ScanRaffael Marty BCS 2006 Jakarta 18
    • 19. Graph Use-Cases Port Scan ? ►Port scan or something else?Raffael Marty BCS 2006 Jakarta 19
    • 20. Graph Use-Cases PortScan SIP DIP DPortRaffael Marty BCS 2006 Jakarta 20
    • 21. Graph Use-Cases Telecom Malicious Code Propagation From Content To Phone# Type|Size Phone#Raffael Marty BCS 2006 Jakarta 21
    • 22. Graph Use-Cases Email Relays Grey out “my domain” invisibleDomain Make emails to From: My From: Other Domain and from “my domain” To: My Domain To: Other Domain Do you run an open relay? From ToRaffael Marty BCS 2006 Jakarta 22
    • 23. Visual Analysis ProcessRaffael Marty BCS 2006 Las Vegas 23
    • 24. Visual Analysis Process Event Feedback Loop Feb 18 13:39:15.598491 rule 71/0(match): pass in on xl0: 195.27.249.139.63263 > 195.141.69.42.80: Device S 492525755:492525755(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 24053 0> (DF) Feb 18 13:39:15.899644 rule 71/0(match): pass in on xl0: 195.27.249.139.63264 > 195.141.69.42.80: S 875844783:875844783(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 24054 0> (DF) Normalization 195.27.249.139,195.141.69.42,80 195.27.249.139,195.141.69.42,80 Filter 195.27.249.139,195.141.69.42,80 Service stopped Correlation VisualRaffael Marty BCS 2006 Jakarta 24
    • 25. Visual Analysis Process Event Feedback Loop Real-time Visual Data Forensic and Detection Processing Historical Analysis Creation of new Filters Visual and Correlation Components Investigation Assign to Content AuthorRaffael Marty BCS 2006 Jakarta 25
    • 26. Visual Analysis Process Visual Detection Beginning of Analyst’s shiftRaffael Marty BCS 2006 Jakarta 26
    • 27. Visual Analysis Process Visual Detection Scanning activity is displayed Firewall Blocks Scan EventsRaffael Marty BCS 2006 Jakarta 27
    • 28. Visual Analysis Process Visual InvestigationRaffael Marty BCS 2006 Jakarta 28
    • 29. Visual Analysis Process Defining New Content 1. Correlation Assign for further analysis if More than 20 firewall drops from an external machine to an internal machine 3. Open a ticket for Operations to quarantine and clean infected machines 2. Filter • Internal machines on white-list • connecting to active directory serversRaffael Marty BCS 2006 Jakarta 29
    • 30. AfterGlow http://afterglow.sourceforge.net ► Two Versions: • AfterGlow 1.x – Perl for Link Graphs • AfterGlow 2.0 – Java for TreeMaps ► Collection of Parsers: • pf2csv.pl BSD PacketFilter (pf) • tcpdump2csv.pl tcpdump 3.9 • sendmail2csv.pl Sendmail transaction logsRaffael Marty BCS 2006 Jakarta 30
    • 31. AfterGlow afterglow.sourceforge.netRaffael Marty BCS 2006 Las Vegas 31
    • 32. AfterGlow Parsers ► tcpdump2csv.pl • Takes care of swapping response source and targets tcpdump -vttttnnelr /tmp/log.tcpdump | ./tcpdump2csv.pl "sip dip sport" ► sendmail_parser.pl • Reassemble email conversations: Jul 24 21:01:16 rmarty sendmail[17072]: j6P41Gqt017072: from=<root@localhost.localdomain>, size=650, class=0, nrcpts=1, Jul 24 21:01:16 rmarty sendmail[17073]: j6P41Gqt017072: to=ram, ctladdr=<root@localhost.localdomain> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30881, dsn=2.0.0, stat=Sent ► pf2csv.pl • Parsing OpenBSD pf outputRaffael Marty BCS 2006 Jakarta 32
    • 33. AfterGlow 1.x - Perl Parser AfterGlow Grapher Graph CSV File LanguageFile ► Supported graphing tools: • GraphViz from AT&T (dot, neato, circo, twopi) http://www.graphviz.org • LGL (Large Graph Layout) by Alex Adai http://bioinformatics.icmb.utexas.edu/lgl/Raffael Marty BCS 2006 Jakarta 33
    • 34. AfterGlow 1.x Features ► Generate Link Graphs ► Filtering Nodes • Based on name Fan Out: 3 • Based on number of occurrences ► Fan Out Filtering ► Coloring • Edges • Nodes ► ClusteringRaffael Marty BCS 2006 Jakarta 34
    • 35. AfterGlow 1.x Hello World Input Data: Command: a,b cat file | ./afterglow –c simple.properties –t neato –Tgif –o test.gif a,c b,c simple.properties: d,e color.source=“green” if ($fields[0] ne “d”) color.target=“blue” if ($fields[1] ne “e”) Output: d color.source=“red” color=“green” b e a cRaffael Marty BCS 2006 Jakarta 35
    • 36. AfterGlow 1.x Property File – Color Definition  Coloring: color.[source|event|target|edge]= <perl expression returning a color name>  Array @fields contains input-line, split into tokens: color.event=“red” if ($fields[1] =~ /^192..*)  Filter nodes with “invisible” color: color.target=“invisible” if ($fields[0] eq “IIS Action”)Raffael Marty BCS 2006 Jakarta 36
    • 37. AfterGlow 1.x Property File - Clustering  Clustering: cluster.[source|event|target]= <perl expression returning a cluster name>Raffael Marty BCS 2006 Jakarta 37
    • 38. AfterGlow 2.0 - Java Parser AfterGlow - Java CSV File ► Command line arguments: -h : help -c file : property file -f file : data fileRaffael Marty BCS 2006 Jakarta 38
    • 39. AfterGlow 2.0 Example ► Data: ## AfterGlow -- JAVA 2.0 AfterGlow JAVA 2.0 ## Properties File Properties File Target System Type,SIP,DIP,User,Outcome Development,192.168.10.1,10.10.2.1,ram,failure ## File to load File to load file.name=/home/ram/afterglow/data/sample.csv VPN,192.168.10.1,10.10.2.1,ram,success file.name=/home/ram/afterglow/data/sample.csv Financial System,192.168.20.1,10.0.3.1,drob,success ## Column Types (default is STRING), start with 0! VPN,192.168.10.1,10.10.2.1,ram,success Column Types (default is STRING), start with 0! ## Valid values: Valid values: VPN,192.168.10.1,10.10.2.1,jmoe,failure ## STRING STRING Financial System,192.168.10.1,10.10.2.1,jmoe,success ## INTEGER INTEGER Financial System,192.168.10.1,10.10.2.1,jmoe,failure ## CATEGORICAL CATEGORICAL column.type.count=4 column.type.count=4 ► Launch: column.type[0].column=0 column.type[0].column=0 column.type[0].type=INTEGER column.type[0].type=INTEGER column.type[1].column=1 column.type[1].column=1 ./afterglow-java.sh –c afterglow.properties column.type[1].type=CATEGORICAL column.type[1].type=CATEGORICAL column.type[2].column=2 column.type[2].column=2 column.type[2].type=CATEGORICAL column.type[2].type=CATEGORICAL column.type[3].column=3 column.type[3].column=3 column.type[3].type=CATEGORICAL column.type[3].type=CATEGORICAL ## Size Column (default is 0) Size Column (default is 0) size.column=0 size.column=0 ## Color Column (default is 0) Color Column (default is 0) color.column=2 color.column=2Raffael Marty BCS 2006 Jakarta 39
    • 40. AfterGlow 2.0 OutputRaffael Marty BCS 2006 Jakarta 40
    • 41. AfterGlow 2.0 Interaction ► Left-click: • Zoom in ► Right-click: • Zoom all the way out ► Middle-click • Change Coloring to current depth (Hack: Use SHIFT for leafs)Raffael Marty BCS 2006 Jakarta 41
    • 42. AfterGlow Firewall Log Analysis Example Input (pflog): Feb 18 13:39:15.598491 rule 71/0(match): pass in on xl0: 195.27.249.139.63263 > 195.141.69.42.80: S 492525755:492525755(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 24053 0> (DF) Feb 18 13:39:15.899644 rule 71/0(match): pass in on xl0: 195.27.249.139.63264 > 195.141.69.42.80: S 875844783:875844783(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 24054 0> (DF) Command: cat pflog | pf2csv.pl “sip dip dport” Output: 195.27.249.139,195.141.69.42,80 195.27.249.139,195.141.69.42,80 AfterGlow Input Visualization: cat pflog | pf2csv.pl “sip dip dport” | afterglow –c properties | neato –Tgif –o foo.gifRaffael Marty BCS 2006 Jakarta 42
    • 43. AfterGlow Firewall Log Analysis Example Command: cat log | grep pass_in | ./afterglow –c properties –d | dot –Tgif –o foo.gif Properties: cluster.source="External" if (!match("^195.141.69")) color=“red” if (field() eq “External”) color.event=“blue" if (regex("^195.141.69")) color.event=“lightblue” color="red" Port 100 accessRaffael Marty BCS 2006 Jakarta 43
    • 44. Summary ► Quickly Visualize Log Files • Understand Relationships • Find Outliers • Spot suspicious activity ► Visual Don’t Read Log Files Don’t Read Log Files Data Analysis Process ► AfterGlow Visualize Them!! Visualize Them!! ► Firewall Log File AnalysisRaffael Marty BCS 2006 Jakarta 44
    • 45. THANKS! raffy@arcsight.comRaffael Marty DefCon 2006 Las Vegas BCS Jakarta 45