Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Applied Detection and Analysis 
Using Network Flow Data 
Chris Sanders and Jason Smith 
TAP Intel-Based Detection 
Mandian...
2 
Chris Sanders 
 Christian & Husband 
 Kentuckian and South Carolinian 
 MS, GSE, et al. 
 Non-Profit Director 
 BB...
3 
Jason Smith 
 Kentuckian 
 Car Aficionado 
 Raspberry Pi enthusiast 
 Junkyard Engineer
4 
Applied Network Security Monitoring 
“This book should be required 
reading for all intrusion analysts 
and those looki...
5 
Agenda 
Flow Data! 
 Why it’s important 
 How you can collect it 
 What you can do with it 
 Tools that can help 
“...
 The collection, detection, and analysis of network security 
data. 
 The goal of NSM is escalation, or to declare that ...
7 
The NSM Cycle 
Collection 
Analysis Detection
8 
Evolution of NSM Emphasis 
Past 
• Detection Era 
Present 
• Collection Era 
Future • Analysis Era
9 
NSM/IR Challenges of the Present 
We All Want Full PCAP… 
 Collection 
− Easy to Capture / Filter Stream Data 
 Detec...
10 
NSM/IR Challenges of the Present 
But, It’s not Feasible for Every Goal… 
 Collection 
− Not Scalable for Extended Re...
11 
Full PCAP vs. Flow Data 
PCAP Data Flow Data
12 
Flow Data 
 Often Called Flow / Session / NetFlow 
 Summary of Network Communications 
 Aggregated Record of Packet...
13 
Flow Data Example 
sTime| sIP|dPort| dIP|dPort|pro|bytes| 
2014/09/22T00:03:58.756| 10.10.120.1| 53| 10.1.179.5| 53| 1...
14 
Building Flow Records 
 Records are Defined by 
Unique 5-tuples 
 Data is added to the 5-tuple 
Record until a termi...
15 
Flow Record Termination Conditions 
 Natural Timeout 
− End of communication per protocol (ex. TCP RST/FIN) 
 Idle T...
Collection with Flow Data
17 
Generating Flow Data 
 Generation 
− Routers 
− Sensors 
 Fprobe 
 YAF 
 Multiple Types: 
− NetFlow (v5,v9) 
− IPF...
18 
Collecting Flow Data 
 Popular Platforms 
− Argus 
+ Reliable + Fast Collection 
- Not Well Supported 
− NFDump 
+ Ea...
19 
SiLK 
 The System for Internet-Level Knowledge 
 CERT NetSA Team 
 Two Major Components: 
− Packing Suite 
 Collec...
20 
SiLK Collection Architecture
21 
SiLK – What You Need 
 Flow Sources 
− Hardware: Routers, Switches 
− Software: YAF, fprobe 
 SiLK Server 
− Rwflowp...
22 
SiLK – Analysis Suite 
 rwfilter - Filters through data based on conditions. 
 rwcut - Converts flow binary data to ...
23 
SiLK Analysis – rwfilter / rwcut (1) 
 Display all records from the beginning the current day 
until the current time...
24 
SiLK Analysis – rwfilter / rwcut (2) 
 Display all records of communication to or from Chinese 
IP addresses over a s...
25 
SiLK Analysis – rwstats (1) 
 Display statistics for the total amount of bytes transferred by protocol 
(top 10): 
rw...
26 
SiLK Analysis – rwstats (2) 
 Show the top 10 sip,dip pairs for valid conversations (top 
10) 
rwfilter --type=all --...
27 
SiLK Analysis – rwstats (3) 
 Show the top 10 outbound destination country codes by 
records: 
rwfilter --type=out,ou...
28 
SiLK Analysis – Real World Examples 
 Rwstats to discover potential ZeroAccess victims 
rwfilter --type=all --dport=1...
29 
SiLK Analysis – Real World Examples 
 Discovering outbound data to applications using nonstandard ports. 
rwfilter Sa...
30 
Collecting Intelligence Data 
 Friendly Intelligence Gathering 
 Identify Services on the Network 
 Identify Normal...
31 
Identifying Services 
 Identify SSH Servers 
rwfilter --type=out --protocol=6 --packets=4- - 
-ack-flag=1 --sport=22 ...
32 
Identifying Friends and Family 
• Identify Friends 
rwfilter --type=out,outweb -- 
saddress=192.168.1.1 --pass=stdout ...
Detection with Flow Data
34 
Flow for Detection 
Signature- 
Based 
Reputation- 
Based 
Behavior- 
Based 
Statistics- 
Based
35 
FlowPlotter 
 Generates Visualizations from the Output of Flow Tools 
 Useful for Detection-Oriented Statistics 
 W...
36 
FlowPlotter - GeoMap 
 rwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout | 
./flowplotter.sh geomap dcc b...
37 
FlowPlotter - LineChart 
 rwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh 
linechart 600 bytes > l...
38 
FlowPlotter - TreeMap 
 rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- -- 
proto=0- --type=out --pass=s...
39 
FlowPlotter - Pie/Bar/Column Chart 
 rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- -- 
proto=0- --type...
40 
FlowPlotter - BubbleChart 
 rwfilter ../Sampledata/sample.rw --type=all --proto=0-255 -- 
pass=stdout | ./flowplotter...
41 
FlowPlotter - Timeline 
 rwfilter --proto=0- --type=out --sport=41142 --pass=stdout | 
./flowplotter.sh timeline dip ...
42 
FlowPlotter - Force Directed 
 rwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all -- 
pass=stdout | ./fl...
43 
FlowPlotter - AssetDiscovery 
 rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout | 
./flowplotter....
Analysis with Flow
45 
Flow in Analysis – PCAP Only 
Validate Signature TP < 1% 
Scoping Relevant Time Range 
Find Related Events in Time Ran...
46 
Flow in Analysis - Improved 
Validate Signature TP < 1% 
Scoping Relevant Time Range 
Find Related Events in Time Rang...
47 
Flow – Barriers to Entry 
 Be Prepared to Look at a LOT of Line-Based Data 
 Very Command Line Oriented 
 Not Welco...
48 
SiLK Data Output
49
50 
 Flow Basic Analysis Tool 
 Graphical Front-End to SiLK 
 Easy Two-Step Install on SiLK Capable Box 
− Install Loca...
51 
Getting Data with FlowBAT (CLI Mode)
Getting Data with FlowBAT (Guided Mode) 
52
53 
Manipulating FlowBAT Data
54 
Pivoting with FlowBAT Data
55 
Generating Stats with FlowBAT
56 
Generating Stats with FlowBAT
57 
Conclusion 
 Flow Data is Underused and Underrated 
 Easy to Collect, Enhances Detection & Analysis 
 Minimal Barri...
58 
Thanks Folks! 
 Questions? 
− Chris Sanders – chris@chrissanders.org 
− Jason Smith – jason.smith.webmail@gmail.com 
...
Upcoming SlideShare
Loading in …5
×

Applied Detection and Analysis Using Flow Data - MIRCon 2014

24,273 views

Published on

In this presentation, Chris Sanders and Jason Smith discuss the importance of using flow data for network security analysis. Flow data is discussed from the viewpoints of collection, detection, and analysis. We also discuss the FlowPlotter tool, and the use of FlowBAT, a graphical flow analysis GUI we've created.

Published in: Technology

Applied Detection and Analysis Using Flow Data - MIRCon 2014

  1. 1. Applied Detection and Analysis Using Network Flow Data Chris Sanders and Jason Smith TAP Intel-Based Detection Mandiant, a FireEye Company
  2. 2. 2 Chris Sanders  Christian & Husband  Kentuckian and South Carolinian  MS, GSE, et al.  Non-Profit Director  BBQ Pit Master
  3. 3. 3 Jason Smith  Kentuckian  Car Aficionado  Raspberry Pi enthusiast  Junkyard Engineer
  4. 4. 4 Applied Network Security Monitoring “This book should be required reading for all intrusion analysts and those looking to develop a security monitoring program.” “Written by analysts, for analysts.” - Amazon Reviewers
  5. 5. 5 Agenda Flow Data!  Why it’s important  How you can collect it  What you can do with it  Tools that can help “Why/How to use Flow Data in NSM/IR”
  6. 6.  The collection, detection, and analysis of network security data.  The goal of NSM is escalation, or to declare that an incident has occurred so that incident response can occur. 6 Network Security Monitoring Network Security Monitoring Incident Response
  7. 7. 7 The NSM Cycle Collection Analysis Detection
  8. 8. 8 Evolution of NSM Emphasis Past • Detection Era Present • Collection Era Future • Analysis Era
  9. 9. 9 NSM/IR Challenges of the Present We All Want Full PCAP…  Collection − Easy to Capture / Filter Stream Data  Detection − Major Detection Tools are PCAP Oriented  Analysis − Gives us Who, Where, When, and What
  10. 10. 10 NSM/IR Challenges of the Present But, It’s not Feasible for Every Goal…  Collection − Not Scalable for Extended Retention  Detection − Not Ideal of Hunting / Rapid Pivoting  Analysis − Not a Great Starting Point
  11. 11. 11 Full PCAP vs. Flow Data PCAP Data Flow Data
  12. 12. 12 Flow Data  Often Called Flow / Session / NetFlow  Summary of Network Communications  Aggregated Record of Packets  Gives Us Who, Where, When  Based on the 5-tuple + Timing/Data Stats Source IP Source Port Dest IP Dest Port Protocol 192.168.5.1 48293 8.8.8.8 53 UDP Start Time End Time Bytes 2014/09/22T00:03:58.756 2014/09/22T00:04:58.756 76
  13. 13. 13 Flow Data Example sTime| sIP|dPort| dIP|dPort|pro|bytes| 2014/09/22T00:03:58.756| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72| 2014/09/22T00:03:58.999 10.10.120.1| 53| 10.1.188.5| 53| 17| 89| 2014/09/22T00:08:59.012| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72| 2014/09/22T00:08:59.466| 10.10.120.1| 53| 10.1.188.5| 53| 17| 89| 2014/09/22T00:03:58.756| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72| 2014/09/22T00:03:58.999 10.10.120.1| 53| 10.1.188.5| 53| 17| 89| 2014/09/22T00:08:59.012| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72| 2014/09/22T00:08:59.466| 10.10.120.1| 53| 10.1.188.5| 53| 17| 89|
  14. 14. 14 Building Flow Records  Records are Defined by Unique 5-tuples  Data is added to the 5-tuple Record until a termination condition is met.
  15. 15. 15 Flow Record Termination Conditions  Natural Timeout − End of communication per protocol (ex. TCP RST/FIN)  Idle Timeout − No data received for 30 seconds  Active Timeout − Thirty minute max timeout (configurable)
  16. 16. Collection with Flow Data
  17. 17. 17 Generating Flow Data  Generation − Routers − Sensors  Fprobe  YAF  Multiple Types: − NetFlow (v5,v9) − IPFIX − jFlow − More…
  18. 18. 18 Collecting Flow Data  Popular Platforms − Argus + Reliable + Fast Collection - Not Well Supported − NFDump + Easy to Setup and Use - Not in Wide Use − SiLK + Exceptional Analysis Tools - More Involved Setup
  19. 19. 19 SiLK  The System for Internet-Level Knowledge  CERT NetSA Team  Two Major Components: − Packing Suite  Collection and parsing of flow data − Analysis Suite  Filter, display, sort, count, group, mate, and more  Excellent Documentation & Community − https://tools.netsa.cert.org/silk/docs.html
  20. 20. 20 SiLK Collection Architecture
  21. 21. 21 SiLK – What You Need  Flow Sources − Hardware: Routers, Switches − Software: YAF, fprobe  SiLK Server − Rwflowpack − Will also have SiLK analysis suite installed  Analyst Workstation − Access SiLK server directly − Locally mirrored database
  22. 22. 22 SiLK – Analysis Suite  rwfilter - Filters through data based on conditions.  rwcut - Converts flow binary data to a human readable format.  rwstats - Generates statistics from flow data  rwcount - Summarizes total network traffic over time
  23. 23. 23 SiLK Analysis – rwfilter / rwcut (1)  Display all records from the beginning the current day until the current time: rwfilter --type=all --proto=0-255 --pass=stdout | rwcut
  24. 24. 24 SiLK Analysis – rwfilter / rwcut (2)  Display all records of communication to or from Chinese IP addresses over a specific week to one local CIDR range: rwfilter --type=all --start-date=2014/08/01 --end-date= 2014/08/07 --any-address=192.168.1.0/24 --any-cc=cn --pass=stdout | rwcut -- fields=stime,sip,dip,sport,dport,type
  25. 25. 25 SiLK Analysis – rwstats (1)  Display statistics for the total amount of bytes transferred by protocol (top 10): rwfilter --type=all --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=proto -- value=bytes
  26. 26. 26 SiLK Analysis – rwstats (2)  Show the top 10 sip,dip pairs for valid conversations (top 10) rwfilter --type=all --proto=0-255 --packets=4, -- pass=stdout | rwstats --top --count=10 -- fields=sip,dip --value=bytes
  27. 27. 27 SiLK Analysis – rwstats (3)  Show the top 10 outbound destination country codes by records: rwfilter --type=out,outweb --proto=0-255 -- pass=stdout | rwstats --top --count=10 --fields=dcc
  28. 28. 28 SiLK Analysis – Real World Examples  Rwstats to discover potential ZeroAccess victims rwfilter --type=all --dport=16464,16465,16470,16471 -- pass=stdout | rwstats --top --fields=sip -- value=distinct:dcc --threshold=3
  29. 29. 29 SiLK Analysis – Real World Examples  Discovering outbound data to applications using nonstandard ports. rwfilter Sampledata/sample.rw --plugin=app-mismatch.so --type=out,outweb --proto=6 --sport=1024- --packets=4- --flags-initial=S/SURFPACE --pass=stdout | rwstats --fields=application,dport --count=100 --distinct:dport
  30. 30. 30 Collecting Intelligence Data  Friendly Intelligence Gathering  Identify Services on the Network  Identify Normal Behaviors of Hosts  Identify “Friends and Family” − Friends: Who a host communicates with outside the network − Family: Who a host communicates with inside the network
  31. 31. 31 Identifying Services  Identify SSH Servers rwfilter --type=out --protocol=6 --packets=4- - -ack-flag=1 --sport=22 --pass=stdout | rwcut -- fields=sip  Identify Web Servers rwfilter --type=outweb --protocol=6 -- packets=4- --ack-flag=1 --sport=80,443,8080 -- pass=stdout | rwcut --fields=sip
  32. 32. 32 Identifying Friends and Family • Identify Friends rwfilter --type=out,outweb -- saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin -- dcidr=192.168.0.0/24 --fail=stdout • Identify Family rwfilter --type=out,outweb -- saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin -- dipset=local --fail=stdout
  33. 33. Detection with Flow Data
  34. 34. 34 Flow for Detection Signature- Based Reputation- Based Behavior- Based Statistics- Based
  35. 35. 35 FlowPlotter  Generates Visualizations from the Output of Flow Tools  Useful for Detection-Oriented Statistics  Written in BASH – Flexible/Tweakable/Minimal  Free/Open Source - Maintained in GitHub  Browser Independent
  36. 36. 36 FlowPlotter - GeoMap  rwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout | ./flowplotter.sh geomap dcc bytes > geomap.html
  37. 37. 37 FlowPlotter - LineChart  rwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh linechart 600 bytes > linechart.html
  38. 38. 38 FlowPlotter - TreeMap  rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- -- proto=0- --type=out --pass=stdout | ./flowplotter.sh treemap dip records > treemap.html
  39. 39. 39 FlowPlotter - Pie/Bar/Column Chart  rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- -- proto=0- --type=all --pass=stdout | ./flowplotter.sh piechart dip bytes > piechart.html
  40. 40. 40 FlowPlotter - BubbleChart  rwfilter ../Sampledata/sample.rw --type=all --proto=0-255 -- pass=stdout | ./flowplotter.sh bubblechart sip > bubblechart.html
  41. 41. 41 FlowPlotter - Timeline  rwfilter --proto=0- --type=out --sport=41142 --pass=stdout | ./flowplotter.sh timeline dip sip > timeline.html
  42. 42. 42 FlowPlotter - Force Directed  rwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all -- pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 > forcetest.html
  43. 43. 43 FlowPlotter - AssetDiscovery  rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout | ./flowplotter.sh assetdiscovery > assettest.html
  44. 44. Analysis with Flow
  45. 45. 45 Flow in Analysis – PCAP Only Validate Signature TP < 1% Scoping Relevant Time Range Find Related Events in Time Range Reduce Data Set Analyze / Make Decisions 5% 10% 35% ~ 50% * Based on the First Hour of Analysis
  46. 46. 46 Flow in Analysis - Improved Validate Signature TP < 1% Scoping Relevant Time Range Find Related Events in Time Range Expand Data Set as Needed Analyze / Make Decisions 5% 10% 5% ~ 80% * Based on the First Hour of Analysis
  47. 47. 47 Flow – Barriers to Entry  Be Prepared to Look at a LOT of Line-Based Data  Very Command Line Oriented  Not Welcoming to Junior-Level Analysts  Hard to Display/Interpret Data Visually
  48. 48. 48 SiLK Data Output
  49. 49. 49
  50. 50. 50  Flow Basic Analysis Tool  Graphical Front-End to SiLK  Easy Two-Step Install on SiLK Capable Box − Install Locally to SiLK Box − Install Remotely and Interact via SSH w/ Keys  Rapid Pivoting Between Data  Graphing Ability  By Analysts, for Analysts
  51. 51. 51 Getting Data with FlowBAT (CLI Mode)
  52. 52. Getting Data with FlowBAT (Guided Mode) 52
  53. 53. 53 Manipulating FlowBAT Data
  54. 54. 54 Pivoting with FlowBAT Data
  55. 55. 55 Generating Stats with FlowBAT
  56. 56. 56 Generating Stats with FlowBAT
  57. 57. 57 Conclusion  Flow Data is Underused and Underrated  Easy to Collect, Enhances Detection & Analysis  Minimal Barriers to Entry − SiLK (Easy to Install on SO) − Argus (Already Installed on SO) − Bro (Already Installed on SO)
  58. 58. 58 Thanks Folks!  Questions? − Chris Sanders – chris@chrissanders.org − Jason Smith – jason.smith.webmail@gmail.com  Blog / Book − http://www.appliednsm.com  FlowPlotter − http://www.github.com/automayt/FlowPlotter/  FlowBAT − http://www.flowbat.com

×