The document discusses Cisco's response to the Heartbleed vulnerability, detailing preparation and monitoring efforts that identified both benign and malicious attacks. It outlines the use of NetFlow for security analytics and forensics, highlighting the challenges and benefits of NetFlow collection and analysis. The document concludes with next steps for organizations to improve their NetFlow integration and monitoring capabilities.

1. © 2014 Lancope, Inc. All rights reserved.
Cisco CSIRT: Security Analytics and Forensics with NetFlow
Presented by:
Michael Scheck, Information Security Manager, Cisco
Paul Eckstein, CSIRT Engineering Manager, Cisco

2. © 2014 Lancope, Inc. All rights reserved.
The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by SSL
April 8, 2014: Heartbleed Vulnerability

3. © 2014 Lancope, Inc. All rights reserved.
Cisco CSIRT Response to Heartbleed
•
Preparation
•
Scanned 1.2M vulnerable servers - 300 needed repair
•
Helped develop signatures for Sourcefire and Cisco IDS
•
Deployed signatures to IDS
•
Monitoring and response
•
Discovered 25 attacks: 21 benign, 4 malicious
•
Researched attack via NetFlow analysis to discern normal connections from those that were anomalous and malicious
3

4. © 2014 Lancope, Inc. All rights reserved.
Heartbleed Benign Host
4

5. © 2014 Lancope, Inc. All rights reserved.
Heartbleed Malicious Host
5

6. © 2014 Lancope, Inc. All rights reserved.
NetFlow@Cisco History

7. © 2014 Lancope, Inc. All rights reserved.
A
B
C
C
B
A
C
A
B
NetFlow Basics
7

8. © 2014 Lancope, Inc. All rights reserved.
NetFlow Collection and Analysis Solutions
8
OSU FlowTools
nfdump
Lancope StealthWatch
License
Open source from Ohio State
Open source from SourceForge
Commercial
NetFlow versions
V5
v5 and up
v5 and up
IPv6 ready?
Yes
Yes
Yes
Syntax
Command-line, like ACLs
Command-line, like tcpdump
GUI
Support
Ad-hoc via Google Code
Up-to-date
Up-to-date

9. © 2014 Lancope, Inc. All rights reserved.
NetFlow at Cisco Before StealthWatch
•
OSU FlowTools
•
25+ systems running in parallel
-
Speeds up query time, but routers have to point at each collector
•20+ Tb of physical storage
-Files were stored in native nfdump/flowtools compressed format
•No flow aggregation
•Some connections passed through multiple devices, causing duplicate flows
•Routers splitting up long running flows
9

10. © 2014 Lancope, Inc. All rights reserved.
NetFlow Challenge:Support
•
Support of open source tools
•
OS support
•
Training staff
•
Feature requests
•
Protocol changes (NetFlow and IP)
•
Difficult to monitor for flow loss
10

11. © 2014 Lancope, Inc. All rights reserved.
NetFlow Investigation with OSU FlowTools Query
bot.acl file uses familiar ACL syntax. create a list named ‘bot’
[mynfchost]$ head bot.acl
ip access-list standard bot permit host 69.50.180.3
ip access-list standard bot permit host 66.182.153.176
[mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* | flow-filter -Sbot -o -...
Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP
0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 31337
0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83 11

12. © 2014 Lancope, Inc. All rights reserved.
NetFlow Investigation with OSU FlowTools Custom NetFlow Report Generator

13. © 2014 Lancope, Inc. All rights reserved.
Our Installation

14. © 2014 Lancope, Inc. All rights reserved.
Internet
Data Center
ISP Gateways
NetFlow Collector
DC Gateways
Corporate Backbone
NetFlow exported at network choke points
NetFlow Export at Cisco
Collect at chokepoints for egress detection
14

15. © 2014 Lancope, Inc. All rights reserved.
NetFlow Collection at Cisco with StealthWatch
15

16. © 2014 Lancope, Inc. All rights reserved.
Common collection infrastructure
•
Redundant forwarding
•
Regional storage
•
Global search
•
Applies to netflow, log collection
16

17. © 2014 Lancope, Inc. All rights reserved.
Lancope Devices and Count
StealthWatch Management Console
FlowReplicator
FlowSensor
FlowCollector
2
2
10
13
17

18. © 2014 Lancope, Inc. All rights reserved.
NetFlow Retention
18
SJC
4-18 months
RCDN
10 months
RTP 4 months
LON
26 months
BGL
5-9 months

19. © 2014 Lancope, Inc. All rights reserved.
Problems Solved

20. © 2014 Lancope, Inc. All rights reserved.
30s
30s
30s
NetFlow Challenge: Flow Timeouts
One 90s flow creates 6 flows
30s timeout 90/30 = 3 x 2 collectors
30s
30s
30s
NetFlow creates 3 flows
NetFlow creates 3 flows
Lab gateway
ISP gateway
20

21. © 2014 Lancope, Inc. All rights reserved.
Business Benefit #1 Storage Capacity
30s
30s
30s
30s
30s
30s
NetFlow creates 3 flows
NetFlow creates 3 flows
Lab gateway
ISP gateway
21

22. © 2014 Lancope, Inc. All rights reserved.
Business Benefit #2 Ease of support
•
IPv4/IPv6 both supported
•
NetFlow v5/v9 both supported
•
All supported on the same system, on the same port!
•
No system administration required
•
Alarms built in for monitoring of lost flows
22

23. © 2014 Lancope, Inc. All rights reserved.
Business Benefit #3 Ease of use
24

24. © 2014 Lancope, Inc. All rights reserved.
•
Other variables: host groups, time range, interfaces, ports
•
Defaults to 2000 flow records returned
•
Much simpler than syntax for CLI (example below)
Flow Table Query
1.
Create a file called‘flow.acl’with a named access list:
linux-machine# cat ip access-list standard botnet permit ip 10.31.33.7 >flow.acl
2. Run a query for the time period you are interested in using the ACL
linux-machine# flow-cat /var/local/flows/data/2006-12-01/ft* | flow-filter -f ~/flow.acl -Sbotnet -o -Dbotnet | flow- print -f5
25

25. © 2014 Lancope, Inc. All rights reserved.
Flow Table Output
26

26. © 2014 Lancope, Inc. All rights reserved.
FlowTable Results
Server, DNS, and Country
Traffic Type & Volume
27

27. © 2014 Lancope, Inc. All rights reserved.
NetFlow Challenge: Limited Detection Capability
•
No concept of host groups for query
•
Effective for forensics
•
Can do basic DOS detection
•
Any other queries required writing algorithms 29

28. © 2014 Lancope, Inc. All rights reserved.
Suspected Data Loss
High File Sharing Index
Max Flows Served
Business Benefit #4: Analytics
30

29. © 2014 Lancope, Inc. All rights reserved.
Use Cases

30. © 2014 Lancope, Inc. All rights reserved.
NetFlow CNC discovery
32
2. Investigate other internal hosts communicating with the same CnC
1. Detect host communicating with external Command-and-Control
3. Uncover other malicious, external entities from the compromised hosts

31. Targeted Monitoring: DoS Detection
33

32. 34
Targeted Monitoring – Data Loss

33. 35
Targeted Monitoring: Data Loss

34. © 2014 Lancope, Inc. All rights reserved.
StealthWatch Host Locking
36
Send syslog for any traffic seen between insides hosts and known C&C servers

35. © 2014 Lancope, Inc. All rights reserved.
StealthWatch Host Locking
37
Modify known C&C server list via API

36. © 2014 Lancope, Inc. All rights reserved.
CRiTs crits@mitre.org
38

37. © 2014 Lancope, Inc. All rights reserved.
CRiTS Indicator Actions
39
Prevent
DNS RPZ
host IDS
BGP
Detect
Syslog
In Progress
passive DNS
Share
Govt
Current
Future
CSIRT
Mandiant
ESA
HIPS
LUPA/ PCAP
WSA
Partner
CRITS
MD5
IPV4
Regkey
AV
SBG
CDSA
Lancope

38. CRiTS NetFlow Alarms
40

39. © 2014 Lancope, Inc. All rights reserved.
Splunk Integration – SMC Alarms
Requirement: integrate flow events with other logs for a single investigation interface
Solution: send relevant alarms as syslog messages to in-house Splunk™ architecture

40. © 2014 Lancope, Inc. All rights reserved.
StealthWatch Splunk Alerts
Link to StealthWatch host snapshot

41. © 2014 Lancope, Inc. All rights reserved.
API Use Cases
Requirement
Problem
API Script Solution
Pull all flows for given time period
SMC Flow Collector query limit
Run consecutive, small queries then concatenate
Keep SMC host groups up to date
Manual configuration, old data
Query internal source of truth, push subnet lists to host groups automatically
Look up events for a particular IP for a specific timeframe
No user attribution (yet)
Find IP and lease time from internal source of truth, query StealthWatch for related events
43

42. © 2014 Lancope, Inc. All rights reserved.
Network Subnets Mapped from IPAM

43. © 2014 Lancope, Inc. All rights reserved.
Network Subnets Map to Lancope Zones
45

44. © 2014 Lancope, Inc. All rights reserved.
Splunk integration: getFlows
Find NetFlow events via Lancope API with the respective src/dst

45. © 2014 Lancope, Inc. All rights reserved.
Splunk Integration - getFlows
47

46. © 2014 Lancope, Inc. All rights reserved.
Next Steps
How to get started:
1.
Find a collection/query system for NetFlow
2.
Export NetFlow from chokepoints
3.
Map your network context from IPAM into zones for query
4.
Configure alarms for specific zones
5.
Setup performance monitoring to mitigate flow loss from exporters
6.
Integrate with your portfolio via API
7.
Train your users and administrators – attend Lancope webinars and training
48

47. © 2014 Lancope, Inc. All rights reserved.
Contact information:
Mike: mscheck@cisco.com
Paul: peckstei@cisco.com