Cisco CSIRT Case Study: Forensic Investigations with NetFlow

3,825 views
3,638 views

Published on

Cisco CSIRT uses NetFlow to collect 16 billion flows from Cisco’s 175TB of traffic observed daily. The data is used to monitor, investigate, and contain incidents using 3 key playbook “plays” each day.

Two leaders from Cisco's Computer Security Incident Response Team (CSIRT) will review a real cyber incident and the resulting investigation leveraging NetFlow collected via the StealthWatch System.

Participants will learn how to use NetFlow and the StealthWatch System to:

Investigate top use cases: C&C discovery, data loss and DOS attacks
Gain contextual awareness of network activity
Accelerate incident response
Minimize costly outages and downtime from threats
Protect the evolving network infrastructure
Provide forensic evidence to prosecute adversaries

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,825
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
95
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cisco CSIRT Case Study: Forensic Investigations with NetFlow

  1. 1. © 2014 Lancope, Inc. All rights reserved. Cisco CSIRT: Security Analytics and Forensics with NetFlow Presented by: Michael Scheck, Information Security Manager, Cisco Paul Eckstein, CSIRT Engineering Manager, Cisco
  2. 2. © 2014 Lancope, Inc. All rights reserved. The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by SSL April 8, 2014: Heartbleed Vulnerability
  3. 3. © 2014 Lancope, Inc. All rights reserved. Cisco CSIRT Response to Heartbleed • Preparation • Scanned 1.2M vulnerable servers - 300 needed repair • Helped develop signatures for Sourcefire and Cisco IDS • Deployed signatures to IDS • Monitoring and response • Discovered 25 attacks: 21 benign, 4 malicious • Researched attack via NetFlow analysis to discern normal connections from those that were anomalous and malicious 3
  4. 4. © 2014 Lancope, Inc. All rights reserved. Heartbleed Benign Host 4
  5. 5. © 2014 Lancope, Inc. All rights reserved. Heartbleed Malicious Host 5
  6. 6. © 2014 Lancope, Inc. All rights reserved. NetFlow@Cisco History
  7. 7. © 2014 Lancope, Inc. All rights reserved. A B C C B A C A B NetFlow Basics 7
  8. 8. © 2014 Lancope, Inc. All rights reserved. NetFlow Collection and Analysis Solutions 8 OSU FlowTools nfdump Lancope StealthWatch License Open source from Ohio State Open source from SourceForge Commercial NetFlow versions V5 v5 and up v5 and up IPv6 ready? Yes Yes Yes Syntax Command-line, like ACLs Command-line, like tcpdump GUI Support Ad-hoc via Google Code Up-to-date Up-to-date
  9. 9. © 2014 Lancope, Inc. All rights reserved. NetFlow at Cisco Before StealthWatch • OSU FlowTools • 25+ systems running in parallel - Speeds up query time, but routers have to point at each collector •20+ Tb of physical storage -Files were stored in native nfdump/flowtools compressed format •No flow aggregation •Some connections passed through multiple devices, causing duplicate flows •Routers splitting up long running flows 9
  10. 10. © 2014 Lancope, Inc. All rights reserved. NetFlow Challenge:Support • Support of open source tools • OS support • Training staff • Feature requests • Protocol changes (NetFlow and IP) • Difficult to monitor for flow loss 10
  11. 11. © 2014 Lancope, Inc. All rights reserved. NetFlow Investigation with OSU FlowTools Query bot.acl file uses familiar ACL syntax. create a list named ‘bot’ [mynfchost]$ head bot.acl ip access-list standard bot permit host 69.50.180.3 ip access-list standard bot permit host 66.182.153.176 [mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* | flow-filter -Sbot -o -... Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP 0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 31337 0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83 11
  12. 12. © 2014 Lancope, Inc. All rights reserved. NetFlow Investigation with OSU FlowTools Custom NetFlow Report Generator
  13. 13. © 2014 Lancope, Inc. All rights reserved. Our Installation
  14. 14. © 2014 Lancope, Inc. All rights reserved. Internet Data Center ISP Gateways NetFlow Collector DC Gateways Corporate Backbone NetFlow exported at network choke points NetFlow Export at Cisco Collect at chokepoints for egress detection 14
  15. 15. © 2014 Lancope, Inc. All rights reserved. NetFlow Collection at Cisco with StealthWatch 15
  16. 16. © 2014 Lancope, Inc. All rights reserved. Common collection infrastructure • Redundant forwarding • Regional storage • Global search • Applies to netflow, log collection 16
  17. 17. © 2014 Lancope, Inc. All rights reserved. Lancope Devices and Count StealthWatch Management Console FlowReplicator FlowSensor FlowCollector 2 2 10 13 17
  18. 18. © 2014 Lancope, Inc. All rights reserved. NetFlow Retention 18 SJC 4-18 months RCDN 10 months RTP 4 months LON 26 months BGL 5-9 months
  19. 19. © 2014 Lancope, Inc. All rights reserved. Problems Solved
  20. 20. © 2014 Lancope, Inc. All rights reserved. 30s 30s 30s NetFlow Challenge: Flow Timeouts One 90s flow creates 6 flows 30s timeout 90/30 = 3 x 2 collectors 30s 30s 30s NetFlow creates 3 flows NetFlow creates 3 flows Lab gateway ISP gateway 20
  21. 21. © 2014 Lancope, Inc. All rights reserved. Business Benefit #1 Storage Capacity 30s 30s 30s 30s 30s 30s NetFlow creates 3 flows NetFlow creates 3 flows Lab gateway ISP gateway 21
  22. 22. © 2014 Lancope, Inc. All rights reserved. Business Benefit #2 Ease of support • IPv4/IPv6 both supported • NetFlow v5/v9 both supported • All supported on the same system, on the same port! • No system administration required • Alarms built in for monitoring of lost flows 22
  23. 23. © 2014 Lancope, Inc. All rights reserved. Business Benefit #3 Ease of use 24
  24. 24. © 2014 Lancope, Inc. All rights reserved. • Other variables: host groups, time range, interfaces, ports • Defaults to 2000 flow records returned • Much simpler than syntax for CLI (example below) Flow Table Query 1. Create a file called‘flow.acl’with a named access list: linux-machine# cat ip access-list standard botnet permit ip 10.31.33.7 >flow.acl 2. Run a query for the time period you are interested in using the ACL linux-machine# flow-cat /var/local/flows/data/2006-12-01/ft* | flow-filter -f ~/flow.acl -Sbotnet -o -Dbotnet | flow- print -f5 25
  25. 25. © 2014 Lancope, Inc. All rights reserved. Flow Table Output 26
  26. 26. © 2014 Lancope, Inc. All rights reserved. FlowTable Results Server, DNS, and Country Traffic Type & Volume 27
  27. 27. © 2014 Lancope, Inc. All rights reserved. NetFlow Challenge: Limited Detection Capability • No concept of host groups for query • Effective for forensics • Can do basic DOS detection • Any other queries required writing algorithms 29
  28. 28. © 2014 Lancope, Inc. All rights reserved. Suspected Data Loss High File Sharing Index Max Flows Served Business Benefit #4: Analytics 30
  29. 29. © 2014 Lancope, Inc. All rights reserved. Use Cases
  30. 30. © 2014 Lancope, Inc. All rights reserved. NetFlow CNC discovery 32 2. Investigate other internal hosts communicating with the same CnC 1. Detect host communicating with external Command-and-Control 3. Uncover other malicious, external entities from the compromised hosts
  31. 31. Targeted Monitoring: DoS Detection 33
  32. 32. 34 Targeted Monitoring – Data Loss
  33. 33. 35 Targeted Monitoring: Data Loss
  34. 34. © 2014 Lancope, Inc. All rights reserved. StealthWatch Host Locking 36 Send syslog for any traffic seen between insides hosts and known C&C servers
  35. 35. © 2014 Lancope, Inc. All rights reserved. StealthWatch Host Locking 37 Modify known C&C server list via API
  36. 36. © 2014 Lancope, Inc. All rights reserved. CRiTs crits@mitre.org 38
  37. 37. © 2014 Lancope, Inc. All rights reserved. CRiTS Indicator Actions 39 Prevent DNS RPZ host IDS BGP Detect Syslog In Progress passive DNS Share Govt Current Future CSIRT Mandiant ESA HIPS LUPA/ PCAP WSA Partner CRITS MD5 IPV4 Regkey AV SBG CDSA Lancope
  38. 38. CRiTS NetFlow Alarms 40
  39. 39. © 2014 Lancope, Inc. All rights reserved. Splunk Integration – SMC Alarms Requirement: integrate flow events with other logs for a single investigation interface Solution: send relevant alarms as syslog messages to in-house Splunk™ architecture
  40. 40. © 2014 Lancope, Inc. All rights reserved. StealthWatch Splunk Alerts Link to StealthWatch host snapshot
  41. 41. © 2014 Lancope, Inc. All rights reserved. API Use Cases Requirement Problem API Script Solution Pull all flows for given time period SMC Flow Collector query limit Run consecutive, small queries then concatenate Keep SMC host groups up to date Manual configuration, old data Query internal source of truth, push subnet lists to host groups automatically Look up events for a particular IP for a specific timeframe No user attribution (yet) Find IP and lease time from internal source of truth, query StealthWatch for related events 43
  42. 42. © 2014 Lancope, Inc. All rights reserved. Network Subnets Mapped from IPAM
  43. 43. © 2014 Lancope, Inc. All rights reserved. Network Subnets Map to Lancope Zones 45
  44. 44. © 2014 Lancope, Inc. All rights reserved. Splunk integration: getFlows Find NetFlow events via Lancope API with the respective src/dst
  45. 45. © 2014 Lancope, Inc. All rights reserved. Splunk Integration - getFlows 47
  46. 46. © 2014 Lancope, Inc. All rights reserved. Next Steps How to get started: 1. Find a collection/query system for NetFlow 2. Export NetFlow from chokepoints 3. Map your network context from IPAM into zones for query 4. Configure alarms for specific zones 5. Setup performance monitoring to mitigate flow loss from exporters 6. Integrate with your portfolio via API 7. Train your users and administrators – attend Lancope webinars and training 48
  47. 47. © 2014 Lancope, Inc. All rights reserved. Contact information: Mike: mscheck@cisco.com Paul: peckstei@cisco.com

×