Download to read offline
![9
history lesson
1996 – Cisco patented NetFlow
2002 – NetFlow v5
2004 – NetFlow v9
NetFlow was designed for application to network management,
but has limitations when applied to threat detection:
•NFv5 has 18 fixed fields (only 10 useful!)
•header information ONLY
•transport layer is UDP only
•no support for: MPLS, IPv6, VLANs, MAC addresses
•[typically 1:50 sampling rates]
Cisco NFv9 supports (most) of these, but is proprietary.
9](https://image.slidesharecdn.com/botprobe-reducingnetworkthreatintelligencebigdata-181127132516/75/Botprobe-Reducing-network-threat-intelligence-big-data-9-2048.jpg)
![15
case studies
further IPFIX templates:
• botprobe : botnets
• smtpprobe : spam traffic
• httpprobe : malicious http streams
• iotprobe : malicious IoT traffic
• icsprobe : malicious Industrial Control Systems traffic
if an attribute is present in a packet [header or payload], we can capture
it.
15](https://image.slidesharecdn.com/botprobe-reducingnetworkthreatintelligencebigdata-181127132516/75/Botprobe-Reducing-network-threat-intelligence-big-data-15-2048.jpg)
![21
new opportunities
template extensibility + big data reduction =
• automated mitigation
• legal interception
• pre-event forensics
• pcap indexing [flow indexing]
• new detection algorithms [not just for botnets]
21](https://image.slidesharecdn.com/botprobe-reducingnetworkthreatintelligencebigdata-181127132516/75/Botprobe-Reducing-network-threat-intelligence-big-data-21-2048.jpg)
Uploaded byDATA SECURITY SOLUTIONS
Botprobe - Reducing network threat intelligence big data
The document presents the Botprobe project, which focuses on a method to capture botnet communication traffic in cloud environments, addressing challenges in threat detection and data capture efficiency. It outlines the evolution of flow protocols from SNMP to IPFIX, emphasizing a significant reduction in data volumes and faster analysis capabilities compared to traditional methods. The paper encourages collaboration and further exploration in utilizing IPFIX templates for various malicious traffic detection applications.
More Related Content
Similar to Botprobe - Reducing network threat intelligence big data
More from DATA SECURITY SOLUTIONS
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Botprobe - Reducing network threat intelligence big data
- 1. ENRICH. ENABLE. #ISC2Summits Botprobe- Reducing Network Threat Intelligence Big Data adrian.winckles@anglia.ac.uk
- 2. 2 project background PhD: “abotnet needle in a virtual haystack” - a mechanism to capture botnet communication traffic in virtualised environments such as Cloud Service Providers. why? •cloud providers are building block for IoT •a great hosting platform for botnets interesting built environment: •tenant isolation, data privacy •internal infrastructure is an attack surface 2
- 3. 3 packet capture 3 vSwitch x CORE SWITCH CORE SWITCH EDGE SWITCH EDGE SWITCH if youwant to capture network traffic for threat detection: use wireshark
- 4. 4 packet capture 4 CORE SWITCH CORE SWITCH EDGE SWITCH EDGE SWITCH A B ThirdParty Capture Probe RSPAN VLAN EDGE SWITCH C Third Party Managed SOC
- 5. 5 packet capture Three drawbacksin this scenario: 1) port mirroring doubles network bandwidth volumes 2) assumes the monitored devices support mirroring 3) big fat pipe to send traffic to a 3rd party SOC 5CORE SWITCH CORE SWITCH EDGE SWITCH EDGE SWITCH A B Third Party Capture Probe RSPAN VLAN EDGE SWITCH C Third Party Managed SOC
- 6. 6 history lesson 1980’s Simple NetworkManagement Protocol (SNMP) • MIB information is limited, so use syslog • Syslog is unstructured 1990’s • 1991 – IETF proposed packet aggregation into flows • 1993 – Disbanded due to lack of interest • 1996 – Cisco patented NetFlow • continued… 6
- 7. 7 history lesson 1996 –Cisco patented NetFlow flow aggregates similar traffic based on an attributes tuple: e.g. 5 field flow tuple: { sIP, dIP, sPort, dPort, protocol } PCAP is a phone call flow is the phone bill (who, when, how long) 7
- 8. 8 flow export architecture 8 Network Packets ProbeCollector Storage Analysis NetFlow / IPFIX Aggregated Flow DBMS Query
- 9. 9 history lesson 1996 –Cisco patented NetFlow 2002 – NetFlow v5 2004 – NetFlow v9 NetFlow was designed for application to network management, but has limitations when applied to threat detection: •NFv5 has 18 fixed fields (only 10 useful!) •header information ONLY •transport layer is UDP only •no support for: MPLS, IPv6, VLANs, MAC addresses •[typically 1:50 sampling rates] Cisco NFv9 supports (most) of these, but is proprietary. 9
- 10. 10 IP Flow InformationeXport 2013 – IPFIX the flow export standard (RFC7011 - RFC7015) IPFIX IS A FLOW EXPORT PROTOCOL IN ITS OWN RIGHT (not NFv10) »Standards-based: vendor neutrality »Extensible: NFv5 – fixed template: 18 fields NFv9 - 79 fields (104 if Cisco) IPFIX - 433 Information Elements (IANA) »EEs create your own bespoke Enterprise Elements »Security: security by design »Future-proof: supports IPv6, MPLS and multi-cast 10
- 11. 11 botprobe template 11 10.0.2.29 210.222.39.8 53542 80 6 184785 187029 7 8 4951234579 192.168.0.1 6542876452 404 NotFound 192.168.0.1 srcIPv4 (8) 0 Bytes (4) dstIPv4 (12) 1 Bytes (4) Template ID (302) Field Count (19) srcPort (7) 4 Bytes (2) dstPort (11) 5 Bytes (2) packetTotal (2) 25 Bytes (8) flowEndMS (154) 21 Bytes (8) flowStartMS (153) 20 Bytes (8) proto (4) 6 Bytes (1) initTCPFlag (6) 29 Bytes (1) tcpSeqNos (184) 37 Bytes (4) collectorIPv4 (211) 80 Bytes (4) Template Records 16_flowKeyHash Bytes (1) 125_ircTextMessage variable 112_httpGet variable 123_httpResponse variable 1_dnsARecord Bytes (1) 6_dnsSOARecord Bytes (4) 162_smtpHello variable 41_sslName variable 10.0.2.29 210.222.39.8 53542 80 6 184785 187029 7 8 4951234579 192.168.0.1 6557765411 302 Found 192.168.0.1 10.0.2.29 210.222.39.8 53542 80 6 184785 187029 7 8 4951234579 192.168.0.1 2557373211 POST /?ptrxcz_FP 200 OK 0_sIP (8) 1_dIP (12) 4_sPort (7) 5_dPort (11) 25_packets (2) 21_eTimeMS (154) 20_sTimeMS (153) 6_protocol (4) 29_iFlags (6) 37_tcpSeq (184) 80_collector (211)
- 12. 12 botprobe performance 12 IPFIX PCAP DATAVOLUME TB MB 97%Reduction pcap botprobe datavolumes: 99.4 MB 3.0 MB load/analysis: 172.5 sec 0.2 sec
- 13. 13 botnet detection 13 repeated 30botnet experiments: »97% less capture data volume »faster capture »no change to algorithm feeds
- 14. 14 ipfix v pcap 14 PCAPIPFIX so what? SPAN mirroring Inline TAP mirroring doubles network bandwidth, TAP is more efficient dedicated infrastructure s/w probe on any device more control over data capture, lower data volumes plain Text encryption, replay protection security by design, can be sent over internet unstructured data structured data easier search TB data volumes MB data volumes 97% reduction data volumes full packet: payload L3/L7 templates- capture privacy, lawful inspection.
- 15. 15 case studies further IPFIXtemplates: • botprobe : botnets • smtpprobe : spam traffic • httpprobe : malicious http streams • iotprobe : malicious IoT traffic • icsprobe : malicious Industrial Control Systems traffic if an attribute is present in a packet [header or payload], we can capture it. 15
- 16. 16 ipfix capture 16 CORE SWITCH IPFIX COLLECTOR end device IPFIX EXPORTER IPFIX IOT sensor IPFIX EXPORTER ICS device modbus SWITCH IPFIX EXPORTER IPFIX EXPORTER IPFIX EXPORTER • softwareprobes • end-point protection • increased visibility for fewer probes • lower capture volumes
- 17. 17 adaptive capture 17 machine learninggenetic algorithms for real- time template adaption as traffic profiles change. AI 97% Less Data SOC ANALYST Network Traffic (PCAP Stream) Passive Inline Capture A B E A B C D E A B C D E A D T1 T2
- 18. 18 threat detection three keyphases of a cyber attack: infection detection response average time to detect a cyberattack is 205 days (Gartner, 2016) the cost of a cyber attack is reputational, not just financial. 18
- 19. 19 big data challenge 19 NetworkBig Data
- 20. 20 big data challenge 20 97%reduction in threat intel. data volumes 1)SOC team reacts faster to cyberattacks 2) protecting business assets and reputation
- 21. 21 new opportunities template extensibility+ big data reduction = • automated mitigation • legal interception • pre-event forensics • pcap indexing [flow indexing] • new detection algorithms [not just for botnets] 21
- 22. 22 we need you… ifyou are interested in collaboration, We’d love to talk with you: adrian.winckles@anglia.ac.uk www.botprobe.co.uk 22
Editor's Notes
- #2 From ARU, Cambridge. Research AI in threat detection. PHD – capture botnet comms traffic (ie. C&C to bot – keep alives, attack traffic). This has morphed into a method to reduce big data volumes threat detection
- #3 Wanted to do something in botnet detection. Already a bunch of v good detection algos, so decided to look into how we can make the capture process more efficient bearing in mind this is a big data environment We set our scenario in CSP because: 1. CSP integral Building block for IOT [ease of access to centralised storage of device data], starting to see device intelligence outsourced to the cloud to reduce cost of end devices (Routers, IoT devices etc) 2. Cloud is a great platform to host bots (spin up VMs with C&C (redeploy if take down), or VM with bot(s) – no AV to bypass and easy to clone) 3. Interesting build environ Got us thinking what would happen if malware could abuse the vulnerabilities in hypervisors (esp QEMU) to attack the infrastructure.
- #4 So how are we going to capture this botnet traffic? Obvious way is via full packet capture… WShark will see A to B if on separate core, but not if B on same edge switch, as wont pass core So need Wireshark on each nw segment –
- #5 Could use Port Mirroring (SPAN in cisco talk) Set up a SPAN port on a switch Mirror takes an EXACT COPY of the packet and forwards it to a collection device (probably via mgmt vlan) This collection device will send it to the analysis device (might be a 3rd party soc) (might be compression in SPAN traffic, but generating a LOAD of extra traffic)
- #6 3 DRAWBACKS: double the bandwidth (because you are mirroring) 1GB edge, what you got in the core? 10GB -> generate TBs of data! 2) assumes the monitored device supports span. What about iot hubs? You’ll probably want to send all this SPAN traffic to a 3rd party SOC (can be compressed but still need a big fat pipe) (encryption to avoid MITM)
- #7 So what else could we use? 1980s – SNMP was standard for network management. Poll a proprietary MIB on a device every x minutes regardless of a change on the device: up/down status, alters etc. SNMP limited – cannot carry large amounts of device data, so if wanted more granular information you would use syslog alongside SNMP Syslog is unstructured, must be rearranged before querying 1991 – Internet Engineering Task Force – aggregation for internet accounting.
- #8 Netflow came from Cisco Express forwarding – route the first packet, switch the rest (a way to speed up the network) It works by grouping together similar traffic streams based on similar attributes: So, if over a fixed collection period, 10 traffic streams match this tuple, they are exported as one aggregated flow. For example, the standard 5 field tuple that is used a lot to aggregate nw magmt traffic is ….. 10 streams in collection period, match tuple => 1 stream. 10 MB to 1 MB NFv5 common in nw mgmt. = 90%+ devices
- #9 Flow EXPORT The probe observes packets via a TAP, and takes a COPY of the fields needed (not the whole packet) –> Not interrupt traffic flow So 2 lots of data reduction TAP not mirror good probe do aggregation (more common in IPFIX, so NF probes aggregate but slower)
- #10 2002, 1st commercial release of netflow. Don’t get me wrong, Netflow was designed for network management statistics and it is good for that. Not so useful for threat detection. 18 fields: nw mgmt. fields – 10 any use! [don’t need input interface, next hop device, ToS, AS (Autonomous systems)] fixed at 48 bytes, you cannot chose what to capture. HEADER only – no L7 <- v imp!!! UDP - fast, but no encryption, susceptible to replay attacks, etc 1:50 = in large NW, NF not efficient (poor aggregation) so typically sample 1:50 - miss botnet comms? NFv9 to overcome this, but still proprietary.
- #11 In 2013, IETF standardised IPFIX. It was developed (using NFv9 as a base) to address the drawbacks of Netflow. export protocol in its own right. Many people call NFv9 IPFIX, or call it NFv10 For threat detection – NFv9 is non standard fore runner to IPFIX Biggest advantage in threat detection is vendor neutral VENDOR NEUTRAL = multi vendor nw EXTENSIBLE EES(L3 – L7) Security = encrypt, replay and drop packet protection Also: bi-directional flows, configurable cache timeouts Protocol: NF is UDP; IPFIX supports UDP / TCP / SCTP (TLS) Transport: TCP/SCTP (stream control transmission protocol) –congestion awareness, protects against replay attack
- #12 25 million pcap streams 30 botnet families (HTTP, IRC, Spam, etc) 300 IEs and EEs Uses rudimentary AI to select common attributes = occurrence, and relevance (basic ML) NFv5 – 48 bytes fixed, 10 fields IPFIX – 43 bytes (core), 19 fields, L7
- #13 97% vol reduction Python search for malware – 3mins v .2 sec
- #14 Repeated 30 expts (used NFv5 and/or PCAP) (2 used NFv9, 0 used IPFIX) NFv5 + PCAP = lots of data Benefits: Data volume reduction NO CHANGE TO ALGO capture ANYTHING in a packet => new detection algorithms.
- #15 IPFIX benefits: Inline tap – reduced data Sw – deploy anywhere Structured data – easier to search, deploy to graph/elk MB data L7
- #16 working on some case studies, with some organisations. More IoT and ICS [Modbus, scada]
- #17 Software on devices. Small footprint devices (tested on PI 0) IPFIX = less data
- #18 Genetic algo Real time adaption – T1 & T2 ABE -> AD What about latency here as AI updates template? (cache pcap?)
- #19 Infection -> detection = 205 days. Longest period in attack cycle, nw most vulnerable We want to reduce this infection to detection time. How much does it cost your business per hour during a cyber breach? Its not just financial, it reputation damage and brand credibility.
- #20 Threat detection is a big data challenge At botprobe, we tackle the challenges of network threat big data. Can you see threat in big data?
- #21 SCRAP the CRAP. (97% reduction) Analyst efficiency.
- #22 So we think our research opens up new opportunities that Use template Extensibility to reduce big data. Automated mitigation = less traffic for AI (needs more traffic when learning, but want less when detecting) Interception = less traffic to store/index Forensics = cannot do with PCAP (too much data) New algos – capture anything in packet [L3 – L7]
- #23 We can do a demo…