Creating Your Own Threat Intel Through Hunting & Visualization
Report
Raffael MartyFollow
Feb. 9, 2016•0 likes•25,179 views
Creating Your Own Threat Intel Through Hunting & Visualization
Feb. 9, 2016•0 likes•25,179 views
Download to read offline
Report
Internet
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start 'hunting' for signs of compromises and anomalies in our own environments. In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company. Visualization. Data science. No machine learning. But pretty pictures. Here is a blog post I wrote a bit ago about the general theme of internal threat intelligence: http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225?
Raffael MartyFollow
Recommended
Threat hunting and achieving security maturityDNIF
OSINT: Open Source Intelligence gatheringJeremiah Tillman
Cyber Threat Hunting with PhirelightHostway|HOSTING
Bsides 2019 - Intelligent Threat HuntingDhruv Majumdar
Threat Intelligence 101 - Steve Lodin - SubmittedSteve Lodin
Cyber Threat Hunting: Identify and Hunt Down IntrudersInfosec
More Related Content
What's hot
BSidesLV 2018 - Katie Nickels and John Wunder - ATT&CKing the Status QuoKatie Nickels
Cyber threat Intelligence and Incident Response by:-Sandeep SinghOWASP Delhi
Cyber Threat Hunting WorkshopDigit Oktavianto
Cyber Threat hunting workshopArpan Raval
Threat hunting foundations: People, process and technology.pptxInfosec
Let’s hunt the target using OSINTChandrapal Badshah
Similar to Creating Your Own Threat Intel Through Hunting & Visualization
Protecting Financial Networks from Cyber CrimeLancope, Inc.
RIoT (Raiding Internet of Things) by Jacob HolcombPriyanka Aash
Splunk Enterpise for Information Security Hands-OnSplunk
Cybersecurity - Jim ButterworthTechBiz Forense Digital
Splunk Enterprise for InfoSec Hands-On Breakout SessionSplunk
Understanding Intrusion Detection & Prevention Systems (1).pptxRineri1
More from Raffael Marty
Exploring the Defender's AdvantageRaffael Marty
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...Raffael Marty
How To Drive Value with Security DataRaffael Marty
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Raffael Marty
Artificial Intelligence – Time Bomb or The Promised Land?Raffael Marty
Understanding the "Intelligence" in AIRaffael Marty
Recently uploaded
Web3_Metaverse_Foundations - DTW Coppenhagen - FINAL - 230919Michael Lesniak
![[FediForum] Reisman FairPay - Rethinking Revenue.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/fediforumreismanfairpay-rethinkingrevenue-230921142132-d55847d0-thumbnail.jpg?width=384&height=256&fit=bounds)
[FediForum] Reisman FairPay - Rethinking Revenue.pdfTeleshuttle Corporation
Networking Basic Refresh.pdfCS Knowledge
CASE STUDY.pdfShivamYadav8517
The value of measuring your accessibility maturityIntopia
Microsoft Blockchain Case Studies.pptxJoelJohn481077
![[FediForum] Reisman FairPay - Rethinking Revenue.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/fediforumreismanfairpay-rethinkingrevenue-230921142132-d55847d0-thumbnail.jpg?width=1600&height=908&fit=bounds)
Creating Your Own Threat Intel Through Hunting & Visualization
- 1. Raffael Marty, CEO Creating Your Own Threat Intel Through Hunting & Visualization Tenerife, Spain February, 2016
- 2. Creating Your Own Threat Intel Through Hunting & Visualization Raffael Marty, CEO
- 3. Security. Analytics. Insight.3 Contents HUNTING AKA INTERNAL THREAT INTELLIGENCE THREAT INTELLIGENCE A PROCESS AND INFRASTRUCTURE VIEW 1 2 VISUALIZATION A THREAT INTELLIGENCE GOLD MINE 3
- 4. Threat Intelligence
- 5. Security. Analytics. Insight.5 • Products / Tools • Firewall - Blocks traffic based on pre-defined rules • Web Application Firewall - Monitors for signs of known malicious activity in Web traffic • Intrusion Prevention System - Looks for ‘signs’ of known attacks in traffic and protocol violations • Anti Virus - Looks for ‘signs’ of known attacks on the end system • Malware Sandbox - Runs new binaries and monitors their behavior for malicious signs • Security Information Management - Uses pre-defined rules to correlate signs from different data streams to augment intelligence • Vulnerability Scanning - Searches for known vulnerabilities and vulnerable software • Rely on pattern matching and signatures based knowledge from the past • Reactive -> always behind • Unknown and new threats -> won’t be detected • ‘Imperfect’ patterns and rules -> cause a lot of false positives We Are Monitoring - What is Going Wrong? Defense Has Been Relying On Past Knowledge
- 6. Security. Analytics. Insight.6 Event Funnel - How We Used To Do It data rule-based correlation prioritization simple statistics attack candidates • What rules do you write? • Do the vendor provided rules work for you? • How do you define a priority 10 event? • High false positive rate! • Unless alerts are VERY focussed • High false negative rate! • Do you know what you don’t know?
- 7. Security. Analytics. Insight.7 Then Came Threat Intelligence • How many hits do you really get? • You are missing most attacks IOCs • How do you match these efficiently against a real-time stream? • How do you de- duplicate and normalize these feeds? attack candidates 70–90% OF MALWARE SAMPLES ARE UNIQUE TO AN ORGANIZATION.
- 8. Security. Analytics. Insight.8 Removing the Event Funnel - Hello Data Lake any data Big Data Lake Rules • Storing more, and more diverse data • Kafka and “dynamic parsing” • Enabling large-scale processing • Spark, SparkStreaming, Storm, Parquet • Using “standard” data access (SQL, REST) • Plug in any other tool! context IOCs This per-se is not new …
- 9. Security. Analytics. Insight.9 Adding Interactive - Analyst Driven Exploration any data Big Data Lake Rules context IOCs … but first we get the human in the loop … Hunting • interactive visualization • analyst driven • machine assisted
- 10. Security. Analytics. Insight.10 Hunting Creates Internal Threat Intelligence any data Big Data Lake Rules context IOCs … then, let’s rethink our rules … Novel, Advanced Attacks internal TI
- 11. Security. Analytics. Insight.11 Hunting Creates Internal Threat Intelligence any data Big Data Lake Rules context IOCs … then, let’s rethink our rules … patterns anyone? internal TI Novel, Advanced Attacks Low False Positive Alerts Patterns
- 12. Security. Analytics. Insight.12 Buzzword Bingo any data Big Data Lake Rules context IOCs … and finally, we are buzzword compliant … behavioral monitoring scoring anomaly detection machine learning artificial intelligence “models” data science internal TI Patterns
- 13. Security. Analytics. Insight.13 How Does All That Architecture Stuff Matter? In the following we’ll explore how this all matters … … but first, let’s see how visualization plays a key role here.
- 14. 14Visualization
- 15. Security. Analytics. Insight.15 “How Can We See, Not To Confirm - But To Learn” - Edward Tufte
- 16. Security. Analytics. Insight.16 Why Visualization?dport time
- 17. Security. Analytics. Insight.17 SELECT count(distinct protocol) FROM flows; SELECT count(distinct port) FROM flows; SELECT count(distinct src_network) FROM flows; SELECT count(distinct dest_network) FROM flows; SELECT port, count(*) FROM flows GROUP BY port; SELECT protocol, count(CASE WHEN flows < 200 THEN 1 END) AS [<200], count(CASE WHEN flows>= 201 AND flows < 300 THEN 1 END) AS [201 - 300], count(CASE WHEN flows>= 301 AND flows < 350 THEN 1 END) AS [301 - 350], count(CASE WHEN flows>= 351 THEN 1 END) AS [>351] FROM flows GROUP BY protocol; SELECT port, count(distinct src_network) FROM flows GROUP BY port; SELECT src_network, count(distinct dest_network) FROM flows GROUP BY port; SELECT src_network, count(distinct dest_network) AS dn, sum(flows) FROM flows GROUP BY port, dn; SELECT port, protocol, count(*) FROM flows GROUP BY port, protocol; SELECT sum(flows), dest_network FROM flows GROUP BY dest_network; … One Graph Summarizes Dozens of Queries port dest_network protocol src_network flows
- 18. Security. Analytics. Insight.18 Visualization To … Present / Communicate Discover / Explore
- 19. Security. Analytics. Insight.19 We will have a look at a couple components from earlier: • Context • Data Science • Clustering • Seriation - Data Science Gone Wrong • Time-series Analysis Analytics Components
- 20. Security. Analytics. Insight.20 Did You Know? Users accessing Sharepoint servers User Sharepoint Server data processing visualization This graph of users accessing sharepoint servers, does not immediately reveal any interesting patterns.
- 21. Security. Analytics. Insight.21 Did You Know - How Context Tells a Story Using HR data as context Remote User San Francisco Office User Sharepoint Server data processing visualization HR data Using color to add context to the graph helps immediately identify outliers and potential problems.
- 22. Security. Analytics. Insight.22 • Simple stuff works! • dc(dest), dc(d_port) • What is normal? • Use data science / data mining to prepare data. Then visualize the output for human analyst. Data Science in Security - Words of Caution
- 23. Security. Analytics. Insight.23 Challenges With Clustering Network Traffic The graph shows an abstract space with colors being machine identified clusters. Hard Questions: • What are these clusters? • Do Web servers cluster? • What are good clusters? • What’s anomalous?
- 24. Security. Analytics. Insight.24 Data Science That Works threshold outliers have different magnitudes
- 25. Security. Analytics. Insight.25 Approximate Curve fitting a curve distance to curve
- 26. Security. Analytics. Insight.26 Data Mining Applied better threshold
- 27. 27Hunting
- 28. Security. Analytics. Insight.28 Hunting - Ready, Fire, Aim • Analysts are your best and most expensive resource • They need the right tools and data • Speed (see earlier architecture) • Interaction (visual!) • Machine-assisted insight Examples • Exploring DNS traffic • High business impact machine analysis • Lateral movement
- 29. Security. Analytics. Insight.29 HBI Metric Analysis Visually learn, Test, Automate
- 30. Security. Analytics. Insight.30 HBI Metric Analysis - If you like Black Backgrounds
- 31. Security. Analytics. Insight.31 We have tried many thing: • Social Network Analysis • Seasonality detection • Entropy over time • Frequent pattern mining • Clustering All kinds of challenges. Simple works! Let’s Get Mathematical U−matrix 4.28e−05 0.0461 0.0921
- 32. Security. Analytics. Insight.32 Simple - Data Abstraction
- 33. Security. Analytics. Insight.33 Lateral Movement - Cross Network Communications Challenges • Scale • You will find one of everything • Defining white-lists and keeping them up to date (i.e., network and asset hygiene) VPN DMZ Office GIA Unknown Internet AWS
- 34. Security. Analytics. Insight.34 http://secviz.org List: secviz.org/mailinglist Twitter: @secviz Share, discuss, challenge, and learn about security visualization. Security Visualization Community
- 35. Security. Analytics. Insight.35 BlackHat Workshop Visual Analytics Delivering Actionable Security Intelligence July 30,31 & August 1,2 - Las Vegas, USA big data | analytics | visualization http://secviz.org
- 36. Security. Analytics. Insight.36 After some exploration … raffael.marty@pixlcloud.com http://slideshare.net/zrlram http://secviz.org and @secviz Further resources: