Creating Your Own Threat Intel Through Hunting & Visualization


Published on

The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out

Published in: Internet
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Creating Your Own Threat Intel Through Hunting & Visualization

  1. 1. Creating Your Own Threat Intel Through Hunting & Visualization Raffael Marty VP Security Analytics May 11, 2016 Honeynet Workshop 2016 – San Antonio, TX
  2. 2. © Raffael Marty 2 "This presentation was prepared solely by Raffael Marty in his personal capacity. The material, views, and opinions expressed in this presentation are the author's own and do not reflect the views of Sophos Ltd. or its affiliates." Disclaimer
  4. 4. Threat Intelligence © Raffael Marty 5
  5. 5. © Raffael Marty 6 • Products / Tools • Firewall - Blocks traffic based on pre-defined rules • Web Application Firewall - Monitors for signs of known malicious activity in Web traffic • Intrusion Prevention System - Looks for ‘signs’ of known attacks intraffic and protocol violations • Anti Virus - Looks for ‘signs’ of known attacks on the end system • Malware Sandbox - Runs new binaries and monitors their behavior for malicious signs • Security Information Management - Uses pre-defined rules to correlate signs from different data streams to augment intelligence • Vulnerability Scanning - Searches for known vulnerabilities and vulnerable software • Rely on pattern matching and signatures based knowledge from the past • Reactive -> always behind • Unknown and new threats -> won’t be detected • ‘Imperfect’ patterns and rules -> cause a lot of false positives We Are Monitoring – With Past Knowledge Verizon 2015 DBIR 70–90% OF MALWARE SAMPLES ARE UNIQUE TO AN ORGANIZATION.
  6. 6. 7 A New Architecture – The Security Data Lake any data Big Data Lake Rules context IOCs
  7. 7. 8 Exploring YOUR Environment - Hunting any data Rules IOCs Hunting • Interactive visualization • Analyst driven • Machine assisted context Big Data Lake
  8. 8. 9 Hunting Creates Internal Threat Intelligence any data Rules IOCs Novel, Advanced Attacks internal TI behavioral monitoring scoring anomaly detection machine learning artificial intelligence “models” data science x new rules context Big Data Lake
  9. 9. 10 How Do We Go Hunting? In the following we’ll explore how this all matters … … but first, let’s see how visualization plays a key role.
  10. 10. Visualization © Raffael Marty 11
  11. 11. S e c u r i t y . A n a l y t i c s . I n s i g h t . 12 “How Can We See, Not To Confirm - But To Learn” - Edward Tufte
  12. 12. 13 Why Visualization? dport time
  13. 13. 14 • SELECT count(distinct protocol) FROM flows; • SELECT count(distinct port) FROM flows; • SELECT count(distinct src_network) FROM flows; • SELECT count(distinct dest_network) FROM flows; • SELECT port, count(*) FROM flows GROUP BY port; • SELECT protocol, count(CASE WHEN flows < 200 THEN 1 END) AS [<200], count(CASE WHEN flows>= 201 AND flows < 300 THEN 1 END) AS [201 - 300], count(CASE WHEN flows>= 301 AND flows < 350 THEN 1 END) AS [301 - 350], count(CASE WHEN flows>= 351 THEN 1 END) AS [>351] FROM flows GROUP BY protocol; • SELECT port, count(distinct src_network) FROM flows GROUP BY port; • SELECT src_network, count(distinct dest_network) FROM flows GROUP BY port; • SELECT src_network, count(distinct dest_network) AS dn, sum(flows) FROM flows GROUP BY port, dn; • SELECT port, protocol, count(*) FROM flows GROUP BY port, protocol; • SELECT sum(flows), dest_network FROM flows GROUP BY dest_network; • etc. One Graph Summarizes Dozens of Queries port dest_network protocol src_network flows
  14. 14. Hunting © Raffael Marty 15
  15. 15. 16 Technical • Visualization • Context • Data Science Non-Technical • Analysts are your best and most expensive resource • They need the right tools and data • Speed (see the data lake) • Interaction (visual!) • Machine-assisted insight (datascience) Core Components To Enable Hunting
  16. 16. 17 Users accessing Sharepoint servers User Sharepoint Server This graph of users accessing sharepoint servers, does not immediately reveal any interesting patterns. data processing visualization
  17. 17. 18 Using HR data as context Remote User San Francisco Office User Sharepoint Server data processing visualization HR data Using color to add context to the graph helps immediately identify outliers and potential problems.
  18. 18. 19 • Simple approaches works! • dc(dest), dc(d_port) • What is normal? • Use data science / data mining to prepare data. Then visualize the output for human analyst. Data Science in Security - Words of Caution
  19. 19. 20 Challenges With Clustering Network Traffic The graph shows an abstract space with colors being machine identified clusters. Hard Questions: • What are these clusters? • Do Web servers cluster? • What are good clusters? • What’s anomalous?
  20. 20. 21 HBI Metric Analysis Visually learn, Test, Automate
  21. 21. 24 • We have tried many thing: o Social Network Analysis o Seasonality detection o Entropy over time o Frequent pattern mining o Clustering • All kinds of challenges • Simple works! Let’s Get Mathematical
  22. 22. 25 Simple - Data Abstraction
  23. 23. 26 Lateral Movement - Cross Network Communications Challenges • Scale • You will find one of everything • Defining white-lists and keeping them up to date (i.e., network and asset hygiene) VPN DMZ Office GIA Unknown Internet AWS
  24. 24. Visual Analytics Delivering Actionable Security Intelligence July 30,31 & August 1,2 - Las Vegas, USA big data | analytics | visualization Want To Dive Deeper? © Raffael Marty 27
  25. 25. @raffaelmarty © Raffael Marty 28