Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Effective Prioritization Through Exploit Prediction

76 views

Published on

Fixing more of what matters, and less of what doesn’t

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Effective Prioritization Through Exploit Prediction

  1. 1. Michael Roytman Jonathan Cran @mroytman @jcran Black Hat 2018 Effective Prioritization Through Exploit Prediction Fixing more of what matters, and less of what doesn’t
  2. 2. 2
  3. 3. 3 Complete Remediation is Infeasible Complexity Abound Multiple patch releases from major vendors, including microcode updates Incompatible Antivirus or Endpoint protection Massive Array of Devices Affected Affects Printers, Thermostats, Door Locks, Cameras, Phones, etc Intel’s Nehalem and Westmere (released in 2008 and 2010) affected Not Just Patches Code “should be recompiled with the /Qspectre switch enabled”
  4. 4. 4 The Modern Stack is COMPLEX Intel / ARM / AMD CPU Hypervisor Java Management Agent Docker Operating System .NET Operating System (Container) Node 3rd party libs Your App Patch Me! Idea Credit: @samnewman Python Ruby PHP App Server / Web Server / etc
  5. 5. Vulnerability Volume Increasing
  6. 6. Exploit Release Dates - Tied to CVE Publish
  7. 7. The Good News
  8. 8. How to find and fix before an event?
  9. 9. 9 “Remember the Recall” Infosec is largely a search problem: 1. We are data rich and signal poor. 2. Multi-stage testing cost-effectively increases both precision and recall. 3. Analyst time is the capacity constraint for most security problems We must aim to create signal for our analysts.
  10. 10. 10 CVSS (alone) Isn’t it
  11. 11. 11 Events Are What Matters 2018: 36 new CVEs with events
  12. 12. 12 Targets Not Created Equal
  13. 13. 13 What Matters for Scoring Is anyone actively targeted? Could we detect success? How much effort is required? What is the attacker payoff? Does a valid attack path exist? score = $CVSS_SCORE score += A if recent_breaches_exist? SCORE += B if exploits_exist? SCORE += C if popular_target? SCORE += D if exploit_will_exist?
  14. 14. How do we know if we are working on the right stuff?
  15. 15. 15 Attack + Defense Detect & Respond Predict & Prevent
  16. 16. 16 Measuring Remediation Strategies Coverage: Of the vulns we fixed, did we pick all (100%) of the correct ones? Efficiency: Of the ones we ended up fixing, did fix any that didn’t matter?
  17. 17. 17 Coverage & Efficiency, Explained OURS NEIGHBORS ROBOT MOWED Coverage =~ 80% Efficiency =~ 60% EFFICIENCY: Out of all the grass mowed, how much of the grass should have been cut COVERAGE: How much of the grass we wanted to cut was actually cut? wasted effort (inefficiency ) not covered
  18. 18. 18 Coverage & Efficiency In Practice CVES with known exploits or events CVEs with no known exploit or event Coverage How many vulnerabilities did we prioritize of those that ended up with a known exploit or event Efficiency (green in the red area green + blue) =~ 9.28% Of all the vulnerabilities we prioritized, how many ended up with a kown exploit or event
  19. 19. 19 Coverage & Efficiency In Practice CVES with known exploits or events CVEs with no known exploit or event Total Prioritized CVEs All CVEs Vulnerabilities prioritized with known exploits or events CVEs prioritized with no known exploits or events Coverage (green / red) How many vulnerabilities did we prioritize of those that ended up with a known exploit or event Efficiency (green / green + blue) Of all the vulnerabilities we prioritized, how many ended up with a known exploit or event
  20. 20. 20 Coverage / Efficiency Tradeoff ● There exists a natural tradeoff between coverage and efficiency. ● We are operating with incomplete information at any given moment. ● Why would you want <100% efficiency? ○ Abundance of caution (if you can afford it!) ● Why would you want <100% coverage? ○ New campaign can spin up or an older one can spin down. The world is not static. Continuous review and adjustment provides the best result.
  21. 21. How to handle quickly escalating threats?
  22. 22. 22 Current Attacker Velocity Average Days from Publish to Exploit (639 / 8%): 19.68 Days Average Days from Publish to Event (36 / 0.5%): 27.36 Days Shortest Window: Adobe Reader (zero days) Longest Window: IE Edge (months)
  23. 23. 23 Recent Popular Targets (2018) Apache Struts 2.3.x - CVE-2017-5638, CVE-2017-9791, CVE-2017-9805 Joomla! 3.7.1 - CVE-2017-8917 Oracle WebLogic 10.3.6, 12.1.x, 12.2.x - CVE-2017-10271 Jenkins 2.56 - CVE-2017-1000353 Microsoft SMBv1 (ETERNALBLUE) - CVE-2017-0143/4/5 MASTER IPCAMERA (hardcoded password) - CVE-2018-5723 Drupal (Drupalgeddon) - CVE-2018-7600 Adobe Flash - CVE-2018-4878
  24. 24. Increasing Risk Factoring in Velocity Created Discovery Disclosure Public Exploit Code Released Exploitation Detected In the Wild Detection Generate d
  25. 25. Exploit Release Dates - Tied to CVE Publish
  26. 26. The Case for Prediction
  27. 27. Enter The Exploit Prediction Model
  28. 28. 28 Future of Data Past Q: “A new vulnerability was just released. Do we scramble? A:
  29. 29. 29 “Prediction is very difficult, especially about the future” -Niels Bohr
  30. 30. Data Sources: CVE Enrichment Projects
  31. 31. Data Sources: Exploit Code & Observations
  32. 32. 32 What IS Machine Learning? • Methods for automatically learning and recognizing complex patterns from data • A set of tools for understanding data by buildings models from data • measure success on coverage and efficiency
  33. 33. 33 Type of Algorithms Do you have labeled data? Supervised Unsupervised What do you want to predict? Classification Regression Category NoYes Quantity
  34. 34. 34 We are current really good at: • “Of my current 300 million vulnerabilities, which ones should I remediate first?” • “Old ones with stable, weaponized exploits, known breaches, high risk meter scores”
  35. 35. 35 Supervised Classification: VS.
  36. 36. 36 Asking the right questions: • Classification: output is qualitative • prediction: “Will this vulnerability have an exploit written for it?” (== cause more risk later)
  37. 37. 37 Input variables Numeric cvss_base cvss_temporal kenna_score breach_count cpe count cpe_vendor_count cpe_product_count cve age days_to_first_exploit first_exploit_age days_to_first_breach first_breach_age vulnerability_count open_vulnerability_count closed_vulnerability_count reference_count Binary any_exploits in metasploit in_exploitdb in_elliot any_breaches popular_target recent_breaches rce description_contains_in_the_wild description_contains_buffer_overflow description_contains_man_in_the_middle exploited_in_7 exploited_in_14 exploited_in_30 exploited_in_60 exploited_in_90 breached_in_/ breached_in_14 breached_in_30 breached_in_60 breached_in_90 library_vuln fix_exists Categorical access_vector access_complexity authentication confidentiality_impact integrity_impact availability_impact wasc_id cwe_id fix_type Text cve summary exploit_authors exploit_platforms exploit types exploit_ports cpe_vendors cpe_products reference_types
  38. 38. 38 Predictive - The Expectations Distribution is not uniform. 77% of dataset is not exploited 1. Accuracy of 77% would be bad Precision matters more than Recall 1. No one would use this model absent actual exploit available data. 2. False Negatives matter less than false positives - wasted effort. We are not modeling when something will be exploited, just IF 1. Could be tomorrow or in 6 months. Re-run the model every day.
  39. 39. 39 Measuring performance of a predictive model The ideal1 10 Precision Recall Returns relevant documents buy misses many useful ones too Returns most relevant documents but includes lots of junk
  40. 40. 40 Coverage Efficiency Tradeoffs
  41. 41. 41 “Somewhat Likely”
  42. 42. 42 “Highly Likely”
  43. 43. 43 “Most Likely”
  44. 44. 44 Characteristics of Predicted CVEs ● Common Phrases ○ “Arbitrary Code Execution ○ “Command Injection” ○ “Remote Attackers” ● Vulnerability Classes ○ SQL Injection (CWE-89) ○ Buffer Overflow (CWE-119) ○ Improper Imput Validation (CWE-20)
  45. 45. 45 The Work Averse Attacker “An attacker massively deploys only one exploit per software version. The only exception we find is for Internet Explorer; the exception is characterised by a very low cost to create an additional exploit, where it is sufficient to essentially copy and paste code from the old exploit, with only few modifications, to obtain the new one.” -The Work-Averse Cyber Attacker Model: Theory and Evidence From Two Million Attack Signatures by Luca Allodi, Fabio Massacci, Julian Williams
  46. 46. 46 ● CVE-2016-10372 - cpe:/o:eir:d1000_modem_firmware: ○ https://www.rapid7.com/db/modules/exploit/linux/http/tr064_ntpserver_cmdinject ● CVE-2017-18046 - cpe:/o:dasannetworks:h640x_firmware:2.77p1-1124 ○ https://blogs.securiteam.com/index.php/archives/3552 ● CVE-2017-8116 - cpe:/o:teltonika:rut900_firmware:00.03.265 ○ https://labs.nettitude.com/blog/cve-2017-8116-teltonika-router-unauthenticated-re mote-code-execution/ ● CVE-2017-16228 - cpe:/a:dulwich_project:dulwich:0.18.4 ○ [no exploit exists] ● CVE-2017-17946 - cpe:/a:novosoft:handy_password:4.9.3 ○ [no exploit exists] Machine Learning Has Side Benefits
  47. 47. 48 ● CVE-2016-10372 - cpe:/o:eir:d1000_modem_firmware: ○ https://www.rapid7.com/db/modules/exploit/linux/http/tr064_ntpserver_cmdinject ● CVE-2017-18046 - cpe:/o:dasannetworks:h640x_firmware:2.77p1-1124 ○ https://blogs.securiteam.com/index.php/archives/3552 ● CVE-2017-8116 - cpe:/o:teltonika:rut900_firmware:00.03.265 ○ https://labs.nettitude.com/blog/cve-2017-8116-teltonika-router-unauthenticated-re mote-code-execution/ ● CVE-2017-16228 - cpe:/a:dulwich_project:dulwich:0.18.4 ○ https://twitter.com/jcran/status/1026533985630007296 ● CVE-2017-17946 - cpe:/a:novosoft:handy_password:4.9.3 ○ [no exploit exists] Machine Learning Has Side Benefits
  48. 48. 49 Interesting, Highly Scored CVEs ● CVE-2016-10717 - MalwareBytes - bypass whitelisting ● CVE-2016-1417 - Snort - DLL hijacking ● CVE-2018-3605 - Trend Micro Control Manager SQLi (RCE) ● CVE-2018-4944 - Adobe Reader Type Confusion ● CVE-2016-7272 - Windows Icon File Integer Overflow
  49. 49. 50 Constraints on the Future Any new rating system must be: ● Simple (in every sense of the word) ● Explainable (cause and effect understandable) ● Defensible (science!) ● an Improvement And every data source is on the table...
  50. 50. The future
  51. 51. 52 Lesson: Less is More New variables aren’t adding much overall
  52. 52. 53 Lesson: Probability is our friend confusing ^ 78% of vulns are < 1% ● While initially confusing, probability offers a very intuitive measure ● Most vulnerabilities are predicted to have < 1% probability of exploitation 2,400+ vulnerabilities are predicted > 10% ● How can we validate probabilistic estimates?
  53. 53. 54 Lesson: Probability is our friend confusing ^ ~450 vulnerabilities (what we say) (what we see) Dashed line is “calibrated”
  54. 54. 55 Real World You can deal with: 44,000,000 Alerts or Fix 299 Vulnerabilities
  55. 55. 56 Takeaways Volume, complexity and speed of both vulnerabilities and threats are modern vulnerability management challenges Coverage and efficiency allow us to measure vuln management strategies For all the new vulnerabilities you’ve seen this week… is it truly critical? Will it be attacked in the future? Future threats should be addressed, but only after immediate / existing threats

×