Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BSidesSF 2014 Fix What Matters:Why CVSS Sucks


Published on

Michael Roytman presentation on CVSS and security prioritization.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

BSidesSF 2014 Fix What Matters:Why CVSS Sucks

  1. 1. Fix What Matters:
 ! Why CVSS Sucks And How To Do Better
  2. 2. Michael Roytman qualifications: Proud Owner of Remote Controlled Airplane Recently a Naive Grad Student Once Jailbroke an Iphone 3G Data Scientist, Risk I/O Does Not Wake Up Before 11 CST
  3. 3. PART 1: ! YOU SUCK AT YOUR JOB ! (and don’t even know it yet)
  4. 4. Why Are We Here? Analytical Failures of CVSS Empirical Failures of CVSS Proper Remediation Frameworks (Yeah, they exist) CVSS SUCKS (+Data Driven Alternatives)
  5. 5. Remediation Accept the Risk Repair the Vulnerability Remove the Threat
  6. 6. C(ommon) V(ulnerability) S(coring) S(ystem) Exploitability/Temporal (Likelihood) Impact/Environmental (Severity) “CVSS is designed to rank information system vulnerabilities” The Good: Open, Standardized Scores
  7. 7. “It is a capital mistake to theorize before one has data. ! ! ! Insensibly, one begins to twist facts to suit theories, instead of theories to suit facts.”
  8. 8. FAIL: Data Fundamentalism Don’t Ignore What a Vulnerability Is: Creation Bias ! Jerico/Sushidude @ BlackHat ! Luca Allodi - CVSS DDOS
  9. 9. F1: Data Fundamentalism Since 2006 Vulnerabilities have declined by 26 percent.” ! ! The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”
  10. 10. FAIL 2: A Priori Modeling “Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”
  11. 11. F3: Logical Inconsistency Temporal Scores Hurt Decision Making Report Confidence is Useless Base Rate Fallacy
  12. 12. F4: Stochastic Ignorance Attackers Change Tactics Daily
  13. 13. F4: Stochastic Ignorance
  14. 14. Empirical Failures of CVSS Objective: Remediate the riskiest vulnerabilities Constraint: Can’t measure impact/priority Need: MOAR DATA!!!
  15. 15. Repair the Vulnerability
  16. 16. I Love It When You Call Me Big Data 50,000,000 Live Vulnerabilities 1,500,000 Assets 2,000 Organizations
  17. 17. I Love It When You Call Me Big Data 3,000,000 Breaches
  18. 18. Baseline Allthethings Probability (You Will Be Breached On A Particular Open Vulnerability)? =(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities) 2%
  19. 19. Probability A Vuln Having Property X Has Observed Breaches RANDOM VULN CVSS 10 CVSS 9 CVSS 8 CVSS 6 CVSS 7 CVSS 5 CVSS 4 Has Patch 0.000 0.010 0.020 0.030 0.040
  20. 20. PART 2: ! FIX WHAT MATTERS
  21. 21. Proper Framework Know which vulnerabilities put you most at risk.
  22. 22. Counterterrorism Known Groups Past Incidents, Close Calls Targets, Layouts Threat Intel, Analysts Surveillance
  23. 23. Uh, Sports? Opposing Teams, Specific Players Learning from Losing Roster, Player Skills Scouting Reports, Gametape Gameplay
  24. 24. InfoSec?
  25. 25. Defend Like You’ve Done It Before Groups, Motivations Learning from Breaches Asset Topology, Actual Vulns on System Vulnerability Definitions Exploits
  26. 26. Work With What You’ve Got: Akamai, Safenet NVD, MITRE ExploitDB, Metasploit
  27. 27. Bad Alternatives Why Don’t I Just Patch The Important Assets?
  28. 28. Good Alternatives
  29. 29. Probability A Vuln Having Property X Has Observed Breaches Random Vuln CVSS 10 Exploit DB Metasploit MSP+EDB 0.0 0.1 0.2 0.2 0.3
  30. 30. Data Is Everything And Everything Is Data
  31. 31. Data Is Everything And Everything Is Data
  32. 32. Be Better Than The Gap
  33. 33. Data is Everything and Everything is Data Spray and Pray = 2% CVSS 10 = 4% Metasploit and Exploit DB = 30%
  34. 34. Holler! @mroytman