The document criticizes the Common Vulnerability Scoring System (CVSS) for prioritizing vulnerabilities based on theoretical models rather than empirical data. It argues that CVSS suffers from analytical and empirical failures, and proposes using a large dataset of vulnerabilities, exploits, and breaches to establish a more accurate prioritization framework. The framework would calculate the probability that an open vulnerability will be exploited based on observed breach data, in order to focus remediation on the vulnerabilities that pose the greatest actual risk.

1. Fix What Matters:
!
Why CVSS Sucks
And How To
Do Better

2. Michael Roytman
qualifications:
Proud Owner of Remote Controlled Airplane
Recently a Naive Grad Student
Once Jailbroke an Iphone 3G
Data Scientist, Risk I/O
Does Not Wake Up Before 11 CST

3. PART 1:
!
YOU SUCK AT
YOUR JOB
!
(and don’t even
know it yet)

4. Why Are We Here?
Analytical Failures of CVSS
Empirical Failures of CVSS
Proper Remediation Frameworks (Yeah, they exist)
CVSS SUCKS
(+Data Driven Alternatives)

5. Remediation
Accept the Risk
Repair the Vulnerability
Remove the Threat

6. C(ommon) V(ulnerability) S(coring) S(ystem)
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
“CVSS is designed to rank information
system vulnerabilities”
The Good: Open, Standardized Scores

7. “It is a capital mistake to theorize
before one has data.
!
!
!
Insensibly, one begins to twist
facts to suit theories, instead of
theories to suit facts.”

8. FAIL: Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias
http://blog.risk.io/2013/04/data-fundamentalism/
!
Jerico/Sushidude @ BlackHat
https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS
http://disi.unitn.it/~allodi/allodi-12-badgers.pdf

9. F1: Data Fundamentalism
Since 2006 Vulnerabilities have declined by 26 percent.”
http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf
!
!
The total number of vulnerabilities in 2013 is up 16 percent
so far when compared to what we saw in the same time
period in 2012. ”
http://www.symantec.com/content/en/us/enterprise/other_resources/bintelligence_report_06-2013.en-us.pdf

10. FAIL 2: A Priori Modeling
“Following up my previous email, I have tweaked my
equation to try to achieve better separation between
adjacent scores and to have CCC have a perfect (storm) 10
score...There is probably a way to optimize the problem
numerically, but doing trial and error gives one plausible set
of parameters...except that the scores of 9.21 and 9.54 are
still too close together. I can adjust x.3 and x.7 to get a
better separation . . .”

11. F3: Logical Inconsistency
Temporal Scores Hurt Decision Making
Report Confidence is Useless
Base Rate Fallacy

12. F4: Stochastic Ignorance
Attackers
Change Tactics
Daily

13. F4: Stochastic Ignorance

14. Empirical Failures of CVSS
Objective:
Remediate the riskiest vulnerabilities
Constraint:
Can’t measure impact/priority
Need:
MOAR DATA!!!

15. Repair the Vulnerability

16. I Love It When You Call Me Big Data
50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations

17. I Love It When You Call Me Big Data
3,000,000 Breaches

18. Baseline Allthethings
Probability
(You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE)
/(Total Open Vulnerabilities)
2%

19. Probability A Vuln Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000
0.010
0.020
0.030
0.040

20. PART 2:
!
FIX WHAT
MATTERS

21. Proper Framework
Know which vulnerabilities
put you most at risk.

28. Counterterrorism
Known Groups
Past
Incidents,
Close
Calls
Targets,
Layouts
Threat Intel,
Analysts
Surveillance

30. Uh, Sports?
Opposing
Teams, Specific
Players
Learning
from
Losing
Roster,
Player
Skills
Scouting
Reports,
Gametape
Gameplay

31. InfoSec?

32. Defend Like You’ve Done It Before
Groups,
Motivations
Learning
from
Breaches
Asset
Topology,
Actual Vulns
on System
Vulnerability
Definitions
Exploits

33. Work With What You’ve Got:
Akamai, Safenet
NVD,
MITRE
ExploitDB,
Metasploit

34. Bad Alternatives
Why Don’t I Just Patch The Important
Assets?

35. Good Alternatives

36. Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0
0.1
0.2
0.2
0.3

37. Data Is Everything And Everything Is Data

38. Data Is Everything And Everything Is Data

39. Be Better Than The Gap

40. Data is Everything and Everything is Data
Spray and Pray = 2%
CVSS 10 = 4%
Metasploit and Exploit DB = 30%

41. Holler!
@mroytman
www.risk.io