The document critiques the Common Vulnerability Scoring System (CVSS) and suggests that it fails to accurately assess risk due to empirical and analytical shortcomings. It emphasizes the need for better data-driven approaches to remediation, focused on the most risky vulnerabilities rather than relying solely on standardized scoring systems. The author advocates for a comprehensive understanding of vulnerabilities, including their practical implications and the necessity of adapting to changing attacker tactics.

1. Fix What Matters:
!
Why CVSS Sucks
And How To
Do Better

2. Once Jailbroke an Iphone 3G
Michael Roytman
Proud Owner of Remote Controlled Airplane
Recently a Naive Grad Student
Data Scientist, Risk I/O
Does Not Wake Up Before 11 CST
qualifications:

3. 15x better than CVSS

4. Probability A Vuln Having Property X Has Observed Breaches
RandomVuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3

5. PART 1:
!
YOU SUCK AT
YOUR JOB
!
(and don’t even
know it yet)

6. Why Are We Here?
Empirical Failures of CVSS
Proper Remediation Frameworks
CVSS SUCKS
Analytical Failures of CVSS
(+Data Driven Alternatives)

7. Remove the Threat
Remediation
Accept the Risk
Repair the Vulnerability

8. C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information
system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores

9. “It is a capital mistake to theorize
before one has data.
!
!
!
Insensibly, one begins to twist
facts to suit theories, instead of
theories to suit facts.”

10. FAIL 1: A Priori Modeling
“Following up my previous email, I have tweaked my
equation to try to achieve better separation between
adjacent scores and to have CCC have a perfect (storm) 10
score...There is probably a way to optimize the problem
numerically, but doing trial and error gives one plausible set
of parameters...except that the scores of 9.21 and 9.54 are
still too close together. I can adjust x.3 and x.7 to get a
better separation . . .”

11. F2: Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias
http://blog.risk.io/2013/04/data-fundamentalism/
!
Jerico/Sushidude @ BlackHat
https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS
http://disi.unitn.it/~allodi/allodi-12-badgers.pdf

12. F2: Data Fundamentalism
Since 2006 Vulnerabilities have declined by 26 percent.”
http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf
!
!
The total number of vulnerabilities in 2013 is up 16 percent
so far when compared to what we saw in the same time
period in 2012. ”
http://www.symantec.com/content/en/us/enterprise/other_resources/b-
intelligence_report_06-2013.en-us.pdf

13. F3: Stochastic Ignorance
Attackers
Change Tactics
Daily

14. F3: Stochastic Ignorance

15. Empirical Failures of CVSS
Objective:
Remediate the riskiest vulnerabilities
Constraint:
Can’t measure impact/priority
Need:
MOAR DATA!!!

16. Repair the Vulnerability

17. I Love It When You Call Me Big Data
50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations

18. I Love It When You Call Me Big Data
3,000,000 Breaches

19. Baseline Allthethings
Probability
(You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE)
/(Total Open Vulnerabilities)
2%

20. Probability A Vuln Having Property X Has Observed Breaches
RANDOMVULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040

21. PART 2:
!
FIX WHAT
MATTERS

22. Empirical Failures of CVSS
Objective:
Remediate the riskiest vulnerabilities
Constraint:
Can’t measure impact/priority
Need:
MOAR DATA!!!

23. Proper Framework
Know which vulnerabilities
put you most at risk.

31. Uh, Sports?
Opposing
Teams, Specific
Players
Gameplay
Scouting
Reports,
Gametape
Roster,
Player
Skills
Learning
from
Losing

32. InfoSec?

33. Defend Like You’ve Done It Before
Groups,
Motivations
Exploits
Vulnerability
Definitions
Asset
Topology,
Actual Vulns
on System
Learning
from
Breaches

34. Work With What You’ve Got:
Akamai, Safenet
ExploitDB,
Metasploit
NVD,
MITRE

35. Alternatives

36. Probability A Vuln Having Property X Has Observed Breaches
RandomVuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3

37. Be Better Than The Gap

38. I Love It When You Call Me Big Data
!
Spray and Pray => 2%
!
CVSS 10 => 4%
!
Metasploit + ExploitDB => 30%

39. Holler!
www.risk.io
@mroytman