Michael Roytman, Data Scientist, Kenna & Ed Bellis, CEO, Kenna (BSidesLV, 2013) While the past isn’t a direct indication of future performance, knowing the past is essential to predicting the future. In security, this requires reviewing large quantities of vulnerability, defect and exploit data to fully understand how attackers are likely to approach their task. This presentation by Kenna CEO Ed Bellis, and Data Scientist Michael Roytman covers vulnerability statistics as it relates to fixing security issues by: Examining the overlapping data generated from 20 of the leading security tools on the market Comparing & contrasting this with the output of multiple breach reports and database Extracting trends that may be important in helping to reduce the number of breaches in the future This is based on research from over 30M vulnerabilities analyzed over the past 12 months, generated across some of the largest corporations in the world. Take a look at this presentation to learn how others are remediating vulnerabilities, how effective their efforts are, and how they could do it a little bit better.

1. Vulnerability & Exploit Trends: A Deep Look Inside the Data
Ed Bellis & Michael Roytman
Vulnerability & Exploit Trends: A
Deep Look Inside the Data
Ed Bellis & Michael Roytman

2. Nice To Meet You
• CoFounder Kenna
About Us
About Risk I/O
• Former CISO Orbitz
• Contributing Author: Beautiful Security
• CSO Magazine/Online Writer
• Data-Driven Vulnerability Intelligence Platform
• DataWeek 2012 Top Security Innovator
• 3 Startups to Watch - Information Week
• InfoSec Island Blogger
• 16 Hot Startups - eWeek
Ed Bellis
• Naive Grad Student
• Still Plays With Legos
• Barely Passed Regression Analysis
• Once Jailbroke His iPhone 3G
• Has Coolest Job In InfoSec
Michael Roytman

3. Starting From Scratch
Academia!
• GScholar!
• JSTOR!
• IEEE!
• ProQuest!
InfoSec Blogs!
• CSIOs!
• Pen Testers!
• Threat Reports!
• SOTI/DBIR!
!
Twitter!
• Thought Leaders (you
know who you are)!
• BlackHats!
• Vuln Researchers!
Primary Sources!
• MITRE!
• OSVDB!
• NIST CVSS
Committee(s)!
• Internal Message
Boards for ^!
Text
CISOs

4. #DoingItWrong
Data Fundamentalism
Don’t Ignore What a Vuln Is: Creation Bias (http://blog.kennasecurity.com/2013/04/data-fundamentalism/) <Shameless(ful) Self-
Promotion
Jerico/Sushidude @ BlackHat (https://www.blackhat.com/us-13/briefings.html#Martin)
Luca Allodi (https://securitylab.disi.unitn.it/lib/exe/fetch.php?media=seminar-unimi-apr-13.pdf):
Protip: http://disi.unitn.it/~allodi/allodi-12-badgers.pdf

5. #DoingItWrong
”Since 2006 Vulnerabilities have declined by 26 percent.”
-http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf
“The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the
same time period in 2012. ”
-http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf

6. What’s Good?
Bad For Vulnerability Statistics:
NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on.
Good For Vulnerability Statistics:
Vulnerabilities.

7. Adding Some Flavor

8. Defend Like You’ve Done It Before

9. Counterterrorism
Known Groups
Surveillance
Threat Intel,
Analysts
Targets,
Layouts
Past
Incidents,
Close
Calls

10. Uh, Sports?
Opposing
Teams, Specific
Players
Gameplay
Scouting
Reports,
Gametape
Roster,
Player
Skills
Learning
from
Losing

11. InfoSec?

12. What It Should Be
Groups,
Motivations
Exploits
Vulnerability
Definitions
Asset
Topology,
Actual Vulns
on System
Learning
from
Breaches

13. Work With What You’ve Got:
Akamai, Safenet
ExploitDB,
Metasploit
NVD,
MITRE

14. Show Me The Money
23,000,000 Vulnerabilities!
Across 1,000,000 Assets!
Representing 9,500 Companies!
Using 22 Unique Scanners!

15. Whatchu Know About Data?
Duplication
Vulnerability Density
Remediation

16. Duplication
0
225,000
450,000
675,000
900,000
1,125,000
1,350,000
1,575,000
1,800,000
2,025,000
2,250,000
2 or more scanners 3 or more 4 or more 5 or more 6 or more

17. Duplication - Lessons From a CISO
We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities
We Want: F(Number of Scanners) => Vulnerability Coverage
Make Decisions At The Margins!
<---------Good Luck!
0.0
25.0
50.0
75.0
100.0
0 1 2 3 4 5 6

18. Density
Type of Asset ~Count
Hostname 20,000
Netbios 1000
IP Address 200,000
File 10,000
Url 5,000
Hostname
Netbios
IP
File
Url
0.0 22.5 45.0 67.5 90.0

19. CVSS And Remediation Metrics
0.0
350.0
700.0
1050.0
1400.0
1 2 3 4 5 6 7 8 9 10
Average Time To Close By Severity OldestVulnerability By Severity

20. CVSS And Remediation - Lessons From A CISO
Remediation/Lack Thereof, by CVSS
1 2 3 4 5 6 7 8 9 10
NVD Distribution by CVSS

21. The Kicker - Live Breach Data
1,500,000 !
Vulnerabilities Related to Live Breaches Recorded!
June, July 2013 !

22. CVSS And Remediation - Nope
0.0
1750.0
3500.0
5250.0
7000.0
1 2 3 4 5 6 7 8 9 10
Oldest BreachedVulnerability By Severity

23. CVSS - A VERY General Guide For Remediation - Yep
0.0
40000.0
80000.0
120000.0
160000.0
1 2 3 4 5 6 7 8 9 10
OpenVulns With Breaches Occuring By Severity

24. The One Billion Dollar Question
Probability(You Will Be Breached On A Particular Open Vulnerability)?
1.98%
=(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities)

25. I Love It When You Call Me Big Data
Probability AVulnerability Having Property X Has Observed Breaches
RANDOMVULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
0.00000 0.01000 0.02000 0.03000 0.04000

26. Enter The Security Mendoza Line
Wouldn’t it be nice if we had something
that helped us divide who we considered
“Amateur” and who we considered
“Professional”?
http://riskmanagementinsight.com/riskanalysis/?
p=294
Josh Corman expands
the Security Mendoza Line
“Compute power grows at the
rate of doubling about every 2
years”
“Casual attacker power grows at
the rate of Metasploit”
http://blog.cognitivedissidents.com/2011/11/01/intro-
to-hdmoores-law/
Alex Hutton comes up
with Security Mendoza
Line

27. I Love It When You Call Me Big Data
Probability AVulnerability Having Property X Has Observed Breaches
RandomVuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.00 0.08 0.15 0.23 0.30

28. I Love It When You Call Me Big Data
P(Breaches Observed On That Vuln | Random Vuln)
1.98%

29. Thank You
Follow Us
Blog: http://blog.kennasecurity.com
Twitter: @mroytman
@ebellis
@riskio
We’re Hiring! http://www.kennasecurity.com/jobs