This is a talk about data science operations and the applications of Risk I/Os insights to the security industry - how we went about mining insights from our large dataset

1. What Your Security
Data Isn’t
Telling You
@mroytman

2. Michael Roytman
Data Scientist, Risk I/O
M.S. Operations Research, Georgia Tech

3. PART 1:
!
DATA SCI OPS:
!
LESS IS MORE
!

4. LESS TOOLS
LESS DATA
LESS MODEL COMPLEXITY
MORE IMPACT
LESS DATA SCIENTISTS

5. SAY
“BIG DATA”
ONE
MORE
TIME

6. EVERYONE
IS A
DATA SCIENTIST

8. TAKE ONLY WHAT YOU
NEED

9. PART 2:
!
FIX WHAT MATTERS

10. Remove the Threat
Remediation
Accept the Risk
Repair the Vulnerability

11. “It is a capital mistake to theorize
before one has data.
!
!
!
!
Insensibly, one begins to twist
facts to suit theories, instead of
theories to suit facts.”

12. C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information
system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores

13. FAIL 1: A Priori Modeling
“Following up my previous email, I have tweaked my
equation to try to achieve better separation between
adjacent scores and to have CCC have a perfect (storm) 10
score...There is probably a way to optimize the problem
numerically, but doing trial and error gives one plausible set
of parameters...except that the scores of 9.21 and 9.54 are
still too close together. I can adjust x.3 and x.7 to get a
better separation . . .”

14. 2: Data Fundamentalism
Since 2006 Vulnerabilities have declined by 26 percent.”
http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf
!
!
The total number of vulnerabilities in 2013 is up 16 percent
so far when compared to what we saw in the same time
period in 2012. ”
http://www.symantec.com/content/en/us/enterprise/other_resources/b-
intelligence_report_06-2013.en-us.pdf

15. 3: Attackers Change Tactics
Daily

22. Repair the Vulnerability

23. I Love It When You Call Me Big Data
50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations

24. I Love It When You Call Me Big Data
15,000,000
Breaches

27. Baseline Allthethings
Probability
(You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE)
/(Total Open Vulnerabilities)
2%

28. Probability A Vuln Having Property X Has Observed Breaches
RANDOMVULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040

30. Counterterrorism
Known Groups
Surveillance
Threat Intel,
Analysts
Targets,
Layouts
Past
Incidents,
Close
Calls

32. Uh, Sports?
Opposing
Teams, Specific
Players
Gameplay
Scouting
Reports,
Gametape
Roster,
Player
Skills
Learning
from
Losing

33. Defend Like You’ve Done It Before
Groups,
Motivations
Exploits
Vulnerability
Definitions
Asset
Topology,
Actual Vulns
on System
Learning
from
Breaches

34. Probability A Vuln Having Property X Has Observed Breaches
RandomVuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3

36. Data is Everything and Everything is Data
Spray and Pray = 2%
CVSS 10 = 4%
Metasploit and Exploit DB = 30%

37. www.risk.io/jobs
@mroytman
THANKS!