Fix What Matters
Michael Roytman
SIRAcon October 21, 2013
Why You Should(n’t) Listen

Michael Roytman
• Data Scientist, Risk I/O
• MS Operations Research, Georgia Tech
• Fraud Dete...
Roadmap

• The Struggle
• What’s Bad?
• What’s Good?
• Framework
• Data Driven Insights

• Decision-Making
Starting From Scratch

“It is a capital mistake to theorize
before one has data. Insensibly one
begins to twist facts to s...
Starting From Scratch
Starting From Scratch

Primary Sources!
Twitter!
InfoSec Blogs!
Academia!
•  GScholar!
•  JSTOR!
•  IEEE!
•  ProQuest!

• ...
Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias !
(http://blog.risk.io/2013/04/data-fundamentalism...
Data Fundamentalism - What’s The Big Deal?

!

”Since 2006 Vulnerabilities have declined by 26 percent.” !
(http://csrc.ni...
What’s Good?

Bad For Vulnerability Statistics:!
!

NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, a...
Data Is Everything And Everything Is Data.
What’s Good?
What’s Good?
What’s Good?
What’s Good?
What’s Good?
What’s Good?
Counterterrorism

Known Groups
Past
Incidents,
Close
Calls

Targets,
Layouts

Threat Intel,
Analysts

Surveillance
What’s Good?
Uh, Sports?
Opposing
Teams, Specific
Players
Learning
from
Losing

Roster,
Player
Skills

Scouting
Reports,
Gametape

Gamep...
InfoSec?
Defend Like You’ve Done It Before
Groups,
Motivations

Learning
from
Breaches

Asset
Topology,
Actual Vulns
on System

Vul...
Work With What You’ve Got:
Akamai, Safenet

NVD,
MITRE

ExploitDB,
Metasploit
Add Some Spice
Show Me The Money

23,000,000 Vulnerabilities!
Across 1,000,000 Assets!
Representing 9,500 Companies!
Using 22 Unique Scan...
Whatchu Know About Dat?(a)
!

Duplication
Vulnerability Density
Remediation
Duplication
2,250,000

2,025,000

1,800,000

1,575,000

1,350,000

1,125,000

900,000

675,000

450,000

225,000

0

2 or ...
Duplication
We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities
We Want: F(Number of Scanners) => Vulner...
Density
Hostname

Type of Asset
Hostname

1000

IP Address

200,000

File

10,000

Url

IP

20,000

Netbios

Netbios

~Cou...
CVSS And Remediation Metrics
Average Time To Close By Severity

Oldest Vulnerability By Severity

1400.0

1050.0

700.0

3...
CVSS And Remediation - Lessons From A CISO
Remediation/Lack Thereof, by CVSS

1

2

3

4

5

6

NVD Distribution by CVSS

...
The Kicker - Live Breach Data

1,500,000 !
Vulnerabilities Related to Live Breaches Recorded!
June, July 2013 !
CVSS And Remediation - Nope
Oldest Breached Vulnerability By Severity

7000.0

5250.0

3500.0

1750.0

0.0
1

2

3

4

5

...
CVSS - A VERY General Guide For Remediation - Yep
Open Vulns With Breaches Occuring By Severity

160000.0

120000.0

80000...
The One Billion Dollar Question

Probability(You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilit...
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
RANDOM VULN
CVSS 1...
What’s the Alternative?
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
Random Vuln

CVSS ...
Data Is Everything And Everything Is Data.
Be Better Than The Gap
I Love It When You Call Me Big Data

Spray and Pray => 2%
!

CVSS 10 => 4%
!

Metasploit + ExploitDB => 30%
!

A Good Mode...
Thank You

Don’t Be A Stranger
Blog: http://blog.risk.io
Twitter: @mroytman
Upcoming SlideShare
Loading in...5
×

Fix What Matters: A Data Driven Approach to Vulnerability Management

309

Published on

Data driven approach to vulnerability management in information security using live breach and vulnerability data.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
309
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Fix What Matters: A Data Driven Approach to Vulnerability Management

  1. 1. Fix What Matters Michael Roytman SIRAcon October 21, 2013
  2. 2. Why You Should(n’t) Listen Michael Roytman • Data Scientist, Risk I/O • MS Operations Research, Georgia Tech • Fraud Detection, Large Bank • Naive Grad Student Not Too Long Ago • Still Plays With Legos • Barely Passed Regression Analysis
  3. 3. Roadmap • The Struggle • What’s Bad? • What’s Good? • Framework • Data Driven Insights • Decision-Making
  4. 4. Starting From Scratch “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” -Sir Arthur Conan Doyle, 1887
  5. 5. Starting From Scratch
  6. 6. Starting From Scratch Primary Sources! Twitter! InfoSec Blogs! Academia! •  GScholar! •  JSTOR! •  IEEE! •  ProQuest! •  CISOs CSIOs! •  Pen Testers! •  Threat Reports! •  SOTI/DBIR! ! Text •  Thought Leaders (you know who you are)! •  BlackHats! •  Vuln Researchers! •  MITRE! •  OSVDB! •  NIST CVSS Committee(s)! •  Internal Message Boards for ^!
  7. 7. Data Fundamentalism Don’t Ignore What a Vulnerability Is: Creation Bias ! (http://blog.risk.io/2013/04/data-fundamentalism/) ! Jerico/Sushidude @ BlackHat ! (https://www.blackhat.com/us-13/briefings.html#Martin)! Luca Allodi - CVSS DDOS ! (http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):!
  8. 8. Data Fundamentalism - What’s The Big Deal? ! ”Since 2006 Vulnerabilities have declined by 26 percent.” ! (http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)! ! ! “The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”! (http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)! ! !
  9. 9. What’s Good? Bad For Vulnerability Statistics:! ! NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on. ! Good For Vulnerability Statistics:! ! Vulnerabilities. !
  10. 10. Data Is Everything And Everything Is Data.
  11. 11. What’s Good?
  12. 12. What’s Good?
  13. 13. What’s Good?
  14. 14. What’s Good?
  15. 15. What’s Good?
  16. 16. What’s Good?
  17. 17. Counterterrorism Known Groups Past Incidents, Close Calls Targets, Layouts Threat Intel, Analysts Surveillance
  18. 18. What’s Good?
  19. 19. Uh, Sports? Opposing Teams, Specific Players Learning from Losing Roster, Player Skills Scouting Reports, Gametape Gameplay
  20. 20. InfoSec?
  21. 21. Defend Like You’ve Done It Before Groups, Motivations Learning from Breaches Asset Topology, Actual Vulns on System Vulnerability Definitions Exploits
  22. 22. Work With What You’ve Got: Akamai, Safenet NVD, MITRE ExploitDB, Metasploit
  23. 23. Add Some Spice
  24. 24. Show Me The Money 23,000,000 Vulnerabilities! Across 1,000,000 Assets! Representing 9,500 Companies! Using 22 Unique Scanners!
  25. 25. Whatchu Know About Dat?(a) ! Duplication Vulnerability Density Remediation
  26. 26. Duplication 2,250,000 2,025,000 1,800,000 1,575,000 1,350,000 1,125,000 900,000 675,000 450,000 225,000 0 2 or more scanners 3 or more 4 or more 5 or more 6 or more
  27. 27. Duplication We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities We Want: F(Number of Scanners) => Vulnerability Coverage <---------Good Luck! Make Decisions At The Margins! 100.0 75.0 50.0 25.0 0.0 0 1 2 3 4 5 6
  28. 28. Density Hostname Type of Asset Hostname 1000 IP Address 200,000 File 10,000 Url IP 20,000 Netbios Netbios ~Count 5,000 File Url 0.0 22.5 45.0 67.5 90.0
  29. 29. CVSS And Remediation Metrics Average Time To Close By Severity Oldest Vulnerability By Severity 1400.0 1050.0 700.0 350.0 0.0 1 2 3 4 5 6 7 8 9 10
  30. 30. CVSS And Remediation - Lessons From A CISO Remediation/Lack Thereof, by CVSS 1 2 3 4 5 6 NVD Distribution by CVSS 7 8 9 10
  31. 31. The Kicker - Live Breach Data 1,500,000 ! Vulnerabilities Related to Live Breaches Recorded! June, July 2013 !
  32. 32. CVSS And Remediation - Nope Oldest Breached Vulnerability By Severity 7000.0 5250.0 3500.0 1750.0 0.0 1 2 3 4 5 6 7 8 9 10
  33. 33. CVSS - A VERY General Guide For Remediation - Yep Open Vulns With Breaches Occuring By Severity 160000.0 120000.0 80000.0 40000.0 0.0 1 2 3 4 5 6 7 8 9 10
  34. 34. The One Billion Dollar Question Probability(You Will Be Breached On A Particular Open Vulnerability)? =(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities) 1.98%
  35. 35. I Love It When You Call Me Big Data Probability A Vulnerability Having Property X Has Observed Breaches RANDOM VULN CVSS 10 CVSS 9 CVSS 8 CVSS 6 CVSS 7 CVSS 5 CVSS 4 Has Patch 0.00000 0.01000 0.02000 0.03000 0.04000
  36. 36. What’s the Alternative?
  37. 37. I Love It When You Call Me Big Data Probability A Vulnerability Having Property X Has Observed Breaches Random Vuln CVSS 10 Exploit DB Metasploit MSP+EDB 0.0 0.1 0.2 0.2 0.3
  38. 38. Data Is Everything And Everything Is Data.
  39. 39. Be Better Than The Gap
  40. 40. I Love It When You Call Me Big Data Spray and Pray => 2% ! CVSS 10 => 4% ! Metasploit + ExploitDB => 30% ! A Good Model That’s Not Built By One Kid Without Hadoop => ???!
  41. 41. Thank You Don’t Be A Stranger Blog: http://blog.risk.io Twitter: @mroytman
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×