Security Metrics are often about the performance of information security professionals - traditional ones are centered around vulnerability close rates, timelines, or criticality ratings. But how does one measure if those metrics are the rights ones? How does one measure risk reduction, or how successful your metrics program is at operationalizing that which is necessary to prevent a breach? The data we'll explore defined the 2016 Verizon DBIR Vulnerabilities section. This talk will borrow concepts from epidemiology, repeated game theory, classical and causal probability theory in order to demonstrate some inventive metrics for evaluating vulnerability management strategies. Not all vulnerabilities are at risk of being breached. Not all people are at risk for catching the flu. By analogy, we are trying to be effective at catching the "disease" of vulnerabilities which are susceptible to breaches, and not all are. How do we determine what is truly critical? How do we determine if we are effective at remediating what is truly critical? Because the incidence of disease is unknown, the absolute risk can not be calculated. This talk will introduce some concepts from other fields for dealing with infosec uncertainty. Attackers are human too - and currently available data allows us to make some predictions about how they'll behave. And to predict is to prevent.

1. MR.HUMAN:
VULNERABILITY MANAGEMENT
FROM THE
ATTACKER’S PERSPECTIVE
@MROYTMAN

3. “Our key to success is
knowing the network better
than the people who set it up.”
-Rob Joyce,
Director, NSA TAO

4. “The NSA tires known
CVEs and SQLi first.”
-Rob Joyce,
Director, NSA TAO

5. “The NSAs worst nightmare is
a sysadmin who pays
attention.”
-Rob Joyce,
Director, NSA TAO

6. INFOSEC:
1. Is slow.
2. Is lazy.
3. Loves hype.

8. SLOW

9. LAZY

10. <3
HYPE

12. ATTACKERS:
1. Automate.
2. Are just as lazy as you are.
3. Focus on what works.

13. 1. ATTACKERS ARE
BETTER AT
AUTOMATION

14. ATTACKERS
ARE
FAST

15. ATTACKERS ARE
BETTER AT
AUTOMATION

16. Q1
Q2
Q3
Q4

17. WE NEED
BETTER
AUTOMATION

18. WE NEED BETTER AUTOMATION
CURRENT VULN MANAGEMENT:
AUTOMATED VULN DISCOVERY
MANUAL-ISH VULN SCANNING
MANUAL THREAT INTELLIGENCE
MANUAL VULN SCORING
MANUAL REMEDIATION PRIORITIZATION

19. MANUAL

20. WE NEED BETTER DATA:
BETTER BASE RATES FOR EXPLOITATION
BETTER EXPLOIT AVAILABILITY
BETTER VULNERABILITY TRENDS
BETTER BREACH DATA
BETTER M E T R I C S

21. SOMETIMES WE MAKE
BAD DECISIONS
SOMETIMES WE HAVE
BAD METRICS

22. METRICS
ARE
DECISION SUPPORT

23. GOOD METRICS
ARE
OBJECTIVE FUNCTIONS FOR
AUTOMATION

24. WHAT MAKES A METRIC
GOOD?

25. HEARTBLEED
CVSS 5
SHELLSHOCK
CVSS 10
POODLE
CVSS 4.3

27. CVSS FOR PRIORITIZATION
IS A SYSTEMIC PROBLEM

28. ATTACKERS CHANGE TACTICS DAILY

29. WHAT DEFINES A GOOD METRIC?
GOOD DATA

30. WHICH SYSTEM IS MORE SECURE?
$1,000 $1,000,000
CONTROL 1 CONTROL 1
ASSET 1 ASSET 2

31. TYPES OF METRICS
-EXCLUDE REAL LIFE
THREAT ENVIRONMENT
TYPE 1
% FALLING FOR SIMULATED
PHISHING EMAIL
CVSS SCORE
-OCCURANCE RATE
CONTROLLED
-INTERACTION WITH
THREAT ENVIRONMENT
TYPE 2
# INFECTED MACHINES OF ISP
% VULNS WITH METASPLOIT
MODULE
-DESCRIBE UNDESIRED
EVENTS

32. WHAT DEFINES A GOOD METRIC?
1. BOUNDED
2. SCALED METRICALLY
3. OBJECTIVE
4. VALID
5. RELIABLE
6. CONTEXT-SPECIFIC - NO GAMING!
7. COMPUTED AUTOMATICALLY

33. 2. ATTACKERS ARE
JUST AS LAZY AS
YOU ARE

34. YOU NEED DATA TO MAKE DATA

35. METASPLOIT PRESENT ON VULN?
1. BOUNDED
2. SCALED METRICALLY
3. OBJECTIVE
4. VALID
5. RELIABLE
6. CONTEXT-SPECIFIC
7. COMPUTED AUTOMATICALLY
✓
✓
✓
✓
✓
✓
✓

36. YOU NEED DATA TO MAKE METRICS
! Probability*
(You*Will*Be*Breached*On*A*Particular*Open*Vulnerability)?
!"#$%&'($#)*+,(,-,#.% /)#*0ℎ#.%!00')#2%3$%4ℎ#,)%5&6)
43-*(%!"#$%&'($#)*+,(,-,#.
6%

37. PROBABILITY A VULNERABILITY HAVING CVSS SCORE >
X HAS OBSERVED BREACHES
0 2 4 6 8 10
0
1
2
3
4
5
6
7
8
9
10
Breach1Probability1(%)
CVSS1Base

38. 0 5 10 15 20 25 30 35
CVSS*10
EDB
MSP
EDB+MSP
Breach*Probability*(%)
Positive Predictive Value (the proportion of positive test results that are
true positives) of remediating a vulnerability with property
X:

39. 3. ATTACKERS
FOCUS ON WHAT
WORKS

41. ATTACKERS:
1. Automate.
2. Are just as lazy as you are.
3. Focus on what works.

42. ATTACKERS:
1. Spray and pray.
2. Use Metasploit.
3. Exploit more than patch
tuesday.

43. KENNASECURITY.COM
@MROYTMAN

44. References
Security Metrics
www.securitymetrics.org
Society of Information Risk Analysts
https://societyinforisk.org/
National Weather Service Research Forum
http://www.nws.noaa.gov/mdl/vlab/forum/VLab_forum.php
Dan Geer’s Full Day Tutorial On Measuring Security
http://geer.tinho.net/measuringsecurity.tutorial.pdf
Yasasin, Emrah, and Guido Schryen. "Derivation of Requirements for IT Security Metrics–An Argumentation Theory Based
Approach." (2015).
Savola, Reijo M. "Towards a taxonomy for information security metrics."Proceedings of the 2007 ACM workshop on
Quality of protection. ACM, 2007.
Böhme, Rainer, et al. "4.3 Testing, Evaluation, Data, Learning (Technical Security Metrics)–Working Group Report." Socio-
Technical Security Metrics(2015): 20.
B. Schneier. Attack trees: Modeling security threats. Dr. Dobb’s journal, 24(12):21–29, 1999.
T. Dimkov, W. Pieters, and P. H. Hartel. Portunes: representing attack scenarios spanning through the physical, digital and
social domain. In Proc. of the Joint Workshop on Automated Reasoning for Security Protocol Analysis and Issues in the
Theory of Security (ARSPA/WITS’10), volume 6186 of LNCS, pp. 112–129. Springer, 2010.
A. Beautement, M. A. Sasse, and M. Wonham. The compliance budget: Managing security behaviour in organisations. In
Proc. of the 2008 Workshop on New Security Paradigms, NSPW’08, pp. 47–58, New York, NY, USA, 2008. ACM.
B. Blakley, E. McDermott, and D. Geer. Information security is information risk management. In Proc. of the 2001 New
Security Paradigms Workshop, pp. 97–104, New York, NY, USA, 2001. ACM.

45. A. Buldas, P. Laud, J. Priisalu, M. Saarepera, and J. Willemson. Rational choice of security measures via multi-parameter attack trees. In Critical Information Infrastructures Security, volume 4347 of LNCS, pp.
235–248. Springer, 2006.
R. Böhme. Security metrics and security investment models. In Isao Echizen, Noboru Kunihiro, and Ryoichi Sasaki, editors, Advances in Information and Computer Security, volume 6434 of LNCS, pp. 10–24.
Springer, 2010.
P. Finn and M. Jakobsson. Designing ethical phishing experiments. Technology and Society Magazine, IEEE, 26(1):46–58, 2007.
M. E. Johnson, E. Goetz, and S. L. Pfleeger. Security through information risk management. IEEE Security & Privacy, 7(3):45–52, May 2009.
R. Langner. Stuxnet: Dissecting a cyberwarfare weapon. Security & Privacy, IEEE, 9(3):49–51, 2011.
E. LeMay, M. D. Ford, K. Keefe, W. H. Sanders, and C. Muehrcke. Model-based security metrics using adversary view security evaluation (ADVISE). In Proc. of the 8th Int’l Conf. on Quantitative Evaluation
of Systems (QEST’11), pp. 191–200, 2011.
B. Littlewood, S. Brocklehurst, N. Fenton, P. Mellor, S. Page, D. Wright, J. Dobson, J. McDermid, and D. Gollmann. Towards operational measures of computer security. Journal of Computer Security, 2(2–3):
211–229, 1993.
H. Molotch. Against security: How we go wrong at airports, subways, and other sites of ambiguous danger. Princeton University Press, 2014.
S. L. Pfleeger. Security measurement steps, missteps, and next steps. IEEE Security & Privacy, 10(4):5–9, 2012.
W. Pieters. Defining “the weakest link”: Comparative security in complex systems of systems. In Proc. of the 5th IEEE Int’l Conf. on Cloud Computing Technology and Science (CloudCom’13), volume 2, pp.
39–44, Dec 2013.
W. Pieters and M. Davarynejad. Calculating adversarial risk from attack trees: Control strength and probabilistic attackers. In Proc. of the 3rd Int’l Workshop on Quantitative Aspects in Security Assurance
(QASA), LNCS, Springer, 2014.
W. Pieters, S. H. G. Van der Ven, and C.W. Probst. A move in the security measurement stalemate: Elo-style ratings to quantify vulnerability. In Proc. of the 2012 New Security Paradigms Workshop, NSPW’12,
pages 1–14. ACM, 2012.
C.W. Probst and R. R. Hansen. An extensible analysable system model. Information security technical report, 13(4):235–246, 2008.
M. J. G. Van Eeten, J. Bauer, H. Asghari, and S. Tabatabaie. The role of internet service providers in botnet mitigation: An empirical analysis based on spam data. OECD STI Working Paper 2010/5, Paris:
OECD, 2010.
Klaus, Tim. "Security Metrics-Replacing Fear, Uncertainty, and Doubt." Journal of Information Privacy and Security 4.2 (2008): 62-63.