Every single risk framework that applies to InfoSec talks about probability but doesn't offer much guidance on how to calculate or apply it. One could borrow from the common financial or manufacturing risk models, but those models do not apply to InfoSec because they are informed from past events. InfoSec risk needs to look forward without being informed by the past, and that drives the risk professional into very specific risk models. These models simplify probability calculations and there are proven ways to apply these models effectively, efficiently, and with high confidence in the results. Originally presented at Vancouver SecSIG, Nov 13, 2015

1. The Application Of
Laplacian Probability Models In
Likelihood Calculations For
Information Security
Risk Assessments
JORDAN SCHROEDER, CISSP, CISM
by
NOVEMBER 13, 2015

2. INTRO
WHO AM I?
▸ Member of the GRC team at Visier, Inc
▸ Moderator of Security StackExchange
▸ Former teacher, actor, singer, director, Coast Guard Officer,
undertaker, database designer, tax preparer, business owner,
day trader
▸ http://www.linkedin.com/in/schroederjordan
▸ http://security.stackexchange.com/users/6253/schroeder
▸ https://gophishyourself.wordpress.com

3. INTRO
WHO AM I?
▸ NOT a formally trained Risk Professional
▸ but deep respect for Risk Professionals
▸ NOT a probabilities expert
▸ but deep respect for probabilities
▸ My experience as a day trader was affected by risk models
and probabilities

4. INTRO
RISK PROFESSIONALS SHROUDED IN MYSTERY

5. INTRO
PROBABILITIES
http://andrewgelman.com/wp-content/uploads/2009/09/univ16.png

6. INTRO
MY INTRO TO INFOSEC RISK
▸ Risk models for organizations I worked in
▸ CISSP
▸ CISM

7. INTRO
RISK WAS TOO RISKY
▸ Big subject
▸ Lots of examples of risk gone wrong (e.g. 2008 credit
crisis)
▸ Simplistic formulas (ALE = ARO x SLE)
▸ Mixed with arcane incantations:
▸ My conclusion: leave it to the professionals
https://en.wikipedia.org/wiki/Black%E2%80%93Scholes_model

8. INTRO
LEAVING IT TO THE PROFESSIONALS
▸ Used existing models in the organization
▸ Tried to understand what the pros had done
▸ Lots of questions that didn’t add up
▸ “What we’ve done isn’t perfect, but it’s what the
organization can handle right now.”
▸ … and then it happened …

9. COULD YOU DEVELOP
AN ERM FOR US?

10. RISKY BUSINESS
RISK FRAMEWORKS GALORE!
▸ COSO
▸ ISO 31000/ ISO 27005
▸ NIST 800-39
▸ RISK IT
▸ FAIR
▸ OCTAVE Allegro

11. RISKY BUSINESS
SIMILARITIES
▸ Frameworks very similar in goal:
▸ Repeatable
▸ Measurable
▸ Consistent
▸ Complete
▸ Cyclical
▸ Maturing

12. RISKY BUSINESS
THE BASICS - THE PROBLEM
▸ Risk = Likelihood × Impact
▸ Annualized Loss Expectancy (ALE) = Likelihood × Value
ISO 31000

13. RISKY BUSINESS
VALUE PROPOSITION
▸ Value/Consequence/Impact is easy to understand and calculate
▸ Risk Frameworks spend large sections guiding the reader on
the useful ways to conceive of and update Value/
Consequence/Impact
▸ But … Likelihood?
▸ Brings to mind complex maths
▸ Where are those formulas?
▸ Non-math options kept being mentioned …

14. RISKY BUSINESS
ONE THING KEPT BUGGING ME
ISO 31000:
5.4.3 (Risk Analysis)
“Consequences and their likelihood can be determined by
modelling the outcomes of an event or set of events, or by
extrapolation from experimental studies or from available
data.”

15. RISKY BUSINESS
ONE THING KEPT BUGGING ME
ISO 31010:
Likelihood:
1. Historical data
2. Forecasts (fault tree analysis, simulations)
3. Opinion

16. RISKY BUSINESS
ONE THING KEPT BUGGING ME
NIST 800-39: “Managing Information Security Risk”
Likelihood determinations can be based on either threat assumptions
or actual threat data (e.g., historical data on cyber attacks, … or
specific information on adversary capabilities, intentions, and
targeting). When specific and credible threat data is available …
organizations can use the empirical data and statistical analyses to
determine more specific probabilities of threat events occurring.
In addition, some organizations prefer quantitative risk assessments
while other organizations, particularly when the assessment involves
a high degree of uncertainty, prefer qualitative risk assessments.

17. RISKY BUSINESS
ONE THING KEPT BUGGING ME
NIST 800-30 r1 “Guide for Conducting Risk Assessments”
Appendix G
The term likelihood, as discussed in this guideline, is not
likelihood in the strict sense of the term; rather, it is a
likelihood score. Risk assessors do not define a likelihood
function in the statistical sense. Instead, risk assessors assign
a score based on available evidence, experience, and expert
judgment.

18. RISKY BUSINESS
ONE THING KEPT BUGGING ME
OCTAVE Allegro
“However, because it is often difficult to accurately quantify
probability, especially with respect to security vulnerabilities
and events, probability is expressed in the OCTAVE Allegro
methodology qualitatively as high, medium, or low. “

19. RISKY BUSINESS
GAO SAVE US!!
General Accounting Office (GAO):
“Estimating the likelihood that such threats will materialize
based on historical information and judgment of
knowledgeable individuals.”

20. RISKY BUSINESS
LET’S RECAP
▸ Modeling
▸ Extrapolation
▸ Fault Tree Analysis
▸ Statistical Analysis
▸ Quantitative
▸ Opinion
▸ Personal judgement
▸ Estimate
▸ Assumption
▸ Qualitative

21. RISKY BUSINESS
DO YOU FEEL LUCKY?
http://andrewgelman.com/wp-content/uploads/2009/09/univ16.png
OR
https://www.pehub.com/wp-content/uploads/2013/02/dartboard.jpg

22. RISKY BUSINESS
THE Q’S
▸ Quantitative or Qualitative?
▸ Is Qualitative analysis just the lazy way out?
▸ I set upon a quest to understand the mathematically valid
approach of calculating likelihood in InfoSec Risk Analysis
▸ If I was confused, I figured others might be, too
▸ What I discovered surprised me

23. MARTIAN COIN
FLIPS

24. MARTIAN COIN FLIPS
THIS PORTION OF THE PRESENTATION FROM
Chapter 18

25. MARTIAN COIN FLIPS
THE LONGEST JOURNEY STARTS WITH THE FIRST STEP
▸ I started my quest looking to develop a mathematical
model that would be appropriate for the InfoSec field
▸ Started to develop a Bayesian Net that would
accommodate the unique challenges of InfoSec risk
▸ evolving threats
▸ 0-days
▸ weaknesses in patch processes

26. MARTIAN COIN FLIPS
DOWN BY THE BAYES
▸ Jaynes text led me to a discussion about the
difference between data and information when it
comes to probability calculation

27. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ What is the probability that a coin will come up tails on the next toss?
▸ P = 0.5
▸ What if you examined the coin (more data) to ensure it was a normal
coin?
▸ P = 0.5
▸ What if you knew more about the design on the coin, when it was
made, etc.?
▸ P = 0.5
▸ What if the coin had already been flipped 5 times and it came up tails
each time?

28. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸More data about the coin’s flip history
does not affect the expectation of
probability
Jaynes pg 558

29. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ What about the likelihood that there used to be life on Mars?
▸ Maybe yes, maybe no
▸ Let’s set it to P = 0.5
▸ New data radically alters the probability one way or the other
▸ One new piece of information can make the difference
▸ What if the coin we were flipping was found in the sands of
Mars?

30. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸New data about Mars affects the
expectation of the probability
Jaynes pg 558

31. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ Calculating probability by itself is not enough
▸ One must know a system’s resilience to new data
▸ How new data affects probability is the key
▸ Requires knowledge of the system
▸ We know how a coin flip works
▸ Uncertainty and probability

32. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ We can apply this to calculating the likelihood
of a web server compromise
▸ WordPress installation
▸ 2 compromises last year
▸ each by a different vulnerable plug-in
▸ now fully patched

33. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ What is the likelihood that my server will be
compromised this year?
▸ How do you set up that calculation?
▸ What data do you use?
▸ What historical data do you use?
▸ What new data might throw off your calculations?
▸ 0-days?

34. MARTIAN COIN FLIPS
A FEW NOTES ON HISTORY …
▸ Can you use historical data when the system
changes?
▸ Patches, configuration changes, new mitigations
▸ The number of 0-days in the past is not indicative of
0-days in the future
▸ Historical data is useful when the system is unaltered:
▸ Threats, costs, unpatched systems

35. MARTIAN COIN FLIPS
WHEN INFORMATION ATTACKS
▸ How can you perform Quantitative probability
analysis when:
▸ there is not enough data (can’t use history in a
changing system)?
▸ every new piece of data radically alters the
probability calculation?
▸ Maybe “more data” is not the point

36. “BUT, STATISTICAL ANALYSIS
DEPENDS ON MORE DATA!”

37. LAPLACE
ENTER, LAPLACE
Laplace Transform
Laplace’s equation using Laplace operator
https://en.wikipedia.org/wiki/Pierre-Simon_Laplace

38. LAPLACE
LAPLACE’S INDUCTIVE PROBABILITY
Laplace and Bayes are the guys who wrote the rules on
how to use historical data to estimate future probability

39. LAPLACE
LAPLACE’S FOLLY
▸ (5000 × 365.25 + 1) : 1 = 1,826,251 : 1 in favour of the sun
rising tomorrow
▸ Under the assumption that the earth was 5000 years old
▸ The whole “sunrise” example was a huge mistake…

40. LAPLACE
LAPLACE’S SANITY
▸ “But [the probability of the sun rising tomorrow] is far
greater for him who, seeing in the totality of
phenomena the principle regulating the days and
seasons, realizes that nothing at the present moment
can arrest the course of it.”
▸ Translation:
▸ Don’t blindly throw numbers into a formula
▸ Knowledge trumps “more data”

41. LAPLACE
LAPLACE’S HOPE
▸ What does this mean for us?
▸ More historical data about the number of days the
sun has risen changes the probability calculation
▸ More historical data does not alter the probability of
the sun rising tomorrow, since we understand the
principles
▸ Knowledge of the “principles” can override the
importance of statistical data

42. LAPLACE
FAILURE ANALYSIS
▸ What’s our risk in regards to email infections?
▸ How many email infection attempts last year?
▸ How many successful email infections last year?
▸ Infections / Attempts = Likelihood per attempt this
year?

43. LAPLACE
FAILURE IN ANALYSIS
▸ Infections / Attempts = Likelihood?
▸ Only valid if the system does not change!
▸ Were the infections due to a specific vulnerability?
▸ Was there a spear-phishing campaign?
▸ Was there a single user clicking every spam email they got?
▸ Knowing the principles of the system overrides a strict
statistical analysis

44. LAPLACE
EQUIVALENT TO A SUNRISE?
▸ What about a situation where, despite a changing
system, failures appear to be predictable?
▸ What about InfoSec failures that are as sure as a
sunrise?

45. LAPLACE
THAT’S NOT A NET, THAT’S A STRING
▸ My Bayes Net was looking shoddy
▸ Historical data disappeared
▸ I was left with the ‘conditionals’ to the statistical
probabilities
▸ borne from expert knowledge of the systems
▸ What I had left looked a lot like a pure Qualitative
analysis

46. CHOSE AN
ANALYSIS METHOD

47. BE CHOOSEY
QUALIFIED ANALYSIS
▸ Qualitative Likelihood Analysis no longer appeared
to be the lazy way out
▸ Perhaps the Qualitative approach might even be the
better way
▸ How to make a valid choice?

48. BE CHOOSEY
QUALIFIED ANALYSIS
▸ Quantitative Likelihood Analysis means using some
form of statistical analysis
▸ For InfoSec Risk in a changing system, this appears
to be invalid
▸ Qualitative Likelihood Analysis allows you to use your
judgement and knowledge of the system
▸ The defensive technologies, the patch history, the
people, configuration changes

49. BE CHOOSEY
QUALITATIVE VS QUANTITATIVE
▸ Risk Frameworks offer the choice
▸ but little guidance on the choice
▸ Quantitative analysis produces numbers
▸ with decimal points!
▸ Qualitative analysis “feels” wrong

50. BE CHOOSEY
ENORMOUS CONFUSION OUT THERE
▸ https://en.wikipedia.org/wiki/Risk_assessment [accessed Oct 30, 2015]
▸ “Risk assessment is the determination of quantitative or qualitative
estimate of risk …”
▸ Talks about quantitative then never mentions qualitative assessments …
▸ http://www.sans.edu/research/leadership-laboratory/article/risk-
assessment
▸ “We now have a quantitative risk assessment value of $15 million and a
qualitative risk level of High.”
▸ author confuses the events in the ALE formula

51. BE CHOOSEY
CLEAR THE CONFUSION
▸ It is important to understand that one can calculate
▸ Likelihood
▸ Impact
▸ using either
▸ Qualitative Analysis
▸ Quantitative Analysis

52. BE CHOOSEY
WHEN TO USE QUANTITATIVE PROBABILITY
▸ It is entirely possible to determine a Quantitative
probability
▸ Used in Finance and Manufacturing all the time
▸ Perfect for when you have lots of historical data
▸ The subjects need to be comparable
▸ 1000 servers all configured the same way
▸ (ISO 31010 Annex B presents a full analysis of methods)
https://www.flickr.com/photos/trevmeister/8413196866

53. BE CHOOSEY
QUANTITATIVELY PROBLEMATIC - STAMP
“… for hazards associated with standard systems with abundant
historical data, it may be possible to define likelihood using a
quantitative probability of occurrence. However, for most complex
socio-technical** systems with little or no historical experience …
a qualitative assessment of likelihood is the best that can be
achieved.”
[Dulac, 2007]
STAMP (Systems-Theoretic Accident Model and Processes) is
based entirely on a Qualitative approach
[**social-technical: technology used by people]

54. BE CHOOSEY
QUANTITATIVELY PROBLEMATIC - NIST
“In addition, some organizations prefer quantitative risk
assessments while other organizations, particularly when the
assessment involves a high degree of uncertainty, prefer
qualitative risk assessments.” [NIST 800-39]
“Consideration of uncertainty is especially important when
organizations consider advanced persistent threats (APT)
since assessments of the likelihood of threat event occurrence
can have a great degree of uncertainty.” [NIST 800-30 r1]

55. BE CHOOSEY
QUANTITATIVELY PROBLEMATIC - ISO 31010
“Full quantitative analysis may not always be possible or
desirable due to insufficient information about the system or
activity being analysed, lack of data, influence of human
factors, etc. ” [5.3.1]

56. BE CHOOSEY
QUANTITATIVELY PROBLEMATIC - GAO
“… the availability of data can affect the extent to which risk
assessment results can be reliably quantified.”
“Reliably assessing information security risks can be more difficult
than assessing other types of risks, because the data on the
likelihood and costs associated with information security risk
factors are often more limited and because risk factors are
constantly changing.”
“Even if precise information were available, it would soon be out
of date due to fast-paced changes in technology and factors such
as improvements in tools available to would-be intruders. ”

57. BE CHOOSEY
MOST INFORMATION SYSTEMS ARE
▸ Complex
▸ Socio-technical (those darn people!)
▸ In a state of change
▸ With high degrees of uncertainty

58. BE CHOOSEY
DATA YOU NEED FOR QUANTITATIVE LIKELIHOOD ANALYSIS
▸ Already stale
▸ About a previous state of the system
▸ Unnecessary (perhaps misleading)
▸ Trumped by knowledge of the system

59. CONCLUSION

60. BE QUALITATIVE
CONCLUSION
Unless you have a valid reason not to,
use a Qualitative approach

61. BE QUALITATIVE
QUALITATIVE LIKELIHOOD ASSESSMENTS
▸ Low
▸ Medium
▸ High
▸ Improbable
▸ Remote
▸ Occasional
▸ Probable
▸ Frequent
▸ Highly unlikely
▸ Unlikely
▸ Somewhat likely
▸ Highly likely
▸ Almost certain
▸ Very Low
▸ Low
▸ Moderate
▸ High
▸ Very High

62. BE QUALITATIVE
CRITIQUES OF QUALITY ASSESSMENTS
▸ Too broad to properly compare
▸ With a detailed assessment, it is easier to rank mitigating options!
▸ Response: be careful that you are not trusting in false accuracy
▸ Response: get stakeholders together and prioritize
▸ Misleading labels between assets or departments can cause
confusion
▸ What does “High” mean to you?
▸ Response: central management and guidance/training can help

63. BE QUALITATIVE
HOW DO I SELL THIS TO OTHERS?
▸ “Someone is going to want a Quantitative Likelihood calculation.
How do I tell them that Qualitative is the valid approach?”
▸ Simply this:
▸ “Since our systems and the threats to those systems are in a
constant state of change, historical statistical data is not
relevant to complete a Quantitative Likelihood calculation.
Risk frameworks for InfoSec systems recommend a Qualitative
approach, which leverages the expert judgement of those
close to the objects of risk, who are best able to estimate
future likelihood.”

64. GATHERING
QUALITATIVE DATA

65. QUALITY
WHEN QUALITATIVE PROBABILITY HAS BEEN USED
▸ GAO did a study on organizations that relied on their ERM,
and where the ERM was perceived by the organization to
be effective (Risk done right)
▸ Risk Assessment methods were:
▸ Simple
▸ Mostly Qualitative

66. QUALITY
GAO TIPS
▸ Segment Risk Assessments into smaller units to limit the scope (easier
to re-assess when necessary)
▸ Make the Risk Assessors the ones who are close to the objects of
risk
▸ System admins, help desk managers, etc.
▸ Use tables, questionnaires (multiple choice) and standardized report
formats
▸ Define the scale to keep it consistent across the organization
▸ Training, guidance documents

67. QUALITY
GAO TIPS
▸ Business Units responsible for
▸ reassessments
▸ following up on recommendations
▸ Encourages a bottom-up approach to Risk Management
▸ Risk becomes a mindset at the bottom
▸ Controls are better understood and adopted by those
implementing them

68. THE END
ACKNOWLEDGEMENTS
▸ Dan Bühler, MSc (Mathematics)
▸ Anton Smessaert, PhD (Data Scientist)
▸ Prof. David Wagner (UC Berkeley)

69. PROBABLY:
THE BEST WAY TO CALCULATE
INFOSEC RISK
JORDAN SCHROEDER, CISSP, CISM