Expert Judgment is the foundation of many risk assessment methodologies. But research is robust on the inaccuracy of Expert Judgment with regards to rare events—and large data breach events are rare. Regression models, which are a statistical characterization of cross-company historical events are substantially more accurate than expert judgment or even models with expert judgment as a foundation.

1. Advantages of Regression
Models Over Expert Judgement
for Characterizing Cyber-Risk
Authors: Thomas Lee PhD, Spencer Graves PhD
Galileo Galilei, father of empirical science and
quantitative observation.

2. Advantages of Regression Models Over Expert Judgment
for Characterizing Cyber-Risk
Authors: Thomas Lee1
, Spencer Graves2
Expert Judgment is the foundation of many risk assessment methodologies. But research is robust on the
inaccuracy of Expert Judgment with regards to rare events—and large data breach events are rare.
Regression models, which are a statistical characterization of cross-company historical events are
substantially more accurate than expert judgment or even models with expert judgment as a foundation.
Much is at stake in accurately assessing risk: business productivity, the expert’s credibility—perhaps
security itself. There are also tens of billions of dollars’ in cross-industry annual security spending which
is guided by expert judgment. Since cross-company regression models can characterize events that are
otherwise rare within any particular company, they can be a powerful new tool and a foundation upon which
the expert can build.
Innaccuracy of Expert Judgment
Nobel laureate, and Princeton University Professor Daniel Kahneman explains in his book Thinking, Fast
and Slow3
that genuine expertise requires regular, high quality feedback on the results of their judgments,
but in cybersecurity, this experience is rare.
For example, an analysis of comprehensive data breach sources such as Health and Human Services (HHS),
reveals a data breach rate of about 2 data breaches
per 100,000 employees per year, for data breaches
that affect more than 500 people. For experts
working within companies with 10,000 employees
or less, a data breach affecting as few as 500
people would not occur more often than twice in
ten years. Familiarity with larger breaches is even
rarer since breach frequency declines
exponentially with the number of people affected.
Therefore, the typical cybersecurity expert does
not receive enough exposure to large data breaches
to form an accurate mental characterization.
Inaccuracy of expert judgment can be
demonstrated by asking experts to estimate the
cost of historical data breaches for which cost
information has been made public. Consistent with
peer reviewed research, we found that experts over
estimated risk by twenty times, on average. The
variability among experts was even larger than the
difference between the expert-average and actual
data breach costs.
1
Thomas Lee, CEO VivoSecurity has degrees in Physics, Electrical Engineering and a PhD in Biophysics from
the University of Chicago.
2
Spencer Graves, Chief Scientist VivoSecurity has degrees Industrial and Aerospace Engineering, an MA in
Mathematics from the University of Missouri at Kansas City, and a PhD in Mathematical Statistics from the
University of Wisconsin.
3
Daniel Kahneman (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux, ISBN 978-0374275631
Compare Expert Estimates with Actual Data Breach Costs
Actual cost (solid black lines) and expert estimates are shown for
four data breaches. Estimates were taken from six experienced
cybersecurity experts, averages (dashed blue line) and upper
range (dotted orange line) are indicated for each breach. Note
that the Y axis must be shown as a log scale to accommodate the
large difference between actual and expert, which visually
diminishes the differences. Breaches are as follows: 1-Malicious
Outsider with 220,000 affected; 2-Malicious Outsider with
30,224 affected and 1 lawsuit; 3-Malicious Insider with 8.5M
affected and 3 lawsuits; 4-Lost/Stolen device with 500,000
affected.
10
100
1,000
10,000
100,000
DataBreachCost
Thousands
1 2 3 4
Actual
Expert Average

3. Regression Models are an Alternative
An alternative to expert judgment are regression models, trained on cross-company historical events. In
fact, Kahneman recommends that “…simple heuristics with an empirical foundation should be given
credence substantially exceeding that of qualitative expert judgment…”. Regression modeling is a mature
methodology used in many industries including banking, insurance and medicine. A regression model
characterizes a set of historical data, discovering factors that are predictive and factors that are not
predictive. Factors can include time, and modeling can therefore reveal historical trends. Regression
modeling can also test commonly held assumptions by including factors for these assumptions during model
development. The degree to which factors correlate with results, will be discovered along with the
magnitude of their contribution.
The banking industry relies upon regression models to such an extent that, to ensure the stability of the
banking system, the Federal Reserve has issued guidance for how these models are to be developed,
evaluated and maintained (see SR 11-7 and SR 15-18). Banks must conform to this guidance, and the
industry has a term for this process: Model Risk Management. Although this guidance was written for the
financial industry, they are best-practices that should be adopted broadly for any regression modeling used
for making important business decisions.
Advantages of Regression Models
The advantages of using regression model over Expert Judgment are numerous and significant. Following
is an examination of a few.
Advantage 1: Characterization of Factors
One of the biggest values of regression modeling is understanding which factors are predictive and
important. The model development often begins with as many as a hundred potentially relevant factors, but
typically ends with just a few factors that are found to be statistically significant. This culling of factors can
provide important insights for risk mitigation in at least two areas. Most obviously, the sensitivity of factors
found to be relevant will be useful. Subtler, but also important, is the list of factors that were eliminated,
either because they were not predictive or because they did not predict as well as factors that were retained
in the model.
For example, a recent regression modeling of historical data breaches found that neither Industry nor Data
Type appeared to be important regarding the impact of a data breach—this allows drawing from a much
larger set of data when making comparisons and forecasting costs. The modeling also found that data
breaches caused by malicious outsiders were five times costlier and that investigation costs are one of the
most significant costs. The cost of a data breach caused by a malicious outsider can therefore be managed
through the incident response plan by ensuring the enterprise is will instrumented to speed post-breach
investigation.

4. Advantage 2: Characterization of Accuracy
Another important value of regression models is characterization of the degree of randomness and
forecasting accuracy and characterization of model stability. There are many ways to test accuracy and
stability, for example randomly dividing a data set
in half then using one-half to train a model that
forecast the other-half.
The degree of randomness and forecasting
accuracy can be characterized with regression
models, because the output is often a mean and
standard deviation. This allows understanding the
range of possible forecast and calculation of
confidence intervals. For example, the graph to the
right is a regression model forecast for the cost of
a data breach caused by a malicious outsider,
affecting 200,000 people. The model forecasts a
median cost of $2.6M with an 80% upper confidence interval is about $13M.
Advantage 3: Repeatability and Scenario Evaluation
Naturally a mathematical model is repeatable, and allows identification of the most important factors and
therefore evaluation of various scenarios. For example, a regression model that characterizes the cost of a
data breach allows projecting the increase in risk with customer growth and allows examining various ex-
ante cost management strategies such as insurance, self-insuring and investments into incident response.
Advantage 4: Credibility
Regression model forecasts are credible—leveraging a century of peer reviewed statistical-science-research
into the characterization of random events. Regression models lend themselves to model validation which
allows peer review of the model development process including suitability of data, variable transformation
and elimination and statistical based conclusions.
Advantage 5: Manage Cyber-Risk in a Model Risk Management Framework
For a financial institution a regression model allows cyber-risk forecasts to be managed within the Model
Risk Management framework, including 1) validation by qualified statisticians, 2) review by all interested
parties, 3) challenger models and 4) periodic model reassessment.
The Model Risk Management framework characterizes and documents the degree of randomness and
limitations of models to forecast, allows all stake holders to review and understand model limitations and
model risk.
Advantage 6: Stronger Risk Management Culture
Regression models are rigorous and allow critical examination through a Model Risk Management
framework. Choosing regression models over expert judgement demonstrates a stronger risk management
culture within an institution. For financial institutions, this can lead to a more favorable review by the
Federal Reserve during stress tests and more lenient capital ratios. For other institutions, it demonstrates
that the corporate board and senior management have used the best tools possible to characterizing and
manage cyber risk.

5. Linear Regression Explained
Linear regression is simply a matter of finding the best-fit line to a set of data: 𝑌 = 𝑚 × 𝑋 + 𝑏. But there
may actually be multiple subsets of data—multiple best fit lines. Modeling is therefore a process of
discovering the subsets of data and factors which distinguish these subsets.
A population appears as a random scatter that follows a straight
line. Regression modeling can discover the best fit line and
therefore a relationship 𝑌 = 𝑚 × 𝑋 + 𝑏
Regression modeling finds that the straight line scatter is
really two populations that can be distinguished using a factor
𝑓2 that was included in the data set. Modeling will find that
the best fit relationship 𝑌 = 𝑚1 × 𝑋 + 𝑏 + 𝑚2 × 𝑓2
The relationship between X and Y may not be simply a straight line. Another part of the modeling process
is therefore discovering an operation that will make the relationship a straight line. For example, the best
fit formula might be 𝑌 = 𝑚 × log 𝑋 + 𝑏, where log 𝑋 is the variable transformation that must be performed
so that linear regression can be used.
Why is Galileo Galilei on the Cover Page?
We emphasize Galileo Galilei, who is considered by many to be the father of empirical science, and
important for moving science away from subjective-argument to quantitative
observation. Aristotle reasoned that heaver objects should fall faster, but
Galileo showed this to be false using quantitative observations. Before
Galileo, we were the center of the universe, the heavenly objects—obviously
revolved around us. After Galileo we learned from quantitative observation
that we were but a mere satellite among many, orbiting the sun.
There are many “obvious” notions about cybersecurity. Quantitative
observations and regression models will help to shed light on which notions
are right and perhaps new discoverers will be made regarding what is effective for security.
About VivoSecurity
VivoSecurity provides data analytics and statistical modeling to Princeton Strategy Group. We are a
Silicon Valley Startup since 2012, with PhD level scientists and statisticians. We use advanced data
analytic techniques to model the probability and cost of cybersecurity events. We have strong
cybersecurity domain knowledge, strong knowledge of software applications, strong knowledge of
operating systems and hardware and a strong understanding of enterprise operations.
For more information, contact:
Thomas Lee, ThomasL@VivoSecurity.com
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30