1. Measuring Risk
What Doesn’t Work and What
Does
aliadocorp.com

2. 



2
Aliado – Name’s Root is ―Ally;‖ How We Value Our
Relationships with our Customers
Leading the Way Since 2008
Professional Service Company /Management
Consultancy
Core Aliado Leaders have over 20 Years Professional
Service Experience from Big 5

3. 
Consultants servicing our technical expertise:
 Security
 Infrastructure
 Risk Management

Our most important Assets - People
 Integrity, Principles, Values
 Reputation, Reputation, Reputation
 Brand

Trusted Advisor

Professional Services – Not Products

4. 


If your risk analysis and risk management don’t work, then that is
your biggest risk.
Risk management (or, for that matter, any management
methodology) itself rarely has performance metrics applied to it.
The most popular methods have no published controlled
experiments where the improvement on forecasts and decisions
was actually measured (although anecdotal case studies are
common).
Mis - Communication – IT Security and Business Managers have
no common language to discuss risk. IT Security talks in terms of
high, medium, low or ordinal scales and Business Managers talks
in terms of quantitative numbers.

5. 
Likelihood




Does your ―Risk Map‖ look more like the top
or bottom chart? If more like the top, how do
the errors mentioned earlier compare to the
variance among the clustered responses?
Clustering means that all the previous errors
mentioned before make up a large part of the
difference between scores of individual risks.
How does this address correlations, common
mode failures, and cascade failures? These
factors can make a few ―low risk‖ items add
up to one very big risk.
The ―math‖ in these methods don’t even
remotely approximate the relationships one
might build in a quantitative model.
Risk maps like this may be ok for initial
brainstorming, but don’t make critical
decisions based on it.
Impact
5

6. 
Extreme
◦ Extreme likelihood of security controls being compromised with the possibility of catastrophic financial losses occurring as a
result. An asset with a vulnerability that was demonstrated to be exploitable and subsequently led to the compromise of
sensitive information would be designated with this rating.

High
◦ High likelihood of security controls being compromised with the potential for significant financial losses occurring as a result. This
rating would be given if a vulnerability was found to be exploitable and potentially affect the confidentiality, availability, and/or
integrity of a given asset.

Elevated
◦ Elevated likelihood of security controls being compromised with the potential for material financial losses occurring as a result.
Assets with a finding that led to information disclosure, for example, but not necessarily a full compromise, would be assigned
this rating.

Moderate
◦ Moderate likelihood of security controls being compromised with the possibility of limited financial losses occurring as a result. A
system with a vulnerability whose impact was reduced by factors such as configuration settings or difficulty of exploitation would
be assigned this rating.

Low
◦ Low likelihood of security controls being compromised with negligible impact as a result. This rating signifies either the nonexistence of vulnerabilities or those that have minimal impact.

7. ▶
•
•
•
•
7
Human expertise is an important input in and it is
hard to completely automate. But there are certain
types of errors in human judgment we know how to
measure and control for:
Overconfidence – Their chance of being right is much less than they believe
Influence by irrelevant factors – Factors like the order in which you consider
projects, whether it is a 5-point scale or a 10-point scale, or how much other
people in the room smile all affect your judgments
Inconsistency – When given the same sets of problems to evaluate, experts have a
hard time giving the same answers; also, their memory is reconstructed so that
they believe they always had one preference when in fact they didn’t
Misinterpretation – We tend to interpret cues about risks, measurements and
decision problems in a way that is logically and mathematically irrational

8. Studies have shown that it is very easy for a decision-making process to
increase confidence in forecasts and decisions even if measured outcomes
(return on decisions, forecasts, etc.) are not improved – or even made worse





Gathering more information makes you feel better but, at some point, begins to reduce
decision quality while confidence continues to increase. (Tsai C. 2008)
Interaction with others also increases decision confidence but, again, at some point
decisions are not improved while confidence continues to increase (Heath C., Gonzalez
R. 1995)
Formal training in detecting lies makes individuals slightly worse at detecting lies in
controlled experiments – but there confidence in their judgments increases dramatically.
(Kassin, S.M., Fong, C.T. 1999)
An experiment with AHP shows confidence increased whether decisions are improved or
degraded. (Williams M. et. al. 2007)
Almost all popular business methodologies show no correlation to financial performance
of the firm (N. Nohria et. al. 2003)

9. Scales are simple. But our response behaviors when we use them are not.
Typical scales combine several complex, subtle errors




The use of scales simply obscures (doesn’t alleviate) the lack of information and
potential disagreements - he calls this an ―illusion of communication‖ (Budescu)
Popular weighted scores add error to unaided human judgment. Scale error is added
even if scales are ―well defined‖ by introducing an extreme rounding error. It is possible
to have one risk 10 or 50 times greater than another risk end up in the same final
group. (Cox)
―Partition dependence‖ creates an unanticipated relationship among choices on a scale.
Two scales that each define a ―1‖ in the same way (e.g. 1=―impact less than $1M), will
elicit different responses for a 1 depending on how many other choices there are.
Treating ordinal scales like linear scales that can be added and multiplied introduces an
error of ―assumed ratios‖. They assume relative values of the scales roughly
approximates real world relationships when an analysis of historical data shows they do
not.

10. 


―Gut Feel‖ is the baseline. Anything that ―works‖ has to show an improvement on this.
Measured sources of error : inconsistency, overconfidence, various biases, inaccurate
estimates
The worst case is not ―gut feel‖ – some methods add more error
The best case isn’t perfection – just measurably reduced error compared to gut feel
Method
Gut
Feel
Weighted Score
Preference
Theory Models
Quant.
Models
Measured
Improvement to
Judgment?
Baseline
No: Remove no errors
and add new errors
No: AHP has known
math problems; might
improve consistency
Yes: Proven
w/controlled tests
Does it quantify
risk?
Only
intuitively
No, it attempts to
describe risk;
No, but it can quantify
risk aversion
Yes
Determines HighPayoff Measures?
No
No: Turns some good
measures into scores
No
Yes (w/AIE)
Net Reduction in
Error?
Baseline
No: Probably Worse
Maybe Slightly Better
– Maybe not
Best

11. 
Aliado provides a Methodology--Applied
Information Economics—that IT Security and
IT Business Leaders can understand
◦ A statistical and probability application that allows
an organization to measure their risk accurately on
an ongoing basis that provides tangible results in
quantifying the risk on any risk landscape
component.

12. 

―Calibrate‖ experts to realistically assess probabilities.
―Do the math‖– don’t rely intuition entirely.
◦ Use the ―calibrated‖ judgments of experts in Monte Carlo simulations.
◦ Simple historical models usually outperform human judges.
◦ Compute the ―Expected Value of Information‖ to identify important
measures.



Improve unaided human judgment with statistical
―smoothing‖.
Try rational incentives to encourage better expert judgment.
Document basic decision criteria - especially risk vs. return.

13. Model The Current State of Uncertainty
– Initially use calibrated estimates and
then actual measurements
Calibration
Training
Define the Decision and Identify
Relevant Variables. Set up the
―Business Case‖ for the decision, using
these variables –
Compute the value of additional Information
– Determine what to measure and how much
effort to spend on measuring it.
No
Is there significant
value to more
information?
Yes
Measure where the information value is
high – Reduce uncertainty using any of
the methods
Optimize Decision – Use the quantified
Risk/Return boundary of the Decision makers to
determine which decision is preferred.

14. Event A
Event B
OR
%Orders
Lost
Demand
Lost
Revenue
•
Performance metrics for decision analysis tools is very sparse, but favors Monte Carlos.
•
One researcher in the oil industry found a correlation between the use of quantitative risk
analysis methods and financial performance – and the improvement in performance started
when they started using the quantitative methods. (F. Macmillan, 2000)
Data at NASA from over 100 space missions showed that Monte Carlo simulations beat
other methods for estimating cost, schedule and risks (Published this in The Failure of Risk
Management and OR/MS Today)
•

15. k
EVI
z
p(ri ) max
i 1
z
V1, j p(
j 1
j
| ri ),
z
V2, j p(
j 1
j
| ri ),... Vl , j p(
j
| ri ),
EV *
j 1
The formula for the value of information has been around for almost 60
years. It is widely used in many parts of industry and government as part of
the “decision analysis” methods – but still mostly unheard of in the parts of
business where it might do the most good.
What it means:
1.Information reduces uncertainty
2.Reduced uncertainty improves decisions
3.Improved decisions have observable consequences with measurable value

16. 
Value of Information

The Priority of Measurements is Reversed: This
calculation reveals that most organizations will
consistently focus on low-value measurements
and ignore high-value measurements - this is
the ―measurement inversion‖
Only a Few Measurements Are Really Needed:
We also found that, if anything, fewer
measurements were required after the
information values were known.
Some Additional Empirical Measurements are
almost always needed: I found that 97% of the
models I built justified further measurement
according to the information values.
Traditional Measurement Priorities


17. •
•
•
Our risk tolerance changes much more frequently than we are aware and for arbitrary
reasons. One study showed that being around people who smile make us more likely to take
risky bets. Others show that simply remembering past events that made us angry make us
more risk tolerant while recalling events where we were afraid made us more risk averse.
The simplest element of Harry Markowitz’s method “Modern Portfolio Theory” is documenting
how much risk an investor accepts for a given return. Documenting our appetite for risk
makes it less vulnerable to capricious change.
The “Investment Boundary” states how much risk an investor is willing to accept for a given
return. For our purposes, we modified Markowitz’s approach a bit.
Acceptable Risk/Return
Boundary
Investment Region
Investment

18. •
•
–
–
–
–
–
–
•
Some managers have told me they wish they could quantify the risks of
their decisions more rigorously
But they cite various reasons why they believe they can’t:
Concepts like “risk” (as well as “quality”, “flexibility”, etc.) are
fundamentally immeasurable
They can better evaluate an investment “by experience” i.e. in relation
to other investments they’ve seen
Some are skeptical about statistics (“you can prove anything with
statistics”)
Any approach that involves statistics will seem too “theoretical” to top
management
“We don’t have enough data”
“We can’t compute a precise probability”
Each of these are refuted by the evidence
Copyright HDR 2008
dwhubbard@hubbardresearch.com

19. 

Business Case:
◦ Retail Firm fined 2.1Billion by FTC for failing Audit
◦ FTC required Retail Firm to have a independent Consulting Firm to provide Risk
Assessment
◦ CSO had a Big 4 Firm as their Auditor and was not confident with them from the fine
from the FTC
◦ CSO heard of our Quantitative Risk Assessments and understood the value of a
tangible assessment that would provide him a real value of the risks they had and
could be accepted by the FTC.
Scope:
◦ Provide Risk Assessment across 4 Major lines of Business
◦ The elements in scope across all their lines of business were:

Customer Applications 
Prescription Applications 
Data Warehouses 
Mobile Data

Mainframe 
Windows Servers 
Financial Applications 
iSeries Systems
Network
Devices 
Internet Applications 
UNIX 
Backup Tapes 
Proprietary Data Applications

Stratus 
Databases 
Personal Systems Applications 
HR
Wireless 
Remote Access

Back Office Applications 
Test Systems
Print Operations 
LDAP Systems

Credential
◦ Threat communities – Cyber Criminals, Privileged Insiders, Non – Privileged Insiders,
Malware

20. ◦ Provide the Information risk for each line of business and the overall aggregate risk
for the Retail Firm.

Our client was able to see that their overall aggregate risk was in line with their expectations
that they were willing to accept for their business.
◦ Identify Risk by Threat Community

Cyber Criminals by far was the biggest risk to our client.

Privileged Insider was second most significant with Non – Privileged Insider than Malware.
◦ Loss Event Type
 Confidentiality consisted most of loss at 290M due to sensitive customer info
 Availability was distant second at $530K
 Integrity was last at $2100
◦ Risk by Asset Group

Personal Systems had the biggest exposure accounting for 48% of risk exposure

Internet Applications at 16%

Windows Servers (Internet facing) at 11%
◦ Best opportunities for risk reduction

Access Privileges – 33M

Personal System Security – 34M

Mobile Media Restrictions – 20M

21. 

Business Case
◦ Federal Executive Agencies face significant management and technical
challenges when measuring the contribution of IT investments to mission
results as required by the Clinger-Cohen Act.
◦ VA Information Security Program had an approved new infrastructure
initiative that will mitigate IT Security – related risk across the department.
The risks include reducing the cost and frequency of virus, unauthorized
access, fraud and other type of losses.
◦ VA wanted to have a methodology that could meet the Clinger
Scope
◦ Provide the best rollout strategy for VA Public Key Infrastructure
investment that will optimize the value of PKI
◦ Combination of its optional investments will reduce the greatest losses at
a reasonable cost.
◦ Determine the Effectiveness of the Information Security Program over time

22. 



Rollout of VAPKI should occur in a particular order and should be
implemented when a certain criterion is met. That criterion is 1.1% is
the expected reduction in fraud cost per person where VAPKI is
implemented. In other words, if the annual fraud cost per person
were $500 then the VAPKI cost per person must be less than $5.50
per person to justify rolling it out to that facility.
VA accelerated the anti-virus roll out by six months
The Information Security Program should reduce by 75% to 95% the
expected losses for all security incidents through 2006 estimated
somewhere between $1.1 billion and $2.4 billion.
One major optional investment (certain parts of Intrusion Detection)
did not reduce losses and therefore should not be made. This is
about a $30 million cost avoidance.

23. 
AIE determined that VA would make the best investment decisions by taken
seven key measurements. Those measurements will allow VA to determine
the effectiveness of the Information Security Program over time.
◦
◦
◦
◦
◦
◦
◦
Annual Fraud losses due to internal unauthorized access - $80M to $180M
Number of Pandemic Virus attacks per year – 2 to 4
Average number of people affected by a virus – 25K to 60K
Percentage productivity loss due to virus outbreak – 15% to 60%
Percentage of veterans affected – 2% to 15%
VAPKI initial cost of VA wide roll out - $1.3M to $2M
VAPKI annual cost of VA wide roll out - $1.1M to $1.3M

24. 


Regardless of how quantitative your environment
might be, be a skeptic about how your organization
assesses decisions and risks – ask how they know
it works (and consider the consequences if it
doesn’t)
Your judgment also has a performance that can –
and must – be measured
Considered against the size and risk of
decisions, better risk analysis will be one of the
best investments in your portfolio

25. Contact:
Information Security Threat Rating Scale
Karen Aldridge
Regional Sales
kaldridge@aliadocorp.com
www.aliadocorp.com