Secure360 on Risk

Challenging
Conventional Wisdom:
A New Approach to
Risk Management
Alex Hutton
Jay Jacobs

What’s this We think you’re getting bad
information!
about? We think our industry can
do better!

We think this will make us
“more secure!”

Security is now so
essential a concern
that we can no longer
use adjectives and
adverbs but must
instead use numbers.
– Dan Geer

How are you making
decisions now?

What’s the quality of
those decisions?

Effective Decisions
need quality data,
models, execution

Our vendors and
standards aren’t
helping us
(-:

hey, why are
you getting
lousy
information
from
standards
and vendors?

The science
hey, why are of information
you getting security & risk
management
lousy is hard

information 1. Pseudo Science &
Proto Science

from 2. Models & Data

standards 3. Complexity

and vendors?

State of the Industry (a)
(Thomas Kuhn is way smarter than we are)

proto-science
somewhat random fact
gathering (mainly of readily
accessible data)
a“morass”of interesting,
trivial, irrelevant
observations
a variety of theories (that are
spawned from what he calls
philosophical speculation) that
provide little guidance to
data gathering

State of the Industry (b)
At our present skill in measurement of
security, we generally have an ordinal
scale at best, not an interval scale and
certainly not a ratio scale. In plain
terms, this means we can say whether
X is better than Y but how much better
and compared to what is not so easy.
– More from Dan Geer

If Science is based on
inductive observations to
derive meaning and
understanding and
measurement on quality
(ratio) scales, how about
InfoSec?

Where do we sit in the
family of sciences?

We’re the Crazy Uncle
with tinfoil hat antennae
used to talk to the space
aliens of Regulus V, has
47 cats, and who too
frequently (but
benignly) forgets to
wear pants.

“the Base Equation multiplies
Impact by 0.6 and
Exploitability by 0.4”

Jet Engine X Peanut Butter = Shiny

decimals aren’t magic.

adding one
willy-nilly doesn’t
suddenly
transform
ordinal rankings
into ratio values.

Data must exist in order to feed our
models...
... but creating the right models are
dependent on understanding what
data is useful!

20

Data, Models, Execution:
Garbage in-Garbage Out

Treat Data Poorly

Adapting to Situations

These “risk”
statements you’re
making...
I don’t think
you’re doing it
right.
- (Chillin’
Friederich Hayek)

A Comforting Thought...
“Given Newton's laws
and the current position
and velocity of every
particle in the universe,
it was possible, in
principle, to predict
everything for all time.”

-- Simon-Pierre LaPlace, 1814

8

4 4

2 2 2 2

Reductionism

8
?
4 4

?
2 2 2 2

Functionalism

Asset Reductionism

Functionalism
Comp. Comp.

Sub. Sub.

Attribute

Attribute

Attribute

Attribute

Awww man...
...even if it were the case that the
natural laws had no longer any
secret for us, we could still only
know the initial situation
approximately. ... small
differences in the initial conditions
produce very great ones in the
final phenomenon. A small error in
the former will produce an
enormous error in the latter.
Prediction becomes impossible...
-- Henri Poincare,
1887

ty non
lexi -l i
p nea
C om r
13

5 6

2 2 2 2

Systems Approach

Holism

Complex systems contain changing
mixtures of failures latent within them.
The complexity of these systems makes it impossible for
them to run without multiple flaws being present.

... individually insufficient to cause failure

...failures change constantly because of
changing technology, work organization,
and efforts to eradicate failures.

Complex systems run in degraded mode.

“How Complex Systems Fail”
- Richard Cook

Security is a characteristic of systems
and not of their components
Security is an emergent property of systems; it does not
reside in a person, device or department of an organization
or system.

... it is not a feature that is separate from
the other components of the system.

...the state of Security in any system is
always dynamic

“How Complex Systems Fail”
- Richard Cook

We may want to
rethink our
approach.

Overcoming the problem
• Medicine uses an “Evidence-
Based” approach to solving
problems in the complex
system that is the body.

• Dr. Peter Tippett (MD, PhD)
applies Evidence-Based
principles to Information
Security.
36

What to study: Sources of Knowledge
Suggested
context:
Capability
to
manage
(skills,
resources,

asset decision
quality…)
landscape
impact
landscape

risk

threat
landscape

controls
landscape

How: Data Quality in Evidence-Based Practice

Evidence
level
D Evidence
level
C Evidence
level
B Evidence
level
A

Evidence
level
A Case-‐series
Consistent
Consistent

“Expert
opinion
study
or
Retrospec8ve
Randomized

without
explicit
extrapola8ons
Cohort,
Exploratory
Controlled
Clinical

cri8cal
appraisal,
from
level
B
Cohort,
Ecological
Trial,
cohort
study,

or
based
on
studies. Study,
Outcomes
all
or
none,
clinical

physiology,
bench
Research,
case-‐ decision
rule

research
or
ﬁrst
control
study;
or
validated
in

principles.” extrapola8ons
from
diﬀerent

level
A
studies. popula8ons.

beNer

Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections

Evidence level B Formal Modeling Decision making
constructs
Evidence level A

Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections
You
are
here

Evidence level B Formal Modeling Decision making
constructs
Evidence level A

So
How
Do
We
Change?

Data
Models…

Standards

START
WITH

THE

OUTCOMES!

Two True Security
Outcomes:
Success and
Failure

Knowing Success in
InfoSec is hard
- Known Success (anti-Threat ops)
- Unknown success (controls work
without us knowing)
- Dumb luck (We’re not targeted, but our
neighbor is)

Getting the
outcomes:
Success

stronger
processes
result in fewer
availability
incidents

Getting the outcomes
- Successes:
- Existences of processes
- Operational (performance) metrics
- Maturity ratings

WHAT WE WANT ARE PATTERNS!

Knowing Failure is
(somewhat) easier

Getting The Outcomes:
Failures
VERIS | Verizon
Enterprise Risk and
Information Sharing

VERIS takes the
incident narrative
and creates metrics
(risk determinants)

VERIS | Verizon
Enterprise Risk and
Information Sharing
A
free
(as
in
beer*)

framework
created
for

metrics,
modeling,
and

compara8ve
analy8cs.
A
security
incident
(or
threat
scenario)
is
modeled
as
a

series
of
events.
Every
event

is
comprised
of
the
following
4
A’s:

Agent:
Whose
acLons
affected
the
asset
AcLon:
What
acLons
affected
the
asset
Asset:
Which
assets
were
affected

AOribute:
How
the
asset
was
affected

VERIS takes this :

INCIDENT REPORT
“An attacker from a Russian IP address
initiated multiple SQL injection attacks
against a public-facing web application.
They were able to introduce keyloggers
and network sniffers onto internal
systems. The keyloggers captured
several domain credentials which the
attackers used to further infiltrate the
corporate network. The packet sniffers
captured data for several months which
the attacker periodically returned to
collect…”

and…

…and translates it to this…
Event 1
Agent: External (Org crime)
Action: Hacking (SQLi)
Asset: Server (Web server, Database)
Attribute: Integrity
Event 2
Action: Malware (Keylogger)
1 > 2 > 3 > 4 >
Asset: Server (Web server)
Attribute: Confidentiality
Event 3
Action: Hacking (Use of stolen creds)
Asset: Server, Network (multiple)
Attribute: Confidentiality, Integrity
Event 4…

Framework

=
∑
∩ ∫√
Models Data

Framework Framework

Data Process
= Process
∑
∩ ∫√
Models =
∑
∩ ∫√
Data Models

Process
Process

Data

Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Get into loss factors (ABC)
- Share data
- Support data sharing efforts

Bring it Home:
your metrics program

Bring it Home:
or

Bring it Home:
or
The Amazing
Technicolor Scorecard

Priority #1:
no more surrogate data

Priority #1: (meaning)
no more risk analysts*

Priority #1: (really)
create data analysts

Data analysts need to
focus on quality data,
models, execution

asset
landscape
A balanced
scorecard of
sorts
threat impact
landscape landscape

risk

controls
landscape

Where to look? The
Two True Security
Outcomes:
Success and
Failure

Failures:
threat
landscape incidents, red/blue team

asset vulnerabilities, misconfigurations,
landscape
unknowns...

gaps in coverage, known lack of
controls
landscape effectiveness, known underskilled/
utilized...

impact Cost-Based Accounting around
landscape
incidents, cost of operations, etc...

Successes:
threat
landscape intel, red/blue teams, SIEM

asset vulnerabilities, misconfigurations,
landscape
unknowns, skills, training

controls positive threat outcomes (tOps), skills,
landscape
training

impact
landscape ROI? ROSI? (ducks to avoid tomatoes)

What to look? Two
types of data to find:
Focus initially
on Visibility,
then look to find
Variability.

How to look? The
GQM Approach:
For each
“where” for each
“what” use the
following “how”

How to look? The
GQM Approach:
For each
“where” for each
“what”, start by
using GQM as
“how.”

Goal, Question,
Metric
Conceptual level (goal)
goals defined for an object for a variety of
reasons, with respect to various models, from
various points of view.

Operational level (question)
questions are used to define models of
the object of study and then focuses on
that object to characterize the assessment
or achievement of a specific goal.
Quantitative level (metric)
Victor Basili
metrics, based on the models, is
associated with every question in order to
answer it in a measurable way.

The Book You
Should Buy
(Jay & Alex aren’t getting a
kickback, in case you’re
wondering)

GQM for Fun & Profit

Goals establish
what we want to Goal 1 Goal 2
accomplish.

Questions help us
understand how to
meet the goal. They Q1 Q2 Q3 Q4 Q5
address context.

Metrics identify the
measurements that
are needed to answer M1 M2 M3 M4 M5 M6 M7
the questions.

GQM for Fun & Profit

Execution Goal 1 Goal 2

Models Q1 Q2 Q3 Q4 Q5

Data M1 M2 M3 M4 M5 M6 M7

data about defined success
and failures
models of assets, controls,
threats contributing to impact
execution by data analysts
...Feeding standards, audits and governance

Questions?
Jay Jacobs Alex Hutton
@jayjacobs @alexhutton
jay@beechplane.com alex@alexhutton.com

Approaching the system
as a system
asset
landscape
impact

Prioritize
landscape

risk

threat
landscape

controls
landscape De-prioritize

Suggested context:
Capability to manage
(skills, resources,
decision quality…)

asset
landscape
impact
landscape

risk

threat
landscape

controls
landscape

Data Sharing:

- Sources:
- Qualify this Intel according to
framework
- Treat with appropriate data quality
listings (let models shape the certainty)

Get Into Accounting

- Use existing models that take
advantage of accounting concepts
(ABC) to Talk to the LOBs

Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Share data
- Support data sharing efforts
- Get into loss factors (ABC)

Challenging
Conventional Wisdom

Conventional Wisdom may not be wrong
- Question current practices
- Seek Evidence and Feedback

Secure360 on Risk

Recommended

Recommended

More Related Content

Similar to Secure360 on Risk

Similar to Secure360 on Risk (20)

More from Alexander Hutton

More from Alexander Hutton (7)

Recently uploaded

Recently uploaded (20)

Secure360 on Risk