2. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?
3. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?
4. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?
9. “…when you can measure
what you are speaking
about, and express it in
numbers, you know
something about it; but when
you cannot express it in
numbers, your knowledge is
of a meagre and
unsatisfactory kind; it may
be the beginning of
knowledge, but you have
scarcely, in your thoughts,
advanced to the stage of
science, whatever the matter
may be.”
William Thomson,
1st Baron Kelvin
& Measurement Badass
10. The Journey Towards Knowledge
(and therefore, security)
1.2 WHAT IS THIS TOPIC
14. Science is based on
inductive observations
to derive meaning and
understanding and
measurement on quality
(ratio) scales, so what
about InfoSec?
Where do we sit in the
family of sciences?
15. We’re the Crazy Uncle
with tinfoil hat antennae
used to talk to the space
aliens of Regulus V, has
47 cats, and who too
frequently (but
benignly) forgets to
wear pants.
21. At our present skill in
measurement of security, we
generally have an ordinal scale
at best, not an interval scale
and certainly not a ratio scale.
In plain terms, this means we
can say whether X is better
than Y but how much better and
compared to what is not so
easy.
– Again, Baddss Dan Geer
22. State of the Industry
- proto-science
- somewhat random fact
gathering (mainly of readily
accessible data)
- a“morass”of interesting,
trivial, irrelevant
observations
- a variety of theories (that are
spawned from what he calls
philosophical speculation) that
provide little guidance to
data gathering
Thomas Kuhn
Philosophy of Science Badass
33. #RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of
Sale
Skimming
Devices
Theft/
Loss
Error
Employee
Misuse
Web
Applications
DBIR Top Patterns:
37. DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using
mobile malware showed that in less than a
year’s time, five variations of mobile
malware for one specific bank could be
detected. Modest estimates suggest that
criminals gained around €50,000 per
week using this specific form of mobile
malware, harvesting over 4,000 user
credentials from 8,500 infected bank
customers in just a few months. Mobile
malware does not move the needle in our
stats as we focus on organizational
security incidents as opposed to consumer
device compromises.
38. DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using
mobile malware showed that in less than a
year’s time, five variations of mobile
malware for one specific bank could be
detected. Modest estimates suggest that
criminals gained around €50,000 per week
using this specific form of mobile malware,
harvesting over 4,000 user credentials from
8,500 infected bank customers in just a
few months. Mobile malware does not
move the needle in our stats as we
focus on organizational security
incidents as opposed to consumer
device compromises.
39. BLIND MAN 2: THE ACCOUNTING-CONSULTANCY
INDUSTRIAL COMPLEX
48. A CSO MUST BECOME “MEASURED” TO
ESCAPE THE MISTAKES OF THE PAST AND PUSH
INTO THE FUTURE
49. SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
50. • What Is a CISO (throne of blood image
WHAT IS A CSO
51. • What Is a CISO (throne of blood image
WHAT IS A MEASURED CSO
52.
53.
54. W.E. DEMING
Father of Total Quality
Management and
inspiration that drove
the Japanese “post-
war economic
miracle.”
55. IT WAS NO MIRACLE.
What Deming taught the
Japanese was
“management by fact.”
56. • Improvements to the
system are never
ending.
• The only people who
really know where the
real potentials for
improvement are the
workers.
• The system is always
changing.
• There are countless
ways for the system to
go wrong.
• Statistics (metrics) are used
to focus the conversation on
fact and improvement
• Goals for quality are cross-
silo
• Theories for improvements
are implemented and tested.
• The management uses the
workers as essential
"instruments" in
understanding what is.
57. A MEASURED CSO:
• Relies on metrics, data, intel for good decisions,
• Invests in improvements to People, Process and Technology,
• Puts innovation for improvements to the system
(improvements = security, cost) in the hands of the operator,
• Ensures that there is a feedback loop for effectiveness
initiatives, and
• Works tirelessly within the bureaucracy to improve all
aspects of the system.
58. THE MEASURED CSO’S MISSION:
• To provide the best and least-cost security for
shareholders, and continuity of employment for
his workers.
• We, as an industry, know that “best” and”least-cost” are
not necessarily contradictory
• We also have a HUGE continuity issue
60. WHAT IS THAT SYSTEM -
That which Defends
(Detects, Responds, & Prevents).
61. THE MEASURED CSO USES METRICS TO:
• Develop and improve the People, Process, and
Technology to Defend
• Plan / Build / Manage those defenses
62. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.
63. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.
Sorry, ISACA
64. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.
• There are two systems which the CSO must
manage across (at least 4 audiences)
• Those that support “defend”
• Those that support Plan/Build/Manage
67. EPIDEMIOLOGY
Risk Factors (Determinants)
Variables associated with increased
frequency of event.
Risk Markers
Variable that is quantitatively associated
with a disease or other outcome, but
direct alteration of the risk marker does
not necessarily alter the risk of the
outcome.
Correlation vs. Causation
Risk factors or determinants are
correlational and not necessarily causal,
because correlation does not prove
causation.
68. EPIDEMIOLOGY
Risk Factors (Determinants)
Variables associated with increased
frequency of event.
Risk Markers
Variable that is quantitatively associated
with a disease or other outcome, but
direct alteration of the risk marker does
not necessarily alter the risk of the
outcome.
Correlation vs. Causation -
Risk factors or determinants are
correlational and not necessarily causal,
because correlation does not prove
causation.
THE MEANS TO FIND PATTERNS
69. Example of a medical approach:
Dr. Peter Tippett & Verizon DBIR
70. A security incident (or threat scenario) is
modeled as a series of events. Every
event
is comprised of the following 4 A’s:
Agent: Whose actions
affected the asset
Action: What actions affected
the asset
Asset: Which assets were
affected
Attribute: How the asset was
affected
VERIS (Vocabulary for
Event Recording &
Incident Sharing)
70
77. #RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of
Sale
Skimming
Devices
Theft/
Loss
Error
Employee
Misuse
Web
Applications
DBIR Top Patterns:
78. THE KEY TO THE MEASURED CSO SYSTEM 1:
FRAMEWORK, DATA, MODELS
83. 83
Data MapReduce Process Analytics & Reporting
Threat Intel Feeds
Control Data
Control Logs
System Logs
Event
History
&
Loss
Loss
Distribu8on
Dev.
B.I.A.
Control Data
Control Logs
System Logs
Configuration Data
Vulnerability Data
HR Information
Process Behaviors
XML
CSV
EDI
LOG
SQL
JSON
Text
Binary
Objects
createmap
reduce
Traditional
RDBMS
Systems
Workflow
Analytics
Reporting
92. MEASURED CSO SYSTEM 1: THE METRICS AND
MODELS THAT “DEFEND” AGAINST THREAT
PATTERNS.
(real and anticipated or forecasted)
93. MEASURED CSO SYSTEM 2: THE METRICS
NEEDED TO PLAN/BUILD/MANAGE
SYSTEMS (OPERATIONS)
94. THE MEASURED CSO MUST ALSO INCLUDE A
KEEN UNDERSTANDING AND PARTNERSHIP
WITH IT OPERATIONS
95. THE MICROMORT
A one in a million
chance of death
Ronald A. Howard
96. Activities that increase the death risk by roughly one micromort, and their
associated cause of death (wikipedia):
Traveling 6 miles by motorbike (accident)
Traveling 17 miles by walking (accident)
Traveling 10 miles by bicycle (accident)
Traveling 230 miles (370 km) by car (accident)
Traveling 1000 miles (1600 km) by jet (accident)
Traveling 6000 miles (9656 km) by train (accident)
Traveling 12,000 miles (19,000 km) by jet in the United States (terrorism)
Increase in death risk for other activities on a per event basis:
Hang gliding – 8 micromorts per trip
Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)
97. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
98. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
Ecstacy (MDMA) – 0.5 micromorts per
tablet (most cases involve other drugs)
99. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
100. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
101. The Measured CSO must know where IT is
overweight, smoking ecstasy, while riding a
rocket-powered bicycle on the railing of a
bridge.
110. 110
But because we
gather what is most
readily available -
most metrics
programs look like
my living room.
How does the
measured CSO get
context?
111. GOAL, QUESTION, METRIC
Conceptual level (goal)
goals defined for an object for a variety of
reasons, with respect to various models, from
various points of view.
Operational level (question)
questions are used to define models of
the object of study and then focuses on
that object to characterize the assessment
or achievement of a specific goal.
Quantitative level (metric)
metrics, based on the models, is
associated with every question in order to
answer it in a measurable way.
Victor Basili
112. GQM FOR FUN & PROFIT
Goals establish
what we want to
accomplish.
Questions help us
understand how to
meet the goal. They
address context.
Metrics identify the
measurements that
are needed to answer
the questions.
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
115. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
% Coverage by Business Units
%Coverage by Asset category
%Coverage by Risk
Unix
Windows Server
Desktop
OS
Components
Likelihood
Impact
Most Significant Failures
Repeat Offenders
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit
116. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
What should our Priorities be for timeliness?
What is Policy for timeliness?
What other Considerations for Timeliness?
What is time to patch like for assets with worst Likelihoods?
What is time to patch like for assets with worst Impacts?
What % are Late by
What are our Repeat Offenders?
likelihood
Impact
by asset category
by business unit
by risk
UNIX
Windows Server
Desktop
likelihood
impact
117. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
Cost
Risk Reduction
Hour per Asset spent Patching
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Cost Per Hour
Hour per Asset, by ALE per Hour
Hour per asset category
118. GQM EXAMPLE: PATCH MANAGEMENT
• The Measured CSO creates a scorecard of
KRI’s & KPI’s that Includes:
• Historical values
• “Triggers”
• “Thresholds”
(each of these?) aren’t perfect, but establish a
hypothesis for testing & optimization.
126. 126
Data MapReduce Process Analytics & Reporting
Threat Intel Feeds
Control Data
Control Logs
System Logs
Event
History
&
Loss
Loss
Distribu8on
Dev.
B.I.A.
Control Data
Control Logs
System Logs
Configuration Data
Vulnerability Data
HR Information
Process Behaviors
XML
CSV
EDI
LOG
SQL
JSON
Text
Binary
Objects
createmap
reduce
Traditional
RDBMS
Systems
Workflow
Analytics
Reporting
129.
“If you do not know
how to ask the right
question, you
discover nothing.”
130. RESOURCES
FOR
GQM
AND
MICROMORTS
-‐
WIKIPEDIA
FOR
DBIR
DATA,
THE
VERIZON
DBIR
FOR
DEMING
QUOTES,
THE
WORKS
OF
MYRON
TRIBUS:
http://www.qla.com.au/papersTribus/Oslo3.pdf
http://www.unreasonable-‐learners.com/wp-‐content/uploads/2011/03/
Germ-‐Theory-‐of-‐Management-‐Myron-‐Tribus1.pdf
http://www.qla.com.au/papersTribus/DEMINGS_.PDF