This document discusses the need for Chief Security Officers (CSOs) to become "measured" by relying on metrics and data to make decisions and continuously improve security programs. It outlines two key systems a measured CSO must manage - one focused on developing metrics and models to detect and prevent threats, and one focused on metrics to plan, build and manage security operations. The document advocates using frameworks like VERIS to classify security information and incidents to identify patterns and risk factors. It also stresses the importance of data warehousing and analytics to enrich security data from various sources. Overall, the document argues that relying on measurable facts rather than subjective standards is critical for CSOs to advance the security field in a scientific manner.

1. THE MEASURED CSO
ALEX
HUTTON
-­‐
A
TOO
BIG
TO
FAIL
BANK
@ALEXHUTTON

2. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?

3. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?

4. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?

5. SECTION 1: BACKGROUND
Who am I? What is this topic? Where are we? How did we get here?

6. 1.1 WHO AM I

7. • Security Engineer
• Security Product Management
• E-Commerce Site Design / Manager
• Risk Consultant
• OCTAVE / NIST
• FAIR
• Verizon DBIR
• IANS Faculty
• Director, Operations / Technology Risk
• Director, Information Security
1.1 WHO AM I

8. 1.2 WHAT IS THIS TOPIC

9. “…when you can measure
what you are speaking
about, and express it in
numbers, you know
something about it; but when
you cannot express it in
numbers, your knowledge is
of a meagre and
unsatisfactory kind; it may
be the beginning of
knowledge, but you have
scarcely, in your thoughts,
advanced to the stage of
science, whatever the matter
may be.”
William Thomson,
1st Baron Kelvin
& Measurement Badass

10. The Journey Towards Knowledge
(and therefore, security)
1.2 WHAT IS THIS TOPIC

11. WHERE ARE WE (OUR INDUSTRY)

12. Security is now so
essential a concern that
we can no longer use
adjectives and adverbs
but must instead use
numbers.
Dan Geer, Security Badass

13. Unfortunately…

14. Science is based on
inductive observations
to derive meaning and
understanding and
measurement on quality
(ratio) scales, so what
about InfoSec?
Where do we sit in the
family of sciences?

15. We’re the Crazy Uncle
with tinfoil hat antennae
used to talk to the space
aliens of Regulus V, has
47 cats, and who too
frequently (but
benignly) forgets to
wear pants.

16. Take, for example, CVSS

17. “the Base Equation multiplies
Impact by 0.6 and
Exploitability by 0.4”

18. = ShinyJet Engine X Peanut Butter

19. “the Base Equation multiplies
Impact by 0.6 and
Exploitability by 0.4”

20. 20
adding one
willy-nilly doesn’t
suddenly
transform
ordinal rankings
into ratio values.
decimals aren’t magic.

21. At our present skill in
measurement of security, we
generally have an ordinal scale
at best, not an interval scale
and certainly not a ratio scale.
In plain terms, this means we
can say whether X is better
than Y but how much better and
compared to what is not so
easy.
– Again, Baddss Dan Geer

22. State of the Industry
- proto-science
- somewhat random fact
gathering (mainly of readily
accessible data)
- a“morass”of interesting,
trivial, irrelevant
observations
- a variety of theories (that are
spawned from what he calls
philosophical speculation) that
provide little guidance to
data gathering
Thomas Kuhn
Philosophy of Science Badass

23. 1.3 HOW DID WE GET HERE

24. 1.3 HOW DID WE GET HERE
The tragedy of two mistakes

25. FIRST MISTAKE: LIMITING OURSELVES
(security is an engineering issue?)

26. • OSI Model
(original version)

27. • OSI Model
(SOA Remix)

28. • OSI Model
(Mika’s
12” Extended
Dance Version)
10: Religion Operator Layer

29. SECOND MISTAKE: BLIND LEADING THE BLIND

30. BLIND MAN 1: THE FUD FACTORY

31. FUD FACTORY EXAMPLE - MOBILE VS WEB

32. Google Trend: Web Security Mobile Malware

33. #RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of
Sale
Skimming
Devices
Theft/
Loss
Error
Employee
Misuse
Web
Applications
DBIR Top Patterns:

34. Web Only:
Web
Applications

35. In FinServ vs. All Industries

36. DBIR Global Representation of Assets in Cases:

37. DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using
mobile malware showed that in less than a
year’s time, five variations of mobile
malware for one specific bank could be
detected. Modest estimates suggest that
criminals gained around €50,000 per
week using this specific form of mobile
malware, harvesting over 4,000 user
credentials from 8,500 infected bank
customers in just a few months. Mobile
malware does not move the needle in our
stats as we focus on organizational
security incidents as opposed to consumer
device compromises.

38. DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using
mobile malware showed that in less than a
year’s time, five variations of mobile
malware for one specific bank could be
detected. Modest estimates suggest that
criminals gained around €50,000 per week
using this specific form of mobile malware,
harvesting over 4,000 user credentials from
8,500 infected bank customers in just a
few months. Mobile malware does not
move the needle in our stats as we
focus on organizational security
incidents as opposed to consumer
device compromises.

39. BLIND MAN 2: THE ACCOUNTING-CONSULTANCY
INDUSTRIAL COMPLEX

42. Complex (adaptive)
Systems
a system
composed of
interconnected
parts that as a
whole exhibit one
or more
properties not
obvious from the
properties of the
individual parts

43. These “risk”
statements you’re
making...
I don’t think
you’re doing it
right.
- (Chillin’
Friederich Hayek)

44. BLIND MAN 3: OUR BROKEN MODELS

45. “the Base Equation multiplies
Impact by 0.6 and
Exploitability by 0.4”

46. ROYTMAN: ON VULNERABILITIES

47. ROYTMAN: ON VULNERABILITIES

48. A CSO MUST BECOME “MEASURED” TO
ESCAPE THE MISTAKES OF THE PAST AND PUSH
INTO THE FUTURE

49. SECTION 2: ON THE ROLE OF THE CSO
What is a CSO? What do they do? What is success? How do they get there?

50. • What Is a CISO (throne of blood image
WHAT IS A CSO

51. • What Is a CISO (throne of blood image
WHAT IS A MEASURED CSO

54. W.E. DEMING
Father of Total Quality
Management and
inspiration that drove
the Japanese “post-
war economic
miracle.”

55. IT WAS NO MIRACLE.
What Deming taught the
Japanese was
“management by fact.”

56. • Improvements to the
system are never
ending.
• The only people who
really know where the
real potentials for
improvement are the
workers.
• The system is always
changing.
• There are countless
ways for the system to
go wrong.
• Statistics (metrics) are used
to focus the conversation on
fact and improvement
• Goals for quality are cross-
silo
• Theories for improvements
are implemented and tested.
• The management uses the
workers as essential
"instruments" in
understanding what is.

57. A MEASURED CSO:
• Relies on metrics, data, intel for good decisions,
• Invests in improvements to People, Process and Technology,
• Puts innovation for improvements to the system
(improvements = security, cost) in the hands of the operator,
• Ensures that there is a feedback loop for effectiveness
initiatives, and
• Works tirelessly within the bureaucracy to improve all
aspects of the system.

58. THE MEASURED CSO’S MISSION:
• To provide the best and least-cost security for
shareholders, and continuity of employment for
his workers.
• We, as an industry, know that “best” and”least-cost” are
not necessarily contradictory
• We also have a HUGE continuity issue

59. THE MEASURED CSO USES METRICS TO
IMPROVE THE SYSTEM.

60. WHAT IS THAT SYSTEM -
That which Defends
(Detects, Responds, & Prevents).

61. THE MEASURED CSO USES METRICS TO:
• Develop and improve the People, Process, and
Technology to Defend
• Plan / Build / Manage those defenses

62. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.

63. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.
Sorry, ISACA

64. THE SYSTEMS FOR DEVELOPING METRICS ARE
MORE IMPORTANT THAN THE SYSTEMS OF
DOGMA THAT DEFINE “STANDARDS” OF
OPERATION.
• There are two systems which the CSO must
manage across (at least 4 audiences)
• Those that support “defend”
• Those that support Plan/Build/Manage

65. MEASURED CSO SYSTEM 1: THE METRICS AND
MODELS THAT “DEFEND”

66. EPIDEMIOLOGY

67. EPIDEMIOLOGY
Risk Factors (Determinants)
Variables associated with increased
frequency of event.
Risk Markers
Variable that is quantitatively associated
with a disease or other outcome, but
direct alteration of the risk marker does
not necessarily alter the risk of the
outcome.
Correlation vs. Causation
Risk factors or determinants are
correlational and not necessarily causal,
because correlation does not prove
causation.

68. EPIDEMIOLOGY
Risk Factors (Determinants)
Variables associated with increased
frequency of event.
Risk Markers
Variable that is quantitatively associated
with a disease or other outcome, but
direct alteration of the risk marker does
not necessarily alter the risk of the
outcome.
Correlation vs. Causation -
Risk factors or determinants are
correlational and not necessarily causal,
because correlation does not prove
causation.
THE MEANS TO FIND PATTERNS

69. Example of a medical approach:
Dr. Peter Tippett & Verizon DBIR

70. A security incident (or threat scenario) is
modeled as a series of events. Every
event
is comprised of the following 4 A’s:
Agent: Whose actions
affected the asset
Action: What actions affected
the asset
Asset: Which assets were
affected
Attribute: How the asset was
affected
VERIS (Vocabulary for
Event Recording &
Incident Sharing)
70

72. 72

73. Object-Oriented Modeling
VERIS (Vocabulary for
Event Recording &
Incident Sharing)
73
1 2 3 4 5>" >" >" >"Incident as a
chain of events>"

74. Object-Oriented Modeling
VERIS (Vocabulary for
Event Recording &
Incident Sharing)
74
1 2 3 4 5>" >" >" >"Incident as a
chain of events>"
A “Pattern”

75. VERIS: Classification of Events by Risk Factor

76. Complex System?
VERIS FOUND PATTERNS!

77. #RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of
Sale
Skimming
Devices
Theft/
Loss
Error
Employee
Misuse
Web
Applications
DBIR Top Patterns:

78. THE KEY TO THE MEASURED CSO SYSTEM 1:
FRAMEWORK, DATA, MODELS

79. √∫∑
Framework
Models Data
=
∩
VERIS+

80. actor
information
asset
information
impact
information
controls
information
risk
Classifying sets of security information

81. √∫∑
Framework
Models Data
=
∩
Data
Warehousing+

82. 82
Apache Storm

83. 83
Data MapReduce Process Analytics & Reporting
Threat Intel Feeds
Control Data
Control Logs
System Logs
Event
History
&
Loss
Loss
Distribu8on
Dev.
B.I.A.
Control Data
Control Logs
System Logs
Configuration Data
Vulnerability Data
HR Information
Process Behaviors
XML
CSV
EDI
LOG
SQL
JSON
Text
Binary
Objects
createmap
reduce
Traditional
RDBMS
Systems
Workflow
Analytics
Reporting

87. ModelssuggestingIOC=true

88. 88
1 2 3 4 5>" >" >" >"Incident as a
chain of events>"

89. 89
1 2 3 4 5>" >" >" >"Incident as a
chain of events>"
X X X

90. 90
Example of data
enrichment:
Asset Intel :
Vendor-owned
SaaS application

91. √∫∑
Framework
Models Data
=
∩

92. MEASURED CSO SYSTEM 1: THE METRICS AND
MODELS THAT “DEFEND” AGAINST THREAT
PATTERNS.
(real and anticipated or forecasted)

93. MEASURED CSO SYSTEM 2: THE METRICS
NEEDED TO PLAN/BUILD/MANAGE
SYSTEMS (OPERATIONS)

94. THE MEASURED CSO MUST ALSO INCLUDE A
KEEN UNDERSTANDING AND PARTNERSHIP
WITH IT OPERATIONS

95. THE MICROMORT
A one in a million
chance of death
Ronald A. Howard

96. Activities that increase the death risk by roughly one micromort, and their
associated cause of death (wikipedia):
Traveling 6 miles by motorbike (accident)
Traveling 17 miles by walking (accident)
Traveling 10 miles by bicycle (accident)
Traveling 230 miles (370 km) by car (accident)
Traveling 1000 miles (1600 km) by jet (accident)
Traveling 6000 miles (9656 km) by train (accident)
Traveling 12,000 miles (19,000 km) by jet in the United States (terrorism)
Increase in death risk for other activities on a per event basis:
Hang gliding – 8 micromorts per trip
Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)

97. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)

98. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
Ecstacy (MDMA) – 0.5 micromorts per
tablet (most cases involve other drugs)

99. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)

100. Modern Risk Management is not only bad at describing risk, but it also is
focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)

101. The Measured CSO must know where IT is
overweight, smoking ecstasy, while riding a
rocket-powered bicycle on the railing of a
bridge.

102. DATA: VISIBLE OPS FOR
SECURITY

104. 104
Example of data
enrichment:
Asset Intel :
Vendor-owned
SaaS application

105. SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?

106. 106
MOST METRICS PROGRAMS

107. If we consider a single metric
as a building block

108. 108
It should be
used by the
CSO to paint
a picture of
the security
program

109. 109
Whose context is the whole of IT.

110. 110
But because we
gather what is most
readily available -
most metrics
programs look like
my living room.
How does the
measured CSO get
context?

111. GOAL, QUESTION, METRIC
Conceptual level (goal)
goals defined for an object for a variety of
reasons, with respect to various models, from
various points of view.
Operational level (question)
questions are used to define models of
the object of study and then focuses on
that object to characterize the assessment
or achievement of a specific goal.
Quantitative level (metric)
metrics, based on the models, is
associated with every question in order to
answer it in a measurable way.
Victor Basili

112. GQM FOR FUN & PROFIT
Goals establish
what we want to
accomplish.
Questions help us
understand how to
meet the goal. They
address context.
Metrics identify the
measurements that
are needed to answer
the questions.
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7

113. Execution
Models
Data
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
GQM FOR FUN & PROFIT

114. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient

115. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
% Coverage by Business Units
%Coverage by Asset category
%Coverage by Risk
Unix
Windows Server
Desktop
OS
Components
Likelihood
Impact
Most Significant Failures
Repeat Offenders
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit

116. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
What should our Priorities be for timeliness?
What is Policy for timeliness?
What other Considerations for Timeliness?
What is time to patch like for assets with worst Likelihoods?
What is time to patch like for assets with worst Impacts?
What % are Late by
What are our Repeat Offenders?
likelihood
Impact
by asset category
by business unit
by risk
UNIX
Windows Server
Desktop
likelihood
impact

117. GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
Cost
Risk Reduction
Hour per Asset spent Patching
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Cost Per Hour
Hour per Asset, by ALE per Hour
Hour per asset category

118. GQM EXAMPLE: PATCH MANAGEMENT
• The Measured CSO creates a scorecard of
KRI’s & KPI’s that Includes:
• Historical values
• “Triggers”
• “Thresholds”
(each of these?) aren’t perfect, but establish a
hypothesis for testing & optimization.

119. Now you’re ready
to come correct,
my Bias!
- (Chillin’
Friederich Hayek)

120. MEASURED CSO FRAMEWORK FOR GQM: NIST CSF
NIST CSF
Identify
Protect
Detect
Respond
Recover
Asset Management
Business Environment
risk assessment
risk management strategy
Governance
Access Control
Awareness and Traininig
Data Security
Information Protection Processes and
Procedures
Maintenance
Protective Technology
Anomalies and Events
Security Continuous Monitoring
Detection Processes
Response Planning
Response Communications
Response Analysis
Response Mitigation
Response Improvements
Recovery Planning
Improvements
Communications

121. SECTION 3: BECOMING MEASURED
What does that mean? What do we need? How do we do it? Where shall we go?

122. √∫∑
Framework
Models Data
=
∩

124. 124
Example of data
enrichment:
Asset Intel :
Vendor-owned
SaaS application

125. ETL AND STORE ALL THE
THINGS!!!

126. 126
Data MapReduce Process Analytics & Reporting
Threat Intel Feeds
Control Data
Control Logs
System Logs
Event
History
&
Loss
Loss
Distribu8on
Dev.
B.I.A.
Control Data
Control Logs
System Logs
Configuration Data
Vulnerability Data
HR Information
Process Behaviors
XML
CSV
EDI
LOG
SQL
JSON
Text
Binary
Objects
createmap
reduce
Traditional
RDBMS
Systems
Workflow
Analytics
Reporting

127. ModelssuggestingIOC=true

129.
“If you do not know
how to ask the right
question, you
discover nothing.”

130. RESOURCES
FOR
GQM
AND
MICROMORTS
-­‐
WIKIPEDIA
FOR
DBIR
DATA,
THE
VERIZON
DBIR
FOR
DEMING
QUOTES,
THE
WORKS
OF
MYRON
TRIBUS:
http://www.qla.com.au/papersTribus/Oslo3.pdf
http://www.unreasonable-­‐learners.com/wp-­‐content/uploads/2011/03/
Germ-­‐Theory-­‐of-­‐Management-­‐Myron-­‐Tribus1.pdf
http://www.qla.com.au/papersTribus/DEMINGS_.PDF