This document discusses security metrics and summarizes several perspectives on developing and using metrics. It describes growing areas of interest including software security, modeling, and benchmarking. It also summarizes experiences with security metrics from organizations like GE, the Department of Veterans Affairs, and Intel. These cases highlight the importance of clear definitions, focus on high impact events, predictive modeling, and gathering substantial baseline data. The document also reviews standards, common metric types, and challenges around managing to metrics and measuring the right things.

1. 14/07/2008
Security Metrics
Phil Huggins

2. 14/07/2008 Security MetricsPage 2
Core Text
Security Metrics : Replacing Fear, Uncertainty and Doubt
Andy Jaquith, 2007
0-321-34998-9

3. 14/07/2008 Security MetricsPage 3
Recommended Texts

4. 14/07/2008 Security MetricsPage 4
Growing field
► Areas of interest
► Software security
► Modelling
► Benchmarking
► Return on investment
► Breach data
► Standards
► ISO / IEC 27004
► NIST SP800-55
► Communities
► Securitymetrics.org
► Metricsexchange.org
► Cybersecurity KTN

5. 14/07/2008 Security MetricsPage 5
Securitymetrics.org
► Open mailing list and wiki
► Active community
► Established by Andy Jaquith
► Runs the US based Metricon and MiniMetricon events
each year

6. 14/07/2008 Security MetricsPage 6
Metricsexchange.org
► New open community group
► Established by Elizabeth Nichols
► Sharing metrics definitions, learning and data
► Early days, big ideas

7. 14/07/2008 Security MetricsPage 7
Cybersecurity KTN – Metrics SIG
► UK Knowledge Trading Networks established by DTI
► Promoting collaboration between industry, academia and
government
► Metrics Special Interest Group has focused on the
delivery of the Internet Threat Exposure (ITE) Index
► Threat and Countermeasure focused metric of exposure
► Appears to be aimed at less sophisticated security
practitioners
► Risk assessment-lite ?
► Currently being developed in an open group

8. 14/07/2008 Security MetricsPage 8
Standards
► NIST SP800 – 55
► Exhaustive list of possible security metrics to measure
► 99 pages
► No real sense of what is a useful metric
► Defines useful characteristics to describe a metric
► Performance Goal, Performance Objective, Metric, Purpose,
Implementation Evidence, Frequency, Formula, Data Source,
Indicator
► ISO/IEC 27004
► Currently in draft / closed group
► Metrics covering the performance of an ISMS as defined in 27001
and 27002

9. 14/07/2008 Security MetricsPage 9
Types of Security Metrics
► Risk Metrics
► Compliance Metrics
► Operational Metrics
► Quality Metrics
► Management Metrics
► Business Metrics
► Confusion among practitioners

10. 14/07/2008 Security MetricsPage 10
Focus problems
► Technical Focus
► “What do we count?”
► Business Focus
► “What do we need to do and why?”
► Counting is the mechanical foundation
► The business wants the story the numbers tell
► Metrics are not the answer to funding problems

11. 14/07/2008 Security MetricsPage 11
Other common problems
► Managing to the metric
► No longer focused on the result
► Measuring emerging threats
► Measuring last years breaches

12. 14/07/2008 Security MetricsPage 12
Questions from the board
► Am I safe?
► Can I take responsibility for the actions of my company?
► Who handles my data?
► Who am I doing business with?
► Are they accountable?

13. 14/07/2008 Security MetricsPage 13
Metricon practitioners top 10 metrics
► Data volumes transmitted to competition
► Coverage metrics
► Availability of business systems
► End user perception of security
► Legal fees paid out
► Total cost of information security
► Information asset value
► Count of events on systems
► Security control success rate
► Cost of security monitoring and reporting

14. 14/07/2008 Security MetricsPage 14
Balanced Security Scorecards
► Complete:
► People, Process, Technology, Budgeting, Innovation,
Organisational Planning, Operations
► Traditionally include four primary perspectives:
► Financial
► Customer
► Internal Processes
► Learning and Growth
► Jaquith has a comprehensive chapter on balanced
security scorecards in his book

15. 14/07/2008 Security MetricsPage 15
Geer’s Scorecard
► Finance
► Cost of data security per transaction
► Downtimes lost to attack by attack class
► Data flow per transaction and source
► Budget correlation with risk measures
► Process
► % of critical systems under a DR plan
► % of critical systems obeying the security policy
► MTBF & MTTR for security incidents
► Frequency of security team internal consultations
► Latency to obey security change orders by department

16. 14/07/2008 Security MetricsPage 16
Geer’s Scorecard
► Learning and growth
► % of job reviews involving security
► % of security workers with training
► Ratio of B.U. security staff to central security staff
► Timely new system security consultations
► % of programs with budgeted security
► Customer
► % of SLAs with security standards
► % of tested external facing applications
► Number of non-employees with access
► % of data secure by default
► % of customer data outside the data centre

17. 14/07/2008 Security MetricsPage 17
GE Global experience
► Metrics to drive behaviour
► Scorecard approach
► Business unit drill down and comparison views
► Communication plan was key
► Built a custom system piecemeal over several years
► Started with manual data, automated over time
► Now moving to a common platform
► Monolithic vs Composite data sources
► Centralised vs Business unit data sources

18. 14/07/2008 Security MetricsPage 18
Dept of Veterans Affairs’ experience
► Didn‟t have common definitions of:
► What IT Security was
► What better IT security looked like
► The value of security
► Identified the security events that drove perception of
security
► Focused on the frequency and impact of those events
► Did not ignore uncertainty!
► Results-focused

19. 14/07/2008 Security MetricsPage 19
Intel’s experience
► Developed predictive model for future security incidents
► Used to provide ROI on „reduce the occurrence‟ controls
NOT „reduce the effect‟ controls
► Needed to gather current state data first in order to
identify „Annual Rate of Occurrence‟
► 2 years of data from 20+ global locations
► Needed to estimate „Single Loss Expectancy‟ value for
target environment
► Identified limited target groups to pilot controls in first to
measure results
► Needed a LOT of data
► 87% accurate predictions over a 12 month period

20. 14/07/2008 Security MetricsPage 20
Verizon 2008 Data Breach Investigations
Report
► 500 Investigations over 4 Years
► 18% of breaches were the result of an unpatched system
► 90% of unpatched breaches had had patches publicly
available for 6 months or more
► No more would have been prevented by a patch cycle
shorter than a month
► There is a lot of useful data in this report

21. 14/07/2008 Security MetricsPage 21
Dan Geer’s counterpoint
► We are losing
► The bad guys are in it for the money
► Attackers costs are continually falling
► Need to start measuring „attack metrics‟
► Focus on increasing their cost of attack
► More cost effective to redirect than to resist

22. 14/07/2008 Security MetricsPage 22
Marcus Ranum’s counterpoint
► Statistics only work where:
► Population is large
► Problems are common and widely shared
► Aggressors act consistently
► The only scores that matter are 0% and 100%
► Security is not „risk management‟ it is „complexity
management‟

23. 14/07/2008
Thank you
phuggins@uk.ey.com