This document summarizes a presentation about overcoming confirmation bias in security. The presentation discusses how security professionals often rely on metrics that don't provide useful information for managing risk. It advocates developing quantitative risk scores linked to business goals to make better risk management decisions. The presentation also warns against signs of confirmation bias, like only looking at past security events rather than probability of vulnerabilities, and provides tips for creating effective security metrics.

1. Copyright © 2010-2011 IANS. The contents of this presentation are confidential . All rights reserved.
Confirmation Bias
How to Stop Doing the Things in
Security That Don't Work
November 2011

2. 2Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Who am I?
» Michael A. Davis
– CEO of Savid Technologies
• IT Security, Risk Assessment, Penetration Testing
– Speaker
• Blackhat, Defcon, CanSecWest, Toorcon, Hack In The Box
– Open Source Software Developer
• Snort
• Nmap
• Dsniff
» Savid Technologies
– Risk Assessments, IT Security Consulting, Audit and
Compliance

3. 3Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Author

4. 4Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
The Issue
“Single biggest security related
problem is a lack of Senior
Level commitment to enterprise
wide security policies.“
Source: 2011 InformationWeek Strategic Security Survey, June 2011

5. 5Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Execs Are Paying Attention
0%
5%
10%
15%
20%
25%
30%
35%
40%
Exec Involvement Budget Constraints
2010
2011
Source: Information Week Data Survey, 2011

6. 6Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
We Protect, They Are Criticized
According to Bloomberg News, Sony has been subpoenaed by New
York attorney general Eric Schneiderman, who is "seeking information
on what Sony told customers about the security of their networks, as
part of a consumer protection inquiry." (Source: informationweek.com)
Rep. Mary Bono Mack (R-Calif.), the subcommittee chair, said that
Sony should have informed its consumers of the breach earlier and
said its efforts were “half-hearted, half-baked.” She was particularly
critical of Sony’s decision to first notify customers of the attack via its
company blog, leaving it up to customers to search for information on
the breach. (Source: washingtonpost.com)

7. 7Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
We All Do Them
Source: 2011 InformationWeek Analytics Strategic Security Survey
0%
10%
20%
30%
40%
50%
60%
70%
80%
Yes No Don't Know
% that perform Risk Assessments
2011
2010

8. 8Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
The Reality
Source: 2011 InformationWeek Analytics Strategic Security Survey
Very
30%
Somewhat
67%
Not At All
3%
Risk Assessment
Effectiveness

9. 9Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Complex IT Projects Fail - A lot
Out Of 200 Multi-nationals:
67% Failed To Terminate Unsuccessful
Projects
61% Reported Major Conflicts
34% Of Projects Were Not Aligned With
Strategy
32% Performed Redundant Work
1 In 6 Projects Had A Cost Overrun Of 200%!
Source: 2011 Harvard Business Review – Berlin Univ Technical survey

10. 10Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
T-Mobile CISO On Metrics
“Security experts can't measure their success
without security metrics, and what can't be
measured can't be effectively managed.”
~ Bill Boni, VP of IS, T-Mobile USA

11. 11Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Why Do We Care?
Management Asks:
–“Are We Secure?”
Without Metrics:
–“Depends How You Look At It”
With Metrics:
–“Look At Our Risk Score Before This
Project, It Dropped 15%. We Are More
Secure Today Than Yesterday”

12. 12Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Metrics, We need metrics!

13. 13Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Where/What to measure
Strategy/Governance
Code Reviews, Project Risk
Assessments,
Exceptions/Waivers
Tactical/Sec Ops
Vuln Management, Patch Management,
Incidents, etc.
IS Budget
Spending/employee
Policy gaps in existence
Industry Standards Adopted
Awareness Plan
% projects going through assessment process
# of policy exceptions
# of risk acceptances
% project doing code reviews
Error rates
Freq of vuln assessment
# outstanding vulns
Rate of fixing
Trend of incident response losses

14. 14Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Who are you?
TCO
Patch
Latency
SPAM/AV Stats

15. 15Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Examples of metrics
Baseline Defenses Coverage (AV, FW, etc)
– Measurement of how well you are protecting your enterprise
against the most basic information security threats.
– 94% to 98%; less than 90% cause for concern
Patch Latency
– Time between a patch’s release and your successful
deployment of that patch.
– Express as averages and criticality
Platform Security Scores
– Measures your hardening guidelines
Compliance
– Measure departments against security standards
– Number of Linux servers at least 90% compliant with the Linux
platform security standard

16. 16Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Phishing Still Works

17. 17Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Stop With The Confirmation Bias
Risk Perception Is Bad
–Tornado V. Kitchen Fire
–Less Familiar Are Perceived As Greater Risk
Favor Info That Match Preconceptions
Cause And Effect Processing
Correlation Does Not Equal Causation
We Manage Risk Using Metrics That Don’t
Matter

18. 18Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.

19. 19Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
The Formula Of Successful Risk Management
PBL = λ1 x p1 + λ2 x p2 + λ3 x p3

20. 20Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Hazard vs. Speculative Risk

21. 21Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Linking to Business Goals
Copyright Carnegie Mellon SETI MOSAIC Whitepaper

22. 22Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Outcome Management
Copyright Carnegie Mellon SETI MOSAIC Whitepaper

23. 23Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
It Is About Risk MANAGEMENT
Effective Metrics Catalog Define:
Category
Metric
How To Measure
Purpose Of This Metric
Target Audience
Reporting Frequency/Period

24. 24Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
5 Signs You Have a Confirmation Bias
Using Quantitative Risk Scores To
Make Decisions
Look At Security Events Instead Of
Probability Of Vulnerabilities
Talk About Risk In Terms Of
“Industry Data”
Lack Of Risk Management
Inability To Communicate Risk

25. 25Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Security Metric Gotchas
Not Tracking Visibility
–What % is the metric representing?
–Develop baseline for acceptance
Not Trending
–Provide at least 4 previous periods and trend
line
Not Providing Forward Guidance
–Red, Green, Yellow (Worse, Better, Same)
Not Mapping To A Business goal
Focusing on Hazard Risk
Not Using Qualitative Metrics

26. 26Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
Contact Information
Michael A. Davis
mdavis@savidtech.com
708-532-2843
Twitter: @mdavisceo