Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Defeating Cognitive Bias 
and 
Developing Analytic Technique 
Chris Sanders 
BSides Augusta 2014
Chris Sanders 
• Christian & Husband 
• Kentuckian and South 
Carolinian 
• MS, GSE, CISSP, et al. 
• Non-Profit Director ...
Chris Sanders
Chris Sanders 
“[Practical Packet Analysis] gives you everything you need, step by step, to become 
proficient in packet a...
Outline 
Objectives: 
 What is Analysis? 
 What is Bias? 
 Recognizing Bias 
 Defeating Bias 
 Analysis Methods 
“How...
**Disclaimer** 
I’m going to talk about matters of the brain, not 
sure the normal tech stuff. 
My research for this prese...
Bias – A very personal story
2 AM
The Pain Begins 
*Dramatization
Ultrasounds == Magic?
At this point… 
So, I went to see a surgeon…
“Let’s Cut it Out!” - Surgeon
Missing Parts
Thus… 
“Would it be accurate to say that I’m a medical 
miracle?” - Me 
“Absolutely.” – Surgeon
Cause and Effect 
• Cause: Bias…lots of it! 
– Confirmation Bias 
– Outcome Bias 
– Congruence Bias 
• Effect: Unnecessary...
Analysis
Analysis is Everywhere 
• Making judgments based upon data 
• Security Analysis Happens for: 
– Malware Analysts 
– Intell...
Network Security Monitoring 
• The collection, detection, and analysis of 
network security data. 
• The goal of NSM is es...
Evolution of NSM Emphasis
The Need for Analytic Technique 
• Kansas State University Anthropological Study 
on SOCs - Key Finding: 
– “SOC analysts ...
Analysis: Thinking About Thinking 
• We need to critically examine how we think 
about information security analysis. 
• W...
Perception vs. Reality 
• Perception: 
– “A way of regarding, understanding, or 
interpreting something.” 
• Reality: 
– “...
RED
GREEN
BLUE
BLACK
YELLOW
Test Results 
• Variation of Stroop Test (John Stroop, 1935) 
• Measures Cognition 
– The Process of Perception 
• Identif...
What is Bias? 
“Prejudice in favor of or against one thing, 
person, or group compared with another, 
usually in a way con...
I’m Going to Show You an Image
I’m Going to Show You a Picture of 
a White Vase.
First Image Results 
• Prompted for Face 
– 88% See Face 
– 12% See Sax Player 
• No Prompt 
– 57% See Face 
– 43% See Sax...
Second Image Results 
• Prompted for Vase 
– 94% See White Vase 
– 6% See Two People 
• No Prompt 
– 62% See White Vase 
–...
Bias Examples
Let’s Hit Closer to Home…
A Recent Example
Anchoring 
• Defined: Heavily relying on a single piece of 
information. 
• Examples: 
– Src/Dst Country -> OMG China! 
– ...
Clustering Illusion 
• Defined: 
Overestimating the 
value of perceived 
patterns in random 
data. 
• Examples: 
– The gre...
Availability Cascade 
• Defined: Strong belief in something due to its 
repetition in public discourse 
• Example: 
– “Chi...
Belief Bias 
• Defined: Occurs when a decision is based on 
the believability of the conclusion. 
• Examples: 
– “We would...
Confirmation Bias 
• Defined: Interpreting data during analysis with 
a focus on confirming one’s preconception. 
• Ego is...
Impact Bias 
• Defined: Tendency to overestimate the 
significance of something based on the 
potential impact. 
• Signatu...
Irrational Escalation 
• Defined: Justifying increased time investment 
based on existing time investment when it 
may not...
Framing Effect 
• Defined: Interpreting information differently 
based on how or from whom it was 
presented. 
• Important...
Overconfidence Effect 
• Defined: Excessive confidence in ones own 
decisions, especially in light of contrasting 
data. 
...
Pro-Innovation Bias 
• Defined: Excessive optimism and biased 
decisions based on an invention of one’s own 
making being ...
There are over 100 types of bias. 
How can we overcome them?
Overcoming Bias
What Can We Do? 
• Preconception and Bias Cannot Be Fully 
Avoided 
• Therefore: 
– Develop Repeatable Analytic Technique ...
Analytic Techniques 
Common Techniques: 
– Relational 
Investigation 
– Differential 
Diagnosis
Relational Investigation 
• “Link Analysis” 
• Commonly Used in Criminal Investigations 
• Focuses on Entities, Relationsh...
Relational Investigation
Setting the Stage – Primary Relationships
Partial Story – Secondary Relationships
Full Attack Diagram – Tertiary Relationships
Differential Diagnosis 
• Commonly Used in 
Medical Diagnosis 
• Relies on Lists of 
Possibilities, and 
Systematically 
E...
Differential Diagnosis
Incident M&M 
• Dr. Ernest Codman at Mass. General Hospital 
• Post-Patient Meetings to Discuss What 
Occurred and How to ...
Alternative Analysis 
• Developed by Richards Heuer Jr. (FBI) 
• Series of Peer Analysis Methods 
• Designed to Help Overc...
Group A / Group B 
• Group A – Presenting Analyst/Team 
• Group B – Secondary Analyst/Team 
• Two Independent Analysis Eff...
Red Cell Analysis 
• Peer Focus on Attacker’s Viewpoint 
• Questioning in Relation to Attackers Perceived 
Goals 
• Requir...
What If Analysis 
• Focus on Cause/Effect of Actions That May 
Not Have Actually Occurred 
– What is the attacker had done...
Key Assumptions Check 
• Presenter Identifies Assumptions During 
Analysis 
• Peers Challenge Assumptions 
• Pairs Well wi...
Incident M&M Best Practices 
• Limit Frequency 
• Set Expectations 
• Require a Strong Mediator 
• Keep it at the Team Lev...
Conclusion 
• The Era of Analysis is Upon Us 
• Bias is Inevitable – Learn to Recognize It 
• Overcome Analysis Hurdles Wi...
Thank You! 
E-Mail: chris@chrissanders.org 
Twitter: @chrissanders88 
Blog: http://www.chrissanders.org 
Book Blog: http:/...
Developing Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in Security
Upcoming SlideShare
Loading in …5
×

Developing Analytic Technique and Defeating Cognitive Bias in Security

20,089 views

Published on

In this presentation, I discuss the evolution to the analysis era in information security and the challenges associated with it. This includes several examples of cognitive biases and the negative effects they can have on the analysis process. I also discuss different analytic techniques that can enhance analysis such as differential diagnosis and relational investigation.

Published in: Technology

Developing Analytic Technique and Defeating Cognitive Bias in Security

  1. 1. Defeating Cognitive Bias and Developing Analytic Technique Chris Sanders BSides Augusta 2014
  2. 2. Chris Sanders • Christian & Husband • Kentuckian and South Carolinian • MS, GSE, CISSP, et al. • Non-Profit Director • BBQ Pit Master
  3. 3. Chris Sanders
  4. 4. Chris Sanders “[Practical Packet Analysis] gives you everything you need, step by step, to become proficient in packet analysis. I could not find a better book.” – Amazon Reviewer
  5. 5. Outline Objectives:  What is Analysis?  What is Bias?  Recognizing Bias  Defeating Bias  Analysis Methods “How to make better technical decisions in any kind of security analysis.“
  6. 6. **Disclaimer** I’m going to talk about matters of the brain, not sure the normal tech stuff. My research for this presentation involved consultation with psychologists. I, however, am not one.
  7. 7. Bias – A very personal story
  8. 8. 2 AM
  9. 9. The Pain Begins *Dramatization
  10. 10. Ultrasounds == Magic?
  11. 11. At this point… So, I went to see a surgeon…
  12. 12. “Let’s Cut it Out!” - Surgeon
  13. 13. Missing Parts
  14. 14. Thus… “Would it be accurate to say that I’m a medical miracle?” - Me “Absolutely.” – Surgeon
  15. 15. Cause and Effect • Cause: Bias…lots of it! – Confirmation Bias – Outcome Bias – Congruence Bias • Effect: Unnecessary Surgery – 1 Week Recovery – Financial Loss – Pessimism Bias
  16. 16. Analysis
  17. 17. Analysis is Everywhere • Making judgments based upon data • Security Analysis Happens for: – Malware Analysts – Intelligence Analysts – Incident Response Analysts – Forensic Analysts – Programming Logic Analysts • My main focus is network intrusion analysis, so this talk will be framed through that.
  18. 18. Network Security Monitoring • The collection, detection, and analysis of network security data. • The goal of NSM is escalation, or to declare that an incident has occurred to that incident response can occur.
  19. 19. Evolution of NSM Emphasis
  20. 20. The Need for Analytic Technique • Kansas State University Anthropological Study on SOCs - Key Finding: – “SOC analysts often perform sophisticated investigations where the process required to connect the dots is unclear even to analysts.” • Analysis == “Tacit Knowledge”
  21. 21. Analysis: Thinking About Thinking • We need to critically examine how we think about information security analysis. • We aren’t alone! – Scientific – Medical – Legal
  22. 22. Perception vs. Reality • Perception: – “A way of regarding, understanding, or interpreting something.” • Reality: – “The state of things as they actually exist.” Let’s take a test…
  23. 23. RED
  24. 24. GREEN
  25. 25. BLUE
  26. 26. BLACK
  27. 27. YELLOW
  28. 28. Test Results • Variation of Stroop Test (John Stroop, 1935) • Measures Cognition – The Process of Perception • Identifies Gap Between Perception & Reality • Used to Measure – Selective Attention – Cognitive Flexibility – Processing Speed
  29. 29. What is Bias? “Prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.” •Perception != Reality •Perception is Everything, but Fallible •We tend to perceive what we expect/are conditioned to perceive
  30. 30. I’m Going to Show You an Image
  31. 31. I’m Going to Show You a Picture of a White Vase.
  32. 32. First Image Results • Prompted for Face – 88% See Face – 12% See Sax Player • No Prompt – 57% See Face – 43% See Sax Player
  33. 33. Second Image Results • Prompted for Vase – 94% See White Vase – 6% See Two People • No Prompt – 62% See White Vase – 38% See Two People
  34. 34. Bias Examples
  35. 35. Let’s Hit Closer to Home…
  36. 36. A Recent Example
  37. 37. Anchoring • Defined: Heavily relying on a single piece of information. • Examples: – Src/Dst Country -> OMG China! – IDS Alert Name -> It say this is X, so it must be X. – Timing -> It’s every 5 minutes!
  38. 38. Clustering Illusion • Defined: Overestimating the value of perceived patterns in random data. • Examples: – The great “beaconing” fallacy – Unguided Visualizations
  39. 39. Availability Cascade • Defined: Strong belief in something due to its repetition in public discourse • Example: – “Chinese Traffic is Bad.” – “That rule generates a lot of false positives.”
  40. 40. Belief Bias • Defined: Occurs when a decision is based on the believability of the conclusion. • Examples: – “We wouldn’t be a target for a nation-state actor.” – “This is probably a false positive because it’s unlikely someone would attack our VoIP system.”
  41. 41. Confirmation Bias • Defined: Interpreting data during analysis with a focus on confirming one’s preconception. • Ego is a big factor here • Examples: – “I think this is nothing.” – “I think there is something going on here.”
  42. 42. Impact Bias • Defined: Tendency to overestimate the significance of something based on the potential impact. • Signature/Alert Naming + Lack of Experience Contribute to this. • Example: – “The alert says this is a known APT1 back door, so I need to spend all day looking at this.”
  43. 43. Irrational Escalation • Defined: Justifying increased time investment based on existing time investment when it may not make sense. • Sunk Cost Fallacy • Example: – “What do you mean this is nothing? I’ve spent all day looking at this. I’ll spend all day tomorrow digging into it; I’m sure I’ll find something else there.”
  44. 44. Framing Effect • Defined: Interpreting information differently based on how or from whom it was presented. • Important in interaction with other analysts • Example: – Old Vet: “Steve doesn’t know what he is doing, so if he is telling me this it probably doesn’t mean much.” – New Guy: “None of the more experienced guys said anything about this, so it must not matter.”
  45. 45. Overconfidence Effect • Defined: Excessive confidence in ones own decisions, especially in light of contrasting data. • Example: • 99% Paradox – “I’m 99% sure this is right.” • One psych study suggest this statement is wrong ~40% of the time.
  46. 46. Pro-Innovation Bias • Defined: Excessive optimism and biased decisions based on an invention of one’s own making being involved in the analysis. • Invention == System / Code / Concept • Example: – “My tool can do that.” – “I wrote that signature so I know it’s accurate.” – “This fits perfectly in my model!”
  47. 47. There are over 100 types of bias. How can we overcome them?
  48. 48. Overcoming Bias
  49. 49. What Can We Do? • Preconception and Bias Cannot Be Fully Avoided • Therefore: – Develop Repeatable Analytic Technique – Recognize Key Assumptions – Allow them to be Challenged
  50. 50. Analytic Techniques Common Techniques: – Relational Investigation – Differential Diagnosis
  51. 51. Relational Investigation • “Link Analysis” • Commonly Used in Criminal Investigations • Focuses on Entities, Relationships, Interactions, and Degrees of Separation
  52. 52. Relational Investigation
  53. 53. Setting the Stage – Primary Relationships
  54. 54. Partial Story – Secondary Relationships
  55. 55. Full Attack Diagram – Tertiary Relationships
  56. 56. Differential Diagnosis • Commonly Used in Medical Diagnosis • Relies on Lists of Possibilities, and Systematically Eliminating Possibilities
  57. 57. Differential Diagnosis
  58. 58. Incident M&M • Dr. Ernest Codman at Mass. General Hospital • Post-Patient Meetings to Discuss What Occurred and How to Better It • Incident M&M 1. Handler/Analyst Presents Case 2. Followed by Alternative Analysis
  59. 59. Alternative Analysis • Developed by Richards Heuer Jr. (FBI) • Series of Peer Analysis Methods • Designed to Help Overcome Bias and Improve Quality of Analysis
  60. 60. Group A / Group B • Group A – Presenting Analyst/Team • Group B – Secondary Analyst/Team • Two Independent Analysis Efforts • Note are Compared During the Presentation • Identify Differing Conclusions from Same Data
  61. 61. Red Cell Analysis • Peer Focus on Attacker’s Viewpoint • Questioning in Relation to Attackers Perceived Goals • Requires Some Offensive Experience • Best Executed by Red Team if Available
  62. 62. What If Analysis • Focus on Cause/Effect of Actions That May Not Have Actually Occurred – What is the attacker had done X? How would you have changed your approach? – What if you didn’t stumble across X in Y data? • Enhances Later Investigations
  63. 63. Key Assumptions Check • Presenter Identifies Assumptions During Analysis • Peers Challenge Assumptions • Pairs Well with “What If” Analysis – “What if it were possible for that malware to escape that virtual machine?” – “Would you come to the same conclusion if you knew this was APT3 instead of APT1?”
  64. 64. Incident M&M Best Practices • Limit Frequency • Set Expectations • Require a Strong Mediator • Keep it at the Team Level – No Sr. Managers • Encourage Servant Leadership • Discourage Personal Attacks • Write it Down!
  65. 65. Conclusion • The Era of Analysis is Upon Us • Bias is Inevitable – Learn to Recognize It • Overcome Analysis Hurdles With: – Analytic Technique – Alternative Analysis
  66. 66. Thank You! E-Mail: chris@chrissanders.org Twitter: @chrissanders88 Blog: http://www.chrissanders.org Book Blog: http://www.appliednsm.com Testimony: http://www.chrissanders.org/mytestimony

×