SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
In this presentation I discuss the need for better understanding of the human investigation process. I demonstrate the tool agnostic investigation simulator I developed to observe and collet investigation data, and discuss results from some of these experiments.
In this presentation I discuss the need for better understanding of the human investigation process. I demonstrate the tool agnostic investigation simulator I developed to observe and collet investigation data, and discuss results from some of these experiments.
3.
Agenda
Era of Analysis
DFIR Cognitive Revolution
Researching the Investigation Process
Data, Data, and more Data
The economics of NSM are not in our favor –
how can we study the investigation process to
make it more efficient?
4.
Economics of Security
“If you want to understand the world of nature,
master physics. If you want to understand the
world of man, master economics.” - Taufiq
Rashid
High
Demand for
Security
Expertise
Low Supply
of Security
Practitioners
Expertise
Services
Software
5.
Evolution of NSM
“The profession
[security] is so
nascent that the how-
tos have not been
fully realized even by
the people who have
the knowledge.”
Every thought-based
profession goes
through a cognitive
crisis and revolution.
Ours is coming.
6.
Symptoms of a Cognitive Crisis
1. Demand for expertise greatly outweights
supply
2. Most information cannot be trusted or
validated
3. Inability to mobilize and tackle big systemic
issues
7.
The Cognitive Revolution in
DFIR
1. Understand the processes
used to perform
investigations and draw
conclusions
2. Develop repeatable
methods and techniques
for performing
investigations
3. Build and advocate
training that teaches
analysts how to think
about investigations, not
just how to use tools.
8.
Investigations as Mental
Labyrinths
The investigation is
the core construct of
information security.
At a high level, an
investigation is a
series of decisions
that begets other
decisions.
Defenders don’t
always know if
they’ve taken the
correct path.
9.
Navigating the Labyrinth
Alert
OSINT
Reputation
File Hash
Sandbox
Behaviors
AV Detections
(VT)
Imphash
More File
Hashes
Friendly Host
Network PCAP
Host
Windows
Logs
Security Log
System Log
App LogRegistry
File System
Hostile Host Network
PCAP
Flow
10.
Studying the Investigation
Process
Goal:
Increase Accuracy
Decrease Time
How do you study
something human
thought?
Challenges:
Creating unique
investigation scenarios
takes time
There is no universal
set of tooling
11.
A Scenario-Based Approach to
Investigation Analysis
Create a tool-agnostic investigation simulator
Make it portable and self contained
Seed it with investigation scenarios where one
variable can be addressed at a time
Allow it to log investigator actions and output a
log of decisions being made
13.
Case Study
Analyzing the Flow of
the Investigation
14.
The Compromise
1. Victim visits friendly
website
2. Redirect to EK landing
page
3. Download flash exploit
4. Exploit is successful and
ransomware file
downloads
5. Ransomware installs and
executes
6. Ransomware begins C2
communication
15.
What data did analysts look at
first?
72%
16%
12%
Observed
PCAP Flow OSINT
Data Suggests:
Analysts prefer a higher context data set…
…even if other data sets are available
…even if lower context data sets can lead to a resolution.
Analysts don’t fully understand their own techniques
49%
28%
23%
Reported
PCAP Flow OSINT
16.
Did the first move affect analysis
speed?
Data Suggests:
While PCAP provides richer context, it may slow down
the investigation if that’s where you start
Starting with a lower context data source can increase
speed when working with higher context data
16
10
9
PCAP Flow OSINT
27
13 13
PCAP Flow OSINT
Avg Time to Close Weighted Time to Close
17.
What happens when Bro data
replaces PCAP?
46%
25%
29%
Observed (Bro)
Bro Flow OSINT
72%
16%
12%
Observed (PCAP)
PCAP Flow OSINT
18.
What happens when Bro data
replaces PCAP?
16
10
9
PCAP Flow OSINT
Avg Time to Close (PCAP)
10 10 11
Bro Flow OSINT
Avg Time to Close (Bro)
Data Suggests:
Better organization of high context data sources
can yield improvements in analysts performance
19.
What data sources were viewed
most and least frequently?
Data Suggests:
Network data is used more frequently than host data…
…even when host data can be used exclusively to resolve.
…even when easy access is provided to host sources.
Revisting data is more prevalent on higher context data
sources
Data Sources Viewed Data Sources Revisited
PCA
P
84%
Flow
11%
OSIN
T
5%
20.
How many steps were taken to
make a disposition judgement?
Data Suggests:
At some point, the number of data sources you
investigate impacts the speed of the investigation
Understanding where data exists and when to use it
can impact analysis speed
6
12
9
3
0
5
10
15
6-10 11-15 16-20 21-25
Number of Steps
9
12
14
24
0
5
10
15
20
25
30
6-10 11-15 16-20 21-25
Avg Time to Close
21.
Did analysts investigate friendly or
hostile systems first?
9%
91%
Observed
Friendly Hostile
Data Suggests:
Analysts are more compelled to investigate unknown
external threats than internal systems
Analysts don’t fully understand their own techniques
41%
59%
Friendly
Friendly Hostile
22.
Do analysts seek to prove or
disprove the alert?
Data Suggests:
Analysts are almost always seek to prove an
alert...
...despite the fact that disproving it is usually faster.
Prove vs. Disprove
Prove
88%
Dispr
ove
12% 19
8
0
5
10
15
20
Prove Disprove
Avg Time to Close
23.
Case Study
What separates novice
and expert analysts?
24.
Mapping the Investigation
Sample:
Novice and expert analysts
Methodology:
30 case studies
Stimulated recall interviews
Focus on individual investigations of varying
types
Perform key phrase anlayis – analyze results
25.
Key Phrase Mapping
Dual Process Theory
Intuition: Implicit, inconscious, fast
Reflection: Explicit, controlled, slow
Intuition
Experimentation
Restructuring
Imagination
Incubation
Metacognition
Evaluation
Goal Setting
Making Plans
Reflection
Analytically Viewing
Data
Rule-Based
Reasoning
Considering
Alternatives
Every town had one doctor and they were also your vet Many home remedies spawn from this time – milk as a treatment for stomach ulcers is an example Major health crises were frequent and impossible to control
How do we research a process that is intrinsically human?
We ended up with an investigation game
TIMECHECK – 15 MINUTES
Sidebar: Analysts looked at the PCAP 100% of the time, even if it wasn’t necessary.
This points to tendencies gained from training. Most shops don’t have easy access to host data.
Anecdotal – Experts I knew took less than 10 steps. Anecdotal – Novices I knew took > 15.