The
Investigation
Labyrinth
Chris Sanders
Security Onion Con 2016

Chris Sanders (@chrissanders88)
 Find Evil @ FireEye
 Founder @ Rural Tech
Fund
 PhD Researcher
 GSE # 64
 BBQ Pit Master
 Author:
 Practical Packet Analysis
 Applied NSM

Agenda
 Era of Analysis
 DFIR Cognitive Revolution
 Researching the Investigation Process
 Data, Data, and more Data
The economics of NSM are not in our favor –
how can we study the investigation process to
make it more efficient?

Economics of Security
“If you want to understand the world of nature,
master physics. If you want to understand the
world of man, master economics.” - Taufiq
Rashid
High
Demand for
Security
Expertise
Low Supply
of Security
Practitioners
Expertise
Services
Software

Evolution of NSM
 “The profession
[security] is so
nascent that the how-
tos have not been
fully realized even by
the people who have
the knowledge.”
 Every thought-based
profession goes
through a cognitive
crisis and revolution.
Ours is coming.

Symptoms of a Cognitive Crisis
1. Demand for expertise greatly outweights
supply
2. Most information cannot be trusted or
validated
3. Inability to mobilize and tackle big systemic
issues

The Cognitive Revolution in
DFIR
1. Understand the processes
used to perform
investigations and draw
conclusions
2. Develop repeatable
methods and techniques
for performing
investigations
3. Build and advocate
training that teaches
analysts how to think
about investigations, not
just how to use tools.

Investigations as Mental
Labyrinths
 The investigation is
the core construct of
information security.
 At a high level, an
investigation is a
series of decisions
that begets other
decisions.
 Defenders don’t
always know if
they’ve taken the
correct path.

Navigating the Labyrinth
Alert
OSINT
Reputation
File Hash
Sandbox
Behaviors
AV Detections
(VT)
Imphash
More File
Hashes
Friendly Host
Network PCAP
Host
Windows
Logs
Security Log
System Log
App LogRegistry
File System
Hostile Host Network
PCAP
Flow

Studying the Investigation
Process
 Goal:
 Increase Accuracy
 Decrease Time
 How do you study
something human
thought?
 Challenges:
 Creating unique
investigation scenarios
takes time
 There is no universal
set of tooling

A Scenario-Based Approach to
Investigation Analysis
 Create a tool-agnostic investigation simulator
 Make it portable and self contained
 Seed it with investigation scenarios where one
variable can be addressed at a time
 Allow it to log investigator actions and output a
log of decisions being made

Additional Data Sources

Case Study
Analyzing the Flow of
the Investigation

The Compromise
1. Victim visits friendly
website
2. Redirect to EK landing
page
3. Download flash exploit
4. Exploit is successful and
ransomware file
downloads
5. Ransomware installs and
executes
6. Ransomware begins C2
communication

What data did analysts look at
first?
72%
16%
12%
Observed
PCAP Flow OSINT
Data Suggests:
 Analysts prefer a higher context data set…
 …even if other data sets are available
 …even if lower context data sets can lead to a resolution.
 Analysts don’t fully understand their own techniques
49%
28%
23%
Reported
PCAP Flow OSINT

Did the first move affect analysis
speed?
Data Suggests:
 While PCAP provides richer context, it may slow down
the investigation if that’s where you start
 Starting with a lower context data source can increase
speed when working with higher context data
16
10
9
PCAP Flow OSINT
27
13 13
PCAP Flow OSINT
Avg Time to Close Weighted Time to Close

What happens when Bro data
replaces PCAP?
46%
25%
29%
Observed (Bro)
Bro Flow OSINT
72%
16%
12%
Observed (PCAP)
PCAP Flow OSINT

What happens when Bro data
replaces PCAP?
16
10
9
PCAP Flow OSINT
Avg Time to Close (PCAP)
10 10 11
Bro Flow OSINT
Avg Time to Close (Bro)
Data Suggests:
 Better organization of high context data sources
can yield improvements in analysts performance

What data sources were viewed
most and least frequently?
Data Suggests:
 Network data is used more frequently than host data…
 …even when host data can be used exclusively to resolve.
 …even when easy access is provided to host sources.
 Revisting data is more prevalent on higher context data
sources
Data Sources Viewed Data Sources Revisited
PCA
P
84%
Flow
11%
OSIN
T
5%

How many steps were taken to
make a disposition judgement?
Data Suggests:
 At some point, the number of data sources you
investigate impacts the speed of the investigation
 Understanding where data exists and when to use it
can impact analysis speed
6
12
9
3
0
5
10
15
6-10 11-15 16-20 21-25
Number of Steps
9
12
14
24
0
5
10
15
20
25
30
6-10 11-15 16-20 21-25
Avg Time to Close

Did analysts investigate friendly or
hostile systems first?
9%
91%
Observed
Friendly Hostile
Data Suggests:
 Analysts are more compelled to investigate unknown
external threats than internal systems
 Analysts don’t fully understand their own techniques
41%
59%
Friendly
Friendly Hostile

Do analysts seek to prove or
disprove the alert?
Data Suggests:
 Analysts are almost always seek to prove an
alert...
 ...despite the fact that disproving it is usually faster.
Prove vs. Disprove
Prove
88%
Dispr
ove
12% 19
8
0
5
10
15
20
Prove Disprove
Avg Time to Close

Case Study
What separates novice
and expert analysts?

Mapping the Investigation
 Sample:
 Novice and expert analysts
 Methodology:
 30 case studies
 Stimulated recall interviews
 Focus on individual investigations of varying
types
 Perform key phrase anlayis – analyze results

Key Phrase Mapping
 Dual Process Theory
 Intuition: Implicit, inconscious, fast
 Reflection: Explicit, controlled, slow
Intuition
Experimentation
Restructuring
Imagination
Incubation
Metacognition
Evaluation
Goal Setting
Making Plans
Reflection
Analytically Viewing
Data
Rule-Based
Reasoning
Considering
Alternatives

Results
Novices Experts
Intuition Metacognition Reflection

Thank You!
Mail: chris@chrissanders.org
Twitter: @chrissanders88
Blog: chrissanders.org