David Veuve, SE, Splunk, walks the audience through automated threat intelligence response, behavioral profiling, anomaly detection, and tracking an attack against the kill chain.
2. Who
Am
I?
2
! David
Veuve
–
Sales
Engineer
for
Major
Accounts
in
Northern
California
! dveuve@splunk.com
! Former
Splunk
Customer
(For
3
years,
3.x
through
4.3)
! Security
Guy
! Primary
author
of
Splunk
Search
Usage
app
! Primary
area
of
Splunk
ExperEse:
Search
Language
! Stands
on
the
shoulders
of
giants
3. Disclaimer
3
During
the
course
of
this
presentaEon,
we
may
make
forward
looking
statements
regarding
future
events
or
the
expected
performance
of
the
company.
We
cauEon
you
that
such
statements
reflect
our
current
expectaEons
and
esEmates
based
on
factors
currently
known
to
us
and
that
actual
events
or
results
could
differ
materially.
For
important
factors
that
may
cause
actual
results
to
differ
from
those
contained
in
our
forward-‐looking
statements,
please
review
our
filings
with
the
SEC.
The
forward-‐looking
statements
made
in
the
this
presentaEon
are
being
made
as
of
the
Eme
and
date
of
its
live
presentaEon.
If
reviewed
a^er
its
live
presentaEon,
this
presentaEon
may
not
contain
current
or
accurate
informaEon.
We
do
not
assume
any
obligaEon
to
update
any
forward
looking
statements
we
may
make.
In
addiEon,
any
informaEon
about
our
roadmap
outlines
our
general
product
direcEon
and
is
subject
to
change
at
any
Eme
without
noEce.
It
is
for
informaEonal
purposes
only
and
shall
not,
be
incorporated
into
any
contract
or
other
commitment.
Splunk
undertakes
no
obligaEon
either
to
develop
the
features
or
funcEonality
described
or
to
include
any
such
feature
or
funcEonality
in
a
future
release.
4. Agenda
• Visibility
–
Analysis
–
AcEon
in
Four
Scenarios
1. Threat
List
IntegraEon
leads
to
Firewall
Blocks
2. Anomaly
DetecEon
leads
to
Opening
a
Ticket
3. Behavioral
Profiling
leads
to
Manager
ConfirmaEon
4. Visual
CorrelaEon
of
Security
Indicators
4
5. Being
Covered
1. Tools
and
Searches
and
Demos
2. All
of
these
examples
and
concepts
come
from
actual
customer
requirements
and
actual
customer
deployments.
No
smoke
and
mirrors.
3. Github
with
data
gens
and
accoutrement
at
end
of
presentaEon
5
6. Who
Are
You?
1. Security
Engineer
/
SOC
Analyst
/
Threat
Analyst
/
Someone
Technical
Who
Cares
about
Security
2. Splunk
skill
level
is
basic-‐advanced
3. No
Enterprise
Security
required
(though
it
can
make
things
easier
at
scale)
6
7. Visibility
–
Analysis
–
AcEon
• Framework
for
evaluaEng
data
and
responding
Splunk
• Applies
to
all
exisEng
frameworks,
as
it’s
the
Splunk
side
of
the
loop.
• For
example,
Let’s
look
at
the
lateral
movement
secEon
of
the
kill
chain.
(Not
familiar
with
the
kill
chain?
It’s
a
great
way
to
understand
the
phases
of
an
agack.
Check
the
URL
below.)
• Visibility:
What
data
will
let
you
detect
Lateral
Movement?
• Analysis:
What
will
you
do
to
that
data
to
come
to
a
decision?
• Ac2on:
What
will
you
do
in
response
to
that
decision?
– Can
we
automate
all
of
this?
• Kill
Chain:
hgp://www.lockheedmarEn.com/content/dam/lockheed/data/corporate/documents/LM-‐White-‐Paper-‐Intel-‐Driven-‐Defense.pdf
7
9. Command
and
Control
DetecEon
and
Blocking
• New
threat
list
intel
(or
any
other
source
of
detecEng
agackers)
has
become
available,
and
we
are
trying
to
block
any
outbound
Command
and
Control.
• The
formal
firewall
policy
can’t
be
pushed
except
every
Wed
night
and
Sunday
night
–
not
fast
enough.
• Goal:
Take
in
the
firewall
logs,
leverage
our
available
intelligence
to
detect
C&C
behavior,
and
then
block
the
desEnaEons,
all
in
near
realEme.
• Visibility:
Firewall
Logs,
Threat
Intel
Sources
• Analysis:
IntersecEon
(lookup)
of
the
two
• Ac2on:
Apply
dynamic
firewall
blocks
9
10. What
/
Where
is
Threat
Intelligence
10
! A
feed
of
known
bad
IPs/DNS
Names/MD5s/URLs/etc
from
a
vendor
or
non-‐profit
that
specializes
in
discovering
Indicators
of
Compromise.
! Great
sources
of
Open
Source
Threat
Intel
include:
– Emerging
Threats:
hgp://rules.emergingthreats.net/
– I-‐Blocklist:
hgps://www.iblocklist.com/lists.php
– MalwareDomains:
hgp://www.malwaredomains.com/
– Zeus
Tracker:
hgps://zeustracker.abuse.ch/
! Many
great
commercial
enEEes
too
(generally
beger
ranking
/
quality):
– Norse
(Splunk
Partner),
iSight
Partners,
Verizon
iDefense,
Commercial
11. Visibility
Palo
Alto
Networks
Firewall
Log
Sep
15
19:02:06
1,2014/09/15
19:02:06,0004C104559,TRAFFIC,end,1,2014/09/15
19:02:05,10.2.2.14,206.16.215.101,206.16.216.158,214.34.245.101,Internet
Traffic,,,
salesforce-‐base,vsys1,Trust,Untrust,ethernet1/8,ethernet1/2,MyLogForwarding,
2014/09/15
19:02:05,24238,1,61845,443,57339,443,0x400000,tcp,allow,
1275,761,514,14,2014/09/15
19:01:31,5,any,0,358477769,0x0,
10.0.0.0-‐10.255.255.255,
United
States,0,8,6
11
ConnecEon
End
Date
Src
and
Dest
IPs
Firewall
Rule
ApplicaEon
To/From
Zone
Dest
Port
Threat
Intel
Lookup:
bad_ip,threat_intel_source
115.29.46.99/32,zeus_c2s
61.155.30.0/24,cymru_hgp
12. Analysis
• First,
we
want
to
pull
out
all
firewall
traffic
coming
from
inside
our
network,
going
outside
our
network.
• Then,
we
want
to
cross-‐reference
that
data
with
our
Threat
Intel
list.
This
is
accomplished
in
the
Splunk
world
via
a
lookup.
• Finally,
we
want
to
pull
just
the
logs
that
have
Threat
Intel
12
index=pan_logs
sourcetype=pan_traffic
src=“10.*”
dest!=“10.*”
|
lookup
ThreatIntel
dest
|
search
ThreatList=*
Name
of
our
lookup,
and
the
key
field
Data
held
in
Lookup
Table
13. Analysis
-‐
Challenges
13
! Performance
–
you
get
lots
of
traffic,
maybe
you
have
lots
of
threat
intel
entries.
– SoluEon:
Enterprise
Security
is
built
to
solve
this
problem
at
scale.
– Alternate
SoluEon:
data
models
help
substanEally
with
the
first
half.
You
can
fragment
the
lookups
if
you
get
to
very
high
numbers.
! MulEple
Threat
Lists
–
DeprioriEze
Open
source
threat
list
vs
Premium
threat
list
– SoluEon:
Enterprise
Security
has
this
fixed
as
well
with
deduping
and
prioriEzing
– Alternate
SoluEon:
|
inputlookup
Premium|
append
[|inputlookup
OpenSource]
|
munge
|
outputlookup
MyList
14. Analysis
–
Value
Adds
14
! Strength
of
AutomaEon
in
Splunk
is
high
fidelity
alerts.
! This
was
a
simple
example,
but
you
could
also
make
it
more
impressive
by
tracking
whether
the
IP
is
in
the
US:
! AlternaEvely,
you
could
look
to
see
whether
that
parEcular
host
had
a
recent
malware
event:
|
join
host
[|
`tstats`
count
from
datamodel=Malware
by
Malware_Agacks.dest
|
stats
count
by
Malware_Agacks.dest
|
rename
Malware_Agacks.dest
as
host]
15. AcEon
• PANBlock!
(Or
other
Network
Response,
see
below)
• Challenges:
– Many
organizaEons
fear
automaEc
response
due
to
potenEal
for
downEme
ê SoluEon:
Start
with
high
confidence
alerts
and
limited
list
of
assets,
verify
success.
ê Alternate
SoluEon:
Don’t
go
automaEc
response.
This
works
through
the
UI
too.
– You
don’t
run
Palo
Alto
Networks
ê SoluEon:
While
PAN/Splunk
have
made
this
work
out
of
the
box,
this
has
been
implemented
many
Emes
with
a
number
of
products,
Incl
but
not
limited
to:
– Cisco
Border
Router:
Expect
Script
to
block
– Check
Point:
R80
Rest
Interface
(Talk
to
me
if
you
want
to
do
this,
I
want
in)
15
21. Where
to
Learn
More
About
PAN
Blocking
21
! Have
a
Palo
Alto
device
and
like
this
parEcular
feature?
Visit
– Docs:
hgps://live.paloaltonetworks.com/docs/DOC-‐6593
– App
Page:
hgp://apps.splunk.com/app/491/
! Or
beger
yet,
go
see
those
talks:
– AutomaEc
Malware
DetecEon,
Analysis
and
MiEgaEon
in
Splunk
Jose
Hernandez,
SoluEons
Security
Architect,
Splunk
You
just
missed
it!
Get
the
PDF
and
watch
the
video
later
– MiEgaEng
Cybersecurity
Risk
with
Palo
Alto
Networks
and
Splunk
Marc
Benoit,
Sr.
Director,
Palo
Alto
Networks
Breakout
Session:
10/09/2014,
2:15-‐3:15
23. Anomaly
DetecEon
EssenEals
• File
audiEng
is
a
common
pracEce,
and
it
can
be
accomplished
quickly
and
easily
in
Splunk.
• It
becomes
harder
at
scale,
but
data
model
acceleraEon
helps.
• UlEmately,
by
conquering
anomaly
detecEon,
you
can
more
effecEvely
find
the
difficult
to
detect
in
your
systems.
• Visibility:
Carbon
Black
Logs
• Analysis:
System
DistribuEon,
accelerated
via
Data
Models
• Ac2on:
Security
Incident
CreaEon
23
24. What
is
Standard
DeviaEon?
24
! A
measure
of
the
variance
for
a
series
of
numbers.
! One
file
is
opened
on
100,
123,
79,
and
145
hosts
per
day
– average
of
111.75
and
a
standard
deviaEon
of
28.53.
! Another
file
is
opened
on
100,
342,
3
and
2
hosts
per
day
– average
of
111.75,
but
a
stdev
of
160.23.
26. How
To
Accelerate
26
• AcceleraEon
facilitates
beger
and
broader
analysis.
• Splunk
has
a
few
ways
of
acceleraEng
content:
• Report
AcceleraEon
• Data
Model
AcceleraEon
• TSCollect
• Summary
Indexing
• Pre-‐processing
of
logs
• Check
out
Gerald
Kanapathy’s
Session
on
Friday:
Title:
Splunk
Search
AcceleraEon
Technologies
Speaker:
Gerald
Kanapathy,
Sr.
Director
Product
Management,
Splunk
When:
10/09/2014,
10:30
AM
–
11:30
AM
28. Analysis
–
Create
Pivot
Search
28
• Create
a
baseline
pivot
search
and
Open
in
Search.
• In
this
case,
split
dc(host)
by
path
• Add
a
filter
for
criEcal
paths
29. Analysis
–
Create
AddiEonal
StaEsEcs
29
Add
addiEonal
stats
command
on
top
of
accelerated
Pivot
search.
31. AcEon
–
Create
a
New
Incident
31
! Will
work
with
essenEally
any
EckeEng
system,
maybe
via
a
scripted
alert.
– Every
TickeEng
System
Accepts
Emails
too!
! Known
to
work
with:
– Remedy:
hgp://wiki.splunk.com/Community:Use_Splunk_alerts_with_scripts_to_create_a_Ecket_in_your_EckeEng_system
– ServiceNow:
hgp://answers.splunk.com/answers/47086/service-‐now-‐Ecket-‐generaEon-‐via-‐splunk-‐alerts.html
– PagerDuty:
hgp://www.pagerduty.com/docs/guides/splunk-‐integraEon-‐guide/
– ArcSight:
hgps://apps.splunk.com/app/1847/
– Q1
– NetCool
– Anything
AccepEng
Email
– Anything
Scriptable:
hgp://docs.splunk.com/DocumentaEon/Splunk/6.1.3/alert/ConfiguringScriptedAlerts
34. Behavioral
Anomaly
DetecEon
• DetecEng
known
bad
is
great,
but
leaves
you
vulnerable.
• Augment
with
syntheEc
checks
of
sensiEve
systems.
• StaEsEcs
can
consume
all
your
Eme
– Generally
easiest
to
leverage
so^
approval
(e.g.,
emails
to
managers)
with
standard
deviaEon.
– AddiEonally,
use
hard
enforcement
for
large
deviaEon
(e.g.,
FW
isolaEon)
• In
this
scenario,
we
are
a
hospital
tracking
paEent
chart
opens.
• Visibility:
CharEng
System
Logs
• Analysis:
Frequency
Analysis
by
User,
Role,
etc.
• Ac2on:
Email
the
employees’
manager
to
invesEgate
34
35. What
is
Standard
DeviaEon?
35
! A
measure
of
the
variance
for
a
series
of
numbers.
In
this
case,
let’s
say
chart
opens.
! Over
a
few
days,
Jane
opens
100,
123,
79,
and
145
charts
per
day
with
an
average
of
111.75
and
a
standard
deviaEon
of
28.53.
! Over
the
same
period,
Jack
opens
100,
342,
3
and
2
charts
per
day,
also
with
an
average
of
111.75,
but
a
stdev
of
160.23.
! When
Jack
and
Jane
both
open
500
records
some
day,
that
will
be
13.6
standard
deviaEons
(z=13.6)
for
Jane
but
only
2.42
for
Jack.
! Z
score
=
number
of
standard
devia2ons
away
from
average
37. Analysis
• Core
Metric:
Chart
Opens
Per
Day,
Per
Employee
• Dimensions
to
Compare:
– Over
Eme
for
the
same
user,
others
with
same
Etle
– Others
with
the
same
Etle
in
the
same
city
or
with
the
same
years
of
experience
• Why
MulEple
Dimensions?
1. Comparing
mulEple
metrics
reduces
false
posiEves.
2. Provides
more
context.
3. If
I
open
25
Emes
as
many
charts,
but
so
does
every
other
nurse
in
my
facility
because
we’re
under
inspecEon,
that
should
be
evident.
• What
about
performance?
– Good
point!
Data
Models
turn
this
into
a
30
seconds
per
5M
events
search
on
my
laptop.
Tscollect
is
manual
but
turns
it
into
a
quarter
second
search.
37
38. Analysis
–
Basic
38
index=cerner
|
eval
EmployeeID=spath(_raw,
"audit_list.prsnl_id")
|
eval
EmployeeName
=
[…]
|
eval
RecordNum=
[…]
|
bucket
_Eme
span=1d
|
stats
dc(RecordNum)
as
NumRecords
by
EmployeeName,
EmployeeID,
_Eme
|
stats
first(NumRecords)
avg(NumRecords)
stdev(NumRecords)
by
EmployeeName,
EmployeeID
|
where
‘first(NumRecords)’
>
‘avg(NumRecords)’
+
‘stdev(NumRecords)’
*
6
! Basic
Data
Set
! Field
Munging
! Pull
the
number
of
stats
per
employee,
per
day
! Pull
the
average,
standard
deviaEon,
and
most
recent
daily
number
per
employee
! Find
instances
where
the
most
recent
number
is
more
than
6
standard
deviaEons
away
from
the
average
40. How
To
Accelerate
40
• AcceleraEon
facilitates
beger
and
broader
analysis.
• Splunk
has
a
few
ways
of
acceleraEng
content:
• Report
AcceleraEon
• Data
Model
AcceleraEon
• TSCollect
• Summary
Indexing
• Pre-‐processing
of
logs
• Check
out
Gerald
Kanapathy’s
Session
on
Friday:
Title:
Splunk
Search
AcceleraEon
Technologies
Speaker:
Gerald
Kanapathy,
Sr.
Director
Product
Management,
Splunk
When:
10/09/2014,
10:30
AM
–
11:30
AM
41. Analysis
–
AcceleraEon
41
index=cerner
|
eval
Role=spath(_raw,
"audit_list.role")
|
eval
RoleID
=
[…]
|
eval
EmployeeID=
[…]
|
eval
EmployeeName
=
[…]
|
eval
PaEentNum=
[…]
|
bucket
_Eme
span=1d
|
stats
dc(PaEentNum)
as
NumRecords
by
EmployeeName,
EmployeeID,
Role,
RoleID
_Eme
|
lookup
HR_IS.csv
EmployeeID
|
tscollect
retain_events=t
Cerner
! Basic
Data
Set
! Field
Munging
! Stats
split
by
as
many
dimensions
as
required,
but
not
more.
! Lookup
occurs
a^er
stats
! Store
the
results
in
a
local
tsidx
(could
also
do
this
with
datamodels)
42. Analysis
–
Find
StaEsEcal
Outliers
Pt
1
42
|
tstats
local=t
first(NumCharts)
as
Recent_NumCharts
avg(NumCharts)
as
Avg_NumCharts
stdev(NumCharts)
as
Stdev_NumCharts
from
Cerner
groupby
EmployeeName,
EmployeeID,
Username,
Role,
RoleID,
City,
YearsAtCompany
|
join
type=outer
RoleID
[|
tstats
local=t
avg(NumCharts)
as
Role_Avg_NumCharts
stdev(NumCharts)
as
Role_Stdev_NumCharts
from
Cerner
groupby
Role,
RoleID
]
! How
many
charts
is
typical
(and
what
is
the
standard
deviaEon)
for
this
person.
Also,
how
many
did
they
open
yesterday?
! How
many
chart
opens
is
standard
for
people
in
this
role?
43. Analysis
–
Find
StaEsEcal
Outliers
Pt
2
43
[…
conEnued
from
previous
slide
…]
|
eval
Personal_Z
=
abs(Recent_NumCharts-‐
Avg_NumCharts)/Stdev_NumCharts
|
eval
Role_Z
=
abs(Recent_NumCharts-‐
Role_Avg_NumCharts)/
Role_Stdev_NumCharts
|
eval
Z_Min
=
min(Role_Z,
Personal_Z)
|
where
Z_Min
>
6
! How
unusual
is
this
acEvity,
for
this
person
or
versus
others
in
this
role?
– Z
score
=
how
many
StDev
away
from
average.
– Consider
other
metrics,
such
as
years
at
the
company,
facility.
– Goal
is
to
capture
normal
across
dimensions,
to
idenEfy
trends
across
organizaEon
(e.g.,
a
facility
audit).
44. AcEon
• Email
the
Manager
• This
opEon
is
mostly
just
forma‡ng.
Join
to
the
HR
/
LDAP
database
and
uElize
sendemail
+
map.
• Could
also
escalate
big
violaEons
to
the
SOC
or
GRC.
|
lookup
LDAPSearch
sAMAccountManager
as
username
OUTPUT
manager
|
lookup
LDAPSearch
dn
as
manager
OUTPUT
mail
as
ManagerEmail
“
44
|
map
maxsearches=100
search=“
|
stats
count
|
eval
ManagerEmail=$ManagerEmail$
|
eval
EmployeeName=$EmployeeName$
|
eval
ZAvg
=
$Z_Avg$
|
sendemail
to=ManagerEmail
sendresults=f
subject=EmployeeName
.
“
excess
Chart
Opens”
message=EmployeeName
.
“
has
opened
more
charts
than
normal
(“
.
ZAvg
.
“
stdev).
_._Please
Follow
Up.”
47. Visual
Event
CorrelaEon
• A^er
conquering
the
essenEals
of
ge‡ng
some
alert
data,
it’s
important
to
be
able
to
understand
an
agacker’s
acEon
plans.
– Progress
through
kill
chain
– Movement
toward
criEcal
assets
– Et
Cetera
• Easiest
with
Enterprise
Security,
but
possible
without
47
48. Visibility
–
Log
Examples
• Anything.
This
should
encompass
all
of
your
log
sources,
correlaEon
rules,
alerts,
and
etc.
• Ideally
include
operaEonal
data
here
too
(e.g.,
website
response
Eme
change)
48
49. Analysis
• Examples
thus
far
have
centered
around
automated
analysis,
but
Splunk
is
also
a
great
tool
for
data
visualizaEon
and
analysis.
• CapabiliEes
here
are
virtually
endless,
but
here
are
a
few
examples.
49
50. AcEon
• Need
more
informaEon?
Enterprise
Security
has
many
built
in
work
flow
acEons
to
go
pull
more
data.
• Go
pull
more
informaEon
from
your
Endpoint
Threat
DetecEon
and
Response
app:
– Tanium:
hgp://apps.splunk.com/app/1862/
– Tripwire
/
nCircle
ip360:
Ask
your
SE
– Bit9
/
Carbon
Black:
hgps://www.bit9.com/soluEons/splunk/
– Many
Others
also
exist
• File
a
Ecket
with
your
EckeEng
– Remedy:
hgp://answers.splunk.com/answers/122019
• Open
a
new
Notable
Event
in
ES
50
53. Demo
–
Visualizing
By
Priority
53
• While
not
as
slick
as
the
ES
version,
you
can
get
much
of
the
same
value
by
leveraging
mulEple
reports
on
one
dashboard,
or
with
stacked
column
charts.