2. Who
am
I?
• Co-‐Founder/CSO
@
Signal
Sciences
• Previously
built/led
the
Security
Engineering
group
@
Etsy
• Prior
to
that,
offensive
research/pentesJng
@
iSEC
Partners
3. This
talk
simply
isn’t
possible
without
a
number
of
people:
– Ben
Hughes
– Brendan
Adamson
– Corey
Benninger
– Kai
Zhong
– Ken
Lee
– Kyle
Barry
– Marcus
Barczak
– Mike
Arpaia
– Omar
Ahmed
4. Also
a
shout
out
to
secteams
we’ve
enjoyed
collaboraJng
with:
Facebook/GitHub/Google/Square/Twi"er
15. Fundamentally
we
have
three
goals:
1) Raise
cost
to
a"ackers
2) Increase
the
odds
of
detecJng
compromise
3) Iterate
defenses
based
on
real
a"ack
pa"erns
25.
From
a
macro
level,
bucket
users
into
technical
vs
non-‐technical
26.
Pa"erns
then
break
down
into:
– Anomalous
if
by
a
non-‐technical
user
– Anomalous
if
by
technical
user
– Always
anomalous
27. Non-‐Technical
bucket:
– Alert
off
any
commands
which
show
technical
capabiliJes
• It’s
either
an
a"acker
or
your
IT
team
Technical
bucket:
– Treat
individual
commands
and
behaviors
as
low
quality
signals
• Aggregate
commands,
look
for
unique
combinaJons
and
bursts
Always
anomalous
(both
buckets):
– Analyze
a"ack
pa"erns
and
idenJfy
commands/behaviors
strongly
indicaJve
of
compromise
• We’re
looking
at
you,
`uname
-‐a`
28. Non-‐Technical
bucket:
– Alert
off
any
commands
which
show
technical
capabiliJes
• It’s
either
an
a"acker
or
your
IT
team
Technical
bucket:
– Treat
individual
commands
and
behaviors
as
low
quality
signals
• Aggregate
commands,
look
for
unique
combinaJons
and
bursts
Always
anomalous
(both
buckets):
– Analyze
a"ack
pa"erns
and
idenJfy
commands/behaviors
strongly
indicaJve
of
compromise
• We’re
looking
at
you,
`uname
-‐a`
29. Non-‐Technical
bucket:
– Alert
off
any
commands
which
show
technical
capabiliJes
• It’s
either
an
a"acker
or
your
IT
team
Technical
bucket:
– Treat
individual
commands
and
behaviors
as
low
quality
signals
• Aggregate
commands,
look
for
unique
combinaJons
and
bursts
Always
anomalous
(both
buckets):
– Analyze
a"ack
pa"erns
and
idenJfy
commands/behaviors
strongly
indicaJve
of
compromise
• We’re
looking
at
you,
`uname
-‐a`
31. Host
level
persistence:
– Look
for
common
pa"erns
of
persistence
via
programs
executed
on
boot,
kernel
modules
loaded,
etc
32. Host
level
persistence:
– Look
for
common
pa"erns
of
persistence
via
programs
executed
on
boot,
kernel
modules
loaded,
etc
– Understand
that
in
pracJce
sophisJcated
persistence
mechanisms
won’t
be
detected
• Aim
to
detect
the
basics,
and
increase
a"acker
cost
by
forcing
use
of
custom
persistence
mechanisms
33.
Shout
out
to
@mimeframe
and
the
FB
secteam
for
their
work
on
BigMac:
h"p://www.slideshare.net/mimeframe/
ruxcon-‐2012-‐15195589
43. Goal:
Detect
a
malicious
kernel
module
loading
on
an
endpoint
– We
thought
kernel
modules
loading
would
be
fairly
rare
aXer
boot
and
we
could
alert
off
that
alone.
We
were
wildly
wrong.
– WhitelisJng/blacklisJng
kernel
module
names
wouldn’t
be
effecJve
– Instead,
analyze
a
kernel
module
being
loaded
for
organizaJonal
uniqueness
44. Goal:
Detect
a
malicious
kernel
module
loading
on
an
endpoint
– We
thought
kernel
modules
loading
would
be
fairly
rare
aXer
boot
and
we
could
alert
off
that
alone.
We
were
wildly
wrong.
– WhitelisJng/blacklisJng
kernel
module
names
wouldn’t
be
effecJve
– Instead,
analyze
a
kernel
module
being
loaded
for
organizaJonal
uniqueness
45. Goal:
Detect
a
malicious
kernel
module
loading
on
an
endpoint
– We
thought
kernel
modules
loading
would
be
fairly
rare
aXer
boot
and
we
could
alert
off
that
alone.
We
were
wildly
wrong.
– WhitelisJng/blacklisJng
kernel
module
names
wouldn’t
be
effecJve
– Instead,
analyze
a
kernel
module
being
loaded
for
organizaJonal
uniqueness
46. Goal:
Detect
a
malicious
kernel
module
loading
on
an
endpoint
– We
thought
kernel
modules
loading
would
be
fairly
rare
aXer
boot
and
we
could
alert
off
that
alone.
We
were
wildly
wrong.
– WhitelisJng/blacklisJng
kernel
module
names
wouldn’t
be
effecJve
– Instead,
analyze
a
kernel
module
being
loaded
for
organizaJonal
uniqueness
47.
“Did
module
X
that
just
got
loaded
on
endpoint
Y
get
loaded
on
less
than
N
systems
across
the
organizaJon
in
the
last
D
days?”
48.
Use
a"ack
post-‐exploitaJon
techniques
in
a
defensive
context
by
separaJng
your
objecJves
from
your
tooling
49.
Specifically,
collect
data
on
the
endpoints
and
analyze/alert
from
that
data
on
the
server-‐side
50.
When
an
a"acker
discovers
and
analyzes
your
endpoint
security
mechanisms
they
shouldn’t
be
able
to
automaJcally
learn
all
alerts
in
place
51. OrganizaJonal
level
persistence:
– LegiJmate
remote
access
mechanisms
or
cloud
systems
with
data
desired
by
a"acker
• Ex:
VPN
and
GMail
– Use
a
mixed
approach
of
manual
and
automated
anomaly
detecJon
for
these
systems
• GeneraJng
daily
rollups
of
new
accounts/keys
created
• AlerJng
off
account
creaJon/modificaJon
at
unusual
Jmes,
from
unusual
locaJons,
etc
52. OrganizaJonal
level
persistence:
– LegiJmate
remote
access
mechanisms
or
cloud
systems
with
data
desired
by
a"acker
• Ex:
VPN
and
GMail
– Use
a
mixed
approach
of
manual
and
automated
anomaly
detecJon
for
these
systems
• GeneraJng
daily
rollups
of
new
accounts/keys
created
• AlerJng
off
account
creaJon/modificaJon
at
unusual
Jmes,
from
unusual
locaJons,
etc
54. Goal:
Instrument
GMail
to
detect
compromise
of
domain
admin
accounts
– GMail
provides
logs
of
interesJng
acJons
via
Admin
Audit
API
and
Email
Audit
API
– Pull
down
logs
via
these
APIs,
store
them
locally
so
you
have
a
record
of
acJons
– Perform
alerJng
on
strong
signals
of
compromise
and
persistence:
• Signins
from
unusual
locaJons/Jmes
• CreaJon
of
new
admin
level
accounts
• CreaJon
of
new
mail-‐forwarding
filters
• Any
change
to
2FA
secngs
• Etc
55. Goal:
Instrument
GMail
to
detect
compromise
of
domain
admin
accounts
– GMail
provides
logs
of
interesJng
acJons
via
Admin
Audit
API
and
Email
Audit
API
– Pull
down
logs
via
these
APIs,
store
them
locally
so
you
have
a
record
of
acJons
– Perform
alerJng
on
strong
signals
of
compromise
and
persistence:
• Signins
from
unusual
locaJons/Jmes
• CreaJon
of
new
admin
level
accounts
• CreaJon
of
new
mail-‐forwarding
filters
• Any
change
to
2FA
secngs
• Etc
56. Goal:
Instrument
GMail
to
detect
compromise
of
domain
admin
accounts
– GMail
provides
logs
of
interesJng
acJons
via
Admin
Audit
API
and
Email
Audit
API
– Pull
down
logs
via
these
APIs,
store
them
locally
so
you
have
a
record
of
acJons
– Perform
alerJng
on
strong
signals
of
compromise
and
persistence:
• Signins
from
unusual
locaJons/Jmes
• CreaJon
of
new
admin
level
accounts
• CreaJon
of
new
mail-‐forwarding
filters
• Any
change
to
2FA
secngs
• Etc
68.
Instrument
these
systems
the
way
you
would
other
high-‐value
pieces
of
infrastructure
69. Alert
off
behavioral
anomalies
such
as:
– Usage
outside
of
normal
hours
• Your
a"ackers
are
rarely
in
your
Jme
zone
– Bursts
of
acJvity
• Viewing
all
security
Jckets
in
the
bug
tracker
isn’t
even
done
by
the
security
team
– Etc
71. Make
compromise
more
expensive
– We’ll
discuss:
• Reducing
trusted
CA
roots
• Removing
cheap
exploitaJon
vectors
• Forcing
updates
without
the
force
• LimiJng
drive-‐by
exposure
• PracJcal
goals
for
security
awareness
training
72. How
can
you
reduce
the
likelihood
of
a
DigiNotar-‐like
MITM
happening
against
your
organizaJon?
73. If
you
remove
unused
CAs,
when
one
is
compromised
it
can’t
silently
MITM
your
endpoints
74. We
performed
several
months
of
anonymized
traffic
analysis
to
record
what
CAs
were
seen
during
our
employees
Internet
usage
75. We
found
less
than
29%
of
SSL
CerJficate
AuthoriJes
trusted
by
our
endpoints
were
actually
used
76. 21.29%
EQUIFAX
SECURE
CERTIFICATE
AUTHORITY
10.37%
ENTRUST.NET
SECURE
SERVER
CERTIFICATION
AUTHORITY
10.07%
DIGICERT
HIGH
ASSURANCE
EV
ROOT
CA
8.97%
GO
DADDY
CLASS
2
CERTIFICATION
AUTHORITY
7.91%
GEOTRUST
GLOBAL
CA
7.23%
ADDTRUST
EXTERNAL
CA
ROOT
6.48%
HTTP://WWW.VALICERT.COM/
6.04%
GTE
CYBERTRUST
GLOBAL
ROOT
4.45%
VERISIGN
CLASS
3
PUBLIC
PRIMARY
CERTIFICATION
AUTHORITY
-‐
G5
4.08%
CLASS
3
PUBLIC
PRIMARY
CERTIFICATION
AUTHORITY
3.82%
BALTIMORE
CYBERTRUST
ROOT
3.22%
CLASS
3
PUBLIC
PRIMARY
CERTIFICATION
AUTHORITY
-‐
G2
1.37%
THAWTE
PRIMARY
ROOT
CA
1.36%
THAWTE
PREMIUM
SERVER
CA
1.33%
ENTRUST.NET
CERTIFICATION
AUTHORITY
(2048)
0.65%
GLOBALSIGN
ROOT
CA
[The
CAs
which
had
<
0.5%
traffic
have
been
edited
out
for
brevity.
More
info
here:
h"p://codeascraX.com/2013/07/16/reducing-‐the-‐roots-‐of-‐some-‐evil/]
Our
raw
results:
77.
By
removing
only
unused
CAs
you
don’t
teach
users
to
click
through
SSL
errors
ConJnue
traffic
analysis,
add/remove
trusted
CAs
as
appropriate
78.
in
the
browser
is:
cheap,
reliable,
and
efficient
(pick
three!)
79.
in
the
browser
is:
cheap,
reliable,
and
efficient
(pick
three!)
…for
a"ackers
80.
81. What
did
we
learn
when
we
removed
Java
web
plugins
from
the
enterprise?
82. • Hardly
any
groups
actually
needed
it
– Network
OperaJons,
for
legacy
networking
equipment
• For
them,
we
built
dedicated
Java
jump
boxes
83. Benefits:
1. No
Java
on
any
laptops/desktops
2. Boxes
with
Java
can’t
hit
Internet
3. Able
to
frequently
re-‐image
jump
boxes
4. Only
a
few
boxes
to
patch
84.
But…
Java
shows
back
up
when
you
apply
Apple
patches.
85.
But…
Java
shows
back
up
when
you
apply
Apple
patches.
Remove
it
on
an
ongoing
basis
86.
87.
Browser
updates
88.
We
wanted
a
less
heavy
handed
approach
to
ensuring
up
to
date
browsers
89.
Built
browser
detecJon
logic
into
our
internal
SSO
point
90.
UX
is
key:
Show
in
screenshots
how
quick
it
is
to
update,
provide
a
bypass
mechanism
91.
92.
Simply
asking
users
to
update
works
shockingly
well
102.
The
metric
to
track/increase
is
the
likelihood
of
phishing
emails
being
reported
to
security
Even
if
36%
sJll
fall
for
phishing,
as
long
as
one
in
the
group
reports
it
IR
can
begin
104.
Problems
with
“pentesJng”
are
well
understood
in
the
offensive
community
but
not
as
well
in
the
defensive
community
105.
Pentests
typically
result
in
a
list
of
enumerated
known
vulnerabiliJes
to
be
patched,
not
data
on
how
a
real
a"acker
would
operate
against
a
given
environment
106.
A"ack
simulaJons
should
be
done
to
learn
how
a"ackers
are
likely
to
achieve
goals
against
your
organizaJon
NOT
to
show
compromise
is
possible
(spoiler
alert:
it
is.)
107.
Use
this
a"ack
data
to
focus
where/how
to
build
detecJon
mechanisms
108.
From
an
organizaJonal
side,
a"ack
simulaJons
compliment
vulnerability
enumeraJon/
compliance/etc
111. Four
keys
to
effecJve
a"ack
simulaJons:
– Goal
oriented
• “Obtain
domain
admin”,
“read
the
CEOs
email”,
“view
credit
card
data”,
…
– Full
organizaJon
in
scope
• Have
a"ack
team
call
a
contact
if
they’re
about
to
do
something
risky
– Simulate
realisJc
compromise
pa"erns
• Start
the
a"ack
team
on
a
standard
laptop/desktop
endpoint
inside
the
organizaJon
to
simulate
phishing/clientside
compromise
• Start
the
a"ack
team
on
a
database
or
web
server
to
simulate
SQL
injecJon/
RCE
• A"ack
team
should
be
encouraged
to
use
0days
– Break
simulaJon
down
into
iteraJons:
• Don’t
spend
the
full
engagement
Jme
on
only
round
of
tesJng,
once
one
team
achieve
goal(s),
then
swap
in
new
a"ack
team
to
achieve
the
same
goal(s)
– Ex:
We
try
to
run
3-‐4
iteraJons
per
several
week
simulaJon
112. Four
keys
to
effecJve
a"ack
simulaJons:
– Goal
oriented
• “Obtain
domain
admin”,
“read
the
CEOs
email”,
“view
credit
card
data”,
…
– Full
organizaJon
in
scope
• Have
a"ack
team
call
a
contact
if
they’re
about
to
do
something
risky
– Simulate
realisJc
compromise
pa"erns
• Start
the
a"ack
team
on
a
standard
laptop/desktop
endpoint
inside
the
organizaJon
to
simulate
phishing/clientside
compromise
• Start
the
a"ack
team
on
a
database
or
web
server
to
simulate
SQL
injecJon/
RCE
• A"ack
team
should
be
encouraged
to
use
0days
– Break
simulaJon
down
into
iteraJons:
• Don’t
spend
the
full
engagement
Jme
on
only
round
of
tesJng,
once
one
team
achieve
goal(s),
then
swap
in
new
a"ack
team
to
achieve
the
same
goal(s)
– Ex:
We
try
to
run
3-‐4
iteraJons
per
several
week
simulaJon
113. Four
keys
to
effecJve
a"ack
simulaJons:
– Goal
oriented
• “Obtain
domain
admin”,
“read
the
CEOs
email”,
“view
credit
card
data”,
…
– Full
organizaJon
in
scope
• Have
a"ack
team
call
a
contact
if
they’re
about
to
do
something
risky
– Simulate
realisJc
compromise
pa"erns
• Start
the
a"ack
team
on
a
standard
laptop/desktop
endpoint
inside
the
organizaJon
to
simulate
phishing/clientside
compromise
• Start
the
a"ack
team
on
a
database
or
web
server
to
simulate
SQL
injecJon/
RCE
• A"ack
team
should
be
encouraged
to
use
0days
throughout
engagement
– Break
simulaJon
down
into
iteraJons:
• Don’t
spend
the
full
engagement
Jme
on
only
round
of
tesJng,
once
one
team
achieve
goal(s),
then
swap
in
new
a"ack
team
to
achieve
the
same
goal(s)
– Ex:
We
try
to
run
3-‐4
iteraJons
per
several
week
simulaJon
114. Four
keys
to
effecJve
a"ack
simulaJons:
– Goal
oriented
• “Obtain
domain
admin”,
“read
the
CEOs
email”,
“view
credit
card
data”,
…
– Full
organizaJon
in
scope
• Have
a"ack
team
call
a
contact
if
they’re
about
to
do
something
risky
– Simulate
realisJc
compromise
pa"erns
• Start
the
a"ack
team
on
a
standard
laptop/desktop
endpoint
inside
the
organizaJon
to
simulate
phishing/clientside
compromise
• Start
the
a"ack
team
on
a
database
or
web
server
to
simulate
SQL
injecJon/
RCE
• A"ack
team
should
be
encouraged
to
use
0days
throughout
engagement
– Break
simulaJon
down
into
iteraJons:
• Don’t
spend
the
full
engagement
Jme
on
only
round
of
tesJng,
once
one
team
achieve
goal(s),
then
swap
in
new
a"ack
team
to
achieve
the
same
goal(s)
– Ex:
We
try
to
run
3-‐4
iteraJons
per
several
week
simulaJon
115.
The
project
output
should
be
a>ack
chains
showing
how
a"ack
team
went
from
A-‐>B-‐>C
to
achieve
goals,
what
steps
they
took
and
why
116.
Just
as
importantly,
what
steps
they
didn’t
take
Ex:
“We
didn’t
try
to
find
internal
network
diagrams
on
your
wiki
because
zone
transfers
were
enabled
so
we
could
got
enough
data
about
your
network
from
that”
117.
Remember,
the
goal
is
to
simulate
realisJc
a"ack
behaviors
and
pa"erns
that
can
be
used
to
enhance
detecJon
118.
In
addiJon,
simulate
varying
a"ack
profiles
from
quick
&
noisy
to
quietly
maintaining
persistence
119.
Over
mulJple
iteraJons
learn
what
behaviors
overlap
between
a"ackers
and
what
strong
signals
of
lateral
movement
in
your
environment
look
like
122.
Where
should
defense
focus?
– Increase
a"acker
cost
by
reducing
cheap
compromise
vectors
– Build
detecJon
mechanisms
around
real
a"ack
pa"erns
• Analyze
past
compromises,
new
offensive
research,
and
conduct
realisJc
a"ack
simulaJons
to
obtain
data
– Depending
on
scale,
have
true
offensive
capabiliJes
on
staff
or
a
call
away
– Have
product/tooling
development
capabiliJes
within
your
security
team
• Roughly
one
quarter
of
our
team
is
soXware
engineers
who
focus
on
building
internal
security
products