cloud security
in an uncertain age

about
•Founder @ Threat Stack
•fun_cuddles@twitter
•jandre@github
•jen.andre@threatstack.com

“There are known knowns. These
are things we know that we know.
!
There are known unknowns. That
is to say, there are things that we
know we don't know.
!
But there are also unknown
unknowns. There are things we
don't know we don't know.”
!
- Satan Donald Rumsfeld
uncertainties

the dream of the
cloud
automation!
scalability!
flexibility!
speed!

reality

uncertainty

automation!

security corollary

!
prevention fails

but amazon will watch
over me!

kind of but no

That’s right:
WITH ONE SET OF
IAM CREDS,
DESTROY ALL OF
THE BUSINESS

shared security model

shared security model
“We’re here to cover our own a**”

a new age of
uncertainties

legacy unknowns
1. what devices are on my network?
2. what are those devices really doing? what applications
are installed, and what network activity is really
happening?
3. where does my important data live and who has
access to it?

the cloud world
1. what devices are on my network?
2. what are those devices really doing? what applications
are installed, and what network activity is really
happening?
3. where does my important data live and who has
access to it?

the cloud world
1. what devices are on my network?
2. what are those devices really doing? what applications
are installed, and what network activity is really
happening?
3. where does my important data live and who has
access to it?
+ everything is online and now there’s a documented API
for it!

shiny new single points
of failure
CI
Config Management
Your ops guy with the SSL keys to push CM changes
AWS and other IaaS/SaaS APIs

shiny new single points
of failure
CI
Config Management
Your ops guy with the SSL keys to push CM changes
AWS and other IaaS/SaaS APIs
who watches the watchmen?

dwell time arms race

time of
breach
dwell time arms race

time of
breach
time of
theft or
damage
dwell time arms race

time of
breach
time of
theft or
damage
dwell time arms race
~3 hrs
reconnaissance

time of
breach
time of
theft or
damage
time
detected
:)
dwell time arms race
~3 hrs
reconnaissance

time of
breach
time of
theft or
damage
dwell time arms race
~3 hrs
reconnaissance

time of
breach
time of
theft or
damage
time
detected
:(
dwell time arms race
~3 hrs
reconnaissance

time of
breach
time of
theft or
damage
dwell time arms race
~3 hrs
reconnaissance

time of
breach
time of
theft or
damage
dwell time arms race
thanks to public APIs, attackers now know where to look!

tools and
standardization that
make our lives easier
also enable attackers

do you care?

do you care?

do you care?
X

do you care?
X

do you care?
X

do you care?
X

do you care?
X ?

response time is critical
if you care about being
hacked

if you aren’t monitoring
yourself, you won’t
know if you are

the future must be
automated

Security + DevOps = ?
•add security protections and automate
them
•continuously verify your protections with
automation
•add security monitoring but automate it
•detect intrusions and automate response
•score yourself on your security posture and
automate the grading

Security + DevOps = ?
•add security protections and automate
them
•continuously verify your protections with
automation
•add security monitoring but automate it
•detect intrusions and automate response
•score yourself on your security posture and
automate the grading

authentications
process activity
network activity
kernel modules
file system actions
appssystems services
authentications
db requests
http requests
AWS api calls
SaaS api calls
what to monitor

authentications
process activity
network activity
kernel modules
file system actions
appssystems services
authentications
db requests
http requests
AWS api calls
SaaS api calls
what to monitor

authentications
process activity
network activity
kernel modules
file system actions
appssystems services
authentications
db requests
http requests
AWS api calls
SaaS api calls
what to monitor

systems monitoring
probably more than you ever wanted to know

questions to answer
“is that guy running commands he shouldn’t be?”
(e.g. why is anyone except chef user MAYBE running
gcc on a prod system)
!
“are accounts logging in from non-standard locations?”
!
“are there anomalies in my traffic?”
!
“did some process suddenly start making outbound
connections?”

step 1: toolbox
process auditing linux audit
network flow libnetfilter_conntrack
logins wtmp/audit/pam_loginuid

!
collect this data from every EC2 host you launch
+
store and correlate in a big data backend
+
build an audit log
+
analyze for suspicious behaviors and insights

linux audit

pros!
!
super powerful
built into your kernel (>=2.3)
“relatively” low overhead
apt-get install audit

it audits all of the things!

(sort of)

userland audit
daemon and tools
!
(e.g. redhat auditd,
auditctl, etc)
kernel
thread
queue
kernel threads
doing things
audit messages
the workings
netlink socket
/var/log/
audit/
audit.log

# files
!
-w /etc/shadow -p wa
!
# syscalls
!
-a always,exit -F arch=ARCH -S init_module -S
delete_module -k modules
!
# follow executable
!
-w /sbin/insmod -p x
configuration

now for the cons…

type=SYSCALL msg=audit(1383252540.673:8711406):
arch=c000003e syscall=59 success=yes exit=0 a0=c27fa8
a1=c24d48 a2=9f8008 a3=7fffc4553ce0 items=2 ppid=46247
pid=56107 auid=0 uid=0 gid=0 eu
type=EXECVE msg=audit(1383252540.673:8711406): argc=2
a0="ping" a1="google.com"
type=CWD msg=audit(1383252540.673:8711406): cwd="/opt/"
!
type=PATH msg=audit(1383252540.673:8711406): item=0
name="/bin/ping" inode=1048904 dev=08:01 mode=0104755
ouid=0 ogid=0 rdev=00:00
!
type=PATH msg=audit(1383252540.673:8711406): item=1
name=(null) inode=1056827 dev=08:01 mode=0100755 ouid=0
ogid=0 rdev=00:00
obtuse logging
execve(‘ping google.com’)

THIS ONE WEIRD TRICK!
!
enable rate limiting or it could ‘crash’ your
box
!
always be listening (or same)
!
…relatively stable otherwise ;)
auditctl -b 1000 -r 15000 # 1000 buffers, 15000 eps max

redhat auditd, the userland
daemon, occasionally wtf-y

yeah…

alternative:!
connect directly to netlink socket
and write your own audit listener

[
{
"exe": "/bin/cat",
"comm": "cat",
"ses": 10,
"fsgid": 0,
"sgid": 0,
"egid": 0,
"fsuid": 0,
"suid": 0,
"euid": 0,
"gid": 0,
"uid": 0,
"auid": 4294967295,
"pid": 31335,
"ppid": 31334,
"items": 2,
"a3": "7fff3480e600",
"a2": "654c88",
"a1": "654bc0",
"a0": "654dc0",
"exit": 0,
"success": "yes",
"syscall": "execve",
"arch": "c000003e",
"milli": 99,
"epoch": 1399248110,
"serial": 855516,
"type": "SYSCALL"
},
{
"a1": "eth0.dhclient",
"a0": "cat",
"argc": 2,
"milli": 99,
"epoch": 1399248110,
"serial": 855516,
"type": "EXECVE"
},
{
"cwd": "/run/resolvconf/interface",
"milli": 99,
"epoch": 1399248110,
"serial": 855516,
"type": "CWD"
},
{
"ogid": 0,
"name": "/bin/cat",
"milli": 99,
"epoch": 1399248110,
"serial": 855516,
"type": "PATH"
},
json output! grouped sanely!

luajit! for filtering, transformation &
alerting

performance improvements,
yay!
!
libevent + filtering + state machine parsing
!
= 120% -> 10% CPU usage with AB 10k connections/sec

+ authentications

# last
jandre pts/1 dev.threatstack. Sun May 4 11:20 - 01:37
(14:17)
jandre pts/0 dev.threatstack. Sun May 4 11:16 still logged
in
!
# in json format
!
{ type: ‘USER_PROCESS',
pid: 777,
line: ‘pts/1',
id: 52,
user: 'jandre',
host: ‘dev.threatstack',
exit_status: { termination: 0, code: 0 },
timestamp: Tue May 06 2014 03:50:03 GMT-0700 (PDT),
address: ’10.0.0.10’ }
}
!
!
wtmp

# if pam is built with audit support…
!
type=USER_AUTH msg=audit(1234877011.791:7731): user
pid=26127 uid=0 1
auid=4294967295 ses=4294967295
msg='op=PAM:authentication acct="root" exe="/usr/sbin/
sshd"
(hostname=jupiter.example.com, addr=192.168.2.100,
terminal=ssh res=success)'
plus audit

!
# /etc/pam.d/login, sshd, wherever you care
!
/etc/pam.d$ grep loginuid *
!
login:session required pam_loginuid.so
sshd:session required pam_loginuid.so
!
# now you get in /var/log/audit/audit.log:
!
type=LOGIN msg=audit(1234877011.799:7734): login pid=26125
uid=0
old auid=4294967295 new auid=0 old ses=4294967295 new ses=1172
!
# cat /proc/<login or sshd pid>/sessionid
!
1172
add pam_loginuid

if you do it correctly…

bonus!

apis: how to monitor

questions to answer
who is using my AWS credentials, and from where?
!
who is engaging anomalous or suspicious API activity
that could indicate a breach?
!
have my credentials been compromised?
!
are IAM credentials assigned with appropriate
permissions?

how?
cloud trail to an s3 bucket for audit logging
AWS APIs for auditing IAM permissions
!
collect periodically and store using IAM role with limited
permissions
group by user account, activity, location
alert by new or suspicious behaviors

idea: audit monitored
data against an
external source of truth

IAM credentials and
permissions
!
store definitions in a repository and audit against AWS
periodically
!

EC2 applications
!
do they match definitions of whats installed in your
config management?
!

build checks for this
auditing automatically
into your CI

Thank You
check out Cloud Sight @ www.threatstack.com