Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Threat stack aws

1,172 views

Published on

  • Be the first to comment

Threat stack aws

  1. 1. cloud security in an uncertain age
  2. 2. about •Founder @ Threat Stack •fun_cuddles@twitter •jandre@github •jen.andre@threatstack.com
  3. 3. “There are known knowns. These are things we know that we know. ! There are known unknowns. That is to say, there are things that we know we don't know. ! But there are also unknown unknowns. There are things we don't know we don't know.” ! - Satan Donald Rumsfeld uncertainties
  4. 4. the dream of the cloud automation! scalability! flexibility! speed!
  5. 5. reality
  6. 6. uncertainty
  7. 7. automation!
  8. 8. security corollary
  9. 9. ! prevention fails
  10. 10. but amazon will watch over me!
  11. 11. kind of but no
  12. 12. That’s right: WITH ONE SET OF IAM CREDS, DESTROY ALL OF THE BUSINESS
  13. 13. shared security model
  14. 14. shared security model “We’re here to cover our own a**”
  15. 15. a new age of uncertainties
  16. 16. legacy unknowns 1. what devices are on my network? 2. what are those devices really doing? what applications are installed, and what network activity is really happening? 3. where does my important data live and who has access to it?
  17. 17. the cloud world 1. what devices are on my network? 2. what are those devices really doing? what applications are installed, and what network activity is really happening? 3. where does my important data live and who has access to it?
  18. 18. the cloud world 1. what devices are on my network? 2. what are those devices really doing? what applications are installed, and what network activity is really happening? 3. where does my important data live and who has access to it? + everything is online and now there’s a documented API for it!
  19. 19. shiny new single points of failure CI Config Management Your ops guy with the SSL keys to push CM changes AWS and other IaaS/SaaS APIs
  20. 20. shiny new single points of failure CI Config Management Your ops guy with the SSL keys to push CM changes AWS and other IaaS/SaaS APIs who watches the watchmen?
  21. 21. dwell time arms race
  22. 22. time of breach dwell time arms race
  23. 23. time of breach time of theft or damage dwell time arms race
  24. 24. time of breach time of theft or damage dwell time arms race ~3 hrs reconnaissance
  25. 25. time of breach time of theft or damage time detected :) dwell time arms race ~3 hrs reconnaissance
  26. 26. time of breach time of theft or damage dwell time arms race ~3 hrs reconnaissance
  27. 27. time of breach time of theft or damage time detected :( dwell time arms race ~3 hrs reconnaissance
  28. 28. time of breach time of theft or damage dwell time arms race ~3 hrs reconnaissance
  29. 29. time of breach time of theft or damage dwell time arms race thanks to public APIs, attackers now know where to look!
  30. 30. tools and standardization that make our lives easier also enable attackers
  31. 31. do you care?
  32. 32. do you care?
  33. 33. do you care? X
  34. 34. do you care? X
  35. 35. do you care? X
  36. 36. do you care? X
  37. 37. do you care? X ?
  38. 38. response time is critical if you care about being hacked
  39. 39. if you aren’t monitoring yourself, you won’t know if you are
  40. 40. the future must be automated
  41. 41. Security + DevOps = ? •add security protections and automate them •continuously verify your protections with automation •add security monitoring but automate it •detect intrusions and automate response •score yourself on your security posture and automate the grading
  42. 42. Security + DevOps = ? •add security protections and automate them •continuously verify your protections with automation •add security monitoring but automate it •detect intrusions and automate response •score yourself on your security posture and automate the grading
  43. 43. authentications process activity network activity kernel modules file system actions appssystems services authentications db requests http requests
 AWS api calls SaaS api calls what to monitor
  44. 44. authentications process activity network activity kernel modules file system actions appssystems services authentications db requests http requests
 AWS api calls SaaS api calls what to monitor
  45. 45. authentications process activity network activity kernel modules file system actions appssystems services authentications db requests http requests
 AWS api calls SaaS api calls what to monitor
  46. 46. systems monitoring probably more than you ever wanted to know
  47. 47. questions to answer “is that guy running commands he shouldn’t be?” (e.g. why is anyone except chef user MAYBE running gcc on a prod system) ! “are accounts logging in from non-standard locations?” ! “are there anomalies in my traffic?” ! “did some process suddenly start making outbound connections?”
  48. 48. step 1: toolbox process auditing linux audit network flow libnetfilter_conntrack logins wtmp/audit/pam_loginuid
  49. 49. ! collect this data from every EC2 host you launch + store and correlate in a big data backend + build an audit log + analyze for suspicious behaviors and insights
  50. 50. linux audit
  51. 51. pros! ! super powerful built into your kernel (>=2.3) “relatively” low overhead apt-get install audit
  52. 52. it audits all of the things!
  53. 53. (sort of)
  54. 54. userland audit daemon and tools ! (e.g. redhat auditd, auditctl, etc) kernel thread queue kernel threads doing things audit messages the workings netlink socket /var/log/ audit/ audit.log
  55. 55. # files ! -w /etc/shadow -p wa ! # syscalls ! -a always,exit -F arch=ARCH -S init_module -S delete_module -k modules ! # follow executable ! -w /sbin/insmod -p x configuration
  56. 56. now for the cons…
  57. 57. type=SYSCALL msg=audit(1383252540.673:8711406): arch=c000003e syscall=59 success=yes exit=0 a0=c27fa8 a1=c24d48 a2=9f8008 a3=7fffc4553ce0 items=2 ppid=46247 pid=56107 auid=0 uid=0 gid=0 eu type=EXECVE msg=audit(1383252540.673:8711406): argc=2 a0="ping" a1="google.com" type=CWD msg=audit(1383252540.673:8711406): cwd="/opt/" ! type=PATH msg=audit(1383252540.673:8711406): item=0 name="/bin/ping" inode=1048904 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 ! type=PATH msg=audit(1383252540.673:8711406): item=1 name=(null) inode=1056827 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obtuse logging execve(‘ping google.com’)
  58. 58. THIS ONE WEIRD TRICK! ! enable rate limiting or it could ‘crash’ your box ! always be listening (or same) ! …relatively stable otherwise ;) auditctl -b 1000 -r 15000 # 1000 buffers, 15000 eps max
  59. 59. redhat auditd, the userland daemon, occasionally wtf-y
  60. 60. yeah…
  61. 61. alternative:! connect directly to netlink socket and write your own audit listener
  62. 62. [ { "exe": "/bin/cat", "comm": "cat", "ses": 10, "fsgid": 0, "sgid": 0, "egid": 0, "fsuid": 0, "suid": 0, "euid": 0, "gid": 0, "uid": 0, "auid": 4294967295, "pid": 31335, "ppid": 31334, "items": 2, "a3": "7fff3480e600", "a2": "654c88", "a1": "654bc0", "a0": "654dc0", "exit": 0, "success": "yes", "syscall": "execve", "arch": "c000003e", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "SYSCALL" }, { "a1": "eth0.dhclient", "a0": "cat", "argc": 2, "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "EXECVE" }, { "cwd": "/run/resolvconf/interface", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "CWD" }, { "ogid": 0, "name": "/bin/cat", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "PATH" }, json output! grouped sanely!
  63. 63. luajit! for filtering, transformation & alerting
  64. 64. performance improvements, yay! ! libevent + filtering + state machine parsing ! = 120% -> 10% CPU usage with AB 10k connections/sec
  65. 65. + authentications
  66. 66. # last jandre pts/1 dev.threatstack. Sun May 4 11:20 - 01:37 (14:17) jandre pts/0 dev.threatstack. Sun May 4 11:16 still logged in ! # in json format ! { type: ‘USER_PROCESS', pid: 777, line: ‘pts/1', id: 52, user: 'jandre', host: ‘dev.threatstack', exit_status: { termination: 0, code: 0 }, timestamp: Tue May 06 2014 03:50:03 GMT-0700 (PDT), address: ’10.0.0.10’ } } ! ! wtmp
  67. 67. # if pam is built with audit support… ! type=USER_AUTH msg=audit(1234877011.791:7731): user pid=26127 uid=0 1 auid=4294967295 ses=4294967295 msg='op=PAM:authentication acct="root" exe="/usr/sbin/ sshd" (hostname=jupiter.example.com, addr=192.168.2.100, terminal=ssh res=success)' plus audit
  68. 68. ! # /etc/pam.d/login, sshd, wherever you care ! /etc/pam.d$ grep loginuid * ! login:session required pam_loginuid.so sshd:session required pam_loginuid.so ! # now you get in /var/log/audit/audit.log: ! type=LOGIN msg=audit(1234877011.799:7734): login pid=26125 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=1172 ! # cat /proc/<login or sshd pid>/sessionid ! 1172 add pam_loginuid
  69. 69. if you do it correctly…
  70. 70. bonus!
  71. 71. apis: how to monitor
  72. 72. questions to answer who is using my AWS credentials, and from where? ! who is engaging anomalous or suspicious API activity that could indicate a breach? ! have my credentials been compromised? ! are IAM credentials assigned with appropriate permissions?
  73. 73. how? cloud trail to an s3 bucket for audit logging AWS APIs for auditing IAM permissions ! collect periodically and store using IAM role with limited permissions group by user account, activity, location alert by new or suspicious behaviors
  74. 74. idea: audit monitored data against an external source of truth
  75. 75. IAM credentials and permissions ! store definitions in a repository and audit against AWS periodically !
  76. 76. EC2 applications ! do they match definitions of whats installed in your config management? !
  77. 77. build checks for this auditing automatically into your CI
  78. 78. Thank You check out Cloud Sight @ www.threatstack.com

×