Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016


Published on

In a follow-up to the duo’s offensive focused talk “DevOops, How I hacked you”, they discuss defensive countermeasures and real experiences in preventing attacks that target flaws in your DevOps environments. In this talk, Chris and Ken describe common ways in which DevOps environments fall prey to malicious actors with a focus on preventative steps. The team will present their recommended approach to hardening for teams using AWS, Continuous Integration, GitHub, and common DevOps tools and processes. More specifically, the following items will be demonstrated:

-AWS Hardening
-AWS Monitoring
-AWS Disaster Recovery
-GitHub Monitoring
-Software Development Practices/Processes
-Secure use of Jenkins/Hudson
-Developer laptop hardening (OS X)

Published in: Internet

DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016

  1. 1. DevOops, Redux Chris Gates, Ken Johnson AppSec USA 2016
  2. 2. Background: KJ • I’m NOT Kevin Johnson
  3. 3. Background: KJ •I’m NOT Ken Bone
  4. 4. Background: KJ •I AM Ken Johnson •CTO of nVisium - @cktricky •Former US Navy •Topics I’ve talked about: – Rails Security (Railsgoat) – Building an AppSec Program – DevOops: Common Flaws in DevOps Tooling – Exploitation of Web Applications
  5. 5. Background: KJ • I run engineering (product) • I work for a security company • I have some concerns...same as you
  6. 6. Background: CG • Chris Gates, Sr. Security Engineer - Uber • Former Army • Topics I’ve talked about: – Breaking into Oracle, Windows, lots of stuff – Phishing – Low to Pwned – Purple Teaming – DevOops – Common Flaws in DevOps Tooling
  7. 7. Background: CG • Was a full time breaker • Now full-ish time fixer • Currently doing Blue Team stuff - <3 Python + REST APIs - Astonished at # of ppl who can’t Internet
  8. 8. About This Talk • Original talk DevOops was about breaking stuff • We were asked about “Proactive” measures in DevOps/Agile/CI-CD environments – Quick Story • We made a solution focused model based on “Common” architecture and needs
  9. 9. Before We Begin • Buckle up, lots of info coming your way • Q&A will be reserved for hallway discussions • Slides will have all the resources you need and will be available • Sections are broken up between Human, Host, and Infrastructure
  10. 10. Employee Intelligence (Human) Making it difficult (for employees) to allow attackers to walk into our environment
  11. 11. Monitoring External Services • Numerous ways for employees to accidently release data –Pastebin-like sites –Github •Gists •Code • Examples: –Slack tokens in github –AWS configs in .dotfiles back ups –Tokens in logs/dumps/snippets
  12. 12. Monitoring GitHub • How you could tackle the problem: –Use GitLab (internal) –Use gitolite (internal) –Use GitHub Enterprise (internal) –Use Phabricator (internal)
  13. 13. Monitoring GitHub • But you won’t, you’ll set up a private GitHub for you org like everyone else. – Now you need to monitor when people post your private stuff on their personal repo – It happens. A lot.
  14. 14. Monitoring GitHub • How you could tackle the problem: –Have employees join the your GitHub organization –Regularly crawl the list of members –Check out all their repos –Run regex against all files looking for known badness
  15. 15. Monitoring GitHub • Gitrob –
  16. 16. Monitoring GitHub • Gitrob
  17. 17. Monitoring GitHub
  18. 18. Monitoring GitHub
  19. 19. AWS Access Keys Example
  20. 20. AWS Access Keys Example
  21. 21. AWS Access Keys Example
  22. 22. Monitoring Pastebin* • Pastebin* – – –
  23. 23. Monitoring Goals • DumpMon
  24. 24. Monitoring Goals • For Pay Services
  25. 25. Monitoring Goals • For Pay Services
  26. 26. Monitoring Goals •For Pay Services -
  27. 27. Monitoring Goals GitMonitor - Some options they provide
  28. 28. Workstation Protection (Host) Protecting and monitoring employees on their development workstations (and servers too)
  29. 29. Host Protections Developer Laptop Hardening • osquery (OS X/Linux) • Doorman • BlockBlock • Little Snitch • CarbonBlack / Sysmon • Splunk • Simian
  30. 30. Host Protections • osquery ( • “osquery is an operating system instrumentation framework for OS X, Linux, and FreeBSD. The tools make low-level operating system analytics and monitoring both performant and intuitive.” • “osquery exposes an operating system as a high-performance relational database. This allows you to write SQL queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.”
  31. 31. Host Protections osquery • Adhoc • Scheduled • Schedule query • Collect logs • Review change • File Integrity Monitoring • Yara rules • Query packs
  32. 32. Host Protections osquery
  33. 33. Host Protections osquery
  34. 34. Host Protections • Doorman ( • “Doorman is an osquery fleet manager that allows administrators to remotely manage the osquery configurations retrieved by nodes.”
  35. 35. Host Protections
  36. 36. Host Protections • BlockBlock ( • Kernel hook to identify any time software wants to persist • Prompt to allow or deny • “The kernel extension tracks process creations, which are consumed by the daemon, which also monitors various persistence locations to detect any new items. Specifically the daemon (currently) watches for new kexts, launch daemon & agents, and new login items via the fsevents device (/dev/fsevents).”
  37. 37. Host Protections
  38. 38. Host Protections • Little Snitch ( • Host based firewall • Prompt to allow or deny and for how long • “Little Snitch intercepts these unwanted connection attempts, and lets you decide how to proceed.”
  39. 39. Host Protections
  40. 40. Host Protections • CarbonBlack ( • Host based agent • Monitor process create, writes, registry queries, net connections • Create rules/watchlist for known bad behavior –Mimikatz-->company_name:*gentilkiwi* –FileVault Encryption Disabled -->process_name:fdesetup cmdline:disable –Unsigned JAR exec-->process_name:*.jar digsig_result: (digsig_result:"Unsigned") –OSX dump user hashes-->process_name:dscl cmdline:ShadowHashData
  41. 41. Host Protections
  42. 42. Host Protections
  43. 43. Host Protections • Sysmon • paper_01.pdf • • • with-sysinternals-sysmon/ •
  44. 44. Host Protections • Splunk
  45. 45. Host Protections
  46. 46. Host Protections • ELK
  47. 47. Host Protections • ELK
  48. 48. Host Protections OSX Patch Management - Simian • “Simian is an enterprise-class Mac OS X software deployment solution.” • Allows you to push munki updates • Free / OSS • Runs on google cloud • Project:
  49. 49. Host Protections Why do we bring this up? • Some people aren’t aware you can perform free OSX patch management • There are a lot of OSX developer shops without an “enterprise budget” • Patch management is a no-brainer and security 101
  50. 50. Host Protections
  51. 51. Host Protections Simian Consists of 2 parts: • Client – Private and Public SSL Keys used to authenticate – Configuration unique per OSX client • Web Application/Server – Runs on Google Cloud – Keep in mind its free but… not for long (eventually costs a little for storage) Takes about a week to learn and get setup
  52. 52. Host Protections Web Application used to Manage Updates
  53. 53. Host Protections Client - DMG File
  54. 54. Host Protections Simian Recap: • Learning curve is moderately difficult IMO • Free-ish (eventually storage costs but still very minimal) • Useful for patch updates and monitoring clients systems for low disk space, uptime, etc.
  55. 55. Production Protection (Infra) Protecting and monitoring production environments (AWS)
  56. 56. My AWS Goals • Harden – Make it difficult to reach your AWS environment • Monitor – If your AWS environment is breached, you need to know and alert yourselves • Restore – Have the ability to reconstruct data/configs after a “hack”
  57. 57. AWS’s Plan • Took the AWS Security Fundamentals Course and… – Fortunately, our strategy lines up with AWS recommendations – You are responsible for leveraging the tools AWS provides (financially) – Your configuration… that is on you – fundamentals/
  58. 58. AWS Hardening Basics Making it difficult (for attackers) to reach our environment
  59. 59. Hardening Checklist 1. Don’t Use The Root Account! 2. Disable Access Keys for Root Account 3. Multi-Factor Authentication 4. API + MFA 5. Strong Password Policy
  60. 60. Don’t Use Root Account • Every AWS env has a root account, only necessary to use for very specific circumstances • When these circumstances arise, notify your team that the account will be used • We will discuss why this is important when we talk about CloudWatch metrics
  61. 61. Disable/Delete Root Account Access Keys • Just delete them if they exist – Disable the access keys in the event you are unable to delete them completely for some reason • Make sure your admins have a (verbal/written) policy that states “we don’t create access keys for the root account”
  62. 62. MFA • If credentials are stolen or guessed, we want a second layer of protection • You can use apps or hardware to do this – Google Authenticator (Apps) – Gemalto (Hardware) • Find the full list of MFA devices here: • This is so ridiculously easy to do, everyone should do it
  63. 63. MFA See the published slide deck for step by step instructions
  64. 64. MFA • At this point, it's worth mentioning that non- administrators or those without IAM privileges cannot enable MFA on their own account • Why is this a problem? Well, they need to be able to enable MFA on their own device… not the administrator’s • Fortunately, we have a solution!
  65. 65. MFA
  66. 66. MFA • Okay so that wasn’t the easiest to read, so here is the link: redentials_delegate-permissions_examples.html#creds- policies-mfa-console • Basically this IAM policy allows a user to manage their *OWN* MFA device
  67. 67. MFA (for Root Account) • Need a shared MFA for root? TOTP! • Recommend using something like 1password for teams, can share the TOTP code:
  68. 68. API + MFA • You have the ability to place a restriction where resources can only be interacted with if the user has authenticated with MFA • This helps prevent (ab)use should someone steal access keys or credentials
  69. 69. API + MFA • This entry enforces MFA for Web/API • Do this for Admin & Power-User groups at a minimum
  70. 70. API + MFA • Truth be told, doing this can be painful at first • Things that used to work, might not (via the API) • Fortunately, we have some answers for you • Firstly, let’s discuss STS or SecurityToken Service
  71. 71. API + MFA • Leverage STS in order to interact with the AWS API should this MFA restriction be placed on resources (and it should ☺ ) • Example of using STS:
  72. 72. API + MFA Output of script
  73. 73. API + MFA Use the creds to leverage tools like ec2-api-tools (-O <access key id>–W <secret> and –T <session token>)
  74. 74. API + MFA And in case you don’t like Ruby…
  75. 75. API + MFA • ElasticBeanstalk does not work with STS. Le Terrible. • However, there is a workaround, use CodePipeline • Very simple process to setup but only works with: – GitHub – AWS CodeCommit – Amazon S3
  76. 76. Password Policy • Password policies are important because historically people do not choose complex passwords • MFA should help, but we’re talking about a layered approach • Again, making our AWS environment harder to reach
  77. 77. Example Password Policy
  78. 78. Hardening Recap • Make credentials hard to guess • If guessed or stolen, we still have MFA • Remember MFA only protects against the web and NOT the API… unless you change your policies and use STS • Root account is King, protect your King
  79. 79. Hardening Recap • Things we did not (and won’t discuss) – S3 bucket policies – Security Group configurations – SSH Key Management – Encrypting Data (Volumes, S3 buckets) • Trusted Advisor – Use it, because it catches a lot of “low hanging fruit” style issues
  80. 80. Hardening Recap • Links to resources that discuss the items we’re not covering: – ty_Checklist.pdf – Security-Check-List_eng.pdf – security-best-practices • Frankly you can’t throw a rock without hitting some basic info regarding AWS Security Checklists
  81. 81. AWS Monitoring Detecting malicious activity
  82. 82. AWS Monitoring • Assuming hardening (prevention) has failed, how would we know? • Luckily, AWS provides several services which alert to anomalies • We will walk through examples of using these services, but ultimately decide what is right for you • Fair warning, some of these services will provide a lot of noise
  83. 83. AWS Monitoring 4 important services: • CloudTrail – Logs • SNS – Notifications • Config – Alerts for modifications & noncompliance • CloudWatch – Alerts for specific types of behavior
  84. 84. AWS Monitoring
  85. 85. AWS Monitoring
  86. 86. AWS CloudTrail
  87. 87. AWS Monitoring (CloudTrail) • CloudTrail is primarily used for log collection • Other services like CloudWatch, for example, use those logs to filter relevant data
  88. 88. AWS Monitoring (CloudTrail) Pretty easy, first turn it on..
  89. 89. AWS Monitoring (CloudTrail) Configure the log group
  90. 90. AWS Monitoring (CloudTrail) Allow the creation of an IAM role by CloudTrail
  91. 91. AWS Monitoring (CloudTrail) • At this point you have cloudtrail enabled • Next step, BEFORE moving to CloudWatch or Config, is configuring SNS topics
  92. 92. AWS SNS
  93. 93. AWS Monitoring (SNS) • Fantastic offering, <3 it – Examples of ways to be notified by SNS • SMS • Email • JSON Post to your Application’s API endpoint
  94. 94. AWS Monitoring (SNS) • Receive SMS/Email/Slack notifications for important events • ^ This is so you get immediate notifications • You can have multiple subscribers, I’d suggest you use that functionality • Basic gist? Receive immediate updates for things you want to see… immediately ☺
  95. 95. AWS Monitoring (SNS) Create a topic
  96. 96. AWS Monitoring (SNS) Create Subscription
  97. 97. AWS Monitoring (SNS) Create SMS (or whatever, but in this case, SMS)
  98. 98. AWS Monitoring (SNS) Example of creating email subscription… bottom line you can have multiple ways of notifying people
  99. 99. AWS Config
  100. 100. AWS Monitoring (Config) • Config: – AWS resource inventory, configuration history, and configuration change notifications – Can either design custom Config rules or use managed (pre-packaged) AWS Config rules – Discovery -Change Management – Compliance -Incident Response
  102. 102. AWS Monitoring (Config) • Examples of things you can have alerts set for: – Change in Firewall (Security Group) ports – Changes in VPC – Any change… at all
  103. 103. AWS Monitoring (Config) Go to the Config service and choose resources to track
  104. 104. AWS Monitoring (Config) Or choose to track everything
  105. 105. AWS Monitoring (Config) Create a bucket, create an SNS topic (…we’ll discuss next)
  106. 106. AWS Monitoring (Config) Allow the role to be created and you’re all set!
  107. 107. AWS CloudWatch
  108. 108. AWS Monitoring (CloudWatch) • We can be very particular here about what it is we want to see • Some very interesting things you can monitor • Some examples: – Billing Alerts (Important for detection of abuse or mistakes) – Track Root Account Usage – Failed login attempts
  109. 109. Billing Alarm
  110. 110. AWS Monitoring (CloudWatch - Billing) • Used to prevent abuse or mistakes from costing your organization money • Analyze and approximate your monthly spend • Configure via CloudWatch • Use SNS for instantaneous alerting
  111. 111. AWS Monitoring (CloudWatch - Billing) Navigate to billing & cost management; enable billing alerts
  112. 112. AWS Monitoring (CloudWatch - Billing) Create an SNS topic
  113. 113. AWS Monitoring (CloudWatch - Billing) Subscribe to Topic
  114. 114. AWS Monitoring (CloudWatch - Billing) Navigate to CloudWatch -> Metrics -> Billing
  115. 115. AWS Monitoring (CloudWatch - Billing) Choose USD/Estimate Charges -> Create Alarm
  116. 116. AWS Monitoring (CloudWatch - Billing) Set price point, SNS topic, and create alarm
  117. 117. AWS Monitoring (CloudWatch - Billing) Exact steps to enable can be found here: v2/free-tier-alarms.html
  118. 118. Root Login Alarm
  119. 119. AWS Monitoring (CloudWatch – Root Login) • Remember how I said don’t use the Root account routinely? • BUT… if this account is used, you should know about it • This is the reason you’ll want to notify others (who receive SNS alerts) of the fact you are about to use the account
  120. 120. AWS Monitoring (CloudWatch – Root Login) Choose log group, create metric
  121. 121. AWS Monitoring (CloudWatch – Root Login) Define Logs Metric Filter
  122. 122. AWS Monitoring (CloudWatch – Root Login) Assign/Create Filter
  123. 123. AWS Monitoring (CloudWatch – Root Login) Click “Create Alarm”
  124. 124. AWS Monitoring (CloudWatch – Root Login) Define Alarm and you’re good…
  125. 125. AWS Monitoring (CloudWatch – Root Login) Exact steps (with pics) exist here: 374D/How-to-Receive-Notifications-When-Your-AWS- Account-s-Root-Access-Keys-Are-Used
  126. 126. Failed Login Alarm
  127. 127. AWS Monitoring (CloudWatch – Failed Logins) • In the event someone is trying to break in, let’s alert ourselves to this! • Failed logins typically suggest either someone forgot their password or… someone is trying to guess yours
  128. 128. AWS Monitoring (CloudWatch – Failed Logins) • In the interest of time… the steps are pretty much the same as the root login alarm • The Regex Filter however, is different
  129. 129. AWS Monitoring (CloudWatch – Failed Logins) Relevant filter pattern
  130. 130. AWS Monitoring (CloudWatch – Failed Logins) • Exact steps exist here: test/userguide/cloudwatch-alarms-for- cloudtrail.html#cloudwatch-alarms-for- cloudtrail-signin
  131. 131. Unauthorized Activity Alarm
  132. 132. AWS Monitoring (Unauthorized Activity) • Remember the aws-interrogate tool? • This alarm is the antidote • Alerts us when someone is trying to access something in AWS, and does not have permissions
  133. 133. AWS Monitoring (Unauthorized Activity) • Again, in the interest of time, steps are same as root login • Regex is of course, different
  134. 134. AWS Monitoring (Unauthorized Activity) Set up regular expression
  135. 135. AWS Monitoring (Unauthorized Activity) What happens when we run interrogate
  136. 136. AWS Monitoring (Unauthorized Activity) The result of doing that is a nice nifty email to the engineering & security team
  137. 137. AWS Monitoring (CloudWatch) – Filter Patterns • If you’d like to create your own custom filter patterns, here is a resource for that: veloperGuide/FilterAndPatternSyntax.html
  138. 138. AWS + Splunk
  139. 139. AWS + Splunk • Splunk is a pretty great resource for monitoring activity – Two separate plugins: • Splunk App for AWS – • Splunk Add-On –
  140. 140. AWS + Splunk • Examples of things you can view: – Billing – Topology – Usage – IAM Activity – SSH Key Pair Activity – User Activity – Network ACL(s) – VPC Activity – and a lot more…
  141. 141. AWS + Splunk
  142. 142. AWS + Splunk
  143. 143. AWS + Splunk
  144. 144. AWS + Splunk • Splunk will need an AWS account in order to retrieve data • Create account(s) for Splunk, grab the necessary permission policy from here: WS/ConfigureAWSpermissions
  145. 145. AWS + Splunk Configure AWS App for Splunk, add account(s), configure each input accordingly:
  146. 146. AWS + Splunk •To view things like IAM Activity… –Subscribe to a cloudtrail log via SNS –Utilize SQS and subscribe SQS to an SNS Topic
  147. 147. Monitoring Recap • Alert yourself when things change • This will get noisy, find a way to filter that which is important – If it’s a high risk event, send an SMS/Slack/Email blast • At a minimum, alert yourself when odd things occur… like: – Billing increases past your normal spend – When somebody authenticates as Root – When someone has a login failure
  148. 148. Monitoring Recap • Interesting Quora thread: – reduce-the-amount-I-need-to-pay • Highlights from the article: – AWS has “a review board of sorts” to determine if you should be refunded – Bots are scouring GitHub searching for exposed access keys – One of the more AWS-seasoned responders mentioned doing part of what we discussed here today to avoid it – A decent number of the people posting on this thread said “Yes, happened to me too”
  149. 149. AWS Restoration & Recovery Plan to fail, just don’t fail to plan
  150. 150. AWS Restoration & Recovery • Do not USE AWS TO BACKUP YOUR AWS • Offsite backups (meaning, off AWS site) • Common things to back-up: – Databases/ Snapshots – S3 Buckets – EBS Volumes – CloudFormation Templates
  151. 151. AWS Restoration & Recovery • Resources: – solutions-for-aws-ec2-instances – – backup-and-recovery.html
  152. 152. AWS Incident Response Plan to fail, just don’t fail to plan
  153. 153. AWS Incident Response • Could be its own talk • Scout 2 -- •Andrew Krug & Alex McCormack – Hardening AWS Environments and Automating Incident Response –
  154. 154. Contact Chris Gates Twitter: @carnal0wnage Blog: Ken Johnson Twitter: @cktricky