Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(SACON) Madhu Akula - Automated Defense Using Cloud Service Aws, Azure, Gcp

1,756 views

Published on

(SACON) Madhu Akula - Automated Defense Using Cloud Service Aws, Azure, Gcp

Published in: Technology
  • Yes you are right. There are many essay writing services available now. But almost services are fake and illegal. Only a genuine service will treat their customer with quality essay papers. HelpWriting.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Discover How to Cure uterine fibroids and PCOS At Any Age, Even If You’ve Tried Everything And Nothing Has Ever Worked For You Before ◆◆◆ http://t.cn/Aig7V1M7
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

(SACON) Madhu Akula - Automated Defense Using Cloud Service Aws, Azure, Gcp

  1. 1. 1.1 2.1 2.2 2.3 2.4 2.5 3.1 3.2 3.2.1 3.2.2 3.2.3 4.1 4.2 4.3 5.1 5.2 5.3 5.4 6.1 6.2 6.3 6.4 6.5 Table of Contents Welcome Introduction Introduction About the trainers - Madhu Akula Disclaimer Agenda Training Preview Getting Started Cloud Accounts Access Cloud accounts setup AWS Azure Google Cloud Cloud Security AWS Security Azure Security GCP Security ELK Stack Setup ELK Stack Alerting Kibana 101 Automation SCENARIO 1 - SSH Bruteforce Introduction Before the Attack Attack Serverless Defence Configuring ELK Stack 2
  2. 2. 6.6 6.7 6.8 6.9 7.1 7.2 7.3 7.4 7.5 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 9.1 9.2 9.3 9.4 9.5 9.6 9.7 10.1 10.2 10.3 10.4 After serverless defence Serverless Explanation Automation Use cases and ideas SCENARIO 2 - Auditing CMS Introduction Configuring ELK Stack Analyzing Wordpress activity Automation Use cases and ideas SCENARIO 3 - IAM Defence Introduction Serverless Defence Before the Attack Attack After the attack Serverless Explanation Automation Use cases and ideas SCENARIO 4 - Container defense Introduction Attack Serverless defence After defence Serverless Explanation Automation Use cases and ideas Tear Down AWS Azure GCP Automation 3
  3. 3. 11.1 11.2 12.1 About Us About Appsecco Upcoming Trainings and Conferences References & Resources References & Resources 4
  4. 4. Welcome 5
  5. 5. Introduction Welcome to "Automated Defense using Cloud Services for AWS, Azure and GCP". This defence focused, hands on training will set you on the path to using serverless and elastic stack with help of cloud services to defend against cloud infrastructure It helps you to get started about building automated defence systems for your environments based on your needs by understanding the approach and methodology The idea behind the Automated Defence is to reduce the bottle-neck of human reaction time to security monitoring by automating defensive actions to have a near real-time response Abstract We live in cloud first era where the cloud is increasingly our first choice of deployment due to its convenience and scalability. Monitoring for attacks and defending against them in real-time is crucial but defending your cloud infrastructure during attacks can prove to be a nightmare even with the solutions currently available in the market. In this training we will teach how to defend your cloud infrastructure using Serverless technologies and Elastic Stack. Elastic Stack collects, analyses logs and triggers alerts based on a pre-configured rule-set and the Serverless stack drives defence to perform automated blocking. The world is advancing towards accelerated deployments using DevOps and Cloud technologies. Automated defence solves modern world security challenges using near real-time alerting system, serverless technologies and centralised monitoring system. Prerequisites Students will need trial accounts in AWS, Azure and GCP with administrative access with billing enabled to have a hand-on experience during the training. Trainers Trainer Details Name : Madhu Akula Twitter : @madhuakula Email : madhu@appsecco.com Introduction 6
  6. 6. Madhu Akula Madhu Akula is a security ninja, published author and Security Automation Engineer at Appsecco. He is passionate about Cloud Native, DevOps and security and is an active member of the international Security and DevOps communities. His research has identified vulnerabilities in over 200 companies and organisations including; Google, Microsoft, LinkedIn, eBay, AT&T, WordPress and Adobe, etc. He is co-author of Security Automation with Ansible2(ISBN-13: 978-1788394512), which is listed as a technical resource by Red Hat Ansible. Madhu frequently speaks and runs technical sessions at security events and conferences around the world including; DEF CON (24 & 26), Blackhat USA 2018, USENIX LISA 2018, Appsec EU 2018, All Day DevOps (2016, 2017 & 2018), DevSecCon (London, Singapore and Boston: 2016, 2017 & 2018), DevOpsDays India, c0c0n (2017 & 2018), Serverless Summit, null and multiple others. Some of the trainings/workshops by Madhu Akula include Automated Defense using Cloud Services for AWS, Azure and GCP - Blackhat USA 2018, 2019 Attacking & Auditing Docker Containers Using Open Source - Defcon 26, OWASP Bay Area Meetup Attacking & Auditing Docker Containers - USENIX LISA 2018, DevSecCon London 2018, c0c0n XI Building visualization platforms for OSINT data using open source solutions - Recon Village 2018 Automated Defense using Serverless for AWS, Azure and GCP - Appsec EU 2018 Breaking and Owning Cloud Servers and Applications - NULLCON Goa 2018 Ninja Level Infrastructure Monitoring - Defcon 24 and DevSecon London 2016 Automated Infrastructure Security Monitoring & Defence - DevSecCon Singapore 2017 Real World Security Monitoring & Automated Defence for almost free - DevSecCon Boston and DevSeCon London 2017 Monitoring & Defending Infrastructure Security Attacks - c0c0n X Linux Container Security - Null Bangalore An Introduction to Containers using Docker and using it for Security Automation - Null Bangalore Automating Documentation, Presentation, KB using Markdown - Null Bangalore Automated infrastructure security monitoring & defence - Null Bangalore Some of the talks given by Madhu Akula include Container Security Monitoring using Open Source - All Day DevOps 2018, Online Webinar Continuous security monitoring in CI and CD pipelines - iwomm 2.5: Continuous Delivery Meetup, London Modern Security Operations aka Secure DevOps - All Day DevOps 2017 Automated Defence for Cloud Security in AWS using Serverless - Serverless Summit 2017 DevOps principles to build your lean startup - Startup Leadership Program Developers guide to security & operations: Introducing DevSecOps - Software Security Bangalore Meetup Automated Infrastructure Security Monitoring using FOSS - All Day DevOps 2016 Infrastructure Security Monitoring - DevOps Days India 2016 Cloud Security for everyone - SDN + IoT + Network Virtualization Enthusiasts Meetup NodeJS Security - Null Bangalore Web & Cloud Security in the Real World - Keynote speaker at CompTIA Bangalore My bug hunting with open source - Hill hacks 2015 Hardening routers & switches - Null Dharamshala About the trainers - Madhu Akula 7
  7. 7. Basics of networking - Null Dharamshala Published Works of Madhu Akula include Cover Details Book - Security Automation with Ansible2, Published by PacktPub December 2017, ISBN 9781788394512 Online Account Details Twitter @madhuakula LinkedIn Madhu Akula About the trainers - Madhu Akula 8
  8. 8. Disclaimer The attacks covered in the training are for educational purposes only. Do not test or attack any system outside of the scope of this training lab unless you have express permission to do so The snippets, commands and scripts used throughout the training are not production-ready, may not be bug-free and are not guaranteed in any way Disclaimer 9
  9. 9. Agenda Here is the high level overview of how next two days will look like Automated Defense using Cloud Serivces for AWS, Azure and GCP Introduction Environment Setup Cloud Account Setup Elastic Stack Setup Scenario-1 : SSH bruteforce detection and defence Scenario-2 : Content management system audit analysis Scenario-3 : IAM CloudTrail logs to defend against stolen credentials Scenario-4 : Container logs to audit Kubernetes security Tear down Wrap up References & Resources Agenda 10
  10. 10. Training Preview Training Preview 11
  11. 11. Cloud service accounts Services we will be using in AWS IAM EC2 S3 Lambda Cloud Watch Cloud Trail VPC API Gateway Dynamo DB Services we will be using in Azure Resource Group Virtual Machine Virtual Network Network Security Group Public IP Address Azure Cosmos DB Azure Functions Services we will be using in GCP Google Cloud Shell Goolge Compute Engine Google Kubernetes Engine IAM Cloud Functions App Engine LoadBalancer Stack driver Search Engine :P Cloud Accounts Access 12
  12. 12. Cloud accounts setup We will now configure our cloud account credentials in the student VM to be able to deploy the services we will be using. Cloud accounts setup 13
  13. 13. Setting up AWS CLI with IAM credentials Introduction The primary AWS account, also called the root account, is very powerful in terms of access. To avoid losing its keys or secrets, we will create a IAM Administrator account which will have the same privileges as a AWS root account except for access to certain features like billing which we can anyways access using the root account. Steps to create an IAM user Search for IAM in the services Click on users > Add user Create a user called iamadmin with the following settings AccessType: Programmatic access and AWS Management Console access Console Password: Select Custom password Provide a strong alphanumeric character Uncheck require password reset We are only asking you uncheck require password reset to do this for the purposes of the training AWS 14
  14. 14. Click on 'Attach existing policies directly' and select 'AdministratorAccess' Click Next and Create User AWS 15
  15. 15. Copy and save the following in your text editor for later use 1. Access key ID 2. Secret access key 3. Unique sign in URL (Bookmark this link) Steps to Configure AWS CLI Run the following command to configure the aws cli Ensure that you run this command in the Training VM aws configure You will need to provide the access key ID and secret access key Type the following values Default region name [None]: us-east-1 (YOU MUST PROVIDE us-east-1 ) Default output format [None]: json These credentials get stored at ~/.aws/credentials Validation Run the following command to validate the AWS configuration to ensure that account is added and set as default Ensure that you run this command in the Training VM aws sts get-caller-identity Additional Information Setting up access using CLI AWS 16
  16. 16. Setting up Azure CLI with credentials Azure CLI is optimized for managing and administering Azure resources from the command line, and for building automation scripts that work against the Azure Resource Manager. Steps to Configure AWS CLI Run the following command to configure the azure cli Ensure that you run this command in the Training VM az login Open the URL in your browser and enter the returned code to go to the next step https://microsoft.com/devicelogin Complete the registration by selecting the free trail training account to confirm Azure 17
  17. 17. After successful authentication, we can see the below output in the command prompt Validation Run the following command to validate the Azure configuration to ensure that account is added and set as default Ensure that you run this command in the Training VM az account list Azure 18
  18. 18. Additional Information Log in with Azure CLI Azure 19
  19. 19. Google Cloud Platform The gcloud auth command group lets you grant and revoke authorization to Cloud SDK (gcloud) to access Google Cloud Platform. Authenticating via gcloud CLI Run the following command in training vm to configure gcloud cli Ensure that you run this command in the Training VM gcloud auth login Copy the link and open in your browser. Make sure you login to the account which you are using the free trail Google Cloud 20
  20. 20. Give the permission by clicking Allow Google Cloud 21
  21. 21. Copy the code for pasting in the console Google Cloud 22
  22. 22. Paste the copied code and press enter to continue Validation Run the following command to validate the gcloud configuration to ensure that account is added and set as default Ensure that you run this command in the Training VM gcloud config list Google Cloud 23
  23. 23. Additional Information gcloud auth login google auth Google Cloud 24
  24. 24. AWS Security Five core areas of Cloud Security According to this whitepaper, security in the cloud is composed of five areas 1. Identity and Access Management 2. Detective Controls 3. Infrastructure Protection 4. Data Protection 5. Incident Response Mapping these areas to AWS Services and Security Concepts we covered in the training Area Services Identity and Access Management AWS IAM Detective Controls AWS Config, AWS CloudWatch, AWS S3, AWS Inspector Infrastructure Protection AWS VPC, AWS S3 Data Protection N/A Incident Response N/A Cloud Security Architecture Building Blocks Block Use Case AWS VPC Logically seperate network AWS IAM Secure access to resources and services for people and computers AWS CloudWatch See logs and take actions AWS CloudTrail Track API requests and monitor and notify AWS Config/Cloud Custodian Validate security policy and remediate automatically Other relevant AWS whitepapers to read and learn from AWS Security Pillar Whitepaper AWS Security Best Practices AWS Auditing use of AWS Checklist AWS Security 25
  25. 25. AWS Security 26
  26. 26. Azure Security The features listed following are capabilities you can review to provide the assurance that the Azure Platform is managed in a secure manner. Links have been provided for further drill-down on how Microsoft addresses customer trust questions in four areas: Secure Platform, Privacy & Controls, Compliance, and Transparency. Available security technical capabilities to fulfil user (Customer) responsibility - Big picture Microsoft Azure provides services that can help customers meet the security, privacy, and compliance needs. The Following picture helps explain various Azure services available for users to build a secure and compliant application infrastructure based on industry standards. The built-in capabilities are organized in six (6) functional areas: Operations Applications Storage Networking Compute Identity Reference https://docs.microsoft.com/en-us/azure/security/azure-security https://docs.microsoft.com/en-us/azure/security/azure-security-technical-capabilities Azure Security 27
  27. 27. Azure Security 28
  28. 28. GCP Security Google cloud infrastructure builds security through progressive layers that deliver true defense in depth. Reference https://cloud.google.com/security/infrastructure/design/ https://cloud.google.com/security/overview/whitepaper https://cloud.google.com/security/ GCP Security 29
  29. 29. ELK Stack Ref: https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html Elasticsearch, Logstash and Kibana Different open source modules working together Helps users/admins to collect, analyse and visualize data in (near) real-time Each module fits based on your use case and environment Components of the stack Elasticsearch Logstash Kibana Beats Elasticsearch ELK Stack 30
  30. 30. Ref: https://www.elastic.co/products Distributed and Highly available search engine, written in Java and uses Groovy (now started painless scripting) Built on top of Lucene Multi Tenant with Multi types and a set of APIs Document Oriented providing (near) real time search Logstash Ref: https://www.elastic.co/products Tool for managing events and logs written in Ruby Centralized data processing of all types of logs Consists of 3 main components Input : Passing logs to process them into machine understandable format Filter : Set of conditions to perform specific action on a event Output : Decision maker for processed events/logs Basic Logstash Configuration input { stdin {} file {} ... } filter { ELK Stack 31
  31. 31. grok {} date {} geoip {} ... } output { elasticsearch {} email {} ... } Kibana Ref: https://www.elastic.co/products Powerful front-end dashboard written in JavaScript Browser based analytics and search dashboard for Elasticsearch Flexible analytics & visualisation platform Provides data in the form of charts, graphs, counts, maps, etc. in real-time Beats Ref: https://www.elastic.co/products Lightweight shippers for Elasticsearch & Logstash Capture all sorts of operational data like logs or network packet data It can send logs to either Elasticsearch, Logstash ELK Stack 32
  32. 32. Different types of Beats Filebeat: Log Files Metricbeat: Metrics Packetbeat: Network Data Winlogbeat: Windows Event Logs Auditbeat: Audit Data Heartbeat: Uptime Monitoring Filebeat sample configuration filebeat.inputs: - type: log enabled: true paths: - /var/log/auth.log tags: ["sshlog"] - type: log enabled: true paths: - /var/log/nginx/access.log tags: ["weblog"] output.logstash: hosts: ["localhost:5044"] ELK Stack for Security Monitoring & Alerting It helps to parse large amount of log data We can aggregate and correlate the data from different types of log formats Centralized way to look into entire logs Provides near real-time search and visualization capabilities ELK Reference Guide We can use the below Gitbook with detailed instructions for references to the ELK stack setup and usage. https://appsecco.com/books/elk-workshop ELK Stack 33
  33. 33. Alerting We can set up a notification system to let users/admins know that a pattern match has occurred. Logstash output plugin alerting via (Email, Pager duty, JIRA, etc.) An open source alerting for elasticsearch by Yelp called elastalert Another open source project by Etsy 411 X-Pack (commerical offering by Elastic) Custom scripts ElastAlert ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. Simple ElastAlert rule to detect ssh bruteforce attacks es_host: localhost es_port: 9200 name: "SSH Bruteforce Login Alert" type: frequency index: filebeat-* num_events: 12 timeframe: minutes: 3 # For more info: # http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html filter: - query: query_string: query: 'tags: "sshlog" AND login: "Failed" AND username: ("root" OR "ubuntu")' alert: - slack - command slack: slack_webhook_url: "https://hooks.slack.com/services/xxxxx" slack_username_override: "attack-bot" slack_emoji_override: "robot_face" command: ["/usr/bin/curl", "https://LAMBDAENDPOINTGOESHERE/%(ip)s"] realert: minutes: 0 Rule Types Any Blacklist Whitelist Change Frequency Spike Flatline New Term Alerting 34
  34. 34. Cardinality Metric Aggregation Percentage Match Alert Types Command, HTTP POST Email, SNS, Stomp Jira, Gitter, ServiceNow OpsGenie, VictorOps, PagerDuty Twilio, Telegram HipChat, Slack, MS Teams Alerting 35
  35. 35. Kibana 101 We will familiarize ourselves with Kibana dashboard now. For now, we already have system logs from the ELK VM. Login to your Kibana dashboard Interactive Hands-On We will practice the following in a hands-on manner. Follow the trainer's instructions and raise any questions you have Index Creation We will create an index pattern for our metricbeat log data so that we can query and build visualisations around them We need to select the timestamp field that we will be using for this index Discovery We will discover and observe the logs Kibana 101 36
  36. 36. Here we see a single log entry in JSON format Custom Search We will use Apache Lucene query to analyze the log data Kibana 101 37
  37. 37. Time Filters We will try out various time filters to restrict our search space Creating Visualization We will create visualizations based on our search queries Selecting the search query for the current visualization Kibana 101 38
  38. 38. Creating Pie Chart We will create a pie chart to represent our visualization Creating Dashboard We will create a dashboard to feature our visualizations and queries Kibana 101 39
  39. 39. Sharing Dashboard We can share the dashboard so that it can be used by others Dev Tools We will explore Dev Tools and try out manual queries to Elastic search Try the following queries to get the cluster status GET _cluster/health GET _cluster/state Management We explore the management tab to manage our custom searches, reports, import and export Kibana 101 40
  40. 40. Generating dashboards for metricbeat We will login to the ELK stack and generate the metricbeat dashboards sudo metricbeat setup --dashboards Now, we can see the system dashboard generated in live Kibana 101 41
  41. 41. Automation deploy-elk-stack-infra Custom Ansible playbook to setup Elasticsearch Logstash Kibana Nginx ElastAlert Beats (Filebeat, Metricbeat) Created a custom AMI using Ansible provisioner and published the final AMI using Packer Used the Terraform to setup AWS infrastructure for ELK stack VPC Subnet Route Tables Internet Gateway Elastic IP Security Group SSH Key pair EC2 Local provisioner Remote provisioner Automation 42
  42. 42. Output Created a simple bash script to Initialise Terraform using stored AWS credentials Deploy the infrastructure using Terraform plan Automation 43
  43. 43. SSH Bruteforce Defence Overview In this scenario, We will setup our infrastructure, which consists of a VM with SSH password authentication We will setup the serverless components required for Automated Defence We will perform a bruteforce attack on the SSH service and see how to defend against the attack using serverless and automated defence approach Introduction 44
  44. 44. Introduction 45
  45. 45. Before the Attack We will look at current state of our infrastructure and logs before the attack. Network ACL Lets observe our Network ACL for our infrastructure VPC subnet Navigate to the VPC -> Network ACL dashboard by going here https://console.aws.amazon.com/vpc/home?region=us-east-1#acls: Please ensure that you are logged in to your aws account before visiting the link above Select the ACL belonging to adef-lab-vpc as shown Observe that everything is allowed at this point Before the Attack 46
  46. 46. Attack We will now attack the Infra VM SSH service by running a bruteforce attack using hydra utility Running the bruteforce attack Run the following command to start the bruteforce attack against the SSH service of the infrastructure vm Ensure that you run this command in the Training VM hydra -V -L /opt/usernames.txt -P /opt/passwords.txt infra.domain.com ssh This script will Run a SSH bruteforce attack with the hydra (Hydra is a brute force password cracking tool) using given wordlists The usernames.txt and passwords.txt files are already placed in your system under /opt/ directory If you see any error, please inform one of the trainers You should see something like this Kibana Dashboard Lets observe the SSH login logs and visualize the attack. We are able to see the logs here because the infra VM has been configured to send logs to the ELK VM by default Navigate to the Kibana dashboard by using the link Create new index pattern in your elk stack and give the index name pattern and select the timestamp. We index the data so that it can be queried and thus visualized Now navigate to discover and select the filebeat pattern to see the near real-time logs. Attack 47
  47. 47. Now, we can see the logs coming in near real-time and we can also use Apache lucene queries to filter the data by selecting the appropriate filters as shown in the screenshots. Query for all login attempts against the users root , ubuntu under sshlog Import the ssh-custom-dashboard.json dashboard. This dashboard will help with visualizing the SSH attack in real-time. Attack 48
  48. 48. References Apache Lucene Query Syntax Kibana Dasboards Attack 49
  49. 49. Serveless Defence We will now deploy the serverless defence that will detect, block and alert us about the attack automatically. Deploying serverless defence Run the following script to deploy serverless defence for the scenario-1 Ensure that you run this command in the Training VM deploy-scenario-1-defence This script will Deploy DynamoDB tables and Lambda Functions used for the serverless defence Print the lambda endpoints that we will use for serverless defence Please note down this information, as we will use this later If you see any error, please inform one of the trainers Serverless Defence 50
  50. 50. Configuring ELK stack We need to configure our ELK stack to trigger a defensive action by making a request to our serverless endpoint. We use ElastAlert, an open source tool, to trigger defensive actions when the conditions defined in the rules are met. Now we have to ensure that the lambda endpoint generated by defence script has to be updated in ElastAlert configuration. Get the endpoint from the student vm by running below command echo $scenario_1_endpoint_ip SSH into the ELK VM Now we'll configure the endpoint in the ElastAlert configuration file to trigger a HTTP request to our serverless endpoint with an ip address to block This command must be run in the ELK VM. If you are not familiar with vi , please use nano instead sudo vi /opt/elastalert/rules/ssh-bruteforce-alert.yml Now we have to restart the ElastAlert service to apply the changes This command must be run in the ELK VM sudo systemctl restart elastalert.service Configuring ELK Stack 51
  51. 51. After serverless defence Let's attack the infra ssh service again to see that serverless defence happening in near real-time Ensure that you run this command in the Training VM hydra -L /opt/usernames.txt -P /opt/passwords.txt infra.domain.com ssh Now we can see the near-real time ssh logs in our Kibana dashboard The attack is now in progress and has most likely been blocked automatically. We shall verify the same. Slack Alert You would've received a slack alert about the IP being blocked You will also receive another slack notification once the IP address has been unblocked Network ACL Lets observe our Network ACL for our infrastructure VPC subnet Navigate to the VPC -> Network ACL dashboard by going here https://console.aws.amazon.com/vpc/home? region=us-east-1#acls: Please ensure that you are logged in to your aws account before visiting the link above Select the ACL belonging to adef-lab-vpc VPC as shown After serverless defence 52
  52. 52. Observe that our student VM IP has been blocked. There may be other IP addresses that have been blocked due to bruteforce attacks on the wild The following is the dynamo DB screenshot of automated defence in action Action history Lets check the actions performed by our serverless defence by invoking the actionhistory endpoint We need to add the accessToken parameter to the URL before we can use it to query for the actions taken Ensure that you run this command in the Training VM After serverless defence 53
  53. 53. echo $scenario_1_endpoint_activity Use the Lambda URL corresponding to the actionhistory function printed when deploying serverless-defence After serverless defence 54
  54. 54. Serverless Explanation We just deployed three lambda functions and the DynamoDB tables used by them for the serverless defence. Let's look at them in more detail A high level diagram on how serverless works blockip - Lambda Function This lambda function is responsible for blocking an IP address from accessing the infrastructure for the configured duration. It can be used by any service to block an IP address by making a HTTP request. It uses the stateTable to store blocking status and historyTable for maintaining a log of all actions taken The ELK stack uses this endpoint to block the IP addresses that go beyond the configured threshold in ElastAlert rule Serverless Explanation 55
  55. 55. handleexpiry - Lambda Function This lambda function runs in regular intervals and ensures that entries in the ACL are removed after the configured expiry time by looking up their expiry timestamp in the stateTable actionhistory - Lambda Function Serverless Explanation 56
  56. 56. This lambda function returns a list of actions that have been performed by the serverless-defence so far by querying the historyTable Parameters configurable before deployment region - AWS Region to deploy in. ACL must be in the same region accessToken - Access token used authorize requests to block IPs aclID - ACL that will be used for blocking stateTableName - DynamoDB table that will be created to maintain current blocking state historyTableName - DynamoDB table that will be created to maintain action history ruleValidity - Time (in minutes) after which the IP is unblocked slackUrl - Slack URL to send alerts slackChannel - Slack channel to send alerts to interval - Time interval between scheduled executions Serverless Explanation 57
  57. 57. Automation deploy-scenario-1-infra Created a custom Ansible playbook to setup Nginx Basic HTML site SSRF Vulnerable Application SSH Service with Login Beats (Filebeat) Created a custom AMI using Ansible provisioner and published the final AMI using Packer Uses Terraform to setup the AWS infrastructure for Scenario 1 Subnet Route Tables Elastic IP Security Group SSH Key pair IAM Policy IAM Role EC2 Local provisioner Remote provisioner Output Automation 58
  58. 58. Created a simple bash script to Initialise Terraform using stored AWS credentials Deploy the infrastructure using Terraform plan deploy-scenario-1-defence Application code performs Blocking and Unblocking of IP addresses in network ACL Triggering slack alert Maintaining block state and history in DynamoDB Used serverless framework to deploy code to AWS Lambda Created a simple bash script to deploy the setup Install required pacakages Use configured AWS credentials to deploy the setup Return the output endpoints and store in bashrc Automation 59
  59. 59. Automation 60
  60. 60. Discussion, Use cases and Limitations Use cases and ideas Preventing bruteforce attacks and limiting bot traffic We can use the same solution with other IDS/IPS that exposes an API Limitations ACL, Security groups have limits on maximum number of rules due to which we have to unblock IP addresses after a while Lets Discuss (15 minutes) Feedback/suggestions on improving this approach How you have been solving a similar issue / plan to solve one If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We shall discuss them in the end of the training. Use cases and ideas 61
  61. 61. Auditing Content Management systems Overview In this scenario, We will setup our infrastructure which consists of a Wordpress CMS which sends logs to ELK stack for analysis We will run an activity generator script to simulate user activity for generating log data We will look at auditing Wordpress to analyze, identify and uncover attacks and suspicious activiy Introduction 62
  62. 62. Configuring ELK stack to receive logs from Wordpress Filebeat in Wordpress machine is already configured to send logs to ELK stack during deployment, but we currently don't see the logs becuase logstash isn't accessible from the Wordpress VM. We will now update the security group for our ELK VM to allow the Wordpress machine in Azure to access logstash running at port 5044. Login to the AWS console Navigate to the EC2 console and Instances https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceId Choose the elk-machine and select the corresponding security-group Now click on elk-sg and click on Edit to add a new inbound rule Get the wordpress machine IP address by running the following command Ensure that you run this command in the Training VM Configuring ELK Stack 63
  63. 63. echo $wordpress_machine_ip Now add the wordpress machine IP to inbound rules in security group Try logging in with incorrect credentials to test whether logging is working We can now see the logs in the Kibana dashboard Configuring ELK Stack 64
  64. 64. Configuring ELK Stack 65
  65. 65. Analyzing Wordpress Activity We will now analyze the activity on our wordpress blog. We will be writing custom queries, creating visualizations and using custom dashboards to understand and audit the activity on the site. Running activity generator We will now simulate user activity on our wordpress site by running the following script. Ensure that you run this command in the Training VM generate-wordpress-activity This script will Use the configured wordpress credentials to generate randomized activity on our wordpress site that we will analyze If you see any error, please inform one of the trainers You should be able to see some activity in the Kibana dashboard Interactive Hands-On We will practice the following in a hands-on manner. Follow the trainer's instructions and raise any questions you have Writing custom search query We will write a custom search query to analyze our wordpress login activity Analyzing Wordpress activity 66
  66. 66. Understanding login patterns with visualisations We will create a pie chart to analyze our wordpress login activity Wordpress CMS Audit Custom Dashboard We will import the scenario-2-wordpress-custom-dashboard.json dashboard and visualize the wordpress login data Analyzing Wordpress activity 67
  67. 67. Analyzing user activity We will now analyze User active time Login locations Weblogs Custom Dashboard We will now simulate user activity on our infrastructure site by running the following command Ensure that you run this command in the Training VM nikto -h infra.domain.com This command will Scan and tests web servers for dangerous files/CGIs, outdated server software and other problems. It performs generic and server type specific checks. We will import the web-custom-dashboard.json dashboard to web application server logs visualisations for analysis Analyzing Wordpress activity 68
  68. 68. Analyzing Wordpress activity 69
  69. 69. Automation deploy-scenario-2-infra Used the Terraform to setup Azure infrastructure for Scenario-2 Resource Group Virtual Network Subnet Public IP Network Security Group Network Interface Storage Account Virtual Machine SSH key pair Local provisioner Output Created the simple bash script to execute this whole setup Initialises Terraform Obtains temporary Azure session token Deploys the infrastructure using Terraform plan deploy-scenario-2-infra-playbook Automation 70
  70. 70. Custom Ansible playbook to setup Nginx MySQL PHP Wordpress CLI Configuration of basic site and initial users Custom plugin setup and configuration Beats (Filebeat) Created a simple bash script to execute the setup Uses IP address and ssh key pair to setup the entire wordpress stack Configures filebeat to send logs to the ELK stack generate-wordpress-activity Python script which performs automated activity on a wordpress site Random browsing Failed logins Correct logins Logouts Random activities Automation 71
  71. 71. Discussion, Use cases and Limitations Use cases and ideas Analyzing and understanding site activity and usage patterns to detect, alert or stop anomalous activity The above method can be used for a wide range of defensive scenarios and other content management suites like Drupal, etc. We could act on the logs automatically using wp-cli Limitations No significant limitations Lets Discuss (10 minutes) Feedback/suggestions on improving this approach How you have been solving a similar issue / plan to solve one If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We shall discuss them in the end of the training. Use cases and ideas 72
  72. 72. IAM CloudTrail logs to defend against stolen credentials Overview In this scenario, We will see how we can use the AWS metadata service to retrieve IAM keys We will exploit an application vulnerable to SSRF and to access the AWS metadata service We will deploy serverless defence that will use Cloud Trail logs to detect and automatically defend our cloud infrastructure Introduction 73
  73. 73. Serveless Defence We will now deploy the serverless defence in the coming steps. Deploying serverless defence Run the following script to deploy serverless defence for the scenario-4 Ensure that you run this command in the Training VM deploy-scenario-4-defence This script will Use the stored AWS credentials to deploy the Lambda function used in the serverless defence If it is successful it will print the information to access the machine If you see any error, please inform one of the trainers Serverless Defence 74
  74. 74. Serverless Defence 75
  75. 75. Before the Attack We will look at current state of our services before the attack Attached IAM Roles Lets confirm that IAM role that has been attached to this instance Navigate to AWS EC2 https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east- 1#Instances:sort=instanceId and select the Select the VM named infra-machine Notice that a role called ec2accesss3 has been attached to the VM. Which gives read-only full access to s3 buckets. Before the Attack 76
  76. 76. Before the Attack 77
  77. 77. Attack We will now exploit the SSRF vulnerability in one of the applications in the Infrastructure VM to gain access to the IAM credentials. Exploiting SSRF to obtain IAM Credentials Lets get the IAM credentials by querying the AWS Metadata service Now, enter the following in the input field of the application file:///etc/passwd As you can see there is a Local File Inclusion vulnerability Lets now try to check for SSRF. Enter the following in the input field http://169.254.169.254/latest/meta-data/ Attack 78
  78. 78. The AWS Metadata service provides meta data about the instance such as IP address, instance details and much more SSRF Attack In a Server-Side Request Forgery (SSRF) attack, the attacker can abuse functionality on the server to read or update internal resources. The attacker can supply or a modify a URL which the code running on the server will read or submit data to, and by carefully selecting the URLs, the attacker may be able to read server configuration such as AWS metadata, connect to internal services like http enabled databases or perform post requests towards internal services which are not intended to be exposed. Let's get some metadata information like instance credentials :P It also provides an endpoint to obtain temporary security credentials for the configured role of the instance. We will now try to get the credentials by accessing http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2acesss3 Store this data for later use. These are the temporary credentials usable by services on the machine due to the attached role SessionTokens The credentials we have obtained are generated by AssumeRole IAM call and are temporary security credentials with a session token. Though these credentials work in the same way IAM keys do for the most part, there are some key differences such as temporary security credentials cannot request for temporary security credentials. The token may have a validity upto 12 hours. To revoke a temporary security credential, one must detach the IAM role and revoke the sessions. To read more about temporary security credentials, please visit https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html Attack 79
  79. 79. Using the stolen IAM Credentials Lets use the stolen IAM credentials. We can do that by adding the credentials under an AWS cli profile in our training VM Configure the credentials found by Ensure that you run this command in the Training VM aws configure --profile ssrfkey As we got temporary session token, we have to edit ~/.aws/credentials and add the session token [ssrfkey] aws_access_key_id = xxxxxxxxxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxx aws_session_token = xxxxxxxxxxxxxxxxxxx Lets try listing the S3 buckets under the AWS account using the found key Ensure that you run this command in the Training VM aws s3 ls --profile ssrfkey You can see that you are able list the s3 buckets under the account Lets try to enumerate further by listing all IAM users Ensure that you run this command in the Training VM aws iam list-users --profile ssrfkey Attack 80
  80. 80. Lets try to list all the ec2 instances Ensure that you run this command in the Training VM aws ec2 describe-instances --profile ssrfkey This command fails because the role does not have the privileges required References AWS Metadata Service SSRF Attack Attack 81
  81. 81. After initiating attack The attack is now in progress and has most likely been blocked automatically. We shall verify the same Slack Alert You will get a slack alert about attack and the action taken. The alert says that the role has been detached and the sessions have been revoked Attached IAM Roles Lets check if the IAM role has been detached Navigate to AWS EC2 https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east- 1#Instances:sort=instanceId Select the VM named infrastructure-vm After the attack 82
  82. 82. Select Actions -> Instance Settings -> Attach/Replace IAM Roles Notice that the role has been detached from the VM and will no longer be available via the AWS metadata endpoint Trying to use the credentials Now that have been alerted that the credentials have been revoked, lets try listing the buckets again Ensure that you run this command in the Training VM aws s3 ls --profile ssrfkey Notice that you are not able to use the key as it has been revoked After the attack 83
  83. 83. Serverless Explanation We just deployed the iamhandler lambda function This lambda function constantly monitors the CloudTrail logs for unauthorized requests to AWS API and detaches the IAM role from a VM along with revoking all the session tokens. The following parameters can be configured in serverless-defence/scenario-4/config.js region - AWS Region to deploy in logGroup - CloudWatch Log Group to monitor interval - Time interval between scheduled executions slackUrl - Slack URL to send alerts slackChannel - Slack channel to send alerts to Serverless Explanation 84
  84. 84. Automation deploy-scenario-4-defence Application code performs Identifying unauthorized API calls in CloudTrail Revokes the existing sessions for the role Detaches the role from the instance Triggers slack alert Used serverless framework to deploy code to AWS Lambda Created the simple bash script to deploy the setup Uses configured AWS default credentials and environment variables Automation 85
  85. 85. Discussion, Use cases and Limitations Use cases and ideas Quarantine the machine after detecting a violation Could send data to ELK stack for analyzing usage and abuse Limitations Not really useful when attached IAM role has AdministratorAccess unless MFA is enabled. CloudTrail logs take anywhere from 5 to 15 minutes to reflect, preventing real-time monitoring and defence Lets Discuss (10 minutes) Feedback/suggestions on improving this approach How you have been solving a similar issue / plan to solve one If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We shall discuss them in the end of the training. Use cases and ideas 86
  86. 86. Container logs to audit Kubernetes security Overview In this scenario we will see how we can detect a senitive file read operation occuring inside a container in our Kubernetes cluster. We will see how to apply serverless defence to automatically stop the attack and apply the fix in near-realtime Introduction 87
  87. 87. Attack We will now exploit the command injection vulnerability in one of the applications in the Kubernetes cluster. Accessing the application We can get the IP address at which the application has been deployed by running the below command echo $scenario_5_endpoint_ip We now try to access the application by vising http://$scenario_5_endpoint . The credentials for the application will are username: adef password: batmanvssuperman Once we authenticate, we can now see the application. Lets try pinging google.com to test the feature. Input the following into the application google.com Exploiting command injection vulnerability in the application Attack 88
  88. 88. It looks like the application is passing the input to ping command and is returning the output. Lets try to exploit this Try the below input ;id As we can see, we are the root user, and lets try to access /etc/shadow . This worked and we are able to see the output Attack 89
  89. 89. We will shortly recieve a slack alert about this activity from sysdig falco logs Viewing the log The entry that triggered this can be found in Stackdriver under the falco logs for this Kubernetes cluster. We can see the logs by choosing GKE Container -> auto-adef -> default -> falco selection under Google Logging Attack 90
  90. 90. Attack 91
  91. 91. Serverless Defence We will now deploy serverless defence Run the following script to deploy serverless defence for the scenario-5 deploy-scenario-5-defence This script will Use the stored gcloud credentials to deploy the cloud function used in the serverless defence If you see any error, please inform one of the trainers Serverless defence 92
  92. 92. After serverless defence Lets try to repeat the same attack again. Try to read /etc/shadow again as shown below The attack succeeds. But within moments of the attack, we receive a slack alert about the attack. Lets try to access the application now that we have deployed automated serverless defence Try accessing /etc/shadow After defence 93
  93. 93. We see an Permission Denied error. Lets try to check the user we are executing commands as by running id again After defence 94
  94. 94. We see that the application is running as app user which does not have permission to access /etc/shadow We will also notice that we are not able to ping anymore After defence 95
  95. 95. The capablity CAP_NET_RAW has also been disabled After defence 96
  96. 96. Serverless Explanation We just deployed the adefscenario5 lambda function This lambda function constantly monitors Stack Driver logs from our falco daemon for the configured rule. If a matching entry exists in the logs, it sends a slack alert and automatically re-deploy the affected application with a more restrictive configuration The following parameters can be configured in serverless-defence/scenario-5/config.js rule - Falco rule to look for in logs slackUrl - Slack URL to send alerts slackChannel - Slack channel to send alerts to Serverless Explanation 97
  97. 97. Automation deploy-scenario-5-infra Created a simple bash script to Spin up new 2-node GKE cluster Installs and enable the helm with service account Installs the node-app deployment installs the sydig falco deploy-scenario-5-defence Application code performs Updating the deployment with security fixes based on logs Triggering slack alert Used gcloud framework to deploy code to cloud functions Automation 98
  98. 98. Created a simple bash script to deploy the setup Deploys the serverless defence code Use configured gcloud credentials to deploy the setup Automation 99
  99. 99. Discussion, Use cases and Limitations Use cases and ideas Detect and act on intrusions and unexpected behaviour Limitations TBA Lets Discuss (10 minutes) Feedback/suggestions on improving this approach How you have been solving a similar issue / plan to solve one If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We shall discuss them in the end of the training. Use cases and ideas 100
  100. 100. AWS We will now delete all services and resources deployed on our AWS account. Though the script only removes the deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account created for the training and do not have any other credentials configured. Ensure that you run this command in the Training VM nuke-destroy-aws-setup This script will Removes the all infrastructure we have setup till now in AWS Ensure you don't have any data in the AWS account before running the script AWS 101
  101. 101. Azure We will now delete all services and resources deployed on our Azure account. Though the script only removes the deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account created for the training and do not have any other credentials configured. Ensure that you run this command in the Training VM nuke-destroy-azure-setup This script will Removes the all infrastructure we have setup till now in Azure Ensure you don't have any data in the Azure account before running the script Azure 102
  102. 102. GCP We will now delete all services and resources deployed on our GCP account. Though the script only removes the deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account created for the training and do not have any other credentials configured. nuke-destroy-gcp-setup Run this command in the Cloud Shell This script will Removes the all infrastructure we have setup till now in GCP Ensure you don't have any data in the GCP account before running the script GCP 103
  103. 103. Automation nuke-destroy-aws-setup Created the simple bash script to execute Removes the AWS s3 buckets using default credentials Terraform destroy the existing infrastructure created for different scenarios in AWS nuke-destroy-azure-setup Created the simple bash script to execute Terraform destroy the existing infrastructure created for scenario-2 nuke-destroy-gcp-setup Created the simple bash script to execute Currently it's not performing anything Automation 104
  104. 104. Automation 105
  105. 105. About Appsecco Appsecco is a specialist application security company, founded in 2015, with physical presence in London, Bangalore, Doha and Boston, providing industry leading security advice that is firmly grounded in commercial reality. Our services cover the entire software development lifecycle from advising on how build and foster a culture of security within development teams and organisations to reviewing and advising on the security of applications and associated infrastructure under development to providing rapid response and advice in the event of a security breach or incident. As a team, we are highly qualified and have many years of extensive experience working with clients across multiple counties and in a wide range of industries and sectors; from financial services to software development, manufacturing to governmental organisations and consumer brands to ecommerce. The solutions, advice and insight we deliver to our clients always follows three core principles: 1. It must be pragmatic; taking into account the specific commercial, organisational and operational realities of each client individually 2. It must genuinely add value; the advice or solutions we provide must addresses the specific problem a client seeks to solve and have actionable insight to enable them to achieve this 3. Never be purely automated; whenever we are testing for security our reports and output always have significant, expert, human input to give the greatest possible value for our clients In addition to their client-facing work our technical team are actively involved in researching and developing new and better ways to stay secure and can regularly be found presenting their findings at industry conferences and events ranging from nullcon in India, DevSecCon in London and Singapore, to DEF CON, the world’s largest security conference held annually in the USA. Structurally we are a UK Limited company with a wholly owned Indian subsidiary (where the majority of our technical resource is based) and raised seed funding for our continuing growth in the UK in late 2016. About Appsecco 106
  106. 106. About Appsecco 107
  107. 107. Upcoming Conferences Nullcon 2019 - Goa, India Black Hat 2019 - Las Vegas, USA Upcoming Trainings and Conferences 108
  108. 108. Upcoming Trainings and Conferences 109
  109. 109. References & Resources Automated Defense using Serverless Computing AWS AWS in Plain English Amazon Web Services - a practical guide AWS CIS Benchmarks AWS Security Best Practices AWS Security Primer Security auditing tool for AWS environments Prowler: AWS CIS Benchmark Tool Nimbostratus -Tools for fingerprinting and exploiting AWS Aardvark is a multi-account AWS IAM Access Advisor API Security Monkey CloudSploit Scans System Shock: How A Cloud Leak Exposed Accenture's Business Fullstop - Audit reporting Abusing the AWS metadata service using SSRF vulnerabilities AWS Vulnerabilities and the Attacker’s Perspective Pivoting in Amazon Clouds Security Tools for AWS https://github.com/nccgroup/PMapper https://github.com/nccgroup/aws-inventory https://yashvier.app.box.com/v/boostawssecurity Azure Microsoft Azure in Plain English Azure Security Centre Azure security technical capabilities Security auditing tool for Azure environments Azure Security Lab Workshop Enumeration and reconnaissance activities in the Microsoft Azure Cloud Azure Security and Compliance Blueprint - FedRAMP Web Applications Automation Secure DevOps Kit for Azure (AzSK) GCP Google Infrastructure Security Design Overview Cloud Security Command Center Forseti Security: Open-source tools for GCP security Google Cloud Platform Security Tool Map AWS services to Google Cloud Platform products References & Resources 110
  110. 110. Serverless Servleress Framework Documentation AWS SDK for Javascript Azure SDK for Node.js GCP Javascript API Documentaion Intrusion and Exfiltration in Server-less Architectures Serverless Architecture Serverless Technologies Awesome Serverless References & Resources 111

×