(SACON) Madhu Akula - Automated Defense Using Cloud Service Aws, Azure, Gcp

2. 1.1
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.2.1
3.2.2
3.2.3
4.1
4.2
4.3
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
Table of Contents
Welcome
Introduction
Introduction
About the trainers - Madhu Akula
Disclaimer
Agenda
Training Preview
Getting Started
Cloud Accounts Access
Cloud accounts setup
AWS
Azure
Google Cloud
Cloud Security
AWS Security
Azure Security
GCP Security
ELK Stack Setup
ELK Stack
Alerting
Kibana 101
Automation
SCENARIO 1 - SSH Bruteforce
Introduction
Before the Attack
Attack
Serverless Defence
Configuring ELK Stack
2

3. 6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
9.2
9.3
9.4
9.5
9.6
9.7
10.1
10.2
10.3
10.4
After serverless defence
Serverless Explanation
Automation
Use cases and ideas
SCENARIO 2 - Auditing CMS
Introduction
Configuring ELK Stack
Analyzing Wordpress activity
Automation
Use cases and ideas
SCENARIO 3 - IAM Defence
Introduction
Serverless Defence
Before the Attack
Attack
After the attack
Serverless Explanation
Automation
Use cases and ideas
SCENARIO 4 - Container defense
Introduction
Attack
Serverless defence
After defence
Serverless Explanation
Automation
Use cases and ideas
Tear Down
AWS
Azure
GCP
Automation
3

4. 11.1
11.2
12.1
About Us
About Appsecco
Upcoming Trainings and Conferences
References & Resources
References & Resources
4

5. Welcome
5

6. Introduction
Welcome to "Automated Defense using Cloud Services for AWS, Azure and GCP".
This defence focused, hands on training will set you on the path to using serverless and elastic stack with help of
cloud services to defend against cloud infrastructure
It helps you to get started about building automated defence systems for your environments based on your needs
by understanding the approach and methodology
The idea behind the Automated Defence is to reduce the bottle-neck of human reaction time to security
monitoring by automating defensive actions to have a near real-time response
Abstract
We live in cloud first era where the cloud is increasingly our first choice of deployment due to its convenience and
scalability. Monitoring for attacks and defending against them in real-time is crucial but defending your cloud
infrastructure during attacks can prove to be a nightmare even with the solutions currently available in the market.
In this training we will teach how to defend your cloud infrastructure using Serverless technologies and Elastic Stack.
Elastic Stack collects, analyses logs and triggers alerts based on a pre-configured rule-set and the Serverless stack
drives defence to perform automated blocking.
The world is advancing towards accelerated deployments using DevOps and Cloud technologies. Automated defence
solves modern world security challenges using near real-time alerting system, serverless technologies and centralised
monitoring system.
Prerequisites
Students will need trial accounts in AWS, Azure and GCP with administrative access with billing enabled to have
a hand-on experience during the training.
Trainers
Trainer Details
Name : Madhu Akula
Twitter : @madhuakula
Email : madhu@appsecco.com
Introduction
6

7. Madhu Akula
Madhu Akula is a security ninja, published author and Security Automation Engineer at Appsecco. He is passionate
about Cloud Native, DevOps and security and is an active member of the international Security and DevOps
communities.
His research has identified vulnerabilities in over 200 companies and organisations including; Google, Microsoft,
LinkedIn, eBay, AT&T, WordPress and Adobe, etc. He is co-author of Security Automation with Ansible2(ISBN-13:
978-1788394512), which is listed as a technical resource by Red Hat Ansible.
Madhu frequently speaks and runs technical sessions at security events and conferences around the world including;
DEF CON (24 & 26), Blackhat USA 2018, USENIX LISA 2018, Appsec EU 2018, All Day DevOps (2016, 2017 &
2018), DevSecCon (London, Singapore and Boston: 2016, 2017 & 2018), DevOpsDays India, c0c0n (2017 & 2018),
Serverless Summit, null and multiple others.
Some of the trainings/workshops by Madhu Akula include
Automated Defense using Cloud Services for AWS, Azure and GCP - Blackhat USA 2018, 2019
Attacking & Auditing Docker Containers Using Open Source - Defcon 26, OWASP Bay Area Meetup
Attacking & Auditing Docker Containers - USENIX LISA 2018, DevSecCon London 2018, c0c0n XI
Building visualization platforms for OSINT data using open source solutions - Recon Village 2018
Automated Defense using Serverless for AWS, Azure and GCP - Appsec EU 2018
Breaking and Owning Cloud Servers and Applications - NULLCON Goa 2018
Ninja Level Infrastructure Monitoring - Defcon 24 and DevSecon London 2016
Automated Infrastructure Security Monitoring & Defence - DevSecCon Singapore 2017
Real World Security Monitoring & Automated Defence for almost free - DevSecCon Boston and DevSeCon
London 2017
Monitoring & Defending Infrastructure Security Attacks - c0c0n X
Linux Container Security - Null Bangalore
An Introduction to Containers using Docker and using it for Security Automation - Null Bangalore
Automating Documentation, Presentation, KB using Markdown - Null Bangalore
Automated infrastructure security monitoring & defence - Null Bangalore
Some of the talks given by Madhu Akula include
Container Security Monitoring using Open Source - All Day DevOps 2018, Online Webinar
Continuous security monitoring in CI and CD pipelines - iwomm 2.5: Continuous Delivery Meetup, London
Modern Security Operations aka Secure DevOps - All Day DevOps 2017
Automated Defence for Cloud Security in AWS using Serverless - Serverless Summit 2017
DevOps principles to build your lean startup - Startup Leadership Program
Developers guide to security & operations: Introducing DevSecOps - Software Security Bangalore Meetup
Automated Infrastructure Security Monitoring using FOSS - All Day DevOps 2016
Infrastructure Security Monitoring - DevOps Days India 2016
Cloud Security for everyone - SDN + IoT + Network Virtualization Enthusiasts Meetup
NodeJS Security - Null Bangalore
Web & Cloud Security in the Real World - Keynote speaker at CompTIA Bangalore
My bug hunting with open source - Hill hacks 2015
Hardening routers & switches - Null Dharamshala
About the trainers - Madhu Akula
7

8. Basics of networking - Null Dharamshala
Published Works of Madhu Akula include
Cover Details
Book - Security Automation with Ansible2, Published by PacktPub December 2017,
ISBN 9781788394512
Online
Account Details
Twitter @madhuakula
LinkedIn Madhu Akula
About the trainers - Madhu Akula
8

9. Disclaimer
The attacks covered in the training are for educational purposes only. Do not test or attack any system outside of
the scope of this training lab unless you have express permission to do so
The snippets, commands and scripts used throughout the training are not production-ready, may not be bug-free
and are not guaranteed in any way
Disclaimer
9

10. Agenda
Here is the high level overview of how next two days will look like
Automated Defense using Cloud Serivces for AWS, Azure and GCP
Introduction
Environment Setup
Cloud Account Setup
Elastic Stack Setup
Scenario-1 : SSH bruteforce detection and defence
Scenario-2 : Content management system audit analysis
Scenario-3 : IAM CloudTrail logs to defend against stolen credentials
Scenario-4 : Container logs to audit Kubernetes security
Tear down
Wrap up
References & Resources
Agenda
10

11. Training Preview
Training Preview
11

12. Cloud service accounts
Services we will be using in AWS
IAM
EC2
S3
Lambda
Cloud Watch
Cloud Trail
VPC
API Gateway
Dynamo DB
Services we will be using in Azure
Resource Group
Virtual Machine
Virtual Network
Network Security Group
Public IP Address
Azure Cosmos DB
Azure Functions
Services we will be using in GCP
Google Cloud Shell
Goolge Compute Engine
Google Kubernetes Engine
IAM
Cloud Functions
App Engine
LoadBalancer
Stack driver
Search Engine :P
Cloud Accounts Access
12

13. Cloud accounts setup
We will now configure our cloud account credentials in the student VM to be able to deploy the services we will be
using.
Cloud accounts setup
13

14. Setting up AWS CLI with IAM credentials
Introduction
The primary AWS account, also called the root account, is very powerful in terms of access. To avoid losing its keys or
secrets, we will create a IAM Administrator account which will have the same privileges as a AWS root account except
for access to certain features like billing which we can anyways access using the root account.
Steps to create an IAM user
Search for IAM in the services
Click on users > Add user
Create a user called iamadmin with the following settings
AccessType: Programmatic access and AWS Management Console access
Console Password: Select Custom password
Provide a strong alphanumeric character
Uncheck require password reset
We are only asking you uncheck require password reset to do this for the purposes of the training
AWS
14

15. Click on 'Attach existing policies directly' and select 'AdministratorAccess'
Click Next and Create User
AWS
15

16. Copy and save the following in your text editor for later use
1. Access key ID
2. Secret access key
3. Unique sign in URL (Bookmark this link)
Steps to Configure AWS CLI
Run the following command to configure the aws cli
Ensure that you run this command in the Training VM
aws configure
You will need to provide the access key ID and secret access key
Type the following values
Default region name [None]: us-east-1 (YOU MUST PROVIDE us-east-1 )
Default output format [None]: json
These credentials get stored at ~/.aws/credentials
Validation
Run the following command to validate the AWS configuration to ensure that account is added and set as default
Ensure that you run this command in the Training VM
aws sts get-caller-identity
Additional Information
Setting up access using CLI
AWS
16

17. Setting up Azure CLI with credentials
Azure CLI is optimized for managing and administering Azure resources from the command line, and for building
automation scripts that work against the Azure Resource Manager.
Steps to Configure AWS CLI
Run the following command to configure the azure cli
Ensure that you run this command in the Training VM
az login
Open the URL in your browser and enter the returned code to go to the next step
https://microsoft.com/devicelogin
Complete the registration by selecting the free trail training account to confirm
Azure
17

18. After successful authentication, we can see the below output in the command prompt
Validation
Run the following command to validate the Azure configuration to ensure that account is added and set as default
Ensure that you run this command in the Training VM
az account list
Azure
18

19. Additional Information
Log in with Azure CLI
Azure
19

20. Google Cloud Platform
The gcloud auth command group lets you grant and revoke authorization to Cloud SDK (gcloud) to access Google
Cloud Platform.
Authenticating via gcloud CLI
Run the following command in training vm to configure gcloud cli
Ensure that you run this command in the Training VM
gcloud auth login
Copy the link and open in your browser. Make sure you login to the account which you are using the free trail
Google Cloud
20

21. Give the permission by clicking Allow
Google Cloud
21

22. Copy the code for pasting in the console
Google Cloud
22

23. Paste the copied code and press enter to continue
Validation
Run the following command to validate the gcloud configuration to ensure that account is added and set as
default
Ensure that you run this command in the Training VM
gcloud config list
Google Cloud
23

24. Additional Information
gcloud auth login
google auth
Google Cloud
24

25. AWS Security
Five core areas of Cloud Security
According to this whitepaper, security in the cloud is composed of five areas
1. Identity and Access Management
2. Detective Controls
3. Infrastructure Protection
4. Data Protection
5. Incident Response
Mapping these areas to AWS Services and Security
Concepts we covered in the training
Area Services
Identity and Access Management AWS IAM
Detective Controls AWS Config, AWS CloudWatch, AWS S3, AWS Inspector
Infrastructure Protection AWS VPC, AWS S3
Data Protection N/A
Incident Response N/A
Cloud Security Architecture Building Blocks
Block Use Case
AWS VPC Logically seperate network
AWS IAM Secure access to resources and services for people and computers
AWS CloudWatch See logs and take actions
AWS CloudTrail Track API requests and monitor and notify
AWS Config/Cloud Custodian Validate security policy and remediate automatically
Other relevant AWS whitepapers to read and learn from
AWS Security Pillar Whitepaper
AWS Security Best Practices
AWS Auditing use of AWS Checklist
AWS Security
25

26. AWS Security
26

27. Azure Security
The features listed following are capabilities you can review to provide the assurance that the Azure Platform is
managed in a secure manner. Links have been provided for further drill-down on how Microsoft addresses customer
trust questions in four areas: Secure Platform, Privacy & Controls, Compliance, and Transparency.
Available security technical capabilities to fulfil user
(Customer) responsibility - Big picture
Microsoft Azure provides services that can help customers meet the security, privacy, and compliance needs. The
Following picture helps explain various Azure services available for users to build a secure and compliant application
infrastructure based on industry standards.
The built-in capabilities are organized in six (6) functional areas:
Operations
Applications
Storage
Networking
Compute
Identity
Reference
https://docs.microsoft.com/en-us/azure/security/azure-security
https://docs.microsoft.com/en-us/azure/security/azure-security-technical-capabilities
Azure Security
27

28. Azure Security
28

29. GCP Security
Google cloud infrastructure builds security through progressive layers that deliver true defense in depth.
Reference
https://cloud.google.com/security/infrastructure/design/
https://cloud.google.com/security/overview/whitepaper
https://cloud.google.com/security/
GCP Security
29

30. ELK Stack
Ref: https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html
Elasticsearch, Logstash and Kibana
Different open source modules working together
Helps users/admins to collect, analyse and visualize data in (near) real-time
Each module fits based on your use case and environment
Components of the stack
Elasticsearch
Logstash
Kibana
Beats
Elasticsearch
ELK Stack
30

31. Ref: https://www.elastic.co/products
Distributed and Highly available search engine, written in Java and uses Groovy (now started painless scripting)
Built on top of Lucene
Multi Tenant with Multi types and a set of APIs
Document Oriented providing (near) real time search
Logstash
Ref: https://www.elastic.co/products
Tool for managing events and logs written in Ruby
Centralized data processing of all types of logs
Consists of 3 main components
Input : Passing logs to process them into machine understandable format
Filter : Set of conditions to perform specific action on a event
Output : Decision maker for processed events/logs
Basic Logstash Configuration
input {
stdin {}
file {}
...
}
filter {
ELK Stack
31

32. grok {}
date {}
geoip {}
...
}
output {
elasticsearch {}
email {}
...
}
Kibana
Ref: https://www.elastic.co/products
Powerful front-end dashboard written in JavaScript
Browser based analytics and search dashboard for Elasticsearch
Flexible analytics & visualisation platform
Provides data in the form of charts, graphs, counts, maps, etc. in real-time
Beats
Ref: https://www.elastic.co/products
Lightweight shippers for Elasticsearch & Logstash
Capture all sorts of operational data like logs or network packet data
It can send logs to either Elasticsearch, Logstash
ELK Stack
32

33. Different types of Beats
Filebeat: Log Files
Metricbeat: Metrics
Packetbeat: Network Data
Winlogbeat: Windows Event Logs
Auditbeat: Audit Data
Heartbeat: Uptime Monitoring
Filebeat sample configuration
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/auth.log
tags: ["sshlog"]
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
tags: ["weblog"]
output.logstash:
hosts: ["localhost:5044"]
ELK Stack for Security Monitoring & Alerting
It helps to parse large amount of log data
We can aggregate and correlate the data from different types of log formats
Centralized way to look into entire logs
Provides near real-time search and visualization capabilities
ELK Reference Guide
We can use the below Gitbook with detailed instructions for references to the ELK stack setup and usage.
https://appsecco.com/books/elk-workshop
ELK Stack
33

34. Alerting
We can set up a notification system to let users/admins know that a pattern match has occurred.
Logstash output plugin alerting via (Email, Pager duty, JIRA, etc.)
An open source alerting for elasticsearch by Yelp called elastalert
Another open source project by Etsy 411
X-Pack (commerical offering by Elastic)
Custom scripts
ElastAlert
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in
Elasticsearch.
Simple ElastAlert rule to detect ssh bruteforce attacks
es_host: localhost
es_port: 9200
name: "SSH Bruteforce Login Alert"
type: frequency
index: filebeat-*
num_events: 12
timeframe:
minutes: 3
# For more info:
# http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- query:
query_string:
query: 'tags: "sshlog" AND login: "Failed" AND username: ("root" OR "ubuntu")'
alert:
- slack
- command
slack:
slack_webhook_url: "https://hooks.slack.com/services/xxxxx"
slack_username_override: "attack-bot"
slack_emoji_override: "robot_face"
command: ["/usr/bin/curl", "https://LAMBDAENDPOINTGOESHERE/%(ip)s"]
realert:
minutes: 0
Rule Types
Any
Blacklist
Whitelist
Change
Frequency
Spike
Flatline
New Term
Alerting
34

35. Cardinality
Metric Aggregation
Percentage Match
Alert Types
Command, HTTP POST
Email, SNS, Stomp
Jira, Gitter, ServiceNow
OpsGenie, VictorOps, PagerDuty
Twilio, Telegram
HipChat, Slack, MS Teams
Alerting
35

36. Kibana 101
We will familiarize ourselves with Kibana dashboard now. For now, we already have system logs from the ELK VM.
Login to your Kibana dashboard
Interactive Hands-On
We will practice the following in a hands-on manner. Follow the trainer's instructions and raise any questions you have
Index Creation
We will create an index pattern for our metricbeat log data so that we can query and build visualisations around
them
We need to select the timestamp field that we will be using for this index
Discovery
We will discover and observe the logs
Kibana 101
36

37. Here we see a single log entry in JSON format
Custom Search
We will use Apache Lucene query to analyze the log data
Kibana 101
37

38. Time Filters
We will try out various time filters to restrict our search space
Creating Visualization
We will create visualizations based on our search queries
Selecting the search query for the current visualization
Kibana 101
38

39. Creating Pie Chart
We will create a pie chart to represent our visualization
Creating Dashboard
We will create a dashboard to feature our visualizations and queries
Kibana 101
39

40. Sharing Dashboard
We can share the dashboard so that it can be used by others
Dev Tools
We will explore Dev Tools and try out manual queries to Elastic search
Try the following queries to get the cluster status
GET _cluster/health
GET _cluster/state
Management
We explore the management tab to manage our custom searches, reports, import and export
Kibana 101
40

41. Generating dashboards for metricbeat
We will login to the ELK stack and generate the metricbeat dashboards
sudo metricbeat setup --dashboards
Now, we can see the system dashboard generated in live
Kibana 101
41

42. Automation
deploy-elk-stack-infra
Custom Ansible playbook to setup
Elasticsearch
Logstash
Kibana
Nginx
ElastAlert
Beats (Filebeat, Metricbeat)
Created a custom AMI using Ansible provisioner and published the final AMI using Packer
Used the Terraform to setup AWS infrastructure for ELK stack
VPC
Subnet
Route Tables
Internet Gateway
Elastic IP
Security Group
SSH Key pair
EC2
Local provisioner
Remote provisioner
Automation
42

43. Output
Created a simple bash script to
Initialise Terraform using stored AWS credentials
Deploy the infrastructure using Terraform plan
Automation
43

44. SSH Bruteforce Defence
Overview
In this scenario,
We will setup our infrastructure, which consists of a VM with SSH password authentication
We will setup the serverless components required for Automated Defence
We will perform a bruteforce attack on the SSH service and see how to defend against the attack using serverless
and automated defence approach
Introduction
44

45. Introduction
45

46. Before the Attack
We will look at current state of our infrastructure and logs before the attack.
Network ACL
Lets observe our Network ACL for our infrastructure VPC subnet
Navigate to the VPC -> Network ACL dashboard by going here
https://console.aws.amazon.com/vpc/home?region=us-east-1#acls:
Please ensure that you are logged in to your aws account before visiting the link above
Select the ACL belonging to adef-lab-vpc as shown
Observe that everything is allowed at this point
Before the Attack
46

47. Attack
We will now attack the Infra VM SSH service by running a bruteforce attack using hydra utility
Running the bruteforce attack
Run the following command to start the bruteforce attack against the SSH service of the infrastructure vm
Ensure that you run this command in the Training VM
hydra -V -L /opt/usernames.txt -P /opt/passwords.txt infra.domain.com ssh
This script will
Run a SSH bruteforce attack with the hydra (Hydra is a brute force password cracking tool) using given
wordlists
The usernames.txt and passwords.txt files are already placed in your system under /opt/ directory
If you see any error, please inform one of the trainers
You should see something like this
Kibana Dashboard
Lets observe the SSH login logs and visualize the attack. We are able to see the logs here because the infra VM has
been configured to send logs to the ELK VM by default
Navigate to the Kibana dashboard by using the link
Create new index pattern in your elk stack and give the index name pattern and select the timestamp. We index
the data so that it can be queried and thus visualized
Now navigate to discover and select the filebeat pattern to see the near real-time logs.
Attack
47

48. Now, we can see the logs coming in near real-time and we can also use Apache lucene queries to filter the data
by selecting the appropriate filters as shown in the screenshots.
Query for all login attempts against the users root , ubuntu under sshlog
Import the ssh-custom-dashboard.json dashboard. This dashboard will help with visualizing the SSH attack in
real-time.
Attack
48

49. References
Apache Lucene Query Syntax
Kibana Dasboards
Attack
49

50. Serveless Defence
We will now deploy the serverless defence that will detect, block and alert us about the attack automatically.
Deploying serverless defence
Run the following script to deploy serverless defence for the scenario-1
Ensure that you run this command in the Training VM
deploy-scenario-1-defence
This script will
Deploy DynamoDB tables and Lambda Functions used for the serverless defence
Print the lambda endpoints that we will use for serverless defence
Please note down this information, as we will use this later
If you see any error, please inform one of the trainers
Serverless Defence
50

51. Configuring ELK stack
We need to configure our ELK stack to trigger a defensive action by making a request to our serverless endpoint.
We use ElastAlert, an open source tool, to trigger defensive actions when the conditions defined in the rules are met.
Now we have to ensure that the lambda endpoint generated by defence script has to be updated in ElastAlert
configuration.
Get the endpoint from the student vm by running below command
echo $scenario_1_endpoint_ip
SSH into the ELK VM
Now we'll configure the endpoint in the ElastAlert configuration file to trigger a HTTP request to our serverless
endpoint with an ip address to block
This command must be run in the ELK VM. If you are not familiar with vi , please use nano instead
sudo vi /opt/elastalert/rules/ssh-bruteforce-alert.yml
Now we have to restart the ElastAlert service to apply the changes
This command must be run in the ELK VM
sudo systemctl restart elastalert.service
Configuring ELK Stack
51

52. After serverless defence
Let's attack the infra ssh service again to see that serverless defence happening in near real-time
Ensure that you run this command in the Training VM
hydra -L /opt/usernames.txt -P /opt/passwords.txt infra.domain.com ssh
Now we can see the near-real time ssh logs in our Kibana dashboard
The attack is now in progress and has most likely been blocked automatically. We shall verify the same.
Slack Alert
You would've received a slack alert about the IP being blocked
You will also receive another slack notification once the IP address has been unblocked
Network ACL
Lets observe our Network ACL for our infrastructure VPC subnet
Navigate to the VPC -> Network ACL dashboard by going here https://console.aws.amazon.com/vpc/home?
region=us-east-1#acls:
Please ensure that you are logged in to your aws account before visiting the link above
Select the ACL belonging to adef-lab-vpc VPC as shown
After serverless defence
52

53. Observe that our student VM IP has been blocked. There may be other IP addresses that have been blocked
due to bruteforce attacks on the wild
The following is the dynamo DB screenshot of automated defence in action
Action history
Lets check the actions performed by our serverless defence by invoking the actionhistory endpoint
We need to add the accessToken parameter to the URL before we can use it to query for the actions taken
Ensure that you run this command in the Training VM
After serverless defence
53

54. echo $scenario_1_endpoint_activity
Use the Lambda URL corresponding to the actionhistory function printed when deploying serverless-defence
After serverless defence
54

55. Serverless Explanation
We just deployed three lambda functions and the DynamoDB tables used by them for the serverless defence. Let's
look at them in more detail
A high level diagram on how serverless works
blockip - Lambda Function
This lambda function is responsible for blocking an IP address from accessing the infrastructure for the configured
duration. It can be used by any service to block an IP address by making a HTTP request. It uses the stateTable to
store blocking status and historyTable for maintaining a log of all actions taken
The ELK stack uses this endpoint to block the IP addresses that go beyond the configured threshold in ElastAlert rule
Serverless Explanation
55

56. handleexpiry - Lambda Function
This lambda function runs in regular intervals and ensures that entries in the ACL are removed after the configured
expiry time by looking up their expiry timestamp in the stateTable
actionhistory - Lambda Function
Serverless Explanation
56

57. This lambda function returns a list of actions that have been performed by the serverless-defence so far by querying
the historyTable
Parameters configurable before deployment
region - AWS Region to deploy in. ACL must be in the same region
accessToken - Access token used authorize requests to block IPs
aclID - ACL that will be used for blocking
stateTableName - DynamoDB table that will be created to maintain current blocking state
historyTableName - DynamoDB table that will be created to maintain action history
ruleValidity - Time (in minutes) after which the IP is unblocked
slackUrl - Slack URL to send alerts
slackChannel - Slack channel to send alerts to
interval - Time interval between scheduled executions
Serverless Explanation
57

58. Automation
deploy-scenario-1-infra
Created a custom Ansible playbook to setup
Nginx
Basic HTML site
SSRF Vulnerable Application
SSH Service with Login
Beats (Filebeat)
Created a custom AMI using Ansible provisioner and published the final AMI using Packer
Uses Terraform to setup the AWS infrastructure for Scenario 1
Subnet
Route Tables
Elastic IP
Security Group
SSH Key pair
IAM Policy
IAM Role
EC2
Local provisioner
Remote provisioner
Output
Automation
58

59. Created a simple bash script to
Initialise Terraform using stored AWS credentials
Deploy the infrastructure using Terraform plan
deploy-scenario-1-defence
Application code performs
Blocking and Unblocking of IP addresses in network ACL
Triggering slack alert
Maintaining block state and history in DynamoDB
Used serverless framework to deploy code to AWS Lambda
Created a simple bash script to deploy the setup
Install required pacakages
Use configured AWS credentials to deploy the setup
Return the output endpoints and store in bashrc
Automation
59

60. Automation
60

61. Discussion, Use cases and Limitations
Use cases and ideas
Preventing bruteforce attacks and limiting bot traffic
We can use the same solution with other IDS/IPS that exposes an API
Limitations
ACL, Security groups have limits on maximum number of rules due to which we have to unblock IP addresses
after a while
Lets Discuss (15 minutes)
Feedback/suggestions on improving this approach
How you have been solving a similar issue / plan to solve one
If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We
shall discuss them in the end of the training.
Use cases and ideas
61

62. Auditing Content Management systems
Overview
In this scenario,
We will setup our infrastructure which consists of a Wordpress CMS which sends logs to ELK stack for analysis
We will run an activity generator script to simulate user activity for generating log data
We will look at auditing Wordpress to analyze, identify and uncover attacks and suspicious activiy
Introduction
62

63. Configuring ELK stack to receive logs from Wordpress
Filebeat in Wordpress machine is already configured to send logs to ELK stack during deployment, but we currently
don't see the logs becuase logstash isn't accessible from the Wordpress VM.
We will now update the security group for our ELK VM to allow the Wordpress machine in Azure to access logstash
running at port 5044.
Login to the AWS console
Navigate to the EC2 console and Instances
https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceId
Choose the elk-machine and select the corresponding security-group
Now click on elk-sg and click on Edit to add a new inbound rule
Get the wordpress machine IP address by running the following command
Ensure that you run this command in the Training VM
Configuring ELK Stack
63

64. echo $wordpress_machine_ip
Now add the wordpress machine IP to inbound rules in security group
Try logging in with incorrect credentials to test whether logging is working
We can now see the logs in the Kibana dashboard
Configuring ELK Stack
64

65. Configuring ELK Stack
65

66. Analyzing Wordpress Activity
We will now analyze the activity on our wordpress blog. We will be writing custom queries, creating visualizations and
using custom dashboards to understand and audit the activity on the site.
Running activity generator
We will now simulate user activity on our wordpress site by running the following script.
Ensure that you run this command in the Training VM
generate-wordpress-activity
This script will
Use the configured wordpress credentials to generate randomized activity on our wordpress site that we will
analyze
If you see any error, please inform one of the trainers
You should be able to see some activity in the Kibana dashboard
Interactive Hands-On
We will practice the following in a hands-on manner. Follow the trainer's instructions and raise any questions you have
Writing custom search query
We will write a custom search query to analyze our wordpress login activity
Analyzing Wordpress activity
66

67. Understanding login patterns with visualisations
We will create a pie chart to analyze our wordpress login activity
Wordpress CMS Audit Custom Dashboard
We will import the scenario-2-wordpress-custom-dashboard.json dashboard and visualize the wordpress login data
Analyzing Wordpress activity
67

68. Analyzing user activity
We will now analyze
User active time
Login locations
Weblogs Custom Dashboard
We will now simulate user activity on our infrastructure site by running the following command
Ensure that you run this command in the Training VM
nikto -h infra.domain.com
This command will
Scan and tests web servers for dangerous files/CGIs, outdated server software and other problems. It
performs generic and server type specific checks.
We will import the web-custom-dashboard.json dashboard to web application server logs visualisations for
analysis
Analyzing Wordpress activity
68

69. Analyzing Wordpress activity
69

70. Automation
deploy-scenario-2-infra
Used the Terraform to setup Azure infrastructure for Scenario-2
Resource Group
Virtual Network
Subnet
Public IP
Network Security Group
Network Interface
Storage Account
Virtual Machine
SSH key pair
Local provisioner
Output
Created the simple bash script to execute this whole setup
Initialises Terraform
Obtains temporary Azure session token
Deploys the infrastructure using Terraform plan
deploy-scenario-2-infra-playbook
Automation
70

71. Custom Ansible playbook to setup
Nginx
MySQL
PHP
Wordpress
CLI
Configuration of basic site and initial users
Custom plugin setup and configuration
Beats (Filebeat)
Created a simple bash script to execute the setup
Uses IP address and ssh key pair to setup the entire wordpress stack
Configures filebeat to send logs to the ELK stack
generate-wordpress-activity
Python script which performs automated activity on a wordpress site
Random browsing
Failed logins
Correct logins
Logouts
Random activities
Automation
71

72. Discussion, Use cases and Limitations
Use cases and ideas
Analyzing and understanding site activity and usage patterns to detect, alert or stop anomalous activity
The above method can be used for a wide range of defensive scenarios and other content management suites
like Drupal, etc.
We could act on the logs automatically using wp-cli
Limitations
No significant limitations
Lets Discuss (10 minutes)
Feedback/suggestions on improving this approach
How you have been solving a similar issue / plan to solve one
If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We
shall discuss them in the end of the training.
Use cases and ideas
72

73. IAM CloudTrail logs to defend against stolen
credentials
Overview
In this scenario,
We will see how we can use the AWS metadata service to retrieve IAM keys
We will exploit an application vulnerable to SSRF and to access the AWS metadata service
We will deploy serverless defence that will use Cloud Trail logs to detect and automatically defend our cloud
infrastructure
Introduction
73

74. Serveless Defence
We will now deploy the serverless defence in the coming steps.
Deploying serverless defence
Run the following script to deploy serverless defence for the scenario-4
Ensure that you run this command in the Training VM
deploy-scenario-4-defence
This script will
Use the stored AWS credentials to deploy the Lambda function used in the serverless defence
If it is successful it will print the information to access the machine
If you see any error, please inform one of the trainers
Serverless Defence
74

75. Serverless Defence
75

76. Before the Attack
We will look at current state of our services before the attack
Attached IAM Roles
Lets confirm that IAM role that has been attached to this instance
Navigate to AWS EC2 https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-
1#Instances:sort=instanceId and select the
Select the VM named infra-machine
Notice that a role called ec2accesss3 has been attached to the VM. Which gives read-only full access to s3
buckets.
Before the Attack
76

77. Before the Attack
77

78. Attack
We will now exploit the SSRF vulnerability in one of the applications in the Infrastructure VM to gain access to the IAM
credentials.
Exploiting SSRF to obtain IAM Credentials
Lets get the IAM credentials by querying the AWS Metadata service
Now, enter the following in the input field of the application
file:///etc/passwd
As you can see there is a Local File Inclusion vulnerability
Lets now try to check for SSRF. Enter the following in the input field
http://169.254.169.254/latest/meta-data/
Attack
78

79. The AWS Metadata service provides meta data about the instance such as IP address, instance details and much
more
SSRF Attack
In a Server-Side Request Forgery (SSRF) attack, the attacker can abuse functionality on the server to read or update
internal resources. The attacker can supply or a modify a URL which the code running on the server will read or
submit data to, and by carefully selecting the URLs, the attacker may be able to read server configuration such as
AWS metadata, connect to internal services like http enabled databases or perform post requests towards internal
services which are not intended to be exposed.
Let's get some metadata information like instance credentials :P
It also provides an endpoint to obtain temporary security credentials for the configured role of the instance. We will
now try to get the credentials by accessing
http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2acesss3
Store this data for later use. These are the temporary credentials usable by services on the machine due to the
attached role
SessionTokens
The credentials we have obtained are generated by AssumeRole IAM call and are temporary security credentials with
a session token. Though these credentials work in the same way IAM keys do for the most part, there are some key
differences such as temporary security credentials cannot request for temporary security credentials. The token may
have a validity upto 12 hours.
To revoke a temporary security credential, one must detach the IAM role and revoke the sessions. To read more about
temporary security credentials, please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html
Attack
79

80. Using the stolen IAM Credentials
Lets use the stolen IAM credentials. We can do that by adding the credentials under an AWS cli profile in our training
VM
Configure the credentials found by
Ensure that you run this command in the Training VM
aws configure --profile ssrfkey
As we got temporary session token, we have to edit ~/.aws/credentials and add the session token
[ssrfkey]
aws_access_key_id = xxxxxxxxxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxx
aws_session_token = xxxxxxxxxxxxxxxxxxx
Lets try listing the S3 buckets under the AWS account using the found key
Ensure that you run this command in the Training VM
aws s3 ls --profile ssrfkey
You can see that you are able list the s3 buckets under the account
Lets try to enumerate further by listing all IAM users
Ensure that you run this command in the Training VM
aws iam list-users --profile ssrfkey
Attack
80

81. Lets try to list all the ec2 instances
Ensure that you run this command in the Training VM
aws ec2 describe-instances --profile ssrfkey
This command fails because the role does not have the privileges required
References
AWS Metadata Service
SSRF Attack
Attack
81

82. After initiating attack
The attack is now in progress and has most likely been blocked automatically. We shall verify the same
Slack Alert
You will get a slack alert about attack and the action taken. The alert says that the role has been detached and the
sessions have been revoked
Attached IAM Roles
Lets check if the IAM role has been detached
Navigate to AWS EC2 https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=us-east-
1#Instances:sort=instanceId
Select the VM named infrastructure-vm
After the attack
82

83. Select Actions -> Instance Settings -> Attach/Replace IAM Roles
Notice that the role has been detached from the VM and will no longer be available via the AWS metadata endpoint
Trying to use the credentials
Now that have been alerted that the credentials have been revoked, lets try listing the buckets again
Ensure that you run this command in the Training VM
aws s3 ls --profile ssrfkey
Notice that you are not able to use the key as it has been revoked
After the attack
83

84. Serverless Explanation
We just deployed the iamhandler lambda function
This lambda function constantly monitors the CloudTrail logs for unauthorized requests to AWS API and detaches the
IAM role from a VM along with revoking all the session tokens.
The following parameters can be configured in serverless-defence/scenario-4/config.js
region - AWS Region to deploy in
logGroup - CloudWatch Log Group to monitor
interval - Time interval between scheduled executions
slackUrl - Slack URL to send alerts
slackChannel - Slack channel to send alerts to
Serverless Explanation
84

85. Automation
deploy-scenario-4-defence
Application code performs
Identifying unauthorized API calls in CloudTrail
Revokes the existing sessions for the role
Detaches the role from the instance
Triggers slack alert
Used serverless framework to deploy code to AWS Lambda
Created the simple bash script to deploy the setup
Uses configured AWS default credentials and environment variables
Automation
85

86. Discussion, Use cases and Limitations
Use cases and ideas
Quarantine the machine after detecting a violation
Could send data to ELK stack for analyzing usage and abuse
Limitations
Not really useful when attached IAM role has AdministratorAccess unless MFA is enabled.
CloudTrail logs take anywhere from 5 to 15 minutes to reflect, preventing real-time monitoring and defence
Lets Discuss (10 minutes)
Feedback/suggestions on improving this approach
How you have been solving a similar issue / plan to solve one
If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We
shall discuss them in the end of the training.
Use cases and ideas
86

87. Container logs to audit Kubernetes security
Overview
In this scenario we will see how we can detect a senitive file read operation occuring inside a container in our
Kubernetes cluster.
We will see how to apply serverless defence to automatically stop the attack and apply the fix in near-realtime
Introduction
87

88. Attack
We will now exploit the command injection vulnerability in one of the applications in the Kubernetes cluster.
Accessing the application
We can get the IP address at which the application has been deployed by running the below command
echo $scenario_5_endpoint_ip
We now try to access the application by vising http://$scenario_5_endpoint .
The credentials for the application will are
username: adef
password: batmanvssuperman
Once we authenticate, we can now see the application. Lets try pinging google.com to test the feature.
Input the following into the application
google.com
Exploiting command injection vulnerability in the
application
Attack
88

89. It looks like the application is passing the input to ping command and is returning the output. Lets try to exploit this
Try the below input
;id
As we can see, we are the root user, and lets try to access /etc/shadow . This worked and we are able to see the
output
Attack
89

90. We will shortly recieve a slack alert about this activity from sysdig falco logs
Viewing the log
The entry that triggered this can be found in Stackdriver under the falco logs for this Kubernetes cluster. We can see
the logs by choosing GKE Container -> auto-adef -> default -> falco selection under Google Logging
Attack
90

91. Attack
91

92. Serverless Defence
We will now deploy serverless defence
Run the following script to deploy serverless defence for the scenario-5
deploy-scenario-5-defence
This script will
Use the stored gcloud credentials to deploy the cloud function used in the serverless defence
If you see any error, please inform one of the trainers
Serverless defence
92

93. After serverless defence
Lets try to repeat the same attack again. Try to read /etc/shadow again as shown below
The attack succeeds. But within moments of the attack, we receive a slack alert about the attack. Lets try to access
the application now that we have deployed automated serverless defence
Try accessing /etc/shadow
After defence
93

94. We see an Permission Denied error. Lets try to check the user we are executing commands as by running id
again
After defence
94

95. We see that the application is running as app user which does not have permission to access /etc/shadow
We will also notice that we are not able to ping anymore
After defence
95

96. The capablity CAP_NET_RAW has also been disabled
After defence
96

97. Serverless Explanation
We just deployed the adefscenario5 lambda function
This lambda function constantly monitors Stack Driver logs from our falco daemon for the configured rule. If a
matching entry exists in the logs, it sends a slack alert and automatically re-deploy the affected application with a
more restrictive configuration
The following parameters can be configured in serverless-defence/scenario-5/config.js
rule - Falco rule to look for in logs
slackUrl - Slack URL to send alerts
slackChannel - Slack channel to send alerts to
Serverless Explanation
97

98. Automation
deploy-scenario-5-infra
Created a simple bash script to
Spin up new 2-node GKE cluster
Installs and enable the helm with service account
Installs the node-app deployment
installs the sydig falco
deploy-scenario-5-defence
Application code performs
Updating the deployment with security fixes based on logs
Triggering slack alert
Used gcloud framework to deploy code to cloud functions
Automation
98

99. Created a simple bash script to deploy the setup
Deploys the serverless defence code
Use configured gcloud credentials to deploy the setup
Automation
99

100. Discussion, Use cases and Limitations
Use cases and ideas
Detect and act on intrusions and unexpected behaviour
Limitations
TBA
Lets Discuss (10 minutes)
Feedback/suggestions on improving this approach
How you have been solving a similar issue / plan to solve one
If you come across any ideas and suggestions later, please send them over at the discussion slack channel. We
shall discuss them in the end of the training.
Use cases and ideas
100

101. AWS
We will now delete all services and resources deployed on our AWS account. Though the script only removes the
deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account
created for the training and do not have any other credentials configured.
Ensure that you run this command in the Training VM
nuke-destroy-aws-setup
This script will
Removes the all infrastructure we have setup till now in AWS
Ensure you don't have any data in the AWS account before running the script
AWS
101

102. Azure
We will now delete all services and resources deployed on our Azure account. Though the script only removes the
deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account
created for the training and do not have any other credentials configured.
Ensure that you run this command in the Training VM
nuke-destroy-azure-setup
This script will
Removes the all infrastructure we have setup till now in Azure
Ensure you don't have any data in the Azure account before running the script
Azure
102

103. GCP
We will now delete all services and resources deployed on our GCP account. Though the script only removes the
deployment done during this training, we do not guarantee that. So please ensure that you are using your trial account
created for the training and do not have any other credentials configured.
nuke-destroy-gcp-setup
Run this command in the Cloud Shell
This script will
Removes the all infrastructure we have setup till now in GCP
Ensure you don't have any data in the GCP account before running the script
GCP
103

104. Automation
nuke-destroy-aws-setup
Created the simple bash script to execute
Removes the AWS s3 buckets using default credentials
Terraform destroy the existing infrastructure created for different scenarios in AWS
nuke-destroy-azure-setup
Created the simple bash script to execute
Terraform destroy the existing infrastructure created for scenario-2
nuke-destroy-gcp-setup
Created the simple bash script to execute
Currently it's not performing anything
Automation
104

105. Automation
105

106. About Appsecco
Appsecco is a specialist application security company, founded in 2015, with physical presence in London, Bangalore,
Doha and Boston, providing industry leading security advice that is firmly grounded in commercial reality.
Our services cover the entire software development lifecycle from advising on how build and foster a culture of
security within development teams and organisations to reviewing and advising on the security of applications and
associated infrastructure under development to providing rapid response and advice in the event of a security breach
or incident.
As a team, we are highly qualified and have many years of extensive experience working with clients across multiple
counties and in a wide range of industries and sectors; from financial services to software development,
manufacturing to governmental organisations and consumer brands to ecommerce.
The solutions, advice and insight we deliver to our clients always follows three core principles:
1. It must be pragmatic; taking into account the specific commercial, organisational and operational realities of each
client individually
2. It must genuinely add value; the advice or solutions we provide must addresses the specific problem a client
seeks to solve and have actionable insight to enable them to achieve this
3. Never be purely automated; whenever we are testing for security our reports and output always have significant,
expert, human input to give the greatest possible value for our clients
In addition to their client-facing work our technical team are actively involved in researching and developing new and
better ways to stay secure and can regularly be found presenting their findings at industry conferences and events
ranging from nullcon in India, DevSecCon in London and Singapore, to DEF CON, the world’s largest security
conference held annually in the USA.
Structurally we are a UK Limited company with a wholly owned Indian subsidiary (where the majority of our technical
resource is based) and raised seed funding for our continuing growth in the UK in late 2016.
About Appsecco
106

107. About Appsecco
107

108. Upcoming Conferences
Nullcon 2019 - Goa, India
Black Hat 2019 - Las Vegas, USA
Upcoming Trainings and Conferences
108

109. Upcoming Trainings and Conferences
109

110. References & Resources
Automated Defense using Serverless Computing
AWS
AWS in Plain English
Amazon Web Services - a practical guide
AWS CIS Benchmarks
AWS Security Best Practices
AWS Security Primer
Security auditing tool for AWS environments
Prowler: AWS CIS Benchmark Tool
Nimbostratus -Tools for fingerprinting and exploiting AWS
Aardvark is a multi-account AWS IAM Access Advisor API
Security Monkey
CloudSploit Scans
System Shock: How A Cloud Leak Exposed Accenture's Business
Fullstop - Audit reporting
Abusing the AWS metadata service using SSRF vulnerabilities
AWS Vulnerabilities and the Attacker’s Perspective
Pivoting in Amazon Clouds
Security Tools for AWS
https://github.com/nccgroup/PMapper
https://github.com/nccgroup/aws-inventory
https://yashvier.app.box.com/v/boostawssecurity
Azure
Microsoft Azure in Plain English
Azure Security Centre
Azure security technical capabilities
Security auditing tool for Azure environments
Azure Security Lab Workshop
Enumeration and reconnaissance activities in the Microsoft Azure Cloud
Azure Security and Compliance Blueprint - FedRAMP Web Applications Automation
Secure DevOps Kit for Azure (AzSK)
GCP
Google Infrastructure Security Design Overview
Cloud Security Command Center
Forseti Security: Open-source tools for GCP security
Google Cloud Platform Security Tool
Map AWS services to Google Cloud Platform products
References & Resources
110

111. Serverless
Servleress Framework Documentation
AWS SDK for Javascript
Azure SDK for Node.js
GCP Javascript API Documentaion
Intrusion and Exfiltration in Server-less Architectures
Serverless Architecture
Serverless Technologies
Awesome Serverless
References & Resources
111