Security in IaaS, attacks, hardening, incident response, forensics and all about its automation. Despite I will talk about general concept related to AWS, Azure and GCP, I will show specific demos and threats in AWS and I will go in detail with some caveats and hazards in AWS.
'Future Evolution of the Internet' delivered by Geoff Huston at Everything Op...
Automate or die! Rootedcon 2017
1. Automate or Die!
How to Survive to an Attack in the Cloud
March 3rd 2017
Toni de la Fuente (@ToniBlyx – blyx.com)
Lead Security Operations / Security Architect
7. Ubiquity
• Regions
• Availability Zones
• AWS:
• 16 regions (+2)
• 42 AZ (+4)
• Azure:
• 32 regions
• GCP:
• 6 regions (+8)
• 18 zones (+16)
*CDN locations not included
8. AWS Region
Amazon RDS MySQL
Master
Internet
gateway
Availability Zone 1 Availability Zone 2
Public subnet Public subnet
NAT gateway
EC2
Bastion
10.0.128.5
NAT gateway
EC2
Bastion
10.0.144.5
Alfresco One
Auto Scaling Group
Elastic Load
Balancing
Amazon RDS MySQL
Slave
S3 for
Shared Content
Store
10.0.0.0/16
10.0.128.0/20 10.0.144.0/20
10.0.0.0/19 10.0.32.0/19
Alfresco Index
Auto Scaling Group
Private SubnetPrivate Subnet
Alfresco Server Alfresco Server Alfresco ServerAlfresco Server
Index Server Index Server Index Server Index Server
* Immutable infrastructure
9. Shared Security Model / Responsibility Zones
IaaS
• Data
• Application
• Operating System
• Virtualization
• Infrastructure
• Physical
PaaS
• Data
• Application
• Operating System
• Virtualization
• Infrastructure
• Physical
SaaS
• Data
• Application
• Operating System
• Virtualization
• Infrastructure
• Physical
10. Shared Security Model / Responsibility Zones
AWS manages the
security OF the Cloud
You
AWS foundation services
Compute Storage Database Networking
AWS global infrastructure
Regions
Availability zones
Edge locations
Client-side data encryption Server-side data Encryption Network traffic protection
Platform, applications, identity & access management
Operating system, network & firewall configuration
Customer applications & content
You define your controls
IN the Cloud
* Similar on other providers / subject to changes depending on the service or product
*
13. Disadvantages and Challenges
Cloud Forensics and Operations
Ubiquity Enumeration
Legal jurisdiction
Elasticity Preservation of evidence
Data integrity
Data persistence (replication) Chain of custody
Evidence integrity
Multi-tenancy Data attribution
Chain of custody
Abstract Determine the best evidence
Preservation and visualization of evidence
Quantity of data and Big Data Systems that cannot be investigated or managed in a traditional manner
Knowledge Trained staff
Continuous evolution and new features almost daily
Providers Service level agreement / service level objectives
Relationship client-provider / transparency
14. Traditional vs Cloud Forensics
Processes Traditional Forensics Cloud Forensics
Identification Identification of an event or incident Multiple tools Few tools
Preservation Securitization and assessment of the scene Yes No
Documentation of the scene Yes No
Evidence collection: origin of the evidence Physical hardware Virtual hardware
Evidence collection: location of the evidence Crime scene Provider’s data center
Marking, packaging and transport Physical Digital through the Internet or physical media
Acquisition /
Extraction
Acquisition time Slow Fast
RAM acquisition Yes Dependant
Hash Slow Fast
Erased data recovery Possible Difficult
Metadata acquisition Yes Yes
Time stamp Precise Complex
Installation (action) of forensic software Expensive Cheap
Configuration and availability of forensic software Expensive Cheap
Transport Yes No
Analysis Analysis Slow Fast (potentially)
Presentation Documentation of evidence Acquired evidence Data from many sources
Declaration Common Difficult to explain to a judge
15. Storage Options
Type AWS Azure GCP
Objects S3 Object Storage
• Buckets
• 5TB max per object
• Encryption In-flight and at-rest
Azure Storage
• Blob storage
• 500TB limit per storage account
• Encryption In-flight and at-rest
Google Cloud Storage
• Buckets
• 5TB max per object
• Encryption In-flight and at-rest
SAN EBS (Volumes)
• Volume size: 1GB to 16TB (in 1GB
increments)
• Magnetic, SSD
• Encryption available
• Snapshots
Azure Block Storage
• Page blobs
• Volume size: 1GB to 1TB
• Standard (Magnetic), SSD premium
• Snapshots
• Encryption available
Google Block Storage
• Volume size: 1GB to 10TB
• Magnetic, SSD
• Snapshots
• Encryption by default
NAS Shared Storage (NFS)
• EFS
File Storage (CIFS) Single Node File Server + Others
Archive Glacier Azure Backup Google Cloud Storage Nearline
Migration Import Export / Snowball Import Export Third Party Solution (Iron
Mountain, etc.)
CDN AWS CloudFront (CDN) Azure CDN Google Cloud CDN
* Ephemeral, DBs, Queues, Caching and Storage GW not included
17. Account and Keys in AWS
• Root account: account owner, full access to all resources in the account, very specific tasks
(transfer domain, billing details, support plan)
– Email and password + MFA code (if enabled)
• IAM (Identity and Access Management)
– User name and password + MFA code (if enabled) to access AWS Management Console,
AWS discussion forums, or AWS support center
– SAML
– Users, Groups, Roles, Policies. Instance profiles (role)
• Access Keys: AWS SDKs, REST, or Query APIs (AWS CLI)
– Access Key i.e: AKIAIOSFODNN7EXAMPLE
– Secret Access Key i.e: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
• STS (Security Token Service):
– temporary, limited-privilege credentials for IAM users or for users that you authenticate
(also for federated users), and for instances (instance profile)
• Key Pairs: Key pairs are used only for:
– Amazon EC2 (SSH) and Amazon CloudFront (sign URLs or sign cookies)
*Become an IAM Ninja: https://youtu.be/Du478i9O_mc
18. AWS Metadata Server
“Instance metadata is data about
your instance that you can use to
configure or manage the running
instance”
“Anyone who can access the instance
can view its metadata. Therefore,
you should take suitable precautions
to protect sensitive data”
# curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
Hostname
iam/
instance-action
instance-id
instance-type
local-hostname
local-ipv4
Mac
metrics/
network/
placement/
product-codes
Profile
public-keys/
reservation-id
security-groups
services/
20. Common incidents
• Access Keys compromise, abuses of unconsciously published keys:
e.g. a developer publishing their keys on SourceCode repo (Github,
Bitbucket, etc) when commit, keys stolen from workstation, keys
hardcoded in application files (bin or config), resources created for
criminal purposes, mining, etc.
• Phishing attacks: your instance is going to be retired (targeted to
admins). Hard to detect phising bc comes with HTTPS, S3, etc.
• Compromised resources: e.g. an unpatched EC2 instance may be
infected with malware and act as a botnet. Poisioned AMI.
• Unintentional abuses: e.g. a crawler-type own process being
classified as a DDoS attack by a third party.
• Abuses committed by users: e.g. malware or other type of illegal
content being published by the end user of an AWS service on a
public S3 bucket.
21. Common incidents
• Application running in a role: can lead to an access to the
application role and stole Access Key or access metadata service
• Abuses related to configuration failures: e.g. a web-based proxy
service incorrectly configured being used as an open proxy. SMTP
relay, etc.
• Infection through 3rd party services: you give them keys to
perform actions (DataDog incident 2016)
• Hybrid attacks: attacks partly carried out from a Cloud-based
system or data storing on S3, even when mobile devices or
personal computers are used.
• Organized crime of all sort
• False positives
• Did I say CONFIGURATION FAILURES*?
22. Create a new one:
$ aws sts get-session-token --duration-seconds 129600
Persistence
• Instance compromised might become Access Keys Compromised
– Metadata service (inside an Instance)
• curl http://169.254.169.254/latest/meta-data/iam/
• curl http://169.254.169.254/latest/meta-data/iam/security-
credentials/<*instanceRole>
{
"Code" : "Success",
"LastUpdated" : "2017-02-02T03:07:42Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE",
"SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC",
"Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED",
"Expiration" : "2017-02-02T09:22:37Z”
}
*If it has been attached to the instance
23. Create a new one:
$ aws sts get-session-token --duration-seconds 129600
Persistence
• Instance compromised might become a Key Compromised
– Metadata service (inside an instance)
• curl http://169.254.169.254/latest/meta-data/iam/
• curl http://169.254.169.254/latest/meta-data/iam/security-
credentials/<instanceRole>
{
"Code" : "Success",
"LastUpdated" : "2017-02-02T03:07:42Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE",
"SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC",
"Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED",
"Expiration" : "2017-02-02T09:22:37Z”
}
24. Serverless!!!
• Who is auditing
serverless?
• Amazon Lambda
• CloudWatch
• Azure Cloud Functions
• WebJobs
• Google Cloud
Functions
25. MadKing Attack
• https://github.com/ThreatResponse/mad-king
• Using stolen access keys. Uses Zappa.io. Creates an API Gateway and
Lambda function
• Features
• Disable CloudTrails
• Encrypt CloudTrails
• Generate New Developer Access Keys
• Stop Instances
• Terminate Instances
• Burn them all (Destroy all instances) – really Mad King m/
https://danielgrzelak.com/backdooring-an-aws-account-da007d36f8f9#.ut0x2bjv5
26.
27. AWS Lambda Infection Toolkit
• https://github.com/Miserlou/mackenzie (zappa.io guy)
• Persistent Lambda Malware PoC
• Features
• Encrypt with pubkey
• Exfil via POST, S3, Email, SMS, Network Resource Tags
• Install Flask backdoor
• Infect old package sources
• Infect all available functions
• Create re-infection handlers
Gone in 60 Millisecons (33c3): https://www.youtube.com/watch?v=YZ058hmLuv0
28. Other Attacks Tools
• Metasploit AWS module
• IAM privilege enumeration module
• Lambda module
• S3 bucket and access enumeration
• AWS pwn
• Reconnaissance, exploitation and exfiltration
• https://github.com/dagrz/aws_pwn
• Comulus Cloud Attack Module (not an attack just vulnerable sample
code)
• Presented at RSA 2017 (Serverless Security)
• https://github.com/devsecops/lambhack
30. Now what?
• Control
• Impact
• Recover
• Investigate
• Improve
• Notifications from AWS
• Access activity (IAM)
• Billing activity (Budget alerts)
• Logs
• Other
• Third parties (dedicated tools)
• NIDS (Snort, Suricata, etc.)
• HIDS (OSSEC, Osquery, rkhunter,
Auditd)
• SIEM
Incident Indicators
31. Sample Task List / Workflow
Compromised
Instance
Start
Forensic
Workstation
Live
or
Dead
Attach the
Tools Volume
Apply
Isolation
SG
Stolen
API Keys
Check new
resources created
Disable Keys
Make API log report if
enabled
If found
Isolate them
Create a report
Attach the
Evidence
Collection
Volume
Isolate
it?
Log in to the
instance
Perform
Evidence
Acquisition
Take
snapshot to
all volumes
Stop it
Make Volumes to
Snapshots
Attach Volumes
to Forensic
Workstation
Attach the
Evidence
Collection
Volume
Log in to the
Forensic
Workstation
Analyze /
Further
Investigation
Incident
Live
Dead
Create
Support Case
with Provider
Revoke Access
Revoke Sessions
Outside Info
Acquisition
(instance
profile,endpo
ints,metadat
a, etc)_
Perform
Timeline
Pre-built
Volatiliy
frofile
Pre-built
LiME krnl
mod
RAM
Acquisition
Yes
NIC Network
Scan
Open an
Internal Case
Separate Network
with Internet
Access to Scan
CAINE /
SIFT / DEFT
/ FCCU /
HELIX3 /
FIRE
Windows_Life_Response
Sysinternals
Nirsoft
FTK Imager
Autopsy
Sleuthkit
Trigger a
Network
Capture
TAG
Resources
under
investigation
*
32. Sample Task List / Workflow
Compromised
Instance
Start
Forensic
Workstation
Live
or
Dead
Attach the
Tools Volume
Apply
Isolation
SG
Stolen
API Keys
Check new
resources created
Disable Keys
Make API log report if
enabled
If found
Isolate them
Create a report
Attach the
Evidence
Collection
Volume
Isolate
it?
Log in to the
instance
Perform
Evidence
Acquisition
Take
snapshot to
all volumes
Stop it
Make Volumes to
Snapshots
Attach Volumes
to Forensic
Workstation
Attach the
Evidence
Collection
Volume
Log in to the
Forensic
Workstation
Analyze /
Further
Investigation
Incident
Live
Dead
Create
Support Case
with Provider
Revoke Access
Revoke Sessions
Outside Info
Acquisition
(instance
profile,endpo
ints,metadat
a, etc)_
Perform
Timeline
Pre-built
Volatiliy
frofile
Pre-built
LiME krnl
mod
RAM
Acquisition
Yes
NIC Network
Scan
Open an
Internal Case
Separate Network
with Internet
Access to Scan
CAINE /
SIFT / DEFT
/ FCCU /
HELIX3 /
FIRE
Windows_Life_Response
Sysinternals
Nirsoft
FTK Imager
Autopsy
Sleuthkit
Trigger a
Network
Capture
TAG
Resources
under
investigation
*
33. Outside Info Acquisition
Perform
Evidence
Acquisition
AWS Infrastructure Logs:
CloudTrail and VPCFlowLogs
AWS Service Logs:
S3 Logs, RDS Logs, Lambda, etc.
Host Based Logs
Messages/System, security, audit, applications, etc.
More Inside:
instance profile, endpoints, syslogs, screen, metadata, etc
More Outside:
Limits, check resources creation from date (all regions)
40. Persistence Prevention (AWS)
• Instance compromised might become a Key Compromised
– UserData in CloudFormation: watch out!
• Prevent it is not very difficult:
• STS tokens can’t be revoked (you only can disable permissions)
• CloudTrail may help to detect it (if enabled!)
– watch the watcher
• It can shutdown your company! (you won’t be the first one: CodeSpaces)
• Use multiple AWS accounts!
iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP
41. Instance / Network / Provider
• Put all what you need in your well known AMI:
• Hardening applied / Tested (Packer/Vagrant)
• CIS Benchmark!
• No config / access needed
• Local tools
• Osquery / OSSEC / rkhunter
• Update rules / serverless
• local configuration (SELinux/AppArmour)
• AuditD
• Collect telemetry host network data (snort/suricata)
• Collect everything your provider allows you
• Networking
• APIs / Accesses
• Red Team / Third party pentesting*
46. TL;DR
• Automation for everything (deployment multi AZ,
hardening, response, recovery/recreation, centralized
logging, log everything!)
• Encryption Everywhere (any layer, any content, on-prem,
on-transit)
• Account Separation and MFA (prod, test, devel, etc.)
• Least Privilege
• Go to Immutability / Ephemeral
• Expect to be Hacked: Buy Bitcoins…
48. References
• Cloud Security Alliance, Mapping the Forensic Standard ISO/IEC 27037 to Cloud Computing, June 2013
• Dr. Keyun Ruan University College Dublin, Designing a Forensic-enabling Cloud Ecosystem, 2013
• International Standard ISO/IEC 27037, Information technology — Security techniques — Guidelines for identification, collection, acquisition, and preservation of digital evidence, October
2012
• Josiah Dykstra, Digital Forensics for IaaS Cloud Computing, June 2012
• Keyun Ruan, Ibrahim Baggili (PhD), Prof Joe Carthy, Prof Tahar Kechadi University College Dublin, Zayed University, Survey on Cloud forensics and critical criteria for Cloud forensic capability:
A preliminary analysis
• Keyun Ruan, Joe Carthy, Tahar Kechadi and Mark Crosbie, Cloud Forensics
• Keyun Ruan, University College Dublin, Cloud Forensics: challenges & opportunitiess, 2010
• NIST Cloud Computing Forensic Science Working Group Information Technology Laboratory, NIST Cloud Computing Forensic Science Challenges, June 2014
• Peter Mell Timothy Grance, NIST Special Publication 800-145, The NIST Definition of Cloud Computing, September 2011
• Report From the First Digital Forensic Research Workshop (DFRWS), A Road Map for Digital Forensic Research, August 2001
• Forensics-as-a-Service (FaaS): Computer Forensic Workflow Management and Processing Using Cloud. Yuanfeng Wen, Xiaoxi Man, Khoa Le and Weidong Shi
• http://static1.squarespace.com/static/5417f7f9e4b0b77770545590/t/56f3c598906340a7f6e78dbd/1458816415654/AWS_Cloud_and_Security.pdf
• https://www.blackhat.com/docs/us-16/materials/us-16-Amiga-Account-Jumping-Post-Infection-Persistency-And-Lateral-Movement-In-AWS-wp.pdf
• https://alestic.com/2015/10/aws-iam-readonly-too-permissive/
• Backdooring an AWS account
• Exploring an AWS account post-compromise
• Disrupting AWS logging
• AWS IAM "ReadOnlyAccess" Managed Policy is Too Permissive (For Us)
• Access Keys will kill you before you kill the password
• Account Jumping Post Infection Persistency and Lateral Movement in AWS
• Disrupt CloudTrail and pwning automation tools
• RSA 2017 talk: Cloud Security Automate or Die, same tittle as mine but a bit different approach
• RSA 2017 talk: Securing Serverless applications in the Cloud
• RSA 2017 talk: DevSecOps on the Offense: Automating Amazon Web Services Account Takeover
49. Thanks!
Special Thanks to:
Alfresco DevOps Team
Andrew K. @andrewkrug & Joel F., ThreatResponse.cloud Team
Daniel Grzelak @dagrz
Lorenzo Martinez @lawwait
Editor's Notes
Intro cloud
Dealing with Incidents
AWS specifics
Attacks
Incident Response
Assessment and Hardening
IaaS but not exclusively
Infrastructure as Code (dev and ops)
Security as Code
APIs, APIs everywhere
You have to look at what you have but also to what you don´t have!
Perimeter!!
PCI-DSS compliance other for NIST, etc.
Kinda Immutable infrastructure / instances (bastion)
Logging externally, config management, monitoring
Blue-green upgrades
Canary upgrades
Attacks with keys are possible if a misconfiguration is in place
FaaS (Function as a Service). Run code without managing servers, pay by consumed compute. Scales automatically, HA, containerized. Real time data and file processing, web apps, transfoms, backend actions, etc.
Lambda 2014: node.js python, java, C# (haskell, shell, etc with hacks)
Azure Cloud Functions 2016: javascript, C#, F#, Python, PHP, Bash, Batch, and PowerShell
Google Cloud Functions: node.js