Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automate or die! Rootedcon 2017

5,955 views

Published on

Security in IaaS, attacks, hardening, incident response, forensics and all about its automation. Despite I will talk about general concept related to AWS, Azure and GCP, I will show specific demos and threats in AWS and I will go in detail with some caveats and hazards in AWS.

Published in: Internet
  • Be the first to comment

Automate or die! Rootedcon 2017

  1. 1. Automate or Die! How to Survive to an Attack in the Cloud March 3rd 2017 Toni de la Fuente (@ToniBlyx – blyx.com) Lead Security Operations / Security Architect
  2. 2. Once upon a time… September 2015
  3. 3. Foundation
  4. 4. First Things First *NIST Definition
  5. 5. Ubiquity • Regions • Availability Zones • AWS: • 16 regions (+2) • 42 AZ (+4) • Azure: • 32 regions • GCP: • 6 regions (+8) • 18 zones (+16) *CDN locations not included
  6. 6. AWS Region Amazon RDS MySQL Master Internet gateway Availability Zone 1 Availability Zone 2 Public subnet Public subnet NAT gateway EC2 Bastion 10.0.128.5 NAT gateway EC2 Bastion 10.0.144.5 Alfresco One Auto Scaling Group Elastic Load Balancing Amazon RDS MySQL Slave S3 for Shared Content Store 10.0.0.0/16 10.0.128.0/20 10.0.144.0/20 10.0.0.0/19 10.0.32.0/19 Alfresco Index Auto Scaling Group Private SubnetPrivate Subnet Alfresco Server Alfresco Server Alfresco ServerAlfresco Server Index Server Index Server Index Server Index Server * Immutable infrastructure
  7. 7. Shared Security Model / Responsibility Zones IaaS • Data • Application • Operating System • Virtualization • Infrastructure • Physical PaaS • Data • Application • Operating System • Virtualization • Infrastructure • Physical SaaS • Data • Application • Operating System • Virtualization • Infrastructure • Physical
  8. 8. Shared Security Model / Responsibility Zones AWS manages the security OF the Cloud You AWS foundation services Compute Storage Database Networking AWS global infrastructure Regions Availability zones Edge locations Client-side data encryption Server-side data Encryption Network traffic protection Platform, applications, identity & access management Operating system, network & firewall configuration Customer applications & content You define your controls IN the Cloud * Similar on other providers / subject to changes depending on the service or product *
  9. 9. Shared Security Model / Responsibility Zones
  10. 10. Challenges in Case of an Incident
  11. 11. Disadvantages and Challenges Cloud Forensics and Operations Ubiquity Enumeration Legal jurisdiction Elasticity Preservation of evidence Data integrity Data persistence (replication) Chain of custody Evidence integrity Multi-tenancy Data attribution Chain of custody Abstract Determine the best evidence Preservation and visualization of evidence Quantity of data and Big Data Systems that cannot be investigated or managed in a traditional manner Knowledge Trained staff Continuous evolution and new features almost daily Providers Service level agreement / service level objectives Relationship client-provider / transparency
  12. 12. Traditional vs Cloud Forensics Processes Traditional Forensics Cloud Forensics Identification Identification of an event or incident Multiple tools Few tools Preservation Securitization and assessment of the scene Yes No Documentation of the scene Yes No Evidence collection: origin of the evidence Physical hardware Virtual hardware Evidence collection: location of the evidence Crime scene Provider’s data center Marking, packaging and transport Physical Digital through the Internet or physical media Acquisition / Extraction Acquisition time Slow Fast RAM acquisition Yes Dependant Hash Slow Fast Erased data recovery Possible Difficult Metadata acquisition Yes Yes Time stamp Precise Complex Installation (action) of forensic software Expensive Cheap Configuration and availability of forensic software Expensive Cheap Transport Yes No Analysis Analysis Slow Fast (potentially) Presentation Documentation of evidence Acquired evidence Data from many sources Declaration Common Difficult to explain to a judge
  13. 13. Storage Options Type AWS Azure GCP Objects S3 Object Storage • Buckets • 5TB max per object • Encryption In-flight and at-rest Azure Storage • Blob storage • 500TB limit per storage account • Encryption In-flight and at-rest Google Cloud Storage • Buckets • 5TB max per object • Encryption In-flight and at-rest SAN EBS (Volumes) • Volume size: 1GB to 16TB (in 1GB increments) • Magnetic, SSD • Encryption available • Snapshots Azure Block Storage • Page blobs • Volume size: 1GB to 1TB • Standard (Magnetic), SSD premium • Snapshots • Encryption available Google Block Storage • Volume size: 1GB to 10TB • Magnetic, SSD • Snapshots • Encryption by default NAS Shared Storage (NFS) • EFS File Storage (CIFS) Single Node File Server + Others Archive Glacier Azure Backup Google Cloud Storage Nearline Migration Import Export / Snowball Import Export Third Party Solution (Iron Mountain, etc.) CDN AWS CloudFront (CDN) Azure CDN Google Cloud CDN * Ephemeral, DBs, Queues, Caching and Storage GW not included
  14. 14. AWS Specifics
  15. 15. Account and Keys in AWS • Root account: account owner, full access to all resources in the account, very specific tasks (transfer domain, billing details, support plan) – Email and password + MFA code (if enabled) • IAM (Identity and Access Management) – User name and password + MFA code (if enabled) to access AWS Management Console, AWS discussion forums, or AWS support center – SAML – Users, Groups, Roles, Policies. Instance profiles (role) • Access Keys: AWS SDKs, REST, or Query APIs (AWS CLI) – Access Key i.e: AKIAIOSFODNN7EXAMPLE – Secret Access Key i.e: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY • STS (Security Token Service): – temporary, limited-privilege credentials for IAM users or for users that you authenticate (also for federated users), and for instances (instance profile) • Key Pairs: Key pairs are used only for: – Amazon EC2 (SSH) and Amazon CloudFront (sign URLs or sign cookies) *Become an IAM Ninja: https://youtu.be/Du478i9O_mc
  16. 16. AWS Metadata Server “Instance metadata is data about your instance that you can use to configure or manage the running instance” “Anyone who can access the instance can view its metadata. Therefore, you should take suitable precautions to protect sensitive data” # curl http://169.254.169.254/latest/meta-data/ ami-id ami-launch-index ami-manifest-path block-device-mapping/ Hostname iam/ instance-action instance-id instance-type local-hostname local-ipv4 Mac metrics/ network/ placement/ product-codes Profile public-keys/ reservation-id security-groups services/
  17. 17. Attacks
  18. 18. Common incidents • Access Keys compromise, abuses of unconsciously published keys: e.g. a developer publishing their keys on SourceCode repo (Github, Bitbucket, etc) when commit, keys stolen from workstation, keys hardcoded in application files (bin or config), resources created for criminal purposes, mining, etc. • Phishing attacks: your instance is going to be retired (targeted to admins). Hard to detect phising bc comes with HTTPS, S3, etc. • Compromised resources: e.g. an unpatched EC2 instance may be infected with malware and act as a botnet. Poisioned AMI. • Unintentional abuses: e.g. a crawler-type own process being classified as a DDoS attack by a third party. • Abuses committed by users: e.g. malware or other type of illegal content being published by the end user of an AWS service on a public S3 bucket.
  19. 19. Common incidents • Application running in a role: can lead to an access to the application role and stole Access Key or access metadata service • Abuses related to configuration failures: e.g. a web-based proxy service incorrectly configured being used as an open proxy. SMTP relay, etc. • Infection through 3rd party services: you give them keys to perform actions (DataDog incident 2016) • Hybrid attacks: attacks partly carried out from a Cloud-based system or data storing on S3, even when mobile devices or personal computers are used. • Organized crime of all sort • False positives • Did I say CONFIGURATION FAILURES*?
  20. 20. Create a new one: $ aws sts get-session-token --duration-seconds 129600 Persistence • Instance compromised might become Access Keys Compromised – Metadata service (inside an Instance) • curl http://169.254.169.254/latest/meta-data/iam/ • curl http://169.254.169.254/latest/meta-data/iam/security- credentials/<*instanceRole> { "Code" : "Success", "LastUpdated" : "2017-02-02T03:07:42Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE", "SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC", "Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED", "Expiration" : "2017-02-02T09:22:37Z” } *If it has been attached to the instance
  21. 21. Create a new one: $ aws sts get-session-token --duration-seconds 129600 Persistence • Instance compromised might become a Key Compromised – Metadata service (inside an instance) • curl http://169.254.169.254/latest/meta-data/iam/ • curl http://169.254.169.254/latest/meta-data/iam/security- credentials/<instanceRole> { "Code" : "Success", "LastUpdated" : "2017-02-02T03:07:42Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE", "SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC", "Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED", "Expiration" : "2017-02-02T09:22:37Z” }
  22. 22. Serverless!!! • Who is auditing serverless? • Amazon Lambda • CloudWatch • Azure Cloud Functions • WebJobs • Google Cloud Functions
  23. 23. MadKing Attack • https://github.com/ThreatResponse/mad-king • Using stolen access keys. Uses Zappa.io. Creates an API Gateway and Lambda function • Features • Disable CloudTrails • Encrypt CloudTrails • Generate New Developer Access Keys • Stop Instances • Terminate Instances • Burn them all (Destroy all instances) – really Mad King m/ https://danielgrzelak.com/backdooring-an-aws-account-da007d36f8f9#.ut0x2bjv5
  24. 24. AWS Lambda Infection Toolkit • https://github.com/Miserlou/mackenzie (zappa.io guy) • Persistent Lambda Malware PoC • Features • Encrypt with pubkey • Exfil via POST, S3, Email, SMS, Network Resource Tags • Install Flask backdoor • Infect old package sources • Infect all available functions • Create re-infection handlers Gone in 60 Millisecons (33c3): https://www.youtube.com/watch?v=YZ058hmLuv0
  25. 25. Other Attacks Tools • Metasploit AWS module • IAM privilege enumeration module • Lambda module • S3 bucket and access enumeration • AWS pwn • Reconnaissance, exploitation and exfiltration • https://github.com/dagrz/aws_pwn • Comulus Cloud Attack Module (not an attack just vulnerable sample code) • Presented at RSA 2017 (Serverless Security) • https://github.com/devsecops/lambhack
  26. 26. Incident Response
  27. 27. Now what? • Control • Impact • Recover • Investigate • Improve • Notifications from AWS • Access activity (IAM) • Billing activity (Budget alerts) • Logs • Other • Third parties (dedicated tools) • NIDS (Snort, Suricata, etc.) • HIDS (OSSEC, Osquery, rkhunter, Auditd) • SIEM Incident Indicators
  28. 28. Sample Task List / Workflow Compromised Instance Start Forensic Workstation Live or Dead Attach the Tools Volume Apply Isolation SG Stolen API Keys Check new resources created Disable Keys Make API log report if enabled If found Isolate them Create a report Attach the Evidence Collection Volume Isolate it? Log in to the instance Perform Evidence Acquisition Take snapshot to all volumes Stop it Make Volumes to Snapshots Attach Volumes to Forensic Workstation Attach the Evidence Collection Volume Log in to the Forensic Workstation Analyze / Further Investigation Incident Live Dead Create Support Case with Provider Revoke Access Revoke Sessions Outside Info Acquisition (instance profile,endpo ints,metadat a, etc)_ Perform Timeline Pre-built Volatiliy frofile Pre-built LiME krnl mod RAM Acquisition Yes NIC Network Scan Open an Internal Case Separate Network with Internet Access to Scan CAINE / SIFT / DEFT / FCCU / HELIX3 / FIRE Windows_Life_Response Sysinternals Nirsoft FTK Imager Autopsy Sleuthkit Trigger a Network Capture TAG Resources under investigation *
  29. 29. Sample Task List / Workflow Compromised Instance Start Forensic Workstation Live or Dead Attach the Tools Volume Apply Isolation SG Stolen API Keys Check new resources created Disable Keys Make API log report if enabled If found Isolate them Create a report Attach the Evidence Collection Volume Isolate it? Log in to the instance Perform Evidence Acquisition Take snapshot to all volumes Stop it Make Volumes to Snapshots Attach Volumes to Forensic Workstation Attach the Evidence Collection Volume Log in to the Forensic Workstation Analyze / Further Investigation Incident Live Dead Create Support Case with Provider Revoke Access Revoke Sessions Outside Info Acquisition (instance profile,endpo ints,metadat a, etc)_ Perform Timeline Pre-built Volatiliy frofile Pre-built LiME krnl mod RAM Acquisition Yes NIC Network Scan Open an Internal Case Separate Network with Internet Access to Scan CAINE / SIFT / DEFT / FCCU / HELIX3 / FIRE Windows_Life_Response Sysinternals Nirsoft FTK Imager Autopsy Sleuthkit Trigger a Network Capture TAG Resources under investigation *
  30. 30. Outside Info Acquisition Perform Evidence Acquisition AWS Infrastructure Logs: CloudTrail and VPCFlowLogs AWS Service Logs: S3 Logs, RDS Logs, Lambda, etc. Host Based Logs Messages/System, security, audit, applications, etc. More Inside: instance profile, endpoints, syslogs, screen, metadata, etc More Outside: Limits, check resources creation from date (all regions)
  31. 31. IRDF Automation Tools
  32. 32. Tools • March 2016: • https://blyx.com/2016/03/11/forensics-in-aws-an-introduction/ • June 2016: • https://blyx.com/2016/06/16/cloud-forensics-caine7-on-aws/ • August 2016: • Threat Response (Presented in BlackHat 2016) • https://s3-us-west-2.amazonaws.com/threatresponse-static/us-16-Krug- Hardening-AWS-Environments-and-Automating-Incident-Response-for-AWS- Compromises-wp.pdf
  33. 33. Pre-Automation POC – AWS CLI (Scripts) # DISABLE STOLEN KEYS aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name Bob aws iam delete-access-key --access-key AKIDPMS9RO4H3FEXAMPLE --user-name Bob # LOOK FOR NEW RESOURCES CREATED aws ec2 describe-instances --region us-east-1 --query 'Reservations[].Instances[?LaunchTime>=`2017-02- 3`][].{id: InstanceId, type: InstanceType, launched: LaunchTime}' # TAG INSTANCE aws ec2 create-tags --resources i-INSTANCE-ID --tags “Key=Environment, Value=Quarantine:REFERENCE-ID” # ISOLATE AN INSTANCE IN A VPC aws ec2 create-security-group --group-name isolation-sg --description “Security group to isolate a EC2-VPC instance” --vpc-id vpc-1a2b3c4d aws ec2 authorize-security-group-ingress --group-id sg-BLOCK-ID --protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32 aws ec2 authorize-security-group-egress --group-id sg-BLOCK-ID --protocol ‘tcp’ --port 80 --cidr ‘0.0.0.0/0’ aws ec2 modify-instance-attribute --instance-id i-INSTANCE-ID --groups sg-BLOCK-ID # CREATE VOLUME SNAPSHOT aws ec2 create-snapshot –-volume-id vol-xxxx –-description “IR-ResponderName- Date-REFERENCE-ID”
  34. 34. Threat Response Tool • Incident Response Tool for AWS • http://threatresponse.cloud/ • Compromised AWS API credentials (Access Keys) • Mitigate compromise: Lock • Compromised EC2 instance • Mitigate compromise • Isolation • Collect evidence • Memory acquisition • Create an Incident Response Workstation in AWS • Start an EC2 instance • Analysis of collected evidences • WebApp (ThreatResponseWeb) • RAM (volatility) • Disk (Log2time + TimeSketch) • AWS hardening • Threatprep • API • Modules
  35. 35. <DEMO> ThreatResponse: aws_ir, margaritashotgun
  36. 36. Assessment and Hardening
  37. 37. Persistence Prevention (AWS) • Instance compromised might become a Key Compromised – UserData in CloudFormation: watch out! • Prevent it is not very difficult: • STS tokens can’t be revoked (you only can disable permissions) • CloudTrail may help to detect it (if enabled!) – watch the watcher • It can shutdown your company! (you won’t be the first one: CodeSpaces) • Use multiple AWS accounts! iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP
  38. 38. Instance / Network / Provider • Put all what you need in your well known AMI: • Hardening applied / Tested (Packer/Vagrant) • CIS Benchmark! • No config / access needed • Local tools • Osquery / OSSEC / rkhunter • Update rules / serverless • local configuration (SELinux/AppArmour) • AuditD • Collect telemetry host network data (snort/suricata) • Collect everything your provider allows you • Networking • APIs / Accesses • Red Team / Third party pentesting*
  39. 39. API calls • Who • When • What call • What resources • Where (from)
  40. 40. Auditing, Assessment and Hardening Tools • AWS Trusted Advisor • AWS CloudTrail / Azure Operational Insights • AWS CloudFormation • AWS Config Rules • Alfresco: Prowler / Automate Hardening CIS Section 3* / OpenSCAP fix (AWS) • Nccgroup: Scout2 (AWS) • Netflix: SecurityMonkey, EDDA, FIDO (AWS) • Capital One: CloudCustodian (AWS) • AWS CIS Benchmark Python code and Lambda functions (AWS) • CloudSploit (AWS) • Widdix Hardening Templates (AWS) • Awslimitchecker (AWS) • OMS Security & Compliance (Azure) • Spotify: gcp-audit (GCP) • *Analytics (ELK, Splunk, Nuix etc) • Git Secrets (AWS)
  41. 41. <DEMO> Hardening Automation with templates, Prowler, Security Monkey
  42. 42. Takeaways Samples, templates, code, links and this presentation is already available at: https://github.com/toniblyx/rootedcon2017
  43. 43. TL;DR • Automation for everything (deployment multi AZ, hardening, response, recovery/recreation, centralized logging, log everything!) • Encryption Everywhere (any layer, any content, on-prem, on-transit) • Account Separation and MFA (prod, test, devel, etc.) • Least Privilege • Go to Immutability / Ephemeral • Expect to be Hacked: Buy Bitcoins…
  44. 44. Questions? toni@blyx.com - @ToniBlyx
  45. 45. References • Cloud Security Alliance, Mapping the Forensic Standard ISO/IEC 27037 to Cloud Computing, June 2013 • Dr. Keyun Ruan University College Dublin, Designing a Forensic-enabling Cloud Ecosystem, 2013 • International Standard ISO/IEC 27037, Information technology — Security techniques — Guidelines for identification, collection, acquisition, and preservation of digital evidence, October 2012 • Josiah Dykstra, Digital Forensics for IaaS Cloud Computing, June 2012 • Keyun Ruan, Ibrahim Baggili (PhD), Prof Joe Carthy, Prof Tahar Kechadi University College Dublin, Zayed University, Survey on Cloud forensics and critical criteria for Cloud forensic capability: A preliminary analysis • Keyun Ruan, Joe Carthy, Tahar Kechadi and Mark Crosbie, Cloud Forensics • Keyun Ruan, University College Dublin, Cloud Forensics: challenges & opportunitiess, 2010 • NIST Cloud Computing Forensic Science Working Group Information Technology Laboratory, NIST Cloud Computing Forensic Science Challenges, June 2014 • Peter Mell Timothy Grance, NIST Special Publication 800-145, The NIST Definition of Cloud Computing, September 2011 • Report From the First Digital Forensic Research Workshop (DFRWS), A Road Map for Digital Forensic Research, August 2001 • Forensics-as-a-Service (FaaS): Computer Forensic Workflow Management and Processing Using Cloud. Yuanfeng Wen, Xiaoxi Man, Khoa Le and Weidong Shi • http://static1.squarespace.com/static/5417f7f9e4b0b77770545590/t/56f3c598906340a7f6e78dbd/1458816415654/AWS_Cloud_and_Security.pdf • https://www.blackhat.com/docs/us-16/materials/us-16-Amiga-Account-Jumping-Post-Infection-Persistency-And-Lateral-Movement-In-AWS-wp.pdf • https://alestic.com/2015/10/aws-iam-readonly-too-permissive/ • Backdooring an AWS account • Exploring an AWS account post-compromise • Disrupting AWS logging • AWS IAM "ReadOnlyAccess" Managed Policy is Too Permissive (For Us) • Access Keys will kill you before you kill the password • Account Jumping Post Infection Persistency and Lateral Movement in AWS • Disrupt CloudTrail and pwning automation tools • RSA 2017 talk: Cloud Security Automate or Die, same tittle as mine but a bit different approach • RSA 2017 talk: Securing Serverless applications in the Cloud • RSA 2017 talk: DevSecOps on the Offense: Automating Amazon Web Services Account Takeover
  46. 46. Thanks! Special Thanks to: Alfresco DevOps Team Andrew K. @andrewkrug & Joel F., ThreatResponse.cloud Team Daniel Grzelak @dagrz Lorenzo Martinez @lawwait

×