Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Anatomy of an AWS account Cryptojack

128 views

Published on

This is a story about a cryptojack security incident involving one of CHT customers’ AWS development accounts. I will discuss not only the incident and response, but also some of the ways to prevent this type of event from occurring, how to detect it, and forensics data for which to look.

It’s a personal story/lesson from the field and I would like to share it with people in the industry to help them avoid this pitfall.

About a year ago I was a part of an incident where one of our customers reported a security event involving one of their numerous AWS accounts used for development purposes. The AWS account in question already generated about $20k worth of EC2 compute charges when they discovered this breach. There were about 200 or so top-end, CPU-heavy Windows machines spread out across the world running at 100% CPU for over 2 days, all apparently mining bitcoin.

No CloudTrail audit configuration was enabled for this account so there was no audit data available to identify what happened and who did what.

Our Cloud governance platform captures the state of AWS accounts and configurations, and historical data for some of these settings. Our team spent a few days reconstructing the state of things and how events transpired by looking at our backup data on a timeline. It was clear from our data that one of their admin employee account AWS credentials got compromised/leaked and were used to spin up all these resources. It also appeared that this attack was entirely automated and only needed sufficient AWS credentials as an input. An attacker also covered up their tracks and tried to “frame” another innocent user!

Luckily, the data we had cleared this user’s name and the customer received full AWS credits for the breach to cover the loss, but this was an important lesson for us and for our customer. This was an entirely preventable incident.

We saw a similar pattern/attack on our own infrastructure. The attempt failed due to some simple security measures we’ve taken. I’ll talk about some of the ways to prevent this event from occurring, how to detect it, and the forensics data to look for: things like enforcing MFA, using external Idp, removing the need for the AWS key and secret key by leveraging roles and instance profiles to grant permissions, etc. I will also talk about the importance of having the CloudTrail audit feature enabled by default in AWS.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Anatomy of an AWS account Cryptojack

  1. 1. Anatomy of an AWS Account Cryptojack DevOpsDays Boston 2018 Presented by Anton Gurov 9/24/18
  2. 2. Introduction
  3. 3. 3 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. About me Anton Gurov • Director of TechOps @ CloudHealth Technologies - Security - Compliance - Operations • Experience in PCI-DSS/SOC2/GDPR compliance - Private/Hybrid/Cloud • Mobile payments • Ad tech • Cloud management • 3 successful exits • Avid car racer, juggler and acro-base! • Contact: www.linkedin.com/in/antongurov
  4. 4. 4 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack summary • Cryptojack - compute resource take-over for the purposes of crypto- mining operations • Browsers • Endpoints/Home routers • Compute resources - physical/cloud • Tesla/Aviva/Gemalto - 2018 - Open Kubernetes clusters - Leaked AWS creds - Many more go unreported • Could be worse! - CodeSpaces - 2014 • Ransom • Company murdered “Money Doesn’t Grow on Trees, but it’s Growing in the Cloud” - RedLock CSI Team
  5. 5. Incident
  6. 6. 6 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Incident info ● Support email from CloudHealth Tech customer requesting assistance ● 200 c4.8xlarge instances in AWS account - multiple regions ○ 100% CPU utilization, $12k per day in EC2 compute $ ● No CloudTrail. No audit logs. ● Innocent user framed!
  7. 7. 7 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Incident handling and response ● AWS Support ○ Account cleanup ○ CloudTrail enabled ○ AWS refund (case dependent) ● CloudHealth ○ Forensics ○ Developed timeline ○ Configured customer CHT security module reporting and best practices
  8. 8. Timeline
  9. 9. 9 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Timeline • 3/14 10:30 UTC A series of public AMIs like ami-6dbd137b created and shared globally across all regions • 3/21 14:43 UTC Customer admin user Access Key 1 Rotated • 3/21 14:44 UTC Framed user console pwd and admin perms set • 3/21 14:45 UTC Instances/VPCs/SGs started getting created • 3/21 14:52 UTC Customer admin user Access Key 1 Last Used - IAM Service • 3/21 14:54 UTC 200 c3.8xlarge Windows instances discovered in customer account by CHT platform • 3/22 Cost spike discovered in CloudHealth app by customer }Less than 10 minutes Pre-stage}
  10. 10. 10 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Timeline summary ● Most useful data came from AWS Credentials Report ○ generate-credential-report ● Compromised AWS Key rotated immediately by an attacker ● Highly automated ○ Pre-baked AMIs ○ SGs/VPCs/EC2 templated ● Limited to 200 machines ○ AWS default account limits
  11. 11. Prevention & Detection
  12. 12. 12 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Protection ● Secure AWS root account ○ Physical MFA - $13 ○ Disable API ● Users/Operators ○ Enable and force MFA for all operations (Console/API) ■ AWS Tutorial: Enable Your Users to Configure Their Own Credentials and MFA Settings ● https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html ■ CLI wrapper for MFA - aws-vault ● https://github.com/99designs/aws-vault ○ Use Idp federation (SAML/SSO) ■ Google, OKTA, Ping Identity, OneLogin, etc ○ No direct permissions to users, use IAM AssumeRole
  13. 13. 13 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Protection - cont. ● Application/Service accounts ○ Say “bye-bye” to AWS Key/Secret keys ■ IAM roles and instance profiles ■ Enable IP whitelisting in IAM policies ○ Limit application and service permissions ■ No blanket *:* ○ Scan your code for AWS keys before they do ■ GitGuardian, keynuker, gitsecrets ● General ○ Keep existing EC2 limits unless required ○ AWS CIS Benchmark ■ ThreatStack, CloudHealth, others ○ Enable CloudTrail and AWS Config!
  14. 14. 14 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Cryptojack incident Detection ● Watch and alert on AWS costs projections ○ Create a Billing Alarm to Monitor Your Estimated AWS Charges ■ https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html ○ CloudHealth - % change from baseline ● CloudTrail monitoring ○ Setup ■ All regions ■ Forward to central secure AWS account ○ Real-time monitoring and alerting ■ ThreatStack, SumoLogic, Splunk, etc ■ Look and flag API credentials changes ● Misc ○ AMI Provenance and VPC Flow Logs
  15. 15. 3 Takeaways
  16. 16. 16 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Key Takeaways ● Enforce MFA on user and root accounts ● Use IAM Roles, ban Key/Secret Keys ● Enable CloudTrail/Config monitoring and Cost alerting

×