1
jhwong@netskope.com
@jenkohwong
defcon cloud-village
August 10, 2019
Here for a good time, not a long time
1
Exploiting AWS loopholes
with temporary credentials
@jenkohwong
• netskope threat research team
• windows security, vulnerability scanning, routers/appliances, av/as, threat intel,
exploits/pen-testing
• product / engineering
2
Agenda
• Attack Scenario: Temporary Tokens
• Defender Viewpoint: Challenges
• Do & Don't
3
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Attack Scenario
AWS Cloud
compromised
access key A
Attacker
1
Compromised Credentials
AWS Cloud
compromised
access key A
Attacker
1
Compromised Credentials
AWS Cloud
compromised
access key A
Attacker
1
Compromised Credentials
AWS Cloud
compromised
access key A
Attacker
1
generate
temp token B
temp token B AWS STS
2
Generate Temp Credentials
AWS Cloud
compromised
access key A
Attacker
1
generate
temp token B
temp token B AWS STS
2
Generate Temp Credentials
Discovery
Discovery
Discovery
can
AssumeRole
belongs
to
compromised
access key A
jenko-bucket6
jenko_test_user JenkoBucketRole
can do
Action: “s3:*”
on
Discovery
can
AssumeRole
belongs
to
compromised
access key A
jenko-bucket6
jenko_test_user JenkoBucketRole
can do
Action: “s3:*”
on
Discovery
AWS Cloud
access key A
jenko-bucket6
AssumeRole
JenkoBucketRole
Privilege Escalation Details
AWS STS
temp token B
temp token A’
temp token B’Attacker
temp token A’
temp token B’
AWS Cloud
access key A
jenko-bucket6
AssumeRole
JenkoBucketRole
Privilege Escalation Details
AWS STS
temp token B
temp token A’
temp token B’Attacker
temp token A’
temp token B’
AWS Cloud
compromised
access key A
Attacker
1
generate
temp token B
temp token B AWS STS
2
escalate
privileges with
compromised
access key A
or temp token B
3
Privilege Escalation
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint
20
Defender Viewpoint
• Assumptions
• AWS experience
• CloudTrail/CloudWatch
• Less knowledge of temp credentials
• Starting Point
• External party re: leaked data
• Events/Alarms
21
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Detect:
1.CloudTrail/Watch
detects Data Exfil
(action/destination)
2. Privilege Escalation?
3. Correlation / Anomaly
detection?
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Investigate:
1. Logs: access key A
AssumeRole
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Investigate:
1. Logs: access key A
AssumeRole
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Investigate:
1. Logs: access key A
AssumeRole
2. JenkoBucketRole
valid, not
overprivileged,
assigned to correct
users
3. user interview =>
compromised key
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Mitigate/Remediate:
1. Delete access key A
2. Key rotation
3. Change Console
password
4. User training
AWS Cloud
compromised
access key A
Attacker
generate
temp token B
temp token B AWS STS
escalate
privileges with
compromised
access key A
or temp token B
S3 Bucket
access S3
Bucket
data
exfiltration
3
1
2
4
5
Defender Viewpoint: Challenges
Mitigate/Remediate:
1. Delete access key A
2. Key rotation
3. Change Console
password
4. User training
Defender Viewpoint: Challenges
• GetSessionToken and the returned
temp token B are troubling
• We're seeing STS and temp tokens
• Same fields in AssumeRole actions
but GetSessionToken is new
• Reading up...we see we have
another set of access keys floating
around
28
STS Temp Tokens
• Expiration/Timing:
• 15 minutes to 36 hours
• +CloudTrail event latency (from API call to logging on S3) of at least 20 minutes
• Temporary tokens generated by AWS (e.g. passing roles to services like EC2) usually have
shorter time frames (1 hour). But automatically refreshed, so an attacker who’s gained control
of an EC2 instance only needs to refresh their tokens every hour.
• API Access
• Can use any service that the original user has privileges for, except…
• Sessions using temporary tokens cannot create more temporary tokens
• Within STS, can only invoke AssumeRole
• Many techniques for Privilege escalation (AssumeRole), not a barrier (follow rhino)
29
Defensive Viewpoint: Temp Tokens
Assess/Analyze
• Untracked, no way to list current active ones or historically generated
• Not in Console, no CLI/API command to ListGeneratedTokens
• They are logged but you would have to parse and persist from CloudTrail
30
Defensive Viewpoint: Temp Tokens
31
Detect
Update CloudWatch/SIEM
filters to detect
• Creation:
GetSessionToken
AssumeRole actions
• Usage: accessKeyId =~
ASIA*
AWS Cloud
Attacker
generate
temp token B
temp token B AWS STS
sessions
temp token B’ S3 Bucket
1
2
4
Defender Viewpoint: Temp Tokens
Mitigate/Remediate:
1. Can’t delete temp
token
2. a) Restrict Role
b) Delete User
3. Update Remediation
Playbook
4. Revoke Active
Session for Role
AssumeRole
JenkoBucketRole
temp token B’
1
belongs
to
all keys/tokens jenko_test_user
2
AWS Cloud
Attacker
generate
temp token B
temp token B AWS STS
sessions
temp token B’ S3 Bucket
1
2
4
Defender Viewpoint: Temp Tokens
Mitigate/Remediate:
1. Can’t delete temp
token
2. a) Restrict Role
b) Delete User
3. Update Remediation
Playbook
4. Revoke Active
Session for Role
AssumeRole
JenkoBucketRole
temp token B’
1
belongs
to
all keys/tokens jenko_test_user
2
AWS Cloud
Attacker
generate
temp token B
temp token B AWS STS
sessions
temp token B’ S3 Bucket
1
Defender Viewpoint: Temp Tokens
Prevention:
1. Can’t prevent
GetSessionToken
AssumeRole
JenkoBucketRole
temp token B’
generated
from
all keys/tokens jenko_test_user
AWS Cloud
Attacker
generate
temp token B
temp token B AWS STS
sessions
temp token B’ S3 Bucket
1
Defender Viewpoint: Temp Tokens
Prevention:
1. Can’t prevent
GetSessionToken
AssumeRole
JenkoBucketRole
temp token B’
generated
from
all keys/tokens jenko_test_user
AWS Cloud
Attacker
generate
temp token B
temp token B AWS STS
sessions
temp token B’ S3 Bucket
1
4
Defender Viewpoint: Temp Tokens
Prevention:
1. Can’t prevent
GetSessionToken
2. No MFA on
GetSessionToken
3. IAM permissions
boundaries
4. Restrict
sts:AssumeRole
5. Revoke active
sessions for Role
AssumeRole
JenkoBucketRole
temp token B’
2
5
3
generated
from
all keys/tokens jenko_test_user
max perms
RED
- Generate temp credentials for backdoor access
- Combine temp credentials with presigned urls, lambdas, log attacks
- Consider lambdas as a means to persist temp credentials
- Assess whether logging/alerting for temp credentials is being done
37
BLUE
• Get a plan in place ASAP to manage temp token usage esp remediation/recovery
• Prevention: lockdown access keys, isolate temp token usage in separate
accounts, minimal privileges for AssumeRole/PassRole
• Detection: alert on GetSessionToken, alert on temp tokens (ASIA*), harden
CloudTrail/CloudWatch/SIEM
• Mitigation/Remediation: review/revise remediation playbook, do not use
GetSessionToken, use AssumeRole, use revoke active sessions for role,
create/test a recovery plan from compromised temp tokens
• Provisioning/Inventory: track temp tokens that are created in a datastore, use
wrapper code for custom apps that need temp tokens, for AWS-generated tokens
(IoT, AssumeRole) have to parse logs
38
Thank you
jhwong@netskope.com
@jenkohwong
Slide deck and recap can be found at:
https://www.netskope.com/blog/aws-loopholes-with-temporary-credentials
39

DEF CON 27 - Exploiting AWS Loopholes

  • 1.
    1 jhwong@netskope.com @jenkohwong defcon cloud-village August 10,2019 Here for a good time, not a long time 1 Exploiting AWS loopholes with temporary credentials
  • 2.
    @jenkohwong • netskope threatresearch team • windows security, vulnerability scanning, routers/appliances, av/as, threat intel, exploits/pen-testing • product / engineering 2
  • 3.
    Agenda • Attack Scenario:Temporary Tokens • Defender Viewpoint: Challenges • Do & Don't 3
  • 4.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Attack Scenario
  • 5.
    AWS Cloud compromised access keyA Attacker 1 Compromised Credentials
  • 6.
    AWS Cloud compromised access keyA Attacker 1 Compromised Credentials
  • 7.
    AWS Cloud compromised access keyA Attacker 1 Compromised Credentials
  • 8.
    AWS Cloud compromised access keyA Attacker 1 generate temp token B temp token B AWS STS 2 Generate Temp Credentials
  • 9.
    AWS Cloud compromised access keyA Attacker 1 generate temp token B temp token B AWS STS 2 Generate Temp Credentials
  • 10.
  • 11.
  • 12.
  • 13.
    can AssumeRole belongs to compromised access key A jenko-bucket6 jenko_test_userJenkoBucketRole can do Action: “s3:*” on Discovery
  • 14.
    can AssumeRole belongs to compromised access key A jenko-bucket6 jenko_test_userJenkoBucketRole can do Action: “s3:*” on Discovery
  • 15.
    AWS Cloud access keyA jenko-bucket6 AssumeRole JenkoBucketRole Privilege Escalation Details AWS STS temp token B temp token A’ temp token B’Attacker temp token A’ temp token B’
  • 16.
    AWS Cloud access keyA jenko-bucket6 AssumeRole JenkoBucketRole Privilege Escalation Details AWS STS temp token B temp token A’ temp token B’Attacker temp token A’ temp token B’
  • 17.
    AWS Cloud compromised access keyA Attacker 1 generate temp token B temp token B AWS STS 2 escalate privileges with compromised access key A or temp token B 3 Privilege Escalation
  • 18.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5
  • 19.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5
  • 20.
  • 21.
    Defender Viewpoint • Assumptions •AWS experience • CloudTrail/CloudWatch • Less knowledge of temp credentials • Starting Point • External party re: leaked data • Events/Alarms 21
  • 22.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Detect: 1.CloudTrail/Watch detects Data Exfil (action/destination) 2. Privilege Escalation? 3. Correlation / Anomaly detection?
  • 23.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Investigate: 1. Logs: access key A AssumeRole
  • 24.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Investigate: 1. Logs: access key A AssumeRole
  • 25.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Investigate: 1. Logs: access key A AssumeRole 2. JenkoBucketRole valid, not overprivileged, assigned to correct users 3. user interview => compromised key
  • 26.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Mitigate/Remediate: 1. Delete access key A 2. Key rotation 3. Change Console password 4. User training
  • 27.
    AWS Cloud compromised access keyA Attacker generate temp token B temp token B AWS STS escalate privileges with compromised access key A or temp token B S3 Bucket access S3 Bucket data exfiltration 3 1 2 4 5 Defender Viewpoint: Challenges Mitigate/Remediate: 1. Delete access key A 2. Key rotation 3. Change Console password 4. User training
  • 28.
    Defender Viewpoint: Challenges •GetSessionToken and the returned temp token B are troubling • We're seeing STS and temp tokens • Same fields in AssumeRole actions but GetSessionToken is new • Reading up...we see we have another set of access keys floating around 28
  • 29.
    STS Temp Tokens •Expiration/Timing: • 15 minutes to 36 hours • +CloudTrail event latency (from API call to logging on S3) of at least 20 minutes • Temporary tokens generated by AWS (e.g. passing roles to services like EC2) usually have shorter time frames (1 hour). But automatically refreshed, so an attacker who’s gained control of an EC2 instance only needs to refresh their tokens every hour. • API Access • Can use any service that the original user has privileges for, except… • Sessions using temporary tokens cannot create more temporary tokens • Within STS, can only invoke AssumeRole • Many techniques for Privilege escalation (AssumeRole), not a barrier (follow rhino) 29
  • 30.
    Defensive Viewpoint: TempTokens Assess/Analyze • Untracked, no way to list current active ones or historically generated • Not in Console, no CLI/API command to ListGeneratedTokens • They are logged but you would have to parse and persist from CloudTrail 30
  • 31.
    Defensive Viewpoint: TempTokens 31 Detect Update CloudWatch/SIEM filters to detect • Creation: GetSessionToken AssumeRole actions • Usage: accessKeyId =~ ASIA*
  • 32.
    AWS Cloud Attacker generate temp tokenB temp token B AWS STS sessions temp token B’ S3 Bucket 1 2 4 Defender Viewpoint: Temp Tokens Mitigate/Remediate: 1. Can’t delete temp token 2. a) Restrict Role b) Delete User 3. Update Remediation Playbook 4. Revoke Active Session for Role AssumeRole JenkoBucketRole temp token B’ 1 belongs to all keys/tokens jenko_test_user 2
  • 33.
    AWS Cloud Attacker generate temp tokenB temp token B AWS STS sessions temp token B’ S3 Bucket 1 2 4 Defender Viewpoint: Temp Tokens Mitigate/Remediate: 1. Can’t delete temp token 2. a) Restrict Role b) Delete User 3. Update Remediation Playbook 4. Revoke Active Session for Role AssumeRole JenkoBucketRole temp token B’ 1 belongs to all keys/tokens jenko_test_user 2
  • 34.
    AWS Cloud Attacker generate temp tokenB temp token B AWS STS sessions temp token B’ S3 Bucket 1 Defender Viewpoint: Temp Tokens Prevention: 1. Can’t prevent GetSessionToken AssumeRole JenkoBucketRole temp token B’ generated from all keys/tokens jenko_test_user
  • 35.
    AWS Cloud Attacker generate temp tokenB temp token B AWS STS sessions temp token B’ S3 Bucket 1 Defender Viewpoint: Temp Tokens Prevention: 1. Can’t prevent GetSessionToken AssumeRole JenkoBucketRole temp token B’ generated from all keys/tokens jenko_test_user
  • 36.
    AWS Cloud Attacker generate temp tokenB temp token B AWS STS sessions temp token B’ S3 Bucket 1 4 Defender Viewpoint: Temp Tokens Prevention: 1. Can’t prevent GetSessionToken 2. No MFA on GetSessionToken 3. IAM permissions boundaries 4. Restrict sts:AssumeRole 5. Revoke active sessions for Role AssumeRole JenkoBucketRole temp token B’ 2 5 3 generated from all keys/tokens jenko_test_user max perms
  • 37.
    RED - Generate tempcredentials for backdoor access - Combine temp credentials with presigned urls, lambdas, log attacks - Consider lambdas as a means to persist temp credentials - Assess whether logging/alerting for temp credentials is being done 37
  • 38.
    BLUE • Get aplan in place ASAP to manage temp token usage esp remediation/recovery • Prevention: lockdown access keys, isolate temp token usage in separate accounts, minimal privileges for AssumeRole/PassRole • Detection: alert on GetSessionToken, alert on temp tokens (ASIA*), harden CloudTrail/CloudWatch/SIEM • Mitigation/Remediation: review/revise remediation playbook, do not use GetSessionToken, use AssumeRole, use revoke active sessions for role, create/test a recovery plan from compromised temp tokens • Provisioning/Inventory: track temp tokens that are created in a datastore, use wrapper code for custom apps that need temp tokens, for AWS-generated tokens (IoT, AssumeRole) have to parse logs 38
  • 39.
    Thank you jhwong@netskope.com @jenkohwong Slide deckand recap can be found at: https://www.netskope.com/blog/aws-loopholes-with-temporary-credentials 39