AWS Architecture Flaws

1. AWS Architecture Flaws
Apollo Clark

3. • Intro
• Multi-tenancy
• Logging
• Alerts
• Secrets Management
• IAM Permissions
• Questions

4. Intro

5. • Web engineer since 2001
• LAMP stacks, PHP, Python, Ruby, Java, …
• AWS user since 2010
• Managed 4TB Elasticsearch, 2TB MongoDB
• Managed 25,000+ servers, 200+ AWS accounts
• Security consultant since 2015

8. Multi-tenancy

9. "AWS designed their account security
for a single deployment, single
environment, in a single region."

11. • VPC (Virtual Private Cloud) is single-region (ex:
us-east, eu-west).
• Most resources are tied to a single VPC.
• IAM Users, Roles, Groups, and Policies are
account-wide.
• S3 Buckets are account-wide.

12. • Use Pacu and Cloudsploit to test AWS Security
• https://github.com/RhinoSecurityLabs/pacu
• https://github.com/cloudsploit/scans

13. “Deploy into dedicated AWS accounts.”

15. Logging

17. • AWS CloudWatch Logs will log AWS resources
like EC2, ELB, RDS.
• AWS CloudTrail logs AWS CLI and SDK call
requests, sometimes the response.
• AWS VPC Flow Logs, logs network 5-tuples
between resources.

18. CloudWatch Log, syslog
Jul 11 04:37:01 ip-172-31-9-199 CRON[9575]: (root) CMD
(/var/awslogs/bin/awslogs-nanny.sh > /dev/null 2>&1)
• You can create a “Log Stream”, to save logs into
an S3 bucket, for easier retrieval.

19. CloudTrail Logs, part 1
{
"eventVersion":"1.05",
"userIdentity":{
"type":"Root",
"principalId":"123456789012",
"arn":"arn:aws:iam::123456789012:apolloclark",
"accountId":"123456789012",
"accessKeyId":"ASIA1234567890ABCD",
"userName":"apolloclark",
"sessionContext":{
"attributes":{
"mfaAuthenticated":"true",
"creationDate":"2017-08-02T17:51:58Z"
}
}
},

20. CloudTrail Logs, part 2
"eventTime":"2017-08-02T18:13:23Z",
"eventSource":"cloudtrail.amazonaws.com",
"eventName":"CreateTrail",
"awsRegion":"us-east-1",
"sourceIPAddress":"73.0.0.1",
"userAgent":"console.amazonaws.com",
"requestParameters":{
"enableLogFileValidation":true,
"name":"test",
"kmsKeyId":"",
"includeGlobalServiceEvents":true,
"isMultiRegionTrail":true,
"s3BucketName":"testbucket"
},

21. CloudTrail Logs, part 3
"responseElements":{
"s3BucketName":"testbucket",
"includeGlobalServiceEvents":true,
"logFileValidationEnabled":true,
"name":"test",
"trailARN":"arn:aws:cloudtrail:us-east-1:123456789012:trail/test",
"isMultiRegionTrail":true
},
"requestID":"4534ea8f-77ae-11e7-93b4-a16934176b9b",
"eventID":"05c2522c-4971-4150-96f5-a45cee763a1b",
"eventType":"AwsApiCall",
"recipientAccountId":"123456789012"
}

22. VPC Flow Logs
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249
1418530010 1418530070 ACCEPT OK
<version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport>
<dstport> <protocol> <packets> <bytes> <start> <end> <action> <log-status>
• You can’t modify a Flow Log’s configuration after creation. You have to
delete it.
• Network interfaces with multiple IP addresses will have data logged only
for the primary IP as the destination address.
• Flow Logs exclude traffic related to DHCP requests and Amazon DNS
activity.

23. Logging Limitations
• Default retention is 90 days.
• CloudWatch Logs must be configured per Service,
ex: Apache, nginx, Tomcat, etc.
• CloudTrail Logs will only detect creation of
insecure objects, after it was enabled, up to 90
days by default. You need to setup retention.
• VPC Flow Logs are unusable for multi-IP EC2
instances.

24. • Use skew to aggregate AWS resource configs,
across multiple accounts, multiple regions.
• https://github.com/scopely-devops/skew

27. Alerts

29. • All of these use CloudWatch, CloudTrail, VPC Flow Logs, and
SSM-Agent or Inspector Agent.
• Sep 2013 - AWS Trusted Advisor, basic hygiene.
• Nov 2014 - AWS Config, AWS resource configuration.
• Oct 2015 - AWS Inspector, EC2 configuration, PCI-DSS, CIS
• Aug 2017 - AWS Macie, S3 configuration.
• Nov 2017 - AWS GuardDuty, network intrusion detection system
• Nov 2018 - AWS Security Hub, aggregates Inspector, Macie,
and GuardDuty.

30. • None of these are free.
• Expensive at scale, with 1000+ servers.
• Still requires a dedicated SIEM, and SOAR
(security orchestration, automation, remediation)

31. • Still need a SIEM to aggregate the alerts.
• ELK, Splunk, LogRhythm, Loggly, SumoLogic,
dome9
• Limited automated remediation.
• DivvyCloud, Uptycs, Rapid7 InsightConnect,
Alienvault, Cyberbit, Swimlane

32. • Use Argos to automate remediation.
• https://github.com/apolloclark/argos

35. Secrets Management

37. cloud-init, user data, startup script
curl http://169.254.169.254/latest/user-data
#!/bin/bash
export POSTGRES_HOST=“54.0.0.1"
export POSTGRES_USERNAME="pg_admin"
export POSTGRES_PASSWORD="5up3r_53cr3t"
export AWS_IAM_ACCESS_KEY="AKIA123456780"
export AWS_IAM_SECRET_KEY="Qbfaui3289bfaf389"
export NESSUS_ACCESS_KEY="abrw31289bfajklfdas"

38. • Use a dedicated configuration management
service.
• Ansible Tower, Chef, Puppet, Hashicorp Vault
• Use AWS Secrets Manager
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-
data.html

40. IAM Permissions

42. IAM Policy, S3 bucket, resource name
{
"Version": "2012-10-17",
"Statement": [
{
“Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion”
],
"Resource": "arn:aws:s3:*:*:CompanyA/Marketing/*"
}
]
}

43. IAM Policy, EBS, resource tag
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:DetachVolume"
],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringEquals": {"ec2:ResourceTag/Department": “Marketing"}
}
}
]
}

44. IAM Policy, RDS
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "rds:DescribeDBInstances",
"Resource": "*"
}
]
}

45. • AWS Managed IAM Policies are too wide-open.
• Inline Policies are tedious to manage.
• IAM Policies for Describe* are Account-Wide
• Unsupported Resource-Level Permissions:
• https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAM.ResourcePer
missions.html#UsingWithRDS.IAM.UnsupportedResourceLevelPermissions
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/ec2-api-permissions.html#ec2-
api-unsupported-resource-permissions
• Check out Netflix/repokid
• https://github.com/Netflix/repokid

47. • Dedicated AWS Accounts
• Logging
• Alerts
• Secrets Management
• IAM Permissions
• https://github.com/toniblyx/my-arsenal-of-aws-security-tools

49. Questions?