- The document summarizes common architecture flaws in AWS environments, including lack of account segmentation, improper logging configuration, insufficient alerts and monitoring, insecure secrets management, and overpermissive IAM policies. It provides examples and recommendations for addressing each issue, such as using separate AWS accounts for different environments, configuring CloudTrail, GuardDuty, and a SIEM, using HashiCorp Vault for secrets, and restricting IAM policies to specific resources. The presenter has extensive experience deploying and securing applications on AWS.

1. AWS Architecture Flaws
Apollo Clark

3. • Intro
• Multi-tenancy
• Logging
• Alerts
• Secrets Management
• IAM Permissions
• Questions

4. Intro

5. • Web engineer since 2001
• LAMP stacks, PHP, Python, Ruby, Java, …
• AWS user since 2010
• Managed 4TB Elasticsearch, 2TB MongoDB
• Managed 25,000+ servers, 200+ AWS accounts
• Security consultant since 2015

8. Multi-tenancy

9. "AWS designed their account security
for a single deployment, single
environment, in a single region."

11. • VPC (Virtual Private Cloud) is single-region (ex:
us-east, eu-west).
• Most resources are tied to a single VPC.
• IAM Users, Roles, Groups, and Policies are
account-wide.
• S3 Buckets are account-wide.

12. • Use Pacu and Cloudsploit to test AWS Security
• https://github.com/RhinoSecurityLabs/pacu
• https://github.com/cloudsploit/scans

13. “Deploy into dedicated AWS accounts.”

15. Logging

17. • AWS CloudWatch Logs will log AWS resources
like EC2, ELB, RDS.
• AWS CloudTrail logs AWS CLI and SDK call
requests, sometimes the response.
• AWS VPC Flow Logs, logs network 5-tuples
between resources.

18. CloudWatch Log, syslog
Jul 11 04:37:01 ip-172-31-9-199 CRON[9575]: (root) CMD
(/var/awslogs/bin/awslogs-nanny.sh > /dev/null 2>&1)
• You can create a “Log Stream”, to save logs into
an S3 bucket, for easier retrieval.

19. CloudTrail Logs, part 1
{
"eventVersion":"1.05",
"userIdentity":{
"type":"Root",
"principalId":"123456789012",
"arn":"arn:aws:iam::123456789012:apolloclark",
"accountId":"123456789012",
"accessKeyId":"ASIA1234567890ABCD",
"userName":"apolloclark",
"sessionContext":{
"attributes":{
"mfaAuthenticated":"true",
"creationDate":"2017-08-02T17:51:58Z"
}
}
},

20. CloudTrail Logs, part 2
"eventTime":"2017-08-02T18:13:23Z",
"eventSource":"cloudtrail.amazonaws.com",
"eventName":"CreateTrail",
"awsRegion":"us-east-1",
"sourceIPAddress":"73.0.0.1",
"userAgent":"console.amazonaws.com",
"requestParameters":{
"enableLogFileValidation":true,
"name":"test",
"kmsKeyId":"",
"includeGlobalServiceEvents":true,
"isMultiRegionTrail":true,
"s3BucketName":"testbucket"
},

21. CloudTrail Logs, part 3
"responseElements":{
"s3BucketName":"testbucket",
"includeGlobalServiceEvents":true,
"logFileValidationEnabled":true,
"name":"test",
"trailARN":"arn:aws:cloudtrail:us-east-1:123456789012:trail/test",
"isMultiRegionTrail":true
},
"requestID":"4534ea8f-77ae-11e7-93b4-a16934176b9b",
"eventID":"05c2522c-4971-4150-96f5-a45cee763a1b",
"eventType":"AwsApiCall",
"recipientAccountId":"123456789012"
}

22. VPC Flow Logs
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249
1418530010 1418530070 ACCEPT OK
<version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport>
<dstport> <protocol> <packets> <bytes> <start> <end> <action> <log-status>
• You can’t modify a Flow Log’s configuration after creation. You have to
delete it.
• Network interfaces with multiple IP addresses will have data logged only
for the primary IP as the destination address.
• Flow Logs exclude traffic related to DHCP requests and Amazon DNS
activity.

23. Logging Limitations
• Default retention is 90 days.
• CloudWatch Logs must be configured per Service,
ex: Apache, nginx, Tomcat, etc.
• CloudTrail Logs will only detect creation of
insecure objects, after it was enabled, up to 90
days by default. You need to setup retention.
• VPC Flow Logs are unusable for multi-IP EC2
instances.

24. • Use skew to aggregate AWS resource configs,
across multiple accounts, multiple regions.
• https://github.com/scopely-devops/skew

27. Alerts

29. • All of these use CloudWatch, CloudTrail, VPC Flow Logs, and
SSM-Agent or Inspector Agent.
• Sep 2013 - AWS Trusted Advisor, basic hygiene.
• Nov 2014 - AWS Config, AWS resource configuration.
• Oct 2015 - AWS Inspector, EC2 configuration, PCI-DSS, CIS
• Aug 2017 - AWS Macie, S3 configuration.
• Nov 2017 - AWS GuardDuty, network intrusion detection system
• Nov 2018 - AWS Security Hub, aggregates Inspector, Macie,
and GuardDuty.

30. • None of these are free.
• Expensive at scale, with 1000+ servers.
• Still requires a dedicated SIEM, and SOAR
(security orchestration, automation, remediation)

31. • Still need a SIEM to aggregate the alerts.
• ELK, Splunk, LogRhythm, Loggly, SumoLogic,
dome9
• Limited automated remediation.
• DivvyCloud, Uptycs, Rapid7 InsightConnect,
Alienvault, Cyberbit, Swimlane

32. • Use Argos to automate remediation.
• https://github.com/apolloclark/argos

35. Secrets Management

37. cloud-init, user data, startup script
curl http://169.254.169.254/latest/user-data
#!/bin/bash
export POSTGRES_HOST=“54.0.0.1"
export POSTGRES_USERNAME="pg_admin"
export POSTGRES_PASSWORD="5up3r_53cr3t"
export AWS_IAM_ACCESS_KEY="AKIA123456780"
export AWS_IAM_SECRET_KEY="Qbfaui3289bfaf389"
export NESSUS_ACCESS_KEY="abrw31289bfajklfdas"

38. • Use a dedicated configuration management
service.
• Ansible Tower, Chef, Puppet, Hashicorp Vault
• Use AWS Secrets Manager
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-
data.html

40. IAM Permissions

42. IAM Policy, S3 bucket, resource name
{
"Version": "2012-10-17",
"Statement": [
{
“Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion”
],
"Resource": "arn:aws:s3:*:*:CompanyA/Marketing/*"
}
]
}

43. IAM Policy, EBS, resource tag
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:DetachVolume"
],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringEquals": {"ec2:ResourceTag/Department": “Marketing"}
}
}
]
}

44. IAM Policy, RDS
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "rds:DescribeDBInstances",
"Resource": "*"
}
]
}

45. • AWS Managed IAM Policies are too wide-open.
• Inline Policies are tedious to manage.
• IAM Policies for Describe* are Account-Wide
• Unsupported Resource-Level Permissions:
• https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAM.ResourcePer
missions.html#UsingWithRDS.IAM.UnsupportedResourceLevelPermissions
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/ec2-api-permissions.html#ec2-
api-unsupported-resource-permissions
• Check out Netflix/repokid
• https://github.com/Netflix/repokid

47. • Dedicated AWS Accounts
• Logging
• Alerts
• Secrets Management
• IAM Permissions
• https://github.com/toniblyx/my-arsenal-of-aws-security-tools

49. Questions?