In the Cloud, nobody can hear you scream: AWS Cloud Security for DevOps
1. AWS Cloud Security for DevOps
In the cloud, nobody can hear you
scream….
Garth Boyd, OWASP Ottawa Co-Lead
Paul Ionescu, OWASP Ottawa Co-Lead
2. AWS Cloud Security for DevOps
In the cloud, nobody can hear you
scream….
OWASP
Ottawa
The OWASP Chapter in Ottawa Region
OWASP Ottawa
3. Mandate
• Web Application Security education and
outreach in the Ottawa Area
• Working with OWASP colleagues from around
the world
• Contributing to OWASP Tool and
Document/Guide Projects
• Leaders: Sherif Koussa, Tanya Janca, Garth
Boyd and Paul Ionescu
• ….and many other volunteers
4. Engagement
• 979 members
• 35-75 attendees per meet Ottawa
• 20-30 attendees per meet Kanata
• CTF, Trivia, Projects
7. Why Cloud Security for DevOps
You are a developer starting
your journey to 100% create
and operate a service in the
cloud, the DevOps way
– Worried about loosing
customer data
– Worried about thousands
of dollars in billing
resources
– Don't know where to start
with security
8. The Treacherous 12
1. Data Breaches
2. Weak Identity, Credential
and Access
Management
3. Insecure APIs
4. System and Application
Vulnerabilities
5. Account Hijacking
6. Malicious Insiders
7. Advanced Persistent Threats
(APTs)
8. Data Loss
9. Insufficient Due Diligence
10. Abuse and Nefarious Use of
Cloud Services
11. Denial of Service
12. Shared Technology Issues
9. The Silly 6
1. Data Breaches
2. Weak Identity, Credential
and Access
Management
3. Insecure APIs
4. System and Application
Vulnerabilities
5. Account Hijacking
6. Malicious Insiders
7. Advanced Persistent Threats
(APTs)
8. Data Loss
9. Insufficient Due Diligence
10. Abuse and Nefarious Use of
Cloud Services
11. Denial of Service
12. Shared Technology Issues
Those that give us thrills and Chills:
12. Or a GDPR nuanced version for Personal Data
Breach:
“a breach of security leading to the accidental or
unlawful destruction, loss, alteration, unauthorised
disclosure of, or access to, personal data transmitted,
stored or otherwise processed”2
Data Breaches
13. • Bad examples include:
– Deep Root Analytics owns the database, and
stores it on an Amazon S3 server.
– Personal information of 191 million
American citizens registered to vote
exposed.3
Data Breaches
14. • Bad examples include:
– Cryptojackers have been
discovered sneaking mining
code on to a LA Times
website through the back
door of a poorly secured
AWS S3 bucket.4
Data Breaches
15. • Bad examples include:
– Washington-based ISP by
the name of Pocket iNet left
73 gigabytes of essential
operational data publicly
exposed in a misconfigured
Amazon S3 storage bucket
for months.5
– contained the “keys to the
kingdom,”
Data Breaches
18. • Challenges
– Configuration management between
different buckets
• Configuration confusion risk
– Private data accidentally stored in
public bucket
• Programmatic or manual upload confusion
risk
– Misconfigured access on ”private”
bucket
Data Breaches
19. Targets:
Data Breaches
1. Where is our sensitive data?
2. Least Privilege
4. Monitoring
- Configuration
- metrics
3. Encryption
5. Trusted Advisor/Inventory
21. 1. Where IS your sensitive data?
Data Breaches
AWS Macie
- Discover and classify – high risk (eg, API Keys)
- Constantly monitor
- Alert
SNS
Topic
- Other tools? IdentityFinder, CUSpider, …
22. Use IAM Roles for fine
grained access controls.
- s3-bucket-public-write-
prohibited
- s3-bucket-public-read-
prohibited
Data Breaches
2. Least Privilege:
23. 3. Encryption8:
Data Breaches
- Enable default encryption
- Use key managed by S3 for
default Encryption
- Or use AWS KMS for key
management
24. 4. Monitor S3 Activity
Data Breaches
• Create a CloudTrail Trail of events for S3 for continuous
ongoing record of events on your S3 bucket
• CloudTrail is automatically enabled on your AWS Account
• S3 integrated with CloudTrail
• Actions taken by user, role, or service in S3
• APIs and Console activity
• Integrate with CloudWatch Log Stream
• Alarm for specific API activity
• Monitor metrics
• Trigger Lambda
• Alert with SNS
• HoneyTokens, (AKA Canary Tokens)
25. - Custom rules
- Enable AWSConfig to monitor S3
bucket ACLs
- CloudWatch Event Rule to trigger
alert when AWSConfig notices
configuration violation
Data Breaches
4. Audit and Monitor
Monitor and Enforce configuration
changes using AWSConfig
26. 4. Audit and Track configuration changes using
AWSConfig6
Data Breaches
Compliant?
AWS ConfigS3
AWS Config
Event
CloudWatch
Event Rule
Trigger
Correct ACL
SNS
Topic
AWS
Lamda
27. 5. Trusted Advisor
Data Breaches
- Free S3 bucket permission checks9
- Integration with CloudWatch Events10
5. Inventory
- Detailed inventory report with encryption status
- Tool to follow best practices
29. Weak Identity, Credential
and Access
Management
• Problems
– Misconfigured roles
• Role management too complex, overhead, easier to use an admin
role with permissions to everything
– Hardcoded AWS IAM keys
• Container needs to speak to AWS services. Where to put the keys?
In the container of course
– Open Ports
• Security groups, such a headache. Open security groups are easier
to manage
– Open Services
• If the security group is open Jenkins can be easily accessible from
our corporate network
30. • Documented cases
of Github project
containing
hardcoded
credentials
• Cryptojackers taking
advantage
Bad Example
31. Common Development
Mistakes Repeated
• These flaws are not new, we have seen them before
because they don't have trivial solutions
– Encryption keys and passwords hard coded
– Keys and passwords stored in clear text or next to
encryption key
• Security can be complicated, requires effort and
resources
• Test/CI environments considered less important
– This changes in the cloud as cryptojackers hunt for
computing resources
32. • AWS KMS and AWS
Secrets Manager
offer a place to store
DB passwords and
IAM keys needed by
computing resources
• AWS Secrets
manager handles key
rotation
• Access controlled
with EC2 roles
Credential Storage
33. Spotting permissive security
groups and expired keys
• AWS Trusted Advisor can help design and monitor the security
of access controls. TA is free for Security!
34. Spotting Breaches
• What if your keys have been stolen?
– AWS Billing alerts can help you detect unusual
usage patterns like for crypto-jacking
– AWS Guard Duty can alert you of network IOCs
– 3rd Party antivirus software available on the AWS
Marketplace can flag common crypto-jackers
36. Interfaces&APIs
Problems:
- API Keys originally designed to identify consumers
- Overloaded authentication and authorization functionality
- Not treated as a critical asset
Randomness
Sequential predictable, trivially short
Spoofable
Keys in URLs
Inconsistent API functionality
Lack of verification
TLS
- 25% of bugcrowd bugs list API in the bug brief
37. Interfaces&APIs
• AWS API Gateway to create, manage and maintain your
private, regional, or edge APIs
• Supports AuthN and AuthZ mechanisms
• IAM
• Cognito
• Resource Policies
• Define a Lambda for processing AuthN Tokens
• AWS Amplify for token signing
• Support OpenAPI specification (AKA Swagger)
• Monitoring support in AWS
• Rate limiting, usage plan
39. System Vulnerabilities
Defined:
System Vulnerability:
Vulnerabilities within the components of the operating
system – kernel, system libraries and application tools.
(a vulnerability is a weakness which can be exploited by a
Threat Actor, such as an attacker, to perform unauthorized
actions within a computer system. )
How do I ensure that my fleet of cloud based systems
are not exposed to the latest set of vulnerabilities?
Question:
41. System Vulnerabilities
AWS Inspector:
• Assists AWS account owners to support their
section of shared security model
• automated
• Provides guidance
• Cloud native
45. Acct Hijacking: Bad Example
• Code Spaces business destroyed by an
attacker who gained access to AWS console
46. Acct Hijacking: Solutions
• MFA
– If you don't have it go set it up
now, we'll give you 5 minutes…
• Use corporate DL for account
id
– More than one person in the
company owns the account
• SAML and IAM Roles
– Use limited IAM roles and
LDAP/AD accounts to login into
console (instead of IAM
accounts)
47. Acct Hijacking: Solutions Cont.
• Disaster Recovery account
– Owned by someone separate
from operations team, trusted
manager
– Data backups with cross-
account role
– Cloud Formation automation
to spin up entire environment
– Devops=Infrastructure as code
48. Malicious Insiders
• No-one can protect you from the AWS account
owner being malicious but maybe we can
reduce the risk from others
• Problems
– Corporate network allowed through Security Groups
– Segmentation is hard
– You trust your colleagues and friends
– Devops = High trust culture, let devs do their own thing
49. Insiders: Access Controls
• Create IAM roles with only needed access for
members of the team
– Need to know/Principle of least privilege
• No IAM accounts, SAML integration ensure
that accounts can be tracked across the
corporate devices and cloud account
• Cloud trail monitoring
– Monitor cloud trail logs in corporate SIEM
50. Insiders: Segmentation
• Separate test/development/staging cloud accounts
from production accounts
• Production like test/development environments
• Immutable production
– Containers = no change no access
– Avoid production access
• SIEM to monitor logs
• Automation pipelines take care of deployment and maintenance
51. Insiders: Code Control
• Github access
• Code review
• Change control
• Malware scans in CICD pipeline
• Code signing
• 3rd Party Component scans
• SAST/DAST
53. References
1. ^ United States Department of Health and Human Services, Administration for Children and
Families. Information Memorandum. Retrieved 2015-09-01.
2. http://ec.europa.eu/newsroom/document.cfm?doc_id=47741, Accessed 23.10.2018,
Article 4 EU General Data Protection Regulation (EU-GDPR)
3. https://www.wired.com/story/voter-records-exposed-database/, Accessed 23.10.2018,
published 19.06.2017
4. https://nakedsecurity.sophos.com/2018/02/27/unsecured-aws-led-to-cryptojacking-attack-
on-la-times/, Accessed 23.10.2018, published 27.02.2018
5. https://motherboard.vice.com/en_us/article/zm9dmj/an-isp-left-corporate-passwords-
keys-and-all-its-data-exposed-on-the-internet, Accessed 24.10.2018, published 23.10.2018
6. https://aws.amazon.com/blogs/security/how-to-use-aws-config-to-monitor-for-and-
respond-to-amazon-s3-buckets-allowing-public-access/
7. https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudtrail-logging.html
8. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/default-bucket-
encryption.html
9. https://aws.amazon.com/about-aws/whats-new/2018/02/aws-trusted-advisors-s3-bucket-
permissions-check-is-now-free/
10. https://docs.aws.amazon.com/awssupport/latest/user/cloudwatch-events-ta.html
Editor's Notes
https://bit.ly/2P8aakr
Paul: add you selections here, perhaps in a different colour
A common pattern is for static, public facing content that has no security classification to be in a public open bucket and is then directly referenced from a client, like a browser thus not costing any performance degradation to the system overall.
Where this gets more complex is when we have both public and private buckets. Or one bucket with separate paths containing different data classifications
This leads to some potential complications.
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html#cloudwatch-alarms-for-cloudtrail-s3-bucket-activity
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html
https://www.opsdash.com/blog/aws-s3-cloudwatch-monitoring.html
Log group
IAM Role granting CloudTrail permission to CloudWatch log stream and deliver events to that log stream.
We can configure a CloudWatch Alarm when an S3 API call
CloudWatch (http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html)
- https://www.opsdash.com/blog/aws-s3-cloudwatch-monitoring.html
Can collect predefined metrics for an AWS Service like S3
Can report and store custom metrics
Can set Threshold-events = alarms can be setup to notify
, as well as take actions
Can graph and create dashboards
AWS Config is a service that enables you to monitor and audit S3 bucket configuration changes.
https://aws.amazon.com/blogs/security/how-to-use-aws-config-to-monitor-for-and-respond-to-amazon-s3-buckets-allowing-public-access/
Enable AWS Config to monitor Amazon S3 bucket ACLs and policies for compliance violations.
Create an IAM Role and Policy that grants a Lambda function permissions to read S3 bucket policies and send alerts through SNS.
Create and configure a CloudWatch Events rule that triggers the Lambda function when AWS Config detects an S3 bucket ACL or policy violation.
Create a Lambda function that uses the IAM role to review S3 bucket ACLs and policies, correct the ACLs, and notify your team of out-of-compliance policies.
Verify the monitoring solution.
https://www.slideshare.net/AmazonWebServices/using-aws-cloudtrail-to-enhance-governance-and-compliance-of-amazon-s3-dev311-reinvent-2017
https://aws.amazon.com/premiumsupport/trustedadvisor/
Trusted Advisor is a tool that scans your aws deployment and compares it to best practices in 5 different categories.
Cost
Security
Performance
Fault tolerance
Service limits
Provides recommended actions.
https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-inventory.html
collect operating system (OS), application, and instance metadata from your Amazon EC2 instances and your on-premises servers or virtual machines (VMs) in hybrid
Amazon API Gateway resource policies are
- JSON policy documents
you attach to an API
to control whether a specified principal (typically an IAM user or role) can invoke the API.
to allow your API to be securely invoked by:
- users from a specified AWS account
- specified source IP address ranges or CIDR blocks
- specified virtual private clouds (VPCs) or VPC endpoints (in any account)
Agent runs on ec2 instance that allows system manager run command. It enumerates EC2 instances to conform to rules you provide in an assessment template