Performing forensics on AWS resources is a new experience for many customers who might have older runbooks based on on-premises workflows using manual steps, or perhaps no processes in place at all. In this session, get a deeper insight into the various runbooks to perform practical forensic tasks on AWS resources like Amazon EC2 instances, using a combination of industry tooling, AWS serverless services like AWS Lambda and AWS Step Functions, and managed services like Amazon Athena.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to Perform Forensics on AWS
Using Serverless Infrastructure
Henrik Johansson
Principal – Office of the CISO
AWS Security
S E C 4 1 6
Andrew Krug
Staff Security Engineer / Identity and Access Management
Mozilla

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Intro/Primer
● Challenges with forensics
● Pre-Work
● Why do you need it?
Tools
● On the fly forensics environment
● State tracking/orchestration
● Capturing
Process/Runbook
● Recommendations
Customer Dialogue
● What requirements do you have today?
● Are you meeting that requirement?
● What challenges do you run into?
Closing Statements
● What to remember
● Next Steps
● More resources to look at

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Breakout repeats
Monday, November 26
How to Perform Forensics on AWS Using Serverless Infrastructure - SEC416-R1
2:30PM PST | Venetian, Level 4, Lando 4305, T1
Wednesday, November 26
How to Perform Forensics on AWS Using Serverless Infrastructure - SEC416-R1
2:30PM PST | Venetian, Level 4, Lando 4305, T1

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud forensics: What
is it?
Forensics can occur during triage to
answer the question:
“Am I having a security incident?”
Or can occur post incident to
answer the who, what, where, when,
why, and how of how a security
incident occurred.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges of cloud forensics
● Cloud can be a big place.
○ Hunting resources that are part of an incident can be challenging.
○ Your tooling and process needs to be as scalable as your application.
● Sometimes the speed of innovation outpaces forensics tools.
○ More on this later.
● Preparation is key to being effective at ensuring effectiveness.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud is a big place
At the time of writing there are:
● 110 distinct AWS services
available in botocore.
● 15 distinct regions in
addition to AWS GovCloud

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sometimes innovation
outpaces tooling
● New security features /
tooling can render analysis
tools obsolete.
● Community supported tools
don’t always move at the
same pace.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Preparation is key

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pre-forensics
Detection and analysis
Containment and
recovery
Evidence Evidence Evidence Evidence
Custody chain
who did x at
0:00GMT
checked out file
x at 0:00GMT

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some tools for detection
• Amazon GuardDuty
• AWS Trusted Advisor
• Your SIEM using threat tntelligence
• AWS CloudTrail anomalies
• Billing alarms
• AWS outreach
• Ad-hoc contact

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Plenty of isolation code out there
#!/bin/bash
aws ec2 authorize-security-group-ingress --group-name isolation-sg
--protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32
aws ec2 revoke-security-group-egress --group-id sg-BLOCK-ID
--protocol ‘-1’ --port all --cidr ‘0.0.0.0/0’
# removed rule that allows all outbound traffic

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional considerations
● Prior to isolation ensure that
evidence is preserved.
● You may need to do
additional work to deregister
from things like autoscale
groups to prevent accidental
termination.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Forensics in AWS : Why do you need it?
● Compliance
○ PCI, GDPR, ISO27001
■ You need to have an incident
plan in place.
● Alignment
○ AWS shared security model
● Your customers
○ Protect them from:
■ The obvious
■ Internally known risk
■ Blind spots
■ and the unknown
■ Trust

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tools
● AWS services
○ Amazon CloudWatch Events
○ AWS Lambda functions
○ AWS Step Functions
○ EC2 API
○ AWS CLI
○ Amazon Athena
● Community provided tooling
○ Rekall framework / Volatility framework
○ ssm_acquire (new) - Released at re: Invent 2018
○ AWS_IR
○ MargaritaShotgun
( Links on final slides for these )

18. Your on the fly forensics environment
Analysis as a service

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What does a forensics environment look like?
• Derived from a standard EC2 image.
• Hardened to an appropriate baseline.
• Throw away environment that includes tooling for analyze:
• Services : Any events that could affect your account.
• Infrastructure : Network traffic, OS related data ( disk, memory, etc ).
• Application Data: Code deployed to Lambda, API Gateway, CloudFront, etc.
• Updated regularly and validated against your current environment.
• Deployed “close” to the affected resources to perform forensics.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Types of evidence
Instance-based API-based
CloudTrail x
Syslog Files x
Instance
metadata
x
Disk snapshot x
Memory
sample
x

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional evidence ( sometimes missed )
• Amazon S3 object logs
• API Gateway logs
• CloudFront logs
• Docker container logs

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenants of a good forensic tool
• Does not give an attacker more privilege during acquisition.
• Performs each action on an instance(s) only once.
• When the environment is mutated ( i.e. attach security group ) keeps an
audit log of the state before and after.
• Alters the instance in the most minimal way possible.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
All about SSM Acquire
What is SSM Acquire?
SSM Acquire is a forensics tool that capitalizes on AWS Systems Manager in order to gather volatile data for use in
triage, forensics, and other applications.
SSM Acquire turns intent into code with an easy to use YAML-based system for driving preservation of incident data
prior to isolation or destructive operations.
Where is ssm_acquire:
https://github.com/mozilla/ssm_acquire

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SSM Acquire : How does it work?
CloudWatch

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Live demo and artifact inspection

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What did we just do?
Received an alert Ran SSM_Acquire
Preserve a
memory sample
Build a Rekall
Profile
Analysis
Netstat ProcessList YaraScan pidhashtable

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How did we embody the tenants of a good tool?
# Template a limited access IAM policy for assumeRole
def get_limited_policy(region, instance_id):
config = get_config()
policy_template = load_policy()
instance_arn = generate_arn_for_instance(region, instance_id)
s3_bucket = config('asset_bucket', namespace='ssm_acquire')
for permission in policy_template['PolicyDocument']['Statement']:
if permission['Action'][0] == 's3:PutObject':
s3_arn = 'arn:aws:s3:::{}/{}'.format(s3_bucket, instance_id)
s3_keys = 'arn:aws:s3:::{}/{}/*'.format(s3_bucket, instance_id)
record_index = policy_template['PolicyDocument']['Statement'].index(permission)
policy_template['PolicyDocument']['Statement'][record_index]['Resource'][0] = s3_arn
policy_template['PolicyDocument']['Statement'][record_index]['Resource'][1] =
s3_keys
elif permission['Action'][0].startswith('ssm:Send'):
record_index = policy_template['PolicyDocument']['Statement'].index(permission)
policy_template['PolicyDocument']['Statement'][record_index]['Resource'][1] =
instance_arn
elif permission['Sid'] == 'STMT4':
s3_arn = 'arn:aws:s3:::{}'.format(s3_bucket)
s3_keys = 'arn:aws:s3:::{}/*'.format(s3_bucket)
record_index = policy_template['PolicyDocument']['Statement'].index(permission)
policy_template['PolicyDocument']['Statement'][record_index]['Resource'][0] = s3_arn
policy_template['PolicyDocument']['Statement'][record_index]['Resource'][1] =
s3_keys
statements = json.dumps(policy_template['PolicyDocument'])
logger.info('Limited scope role generated for assumeRole: {}'.format(statements))
return statements
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Sid: "STMT1"
Effect: "Allow"
Action:
- "s3:PutObject"
- "s3:GetObject"
- "s3:ListObjects"
Resource:
- None
- None
-
Sid: "STMT2"
Effect: "Allow"
Action:
- "ssm:ListDocuments"
- "ssm:ListDocumentsVersions"
- "ssm:DescribeDocument"
- "ssm:GetDocument"
- "ssm:DescribeInstanceInformation"
- "ssm:DescribeDocumentParameters"
- "ssm:DescribeInstanceProperties"
- "ssm:GetCommandInvocation"
Resource: '*'

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resultant policy
"Version": "2012-10-17", "Statement": [ { "Sid": "STMT1", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject",
"s3:ListObjects" ], "Resource": [ "arn:aws:s3:::ssmacquireroles-assetbucket-wuno61xfxj4k/i-
02b11a4e45295e95c", "arn:aws:s3:::ssmacquireroles-assetbucket-wuno61xfxj4k/i-02b11a4e45295e95c/*" ]
}, { "Sid": "STMT2", "Effect": "Allow", "Action": [ "ssm:ListDocuments", "ssm:ListDocumentsVersions",
"ssm:DescribeDocument", "ssm:GetDocument", "ssm:DescribeInstanceInformation",
"ssm:DescribeDocumentParameters", "ssm:DescribeInstanceProperties", "ssm:GetCommandInvocation" ],
"Resource": "*" }, { "Sid": "STMT3", "Effect": "Allow", "Action": [ "ssm:SendCommand",
"ec2:DescribeInstanceStatus" ], "Resource": [ "arn:aws:ssm:*:*:document/*", "arn:aws:ec2:*:*:instance/i-
02b11a4e45295e95c" ] }, { "Sid": "STMT4", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [
"arn:aws:s3:::ssmacquireroles-assetbucket-wuno61xfxj4k", "arn:aws:s3:::ssmacquireroles-assetbucket-
wuno61xfxj4k/*" ] } ] }

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The result
• A time limited scoped token that can only runCommand on the
instance in question and putObject to the asset store.
• The asset store in Amazon S3 has bucket versioning enabled.
• If an attacker exfiltrates the credential the blast radius is limited to
uploading versions of assets and SSM runCommand on the instance
they already gained access to.

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do I begin?
At this stage:
● You’ve acknowledged being able to perform forensics provides value.
● Understand the pre-work of reaching isolation / containment.
● Have a firm grasp of what evidence preservation and chain of custody
or audit logs look like.
What’s next?
● Start writing runbooks for potential incidents.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why runbooks?
● Runbooks ensure reliable / consistent process when you most need it
● Runbooks provide a baseline for iteration and continuous improvement

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A starting point
If you have ever wondered about
what kind of security incidents
you could have in Amazon
GuardDuty is an excellent starting
point. The Generate Sample
Findings feature will populate the
console with every type of finding
GuardDuty knows how to detect.
You can then decided which are
relevant to you and write a run
book for each type of finding.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Anatomy of a runbook
Purpose
Title The title of your runbook.
CreateDate When was this written?
Owner Who owns the process?
Objective What does the process seek to achieve?
Scope What does the runbook apply to ( People, process, technology )
Methodology What are our IOCs? Who should be notified? What is the kill chain?

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample runbook : Backdoor:EC2/Spambot
Purpose
Title Unusual Behavior Detected on Port 25
CreateDate 11-20-18
Owner John Smith
Objective
Determine if the instance is sending out spam. Mitigate, preserve evidence,
inform stakeholders.
Scope Compromised EC2 instances.
Methodology ( continued next slide )

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Methodology : automate what is possible
Humans are awesome…
unless you want consistent results - Henrik

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

40. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Henrik Johansson
@henrikjay - twitter
henrikj@amazon.com
Andrew Krug
@andrewkrug - twitter
akrug@mozilla.com

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.