by Nathan Case, Sr. Consultant, AWS Events are precursor to incidents, but how do you decide if an event is harmful? Tuning the signal to noise means that every event needs to be inspected and its impact calculated in as short amount of time as possible to stop bad things from happening. In this session, we will dive deep into a few event types to do advanced analysis in pursuit of deciding if it is a security incident, and how to resolve it by the time the alert hits your inbox.

1. Incident Response on AWS
A practical look

2. WHAT SHOULD I LOOK FOR?

3. Configuration vs Behavior
Configuration
Security Group rules
NACL
Bucket Policy
Behavior
Login attempts
New credentials
Unusual access patterns

4. Questions you will need to answer
• What is my expressed security objective in words?
• Is this configuration or behavior related?
• What data, where, could help inform me?
• Do I have requisite ownership or visibility?
• What are my performance requirements?
• What mechanisms support the above?
• What is my expressed security objective in code?

5. APPLYING WHAT WE KNOW

6. The high-level playbook
CloudWatch EventAdversary Your environment Responder

7. “IF SOMEONE TURNS CLOUDTRAIL
OFF, TURN IT BACK ON.”
Security Objective

8. cloudtrail:StopLogging
Incident: CloudTrail gets turned off
Adversary API Call

9. CloudWatch
Events event
{
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [ "cloudtrail.amazonaws.com" ],
"eventName": [ "StopLogging" ]
}
}
Incident: CloudTrail gets turned off
Adversary CloudWatch
Event
API Call

10. Incident: CloudTrail gets turned off
Adversary ResponderCloudWatch
Event
API Call
cloudtrail.start_logging

11. “I ONLY WANT APPROVED MANAGED
POLICIES ATTACHED TO IAM USERS.”
Security Objective

12. Adversary
iam.attach_user_policy(
UserName='Bill',
PolicyArn='arn:aws:iam::aws:policy/PowerUserAccess'
)
IAM

13. CloudWatch
Events event
Adversary
{
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [ "iam.amazonaws.com" ],
"eventName": [
"AttachGroupPolicy”,
"AttachRolePolicy",
"AttachUserPolicy"
]
}
}

14. Adversary Responder
iam.detach_user_policy

15. “DO NOT ALLOW INLINE IAM
POLICIES.”
Security Objective

16. Adversary
iam.put_user_policy(
UserName='Bill',
PolicyName='AdministratorAccess',
PolicyDocument=adminpolicy
)
IAM
adminpolicy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}

17. CloudWatch
Events event
Adversary
{
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [ "iam.amazonaws.com" ],
"eventName": [
”PutGroupPolicy",
”PutRolePolicy",
”PutUserPolicy"
]
}
}

18. Adversary Responder
iam.delete_user_policy

19. “ONLY ALLOW EC2 INSTANCES LAUNCHED FROM
APPROVED AMIS AND WITH APPROPRIATE SUBNETS
AND SECURITY GROUPS.”
Security Objective

20. ImageId=ami-f9dd458a
SubnetId=subnet-a8aa4ef0
SecurityGroups=[
GroupId=sg-45533823
]
EC2

21. CloudWatch
Events event
{
"detail-type": [
"EC2 Instance State-change Notification"
],
"detail": {
"state": [ "pending" ]
},
"source": [ "aws.ec2" ]
}

22. Responder
# check if the AMI is approved
# check if AMI is used in correct subnet
# check if AMI was launched with approved security group

23. {
"ami": "ami-0d77397e",
"region": "eu-west-1",
"security_groups": [
"sg-cc9a3aaa"
],
"subnets": [
"subnet-ac3d7cda",
"subnet-2f9c1677"
]
},
{
"ami": "ami-f9dd458a",
"region": "eu-west-1",
"security_groups": [
"sg-ee9a3a88"
],
"subnets": [
"subnet-ad3d7cdb",
"subnet-2e9c1676"
]
}
DynamoDB

24. {
'Time': int(time.time()),
'Source': 'auto.responder.level1',
'Resources': [ str(instance_id) ],
'DetailType': 'activeResponse',
'Detail': {
'instance': instance_id,
'actionsRequested': 'instanceTermination'
}
}
Event
DynamoDB

25. CloudWatch Event
{
"detail-type": [
"activeResponse"
],
"source": [
"auto.responder.level1"
]
}

26. L2 responder
ec2.terminate_instances

27. ON-INSTANCE

28. Shortest Route to Lambda
CloudWatch EventEC2 CloudWatch Logs Lambda

29. On-instance…now what?
CloudWatch EventEC2 CloudWatch Logs Lambda
?

30. Introducing Amazon EC2 Systems Manager
A set of capabilities that enable automated configuration and
ongoing management of systems at scale, across all your Windows
and Linux workloads, running in Amazon EC2 or on-premises

31. Systems Manager Capabilities
Run Command Maintenance
Windows
Inventory
State Manager Parameter Store
Patch Manager
Automation
Configuration,
Administration
Update and
Track
Shared
Capabilities

32. Automation
EC2 Systems Manager
• Simplified automation solution
• Perfect for AMI updates, instance deployment & config
• Pro-active event notifications
• AWS optimized (EC2 Run Command, AWS Lambda,
AWS CloudTrail, IAM, and Amazon CloudWatch
integrations)

33. Automation – Getting Started
1. Create an
automation
document
2. Run automation 3. Monitor your
automation

34. Automate Using Extensible Framework
• Generic framework to convert
manual and repetitive tasks into
automated steps
• Use predefined automation tasks
or create custom automation
• Safely perform management
operations at scale using
delegated administration
Automation document
Run the automation
Role and permissioninput

35. On-instance…now what?
CloudWatch EventEC2 CloudWatch Logs Lambda
?

36. Automate!
CloudWatch EventEC2 CloudWatch Logs Lambda
Run Command

37. Go forth and respond!
• Understand what normal looks like
• Express your security objectives in a clear way
• Know where to find the right information
• Have a plan
• Practice

38. Incident Response on AWS
Any questions?