Evolve Your IR Process with AWS Powers

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Henrik Johansson
AWS Security
SID306
Evolve Your Incident Response Process and
Powers for AWS

What to Expect from This Session
• Quick sync on incident response, how runbooks support
• Overview of empowering AWS capabilities for IR process
• Discussion of traditional IR analogs in AWS environments
• Reminder of key IR pre-reqs for AWS-oriented IR success
• High-level runbook example, evolved for AWS
• Additional resources for security heroes

Incident Response (IR) at a Glance
Establish
control
Determine
impact
Recover as
needed
Investigate
root cause
Improve

Process, People, and … Powers
• Process = The How
• Informal knowledge -> Runbooks
• People = The Who
• Security Army of One -> Security Operations Team
• Powers = Capabilities / Tools
• COTS / OSS for low #s -> Bespoke / Automated / Scalable
This will vary ... And that’s OK!

IR Where / How?
• Nowadays, IR should not be a manual process
• Effective incident response blends automation and manual
abilities where applicable.
• Re-evaluate any manual process for automation opportunities
• Natural efficiencies of cloud-based IR for cloud concerns
• Also look for opportunities to leverage powers of the cloud for
on-premises IR

Reality of now vs. ….?
You (likely | hopefully) have established
IR runbooks
You are using AWS
Your IR process, people, powers needs
to be informed and account for that!

This talk is new, but this topic is not!
• Previous related talks …
• YouTube search “automating event response AWS”
• AWS specific features & empowering capabilities
• Event detection, logging, automation triggers, rollback
• Various pre-requisite knowledge
• What security team access to enable, what to turn on, where
Let’s go over some of that!

Empowering AWS Capabilities
Let’s make IR easier!
• Amazon GuardDuty
• AWS CloudTrail
• Amazon CloudWatch
• AWS Config

Amazon GuardDuty
Intelligent threat detection and
continuous monitoring to protect
your AWS accounts and
workloads

What Can GuardDuty Detect?
RDP brute
force
RAT Installed
Exfiltrate temp
IAM creds over
DNS
Probe API with
temp creds
Attempt to
compromise
account
Malicious or
suspicious IP
Unusual ports
DNS exfiltration
Unusual traffic volume
Connect to blacklisted site
Recon
Anonymizing proxy
Temp credentials
used off-instance
Unusual ISP caller
Bitcoin activity
Unusual instance launch

Finding Types
Recon
• Port probe on unprotected port
• Outbound port scans
• Callers from anonymizing proxies
Backdoor
• Spambot or C&C activity
• Exfiltration over DNS channel
• Suspicious domain request
Trojan
• Domain Generation Algorithm (DGA)
domain request
• Blackhole traffic
• Drop point
Unauthorized Access
• Unusual ISP caller
• SSH/RDP brute force
Stealth
• Password policy change
• AWS CloudTrail logging disabled
• Amazon GuardDuty disabled in member
account
Cryptocurrency
• Communication with bitcoin DNS pools
• Cryptocurrency related DNS calls
• Connections to bitcoin mining pool

Multi-Account Support
Account B Account C
Security team account
Account A
CloudWatch Events
GuardDuty GuardDuty GuardDuty
GuardDuty

AWS CloudTrail
• CloudTrail is a service that enables governance, compliance,
operational auditing, and risk auditing of your AWS account
• Captures account activity and events for supported services
made in your AWS account and sends the event log files to
Amazon Simple Storage Service (Amazon S3), CloudWatch Logs,
and CloudWatch Events.
• Visibility Into User and Resource Activity
• Log File Encryption, Integrity Validation, other features

CloudTrail Example
{
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [ "cloudtrail.amazonaws.com" ],
"eventName": [ "StopLogging" ]
}
}

Amazon CloudWatch
CloudWatch Events delivers a near real-time stream of system
events that describe changes in AWS resources.
CloudWatch Events becomes aware of operational changes as
they occur and allows you to direct them to suitable targets.

CloudWatch Events
{
"source": [
"aws.guardduty"
]
}
CloudWatch
Event
GuardDuty
findings
Lambda
function

AWS Config / AWS Config Rules
Continuously tracks your resource
configuration changes and if they violate
any of the conditions in your rules

AWS Config Rules
A continuous recording and assessment service
Changing resources
AWS Config
AWS Config rules
History
snapshot
Notifications
API access
Normalized
• How are my resources configured over time?
• Is a change that just occurred to a resource, compliant?

IR via AWS … Traditional Analogs
• Log gathering
• Write once media
• Network isolation
• Disk capture
Can you do all of this in AWS environments? YES!

AWS Based IR Triggers ... DIY or aaS!
• DIY anomaly detection
• FlowLogs, CloudTrail, CloudWatch Logs
• Managed services
• GuardDuty, AWS Trusted Advisor, Amazon Macie, CloudWatch
Events, AWS Config

Amazon
CloudWatch
CloudTrail
AWS Config
Lambda
function
AWS APIs
AWS WAF
AWS Shield
Detection
Alerting
Remediation
Countermeasures
Forensics
Team
collaboration
(Slack etc.)
GuardDuty
VPC Flow Logs
AWS-Oriented IR at A High Level

Host-Based Cloud Triggers
Integrate cloud controls with host-based IDS/IDP
Trigger visible events
Doesn’t require native cloud support
Use Amazon EC2 roles
Example:
SSH PAM modules
Agents with script support

Host-Based Cloud Triggers - Example
Host script in SSH PAM module:
#!/bin/bash
INSTANCE_ID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone|sed
's/.{1}$//')DATE=$(date)
aws ec2 --region $REGION create-tags --resources $INSTANCE_ID --tags "Key=Tainted,Value=$DATE
{
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [ "ec2.amazonaws.com" ],
"eventName": [ "CreateTags" ],
"errorCode": [ "Client.UnauthorizedOperation" ]
}
CloudTrail event:

On-premises vs. Cloud IR …
All the same, right?

On-Premises Network Isolation Options
Switches/Routers
Centralized management/logs?
Tamper evident?
Mixed vendor/brand?
Firewalls
Brand knowledge requirements?
Running shoes, a scissor and CAT5 cables

Network Isolation in AWS
VPC – 100% API based – Centralized integrity validated logging
Security Groups
Network ACL
PrivateLink
VPC Endpoints
Non-AWS constructs
Proxies
Gateways

On-Premises Disk Capture Options
- Require physical (snapshot)
+ Established industry processes
+ Tool driven (Lime, require trigger engine)

Disk Capture in AWS
- New method for many companies
+ Fully remote
+ Snapshot (AWS API) or tool driven (Lime)
+ Trigger using automation (AWS Step Functions / AWS Lambda)

IR via AWS Advantage
• Never run out of disk space, compute
• Never feel guilty about not using it, either
• Streaming data processing
• Reduce event -> response, MTTR
• Automation
• API driven with complete audit trail
• Allow rapid response, isolation of resources
• Self-healing capabilities for availability

Data Gathering / Fusion / Analysis / Query
Systems Manager
documents
CloudWatch
rule
EC2 instance contents
Instance:~ ec2-user$ top
Instance:~ ec2-user$ pcap
Instance:~ ec2-user$ lime
LambdaGuardDuty
Elastic Network
Adapter
Elastic Network
Adapter
Lambda
function
EBS Volume

Detach instance from Auto Scaling Group (self-heal) and isolate
# Detach instance from autoscaling group
CLIENT_AS.detach_instances(
InstanceIds=[instances[i]['instanceId’]], AutoScalingGroupName=instances[i]['asGroup’], ShouldDecrementDesiredCapacity=False
)
# Get correct security group
response = CLIENT_EC2.describe_instances(InstanceIds=[instances[i]['instanceId']])
vpcId = response['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['VpcId’]
# Get Security Group groupID for the Isolation group
CLIENT_EC2.describe_security_groups(
DryRun=False,
Filters=[
{'Name': 'description','Values': ['SG-Isolation']},
{'Name': 'vpc-id','Values': [vpcId]}
]
)
try:
# Remove egress rule on security group if exists
if response['SecurityGroups'][0]['IpPermissionsEgress'][0]:
client.revoke_security_group_egress(GroupId=sg['SecurityGroups'][0]['GroupId’],
IpPermissions=sg['SecurityGroups'][0]['IpPermissionsEgress’])
except:
pass
try:
sgGroup = response['SecurityGroups'][0]['GroupId’]
# Isolate instance by applying empty security group
CLIENT_EC2.modify_instance_attribute(InstanceId=instances[i]['instanceId’], Groups=[sgGroup])
print "Isolating instance: ", instances[i]['instanceId']

# Detach instance from autoscaling group
CLIENT_AS.detach_instances(
InstanceIds=[instances[i]['instanceId’]], AutoScalingGroupName=instances[i]['asGroup’], ShouldDecrementDesiredCapacity=False
)
# Get correct security group
response = CLIENT_EC2.describe_instances(InstanceIds=[instances[i]['instanceId']])
vpcId = response['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['VpcId’]
# Get Security Group groupID for the Isolation group
CLIENT_EC2.describe_security_groups(
DryRun=False,
Filters=[
{'Name': 'description','Values': ['SG-Isolation']},
{'Name': 'vpc-id','Values': [vpcId]}
]
)
try:
# Remove egress rule on security group if exists
if response['SecurityGroups'][0]['IpPermissionsEgress'][0]:
client.revoke_security_group_egress(GroupId=sg['SecurityGroups'][0]['GroupId’],
IpPermissions=sg['SecurityGroups'][0]['IpPermissionsEgress’])
except:
pass
try:
sgGroup = response['SecurityGroups'][0]['GroupId’]
# Isolate instance by applying empty security group
CLIENT_EC2.modify_instance_attribute(InstanceId=instances[i]['instanceId’], Groups=[sgGroup])
print "Isolating instance: ", instances[i]['instanceId']
Important!
Rate limit / Guardrails
Don’t nuke yourself!
Detach instance from Auto Scaling Group (self-heal) and isolate

Wait! Did you make sure to …? AWS IR Pre-Reqs
• Audit / IR role
• CloudTrail “On”
• Centralized logging / Alerting
• Amazon S3 bucket logging (do we need this with Amazon S3
event access > CloudTrail?)
• Resource backup / versioning
• Pre-built IR environments

Enforcing Security Policy at Scale
Use managed services to offload
GuardDuty, Trusted Advisor
Serverless for rapid scalability
Doesn’t require cold resources
Multi-account
CloudWatch Events Hub

Enough! Gimme runbooks.

What’s in a runbook?
Wikipedia:
In a computer system or network, a runbook is a compilation of
routine procedures and operations that the system administrator
or operator carries out. System administrators in IT departments
and NOCs use runbooks as a reference. Runbooks can be in either
electronic or in physical book form.
Or…

On-premises vs. AWS vs. Hybrid IR Runbooks
• Relies on multiple control
planes
• Possible physical access
required
• Certain resources allow
automation
On-premises
• Single API driven control
plane
• 100% network-based control
• All resources allow
automation
AWS

In terms for today:
1. Definition of Application Normal
2. Statement of Problem or Event
3. Response for Problem or event
Or…

In terms for today:
A place for the beginning of:
• automation and process
• Security Incident Response Simulations
and…in the event of an emergency
A place to start corrective action.

In terms for today:
A runbook provides needed confidence and stability in a moment
that would normally be chaotic.

What’s in a runbook - Reality
Process – Written to be tested and modified, Not set in stone.
People – Trained on those processes and in the tools to provide stability and
confidence
Capabilities / Tools - built and tested with the people and processes
routinely
=
Power

On-Premises IR
• Focused on infrastructure or
application surfaces
• Focused on utilization of
internal resources, system
that were pre-defined, and
limited by cost and
maintenance
• Limited ability to automate a
response.

Cloud IR Runbook
• Services-based threat detection
and automated remediation.
• Focused on multiple surface
interaction.
• Forensics automation is
possible.
• System recovery and regional DR
is possible with automated
procedures.

Hybrid IR Runbook
• Working with cloud tools, server
automation is possible.
• (VMWare, SSM)
• Centralized SIEM is possible in the
cloud, expanding as it is needed.
• SSM is able to automate the forensics
of systems on premises and copy
them to the forensics environments in
the cloud
• Allowing for safe and secure review

On-premises IR Runbook: Botnet computer detected
Source: Log/traffic analysis or abuse alert
Process:
1. Go to computer/server
2. Unplug network
3. Image hard drive
1. If Lime/process present: Live snapshot
2. If no Lime/process present: Processes terminated
4. Inventory credentials at risk
5. Rotate/re-issue credentials
6. Re-issue clean computer/server
7. Extract credential usage from local/central/identified logs

Cloud IR Runbook: Botnet instance detected
Source: GuardDuty automatic alert
Process:
1. Describe instance
2. Isolate instance (Isolate script: Security group, network ACL, ALB
remove)
1. Auto Scaling group self-heals with new clean instance
3. Snapshot live/isolated instance
4. Inventory credentials used
5. Rotate credentials
6. Extract all credential activities (Aurora script for AWS CloudTrail)

Cloud IR Runbook: Botnet instance detected
Source: GuardDuty automatic alert
Process:
1. Describe instance
2. Isolate instance (Isolate script: Security group, NACL, ALB remove)
1. AutoScalingGroup self heals with new clean instance
3. Snapshot live/isolated instance
6. Extract all credential activities (Aurora script for CloudTrail)
Note:
Step 1-6 can be fully automated with
GuardDuty -> CloudWatch Events -> Lambda/Step Functions

Hybrid IR Runbook: Botnet instance detected
Source: <Depends on hybrid model>
1. If On-premises resources
3. Extract all credential activities (Aurora script for CloudTrail)
Follows suitable runbook but contains AWS specific parts like IAM
credential management

IR-Related Partner Solutions

Open-Source IR Solutions
• AWS Security Automation
https://github.com/awslabs/aws-security-automation
• ThreatResponse
https://threatresponse.cloud
https://github.com/ThreatResponse/aws_ir
• Wazuh
https://documentation.wazuh.com/current/amazon/

Open Source IR Solutions, Continued
• Cloud Custodian
https://github.com/capitalone/cloud-custodian
• Fido
https://github.com/Netflix/Fido
• Security Monkey
https://github.com/Netflix/security_monkey
• StreamAlert
https://github.com/airbnb/streamalert

Community / Industry Resources
• FIRST
https://first.org/
• Cloud.gov
https://cloud.gov/docs/ops/security-ir/

Summary
• If you don’t have runbooks, write them. If you already have
runbooks, ensure they account for AWS.
• Leverage AWS-specific capabilities for basic -> advanced IR,
including automation.
• Challenges are opportunities to learn more about the platform
and partner solutions, perhaps next-level IR
• Practice, and engage AWS Support, as needed

Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

Thanks!

Evolve Your IR Process with AWS Powers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Evolve Your IR Process with AWS Powers

Similar to Evolve Your IR Process with AWS Powers (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Evolve Your IR Process with AWS Powers