How Zocdoc Achieves Automatic Threat Detection & Remediation with Security as Code (SEC321-R1) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automating Threat Detection &
Remediation
Brian Lozada
CISO
Zocdoc
S e s s i o n I D : 3 2 1
Jay Ball
Head of Application Security
Zocdoc

The 21st Century Patient

Solving the Access Problem
(Avg. Wait Time for Primary Care, Days)
Portland39
Los Angeles42
Dallas12
Houston21
Miami27
Philadelphia17
Denver27
New York26
Albany
122
Boston
109
Washington DC17
Atlanta27
Seattle26
San Diego13
Detroit27

Portland39
Los Angeles42
Miami27
Philadelphia17
Denver27
New York26
Albany
122
Boston
109
Washington DC17
Atlanta27
Seattle26
Detroit27
Reduced Wait Time 30% unbooked,
cancelled or
rescheduled
Dallas12
Houston21
San Diego13
24
Hours

Supplying Care to the 21st Century Patient
2017
ALL IN
Zocdoc 2.0
Private Medical
Practices
Larger Health
Systems

Zocdoc AWS Goals
Scale
Horizontally
Diversify
Tech Stack
Open
Source
Data
Liberation
Elevate
Security

Securing Cloud Infrastructure

Security Controls
Monitor
Alert
Investigate
Prevent/Block

Amazon
CloudWatch
AWS
Trusted
Advisor
Lambda
Function
AWS
GuardDuty
AWS
CloudTrail
AWS WAF /
Shield
Amazon
Inspector

Automation in AWS Today
Automated
Remediation
Automated
Alerting
Automated
Monitoring

Issue: How do we make
sure S3 buckets are
always encrypted?
Decision: Automatic
encryption upon detection
where lacking.

Scenarios

Encryption Automation Process
Lambda
FunctionS3 operation detected,
calls function
Validate
Crypto;
Encrypts
S3 bucket
events logged
S3
Bucket
AWS
CloudTrail
Amazon
CloudWatch
Processed in
CloudWatch

Automated Detection in CloudTrail

CloudWatch Configuration
{
"source": [ "aws.s3" ],
"detail-type": [ "AWS API Call via CloudTrail" ],
"detail": {
"eventSource": [
"s3.amazonaws.com"
],
"eventName": [
"CreateBucket",
"PutBucketAcl",
"PutBucketPolicy",
"PutBucketEncryption",
"DeleteBucketEncryption"
]
}
}

Code: Setup & Test
s3 = boto3.client('s3')
sns = boto3.client('sns')
def lambda_handler(event, context):
bucket_name = event['detail']['requestParameters']['bucketName']
bucket_creator = event['detail']['userIdentity']['principalId']
bucket_user = bucket_creator.split(":",1)
try:
currEncrypt = s3.get_bucket_encryption (Bucket=bucket_name)

Code: Encrypt
except ClientError as e:
toEncrypt = s3.put_bucket_encryption (
Bucket=bucket_name,
ServerSideEncryptionConfiguration = {
'Rules':[ {
'ApplyServerSideEncryptionByDefault':
{ 'SSEAlgorithm': 'AES256' }
}]
} )
response = sns.publish(
TopicArn= 'arn:aws:sns:us-east-1: …
000012345678:unEncryptedS3BucketCreated',
Message= 'Unencrypted Bucket by '+ bucket_user[1] )
else:
return currEncrypt

Issue: Malware infection
is calling out to
command & control.
Decision: Kill the instance.

Automated Detection & Response
AWS
GuardDuty
Amazon
CloudWatch
Lambda
Function
Processed by
CloudWatch
Call Lambda on
malicious trigger
Kills EC2
instance
DNS lookup of malicious
hostname detected
AWS
EC2

Post Automation Review
• CloudWatch and CloudTrail log analysis
• Machine forensics on attached storage
• Automated instance rebuild with CloudFormation

Automated Detection

GuardDuty Configuration
Configure IAM
Permissions for
GuardDuty
Sign in &
Enable
GuardDuty
Watch the
Data Flow
1 2 3

{
"source": [
"aws.guardduty"
],
"detail-type": [
"GuardDuty Finding"
],
"detail": {
"type": [
"Backdoor:EC2/C&CActivity.B!DNS"
]
}
}

Code: Initial Sanity Checks
# lots of imports
source = event['source']
if source != "aws.guardduty":
# wrong caller, just silently return
return
detail_type = event['detail-type']
if detail_type != 'GuardDuty Finding':
return
event_type = event['detail']['type’]
bad_event_list = [ 'Backdoor:EC2/C&CActivity.B!DNS’ ]
if not(event_type in bad_event_list):
# wrong event type, silently return
return

Code: Event Processing
instance_id =
event['detail']['resource']['instanceDetails']['instanceId']
# only shutdown certain tags:
taglist= event['detail']['resource']['instanceDetails']['tags']
runme=False
for nvp in taglist:
if "security_guillotine" == nvp['key'].lower():
runme=True
if not(runme):
# We are not part of our auto-shutdown EC2 group, return
return

Code: Euthanize It
print("Shutting down instance ", instance_id)
ec2 = boto3.client('ec2')
try:
response = ec2.stop_instances(
InstanceIds=[ instance_id ]
# , DryRun=True
)
if 'DryRunOperation' not in str(e):
print("No permission to reboot instances.")
raise
else:
print('Error', e)

GuardDuty Event Detection
'Backdoor:EC2/XORDDOS',
'Backdoor:EC2/Spambot',
'Backdoor:EC2/C&CActivity.B!DNS',
'CryptoCurrency:EC2/BitcoinTool.B!DNS',
'Trojan:EC2/BlackholeTraffic',
'Trojan:EC2/DropPoint',
'Trojan:EC2/BlackholeTraffic!DNS',
'Trojan:EC2/DriveBySourceTraffic!DNS',
'Trojan:EC2/DropPoint!DNS',
'Trojan:EC2/DGADomainRequest.B',
'Trojan:EC2/DGADomainRequest.C!DNS',
'Trojan:EC2/DNSDataExfiltration',
'Trojan:EC2/PhishingDomainRequest!DNS'
GuardDuty
reports on many
events for which
we want to kill
the instance.

Issue: Someone opens
sensitive ports to the
entire Internet.
Decision: Modify the security
group to remove the rule
that opens the sensitive
ports.

Automated Response
Amazon
CloudWatch
Lambda
Function
Alert raised in
CloudWatch
Call to run
Lambda
Remove bad CIDR
from security group
Disallowed
CIDR used
Security
Group
server
AWS
CloudTrail

{
"source": [
"aws.cloudtrail"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"CreateSecurityGroup",
"AuthorizeSecurityGroupIngress"
]
}
}

Code: Initial Sanity Checks
source = event['detail']['eventSource']
if source != "ec2.amazonaws.com":
return
allowed_event_list = [
'CreateSecurityGroup',
"AuthorizeSecurityGroupIngress"
]
event_name = event['detail']['eventName']
if not(event_name in allowed_event_list):
# wrong event, just silently return
print("Wrong Filter: source=", source, " / event_name=", event_name)
return

Code: GetSecurityGroupID
resp = event['detail']['responseElements']
if (resp["_return"] != True):
# event was not a successful update, so we can ignore it.
return
SG_id = 'invalid'
if event_name == 'CreateSecurityGroup':
SG_id = resp["groupId"]
elif event_name == 'AuthorizeSecurityGroupIngress':
SG_id = event['detail']['requestParameters']['groupId']
else:
# We shouldn't actually get here.
return

Code: Group Sensitivity Training
ec2 = boto3.resource('ec2')
security_group = ec2.SecurityGroup(SG_id)
sensitive_ports = [ 22, 3389, 54321 ]
ingress_list = security_group.ip_permissions
for perm in ingress_list:
fromport=0 ; toport=0 ; ipprot=0 ; sensitive=False
if 'FromPort' in perm:
fromport = perm['FromPort']
if 'ToPort' in perm:
toport = perm['ToPort']
if 'IpProtocol' in perm:
ipprot = perm['IpProtocol']
if ipprot == "-1":
sensitive = True
if fromport > 0:
for p in sensitive_ports:
if fromport <= p and p <= toport:
sensitive = True

Code: Test for Zero CIDR
if sensitive:
for r in perm['IpRanges']:
# this could be more complex, but 0000/0 catches 95% of the cases
if r['CidrIp'] == "0.0.0.0/0":
print("Removing Ingress Rule violation: ", json.dumps(perm))
try:
security_group.revoke_ingress(
CidrIp = r['CidrIp'],
IpProtocol = perm['IpProtocol'],
FromPort = fromport,
ToPort = toport # , DryRun = True
)
if 'DryRunOperation' not in str(e):
print("Error: ", e)
raise
else:
print('DryRun: ', e)

Issue: Spend too much on
an underwhelming WAF
solution.
Decision: Leverage AWS WAF
& Shield Protection.

Traditional WAF Configuration Inefficiencies
1) Each site / ALB requires separate WAF configuration
• Ex: www and espanol, different html, same endpoints
2) Need to manage 9 different configurations, one per site
3) No ability to share rules between each configuration
• Ex: blacklist should be the same for all sites
4) No IP reputation lists
So, how did we move forward…

AWS WAF Security Automations

AWS WAF Security Automations with Modifications
Add: Country-level blocks
Add: Specialty whitelisting for some
URIs/parameters
Add: Specialty blacklisting for obsolete endpoints
Update: Reputation List processing logic
Remove: Bad bot honey pot

Dual WAFs
Zocdoc has two types of
systems
• Each has a different set of
AWS WAF rules (WebACL)
• Each type points to
multiple ALBs, like www &
espanol

WAF Configuration
Our WAFs share
some rules
Exclusions on
XSS/SQL blocks
Some rules are
unneeded on
api
ẅŵẘ ầṗĩ
1 Whitelist by IP
2 Blacklist by IP
3 IP Reputational Blacklist
4 Country Blacklist
5 Whitelist by Criteria
6 Blacklist URL ---
7 Flood
8 Scan
9 SQLi Blocks ---
10 XSS Blocks ---
Default  Allow

WAF Comparison
• 9 different rule sets
• No shared rules
• Geoblocking rules
• Many possible rules per set
• Lots of string and regex
match sets
• Specialty “Good Bot” rules
• 2 different rule sets
(WebACLs)
• Many shared rules
• Geoblocking rules
• Maximum 10 rules per set
• Maximum 5 regex match sets
total
• … phase two, “Good Bot”,
maybe
OLD WAF AWS WAF

What Security is Automated
• Bad Reputation IP blocklists updated hourly
• Floods protection from high rate IPs update periodically
• Probing IPs blocklist is update periodically
• Bad-bots IP blocklist updated instantly after trigger

AWS WAF Code
// Our Code is Amazon’s Code, thus it can be Your Code too!
Zocdoc_Customizations = {
“Special Regex Rules”,
“Double WAF with shared rules”,
“Reordering of existing rules”,
“Complex deployment process using Ansible”
}
// But actual automations are all Amazon’s own CloudFormation code!

AWS WAF Code – Zocdoc Updates
Expanded Reputation List CIDR range processing Lambda code:
https://github.com/awslabs/aws-waf-security-automations/pull/64
This change eliminates the need for one rule
... and removes up to 15,000 comparisons for each web request, thus faster
Once less processing rule saves money for all users of this Security Automation template

Issue: It’s vulnerability
time.
Decision: Leverage Amazon
Inspector for visibility and
compliance of AWS
environments to industry
standard requirements.

Basic Requirements
Amazon Inspector agent Amazon Inspector Generated
findings for
review

Zocdoc Compliance Measures
Infosec reviews the findings and submits patch requests to
teams
Prioritize by severity and compliance requirements
• SOC 2 Type II – Mostly Process
• PHI / HIPPA – Mostly Policy
• CIS – Center for Internet Security – The ever-evolving
system hardening checklist

But, what might the future be?
Generate
findings
for review
AWS
Systems
Manager
Patch
Source AMI
Kill
noncompliant
systems
Amazon Inspector
Agent runs on OS
Automatic
findings
processing
Redeploy
system

Divinations & Seering
We are not there yet, but we can start with baby steps
• OS patches are generally better defined that
Application patching
Some commercial solutions do parts of this idea
• None are perfect and all are pricy
• Amazon gives us , let’s use them
Start with CIS compliance

• Reduce inefficiencies of manual work through
automation
• Reduce alert traffic through automation
• Drives business enablement

Interested in Code?
github.com/Zocdoc/
ZocSec.SecurityAsCode.AWS

Thank you!
Brian Lozada
CISO
Twitter: @BrianL1775
LinkedIn: blozada
Jay Ball
Head of Application Security
Twitter: @veggiespam
LinkedIn: veggiespam
Security Blog: www.veggiespam.com
Instagram: woolpictures

How Zocdoc Achieves Automatic Threat Detection & Remediation with Security as Code (SEC321-R1) - AWS re:Invent 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How Zocdoc Achieves Automatic Threat Detection & Remediation with Security as Code (SEC321-R1) - AWS re:Invent 2018

Similar to How Zocdoc Achieves Automatic Threat Detection & Remediation with Security as Code (SEC321-R1) - AWS re:Invent 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

How Zocdoc Achieves Automatic Threat Detection & Remediation with Security as Code (SEC321-R1) - AWS re:Invent 2018