NET203_Using Amazon VPC Flow Logs to Do Predictive Security Analytics

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
N E T 2 0 3
AWS re:INVENT
Using Amazon VPC Flow Logs to Do
Predictive Security Analytics
N o v e m b e r 2 7 , 2 0 1 7

Introductions and Welcome
Carl Johnson
Enterprise Solutions Architect
carlpjoh@amazon.com
Grant McCarthy
Enterprise Solutions Architect
gmccarth@amazon.com

Prerequisites
Full Participation:
• A laptop with Internet Access and a Web Browser.
• AWS Account with full IAM privileges, access to EU-WEST-1 region.
• EC2 Keypair in the EU-WEST-1 region.
• Setup your web stack using this AWS CloudFormation:
http://amzn.to/2yX3rSb
• A basic understanding of ANSI SQL, Amazon VPC, AWS Lambda, and AWS WAF.
Follow Along:
• A neighbor that loves to share!
$25 AWS Credits are available at the end of the workshop

Reference Architecture
EC2
EC2
VPC
Internet
Gateway
Application
Load Balancer
VPC Flow
Logs
S3
Amazon
Elasticsearch Service
Amazon
Kinesis Analytics
Amazon
Machine Learning
Amazon
Kinesis
Firehose

AWS Services used?
Primary to this solution:
AWS Web Application Firewall (WAF)
Amazon VPC Flow Logs
Amazon Elasticsearch Service
Amazon Kinesis Analytics
Amazon Machine Learning
Helpers:
Amazon Kinesis Firehose
Amazon Simple Storage Service (S3)
Amazon Lambda
Amazon CloudWatch Logs

What are we doing and why?
Users
Attackers
EC2
Application

What are we doing and why?
Users
Attackers
EC2
Application
WAF

AWS Web Application Firewall

What is a web application firewall?
• Web application firewall (WAF) is an appliance,
server plugin, or filter that applies a set of rules
to HTTP traffic
• WAFs come in four flavors
• Pure play: Standalone appliance or software
• CDN: bundled with content delivery network
• Load balancer: bundled with a load balancer
• Universal threat manager (UTM): catch-all for
misc. security

Why use a WAF?
Application vulnerabilities:
Good users
Bad folks
Web server Database
Exploit
code
Your application

Why use a WAF?
Abuse detection and prevention:
Good users
Bad folks
Web server Database
Your applicationData
leaks

Why use a WAF?
Distributed denial of service (DDOS) attacks:
Good users
Bad folks
Web server Database
Your application

AWS
WAF
Why use a WAF?
AWS WAF  block the bad folks and allow the good users:
Good users
Bad folks
Web server Database
Your application

Why use a WAF?
• WAFs help protect websites and applications against
attacks that cause data breaches and downtime
• General WAF use cases
• Protect from SQL injection (SQLi) and cross-site scripting (XSS)
• Prevent website scraping, crawlers, and BOTs
• Mitigate DDOS (HTTP/HTTPS floods)
• Gartner reports that main driver of WAF purchases (25-
30%) is PCI compliance

How - Analyze.
Users
Attackers
EC2
Application
Human: Analyze

 Stores log in AWS CloudWatch Logs
 Can be enabled on
• Amazon VPC, a subnet, or a network interface
• Amazon VPC & Subnet enables logging for all interfaces in the VPC/subnet
• Each network interface has a unique log stream
 Flow logs do not capture real-time log streams for your network interfaces
 Filter desired result based on need
• All, Reject, Accept
• Troubleshooting or security related with alerting needs?
• Think before enabling All on VPC, will you use it?

VPC Flow Logs
• Agentless
• Enable per ENI, per subnet, or per VPC
• Logged to AWS CloudWatch Logs
• Create CloudWatch metrics from log data
• Alarm on those metrics
AWS
account
Source IP
Destination IP
Source port
Destination port
Interface Protocol Packets
Bytes Start/end time
Accept
or reject

VPC Flow Logs
• Amazon
Elasticsearch
Service
• Amazon
CloudWatch
Logs
subscriptions

How – Protect.
Users
Attackers
EC2
Application
WAF
Human: Protect

Amazon
Elasticsearch
Service
Amazon Kinesis
Firehose
VPC Flow
Logs
AWS WAF
Amazon Elasticsearch Service

Amazon Elasticsearch Service – Use Case 1
http://amzn.to/2yX9dTR

How - Analyze.
Users
Attackers
EC2
Application
Machine: Analyze

Pay for only what you use
Automatic elasticity
Standard SQL for analytics
Real-time processing
Easy to use

Use SQL to build real-time applications
Easily write SQL code to process
streaming data
Connect to streaming source
Continuously deliver SQL results

Connect to streaming source
• Streaming data sources include Firehose or
Streams
• Automatic ingestion of JSON and CSV
formats; Other formats supported through
Lambda Pre-processing.
• Each input has a schema; schema is inferred,
but you can edit
• Reference data sources (S3) for data
enrichment

Write SQL code
• Build streaming applications with one-to-many
SQL statements
• Robust SQL support and advanced analytic
functions
• Extensions to the SQL standard to work
seamlessly with streaming data
• Support for at-least-once processing
semantics
• Support for the majority of ANSI SQL 2011
standard

Continuously deliver SQL results
• Send processed data to multiple destinations
• S3, Amazon Redshift, Amazon ES (through
Firehose)
• Streams (with AWS Lambda integration for
custom destinations)
• End-to-end processing speed as low as sub-
second
• Separation of processing and data delivery

Generate time series analytics
• Compute key performance indicates over-time windows
• Combine with historical data in S3 or Amazon Redshift
Amazon
Kinesis
AnalyticsStreams
Firehose
Amazon
Redshift
S3
Streams
Firehose
Custom,real-
time
destinations

Feed real-time dashboards
• Validate and transform raw data, and then process to calculate
meaningful statistics
• Send processed data downstream for visualization in BI and
visualization services
Amazon
QuickSightAnalytics
Amazon ES
Amazon
Redshift
Amazon
RDS
Streams
Firehose

Create real-time alarms and notifications
• Build sequences of events from the stream, like user sessions in a
clickstream or app behavior through logs
• Identify events (or a series of events) of interest, and react to the
data through alarms and notifications
Analytics
Streams
Firehose
Streams
Amazon
SNS
Amazon
CloudWatch
Lambda

How – Detect & Protect.
Users
Attackers
EC2
Application
WAF
Machine: Analyze, Detect, Alert, Protect

Amazon Kinesis
Analytics
Amazon Kinesis
Firehose AWS WAF
Amazon SNSVPC Flow
Logs
AWS Lambda
Alert?

Amazon Kinesis Analytics - Use Case 1
High Frequency abuse, content crawlers. Lets block them!
http://amzn.to/2zDsgPY

How - Learn.
Users
Attackers
EC2
Application
Machine: Learn

AI Solutions for Every DeveloperUSABILITY&
SIMPLICITY
CONTROL

What is Amazon Machine Learning?
• Easy-to-use, managed machine learning service built for developers
• Robust, powerful machine learning technology based on Amazon’s
internal systems
• Create models using your data already stored in the AWS cloud
• Deploy models to production in seconds

Integrated with the AWS data ecosystem
Access data that is stored in Amazon S3, Amazon Redshift, or MySQL databases in
Amazon RDS
Output predictions to Amazon S3 for easy integration with your data flows
Use AWS Identity and Access Management (IAM) for fine-grained data access
permission policies

Binary classification
Predict the answer to a Yes/No question
Multiclass classification
Predict the correct category from a list
Regression
Predict the value of a numeric variable
Three supported types of predictions

Explore and understand your data

Explore model quality

Batch predictions
Asynchronous, large-volume prediction generation
Request through service console or API
Best for applications that deal with batches of data records
>>> import boto
>>> ml = boto.connect_machinelearning()
>>> model = ml.create_batch_prediction(
batch_prediction_id = 'my_batch_prediction’,
batch_prediction_data_source_id = ’my_datasource’,
ml_model_id = ’my_model',
output_uri = 's3://examplebucket/output/’)

Real-time predictions
Synchronous, low-latency, high-throughput prediction generation
Request through service API, server, or mobile SDKs
Best for interaction applications that deal with individual data records
>>> import boto
>>> ml = boto.connect_machinelearning()
>>> ml.predict(
ml_model_id = ’my_model',
predict_endpoint = ’example_endpoint’,
record = {’key1':’value1’, ’key2':’value2’})
{
'Prediction': {
'predictedValue': 13.284348,
'details': {
'Algorithm': 'SGD',
'PredictiveModelType': 'REGRESSION’
}
}
}

How – Predict & Protect.
Users
Attackers
EC2
Application
WAF
Machine: Learn, Predict, Alert, Protect

Amazon Machine
Learning
Amazon S3
Amazon Kinesis
Firehose
AWS WAF
Amazon SNS
VPC Flow
Logs
AWS Lambda

Amazon Machine Learning - Use Case 1
Test the model and lets rate limit them!
http://amzn.to/2yYnsHR

Build an Amazon Machine Learning
Collect
& Organize
Augment
& Enrich
Experiment
& Learn

Amazon
Machine
Learning
AWS WAF
Amazon
SNS
VPC Flow
Logs
Amazon
Machine
Learning
Good Traffic?
Yes
No Within
acceptable
limits?
Yes
No
Extra Credit: Using multiple ML models

Expected Outcome.
Users
Attackers
EC2
Application
WAF
Automated: Analyze, Predict, Alert, Protect

Don’t Forget: Clean up
• Ensure any resources used in this Workshop are
terminated
• Delete the CloudFormation Stack
AND/OR
• Terminate EC2 instances
• Delete S3 objects
• Delete Kinesis Delivery streams/Analytics
• Remove Lambda triggers on S3 buckets
• Etc.
$25 AWS Credits are available at the end of the workshop

Closing & Questions
Please complete the survey!

THANK YOU!

NET203_Using Amazon VPC Flow Logs to Do Predictive Security Analytics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to NET203_Using Amazon VPC Flow Logs to Do Predictive Security Analytics

Similar to NET203_Using Amazon VPC Flow Logs to Do Predictive Security Analytics (20)

More from Amazon Web Services

More from Amazon Web Services (20)

NET203_Using Amazon VPC Flow Logs to Do Predictive Security Analytics