In a rapidly changing IT environment, detecting and responding to new threats is more important than ever. This session shows you how to build a predictive analytics stack on AWS, which harnesses the power of Amazon Machine Learning in conjunction with Amazon Elasticsearch Service, AWS CloudTrail, and VPC Flow Logs to perform tasks such as anomaly detection and log analysis. We also demonstrate how you can use AWS Lambda to act on this information in an automated fashion, such as performing updates to AWS WAF and security groups, leading to an improved security posture and alleviating operational burden on your security teams.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Michael Capicotto, Solutions Architect
Matt Nowina, Solutions Architect
November 30, 2016
SAC304
Predictive Security
Using Big Data to Fortify Your Defenses

2. Cybersecurity headlines from 2015…
...Over 169 million personal records were exposed, stemming from 781
publicized breaches across the financial, business, education,
government and healthcare sectors.
...There were 38 percent more security incidents detected than in 2014.
...The median number of days that attackers stay dormant within a
network before detection is over 200.
... 81 percent reported they had neither a system nor a managed security
service in place to ensure they could self-detect data breaches, relying
instead on notification from an external party.
... Only 38 percent of global organizations claim they are prepared to
handle a sophisticated cyberattack.

3. You will learn how to…
 Build a log analytics stack with Amazon Elasticsearch
Service
 Utilize Amazon Machine Learning to predict bad actors
 Perform forensic analysis on your network paths
 Implement advanced options in your continuous,
predictive security stack

4. Big Data – Logs, logs everywhere

5. ?Nobody looks at
them!
Big Data – Logs, logs everywhere…isn’t always good

6. Build a log analytics stack

7. Log sources in AWS
AWS CloudTrail logs OS and application
logs
VPC flow logs Amazon CloudWatch Logs

8. Setting up a log analytics stack
CloudWatch Logs Amazon Elasticsearch
Service
AWS Lambda

9. Demo #1 – Elasticsearch and Kibana

10. Awesome, we can see stuff!
 Now we have real-time visualization of all logs
Great for risk scenarios we
already know about!
Example – Single user logging in from
several IP addresses
Not so great for unknown
scenarios
There are many of these!
How do we protect against these risks?

11. Integrating machine learning

12. Amazon Machine Learning
Easy to use,
managed machine
learning service built
for developers
Robust, powerful
machine learning
technology based on
Amazon’s internal
systems
One-click production
model deployment
Binary classification
Multiclass classification
Regression

13. Using Amazon Machine Learning’s real-time predictions, we
can drastically shorten how long it takes you to become aware
of a threat

14. Training your model (daily)
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
Amazon Machine
Learning
Build model from
dataset
Log analytics
stack
AWS Lambda
Transform and
store logs in S3

15. Using Big Data – Example dataset
{
"datetime": "7/30/16 0:20",
"AWSregion": "aws-sa-east-1",
"IP": "69.90.60.155",
"protocol": "TCP",
"source": "6000",
"destination": "1433",
"country": ”BrVirginIslands",
"region": ”PricklyPear",
"postalcode": ”VG1120",
"Lat": ”18.5000",
"Long": ”64.3667”,
"Threat": 94
}

16. Real-time predictions
Amazon Machine
Learning
Endpoint for real-
time predictions
Log analytics
stack
AWS Lambda
Trigger on each
new log entry
Amazon SNS
notification

17. Demo #2 – Real-time ML predictions

18. Security stack
Amazon Machine
Learning
Trained model and
endpoint for real-
time predictions
Log analytics
stack
AWS Lambda
Trigger on each
new log entry
Amazon SNS
notification
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
AWS Lambda
Transform and
store logs in S3

19. Close, but not perfect!
We still wont catch every potential breach
 Machine learning cannot predict every possible threat
 Attackers are getting smarter and more sophisticated every day
When one does occur, we want to know why
 This helps us prevent it from happening again!

20. Forensic analysis

21. AWS Production Account
us-east-1a
us-east-1b
Proxies
NAT
RDS DB
DMZSubnet
PrivateSubnet
PrivateSubnet
Proxies
Bastion
RDS DB
PrivateSubnet
PrivateSubnet
Virtual Private Cloud (VPC)
Network sprawl
AWS API Account
us-east-1a
us-east-1b
PrivateSubnetPrivateSubnet
Virtual Private Cloud (VPC)

22. Reasoning about networks
Web service and CLI
available in private
beta
Answers questions
about your network
No packets sent
?

23. Demo #3 – Network reasoning

24. Demo

25. Advanced options

26. Evolving the practice of security architecture
Security architecture as a separate function can no longer
exist
Static position papers,
architecture diagrams, and
documents
UI-dependent consoles and
technologies
Auditing, assurance, and
compliance are decoupled,
separate processes
Current security
architecture
practice

27. Evolving the practice of security architecture
Architecture artifacts
(design choices, narrative,
etc.) committed to common
repositories
Complete solutions account
for automation
Solution architectures are
living audit/compliance
artifacts and evidence in a
closed loop
Evolved security
architecture
practice
AWS
CodeCommit
AWS
CodePipeline Jenkins
Security architecture can now be part of the “maker” team

28. Continuous monitoring and auto-remediation
Self-managed
 AWS CloudTrail -> Amazon CloudWatch Logs -> Amazon CloudWatch Alerts
 AWS CloudTrail -> Amazon SNS -> AWS Lambda -> Network reasoning
Compliance validation
 AWS Config Rules
Host-based compliance validation
 Amazon Inspector
Active change remediation
 Amazon CloudWatch Events

29. More sophisticated machine learning models
Train your model with your data
 Real-world data specific to your application
 Previous threats you have dealt with
Considering modeling threats by clusters of logs
 Identify threats more accurately than just a single log entry
Build threat profiles that pattern typical attack stages
 Reconnaissance, scanning, gaining access, maintaining access, and
covering tracks

30. Tying it all together
Amazon Machine
Learning
Trained model and
endpoint for real-
time predictions
Log analytics
stack
AWS Lambda
Trigger on each
new log entry
Amazon SNS
notification
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
AWS Lambda
Transform and
store logs in S3
AWS Config Rules
Network
reasoning
VPC, security groups,
network ACLs

31. Next steps
 Set up your log analytics stack: http://amzn.to/2dIZjIz
 Blog post and AWS CloudFormation template
 Build your first Amazon ML machine learning model:
http://amzn.to/1K8HfRu
 Stay tuned on the AWS Security Blog for more on this
topic
 We’re here all week! Come chat with us.

32. Thank you!

33. Remember to complete
your evaluations!