Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defenses (SAC304)

791 views

Published on

In a rapidly changing IT environment, detecting and responding to new threats is more important than ever. This session shows you how to build a predictive analytics stack on AWS, which harnesses the power of Amazon Machine Learning in conjunction with Amazon Elasticsearch Service, AWS CloudTrail, and VPC Flow Logs to perform tasks such as anomaly detection and log analysis. We also demonstrate how you can use AWS Lambda to act on this information in an automated fashion, such as performing updates to AWS WAF and security groups, leading to an improved security posture and alleviating operational burden on your security teams.

Published in: Technology

AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defenses (SAC304)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Michael Capicotto, Solutions Architect Matt Nowina, Solutions Architect November 30, 2016 SAC304 Predictive Security Using Big Data to Fortify Your Defenses
  2. 2. Cybersecurity headlines from 2015… ...Over 169 million personal records were exposed, stemming from 781 publicized breaches across the financial, business, education, government and healthcare sectors. ...There were 38 percent more security incidents detected than in 2014. ...The median number of days that attackers stay dormant within a network before detection is over 200. ... 81 percent reported they had neither a system nor a managed security service in place to ensure they could self-detect data breaches, relying instead on notification from an external party. ... Only 38 percent of global organizations claim they are prepared to handle a sophisticated cyberattack.
  3. 3. You will learn how to…  Build a log analytics stack with Amazon Elasticsearch Service  Utilize Amazon Machine Learning to predict bad actors  Perform forensic analysis on your network paths  Implement advanced options in your continuous, predictive security stack
  4. 4. Big Data – Logs, logs everywhere
  5. 5. ?Nobody looks at them! Big Data – Logs, logs everywhere…isn’t always good
  6. 6. Build a log analytics stack
  7. 7. Log sources in AWS AWS CloudTrail logs OS and application logs VPC flow logs Amazon CloudWatch Logs
  8. 8. Setting up a log analytics stack CloudWatch Logs Amazon Elasticsearch Service AWS Lambda
  9. 9. Demo #1 – Elasticsearch and Kibana
  10. 10. Awesome, we can see stuff!  Now we have real-time visualization of all logs Great for risk scenarios we already know about! Example – Single user logging in from several IP addresses Not so great for unknown scenarios There are many of these! How do we protect against these risks?
  11. 11. Integrating machine learning
  12. 12. Amazon Machine Learning Easy to use, managed machine learning service built for developers Robust, powerful machine learning technology based on Amazon’s internal systems One-click production model deployment Binary classification Multiclass classification Regression
  13. 13. Using Amazon Machine Learning’s real-time predictions, we can drastically shorten how long it takes you to become aware of a threat
  14. 14. Training your model (daily) Amazon S3 Stores machine learning dataset AWS Lambda Daily machine learning model training Amazon Machine Learning Build model from dataset Log analytics stack AWS Lambda Transform and store logs in S3
  15. 15. Using Big Data – Example dataset { "datetime": "7/30/16 0:20", "AWSregion": "aws-sa-east-1", "IP": "69.90.60.155", "protocol": "TCP", "source": "6000", "destination": "1433", "country": ”BrVirginIslands", "region": ”PricklyPear", "postalcode": ”VG1120", "Lat": ”18.5000", "Long": ”64.3667”, "Threat": 94 }
  16. 16. Real-time predictions Amazon Machine Learning Endpoint for real- time predictions Log analytics stack AWS Lambda Trigger on each new log entry Amazon SNS notification
  17. 17. Demo #2 – Real-time ML predictions
  18. 18. Security stack Amazon Machine Learning Trained model and endpoint for real- time predictions Log analytics stack AWS Lambda Trigger on each new log entry Amazon SNS notification Amazon S3 Stores machine learning dataset AWS Lambda Daily machine learning model training AWS Lambda Transform and store logs in S3
  19. 19. Close, but not perfect! We still wont catch every potential breach  Machine learning cannot predict every possible threat  Attackers are getting smarter and more sophisticated every day When one does occur, we want to know why  This helps us prevent it from happening again!
  20. 20. Forensic analysis
  21. 21. AWS Production Account us-east-1a us-east-1b Proxies NAT RDS DB DMZSubnet PrivateSubnet PrivateSubnet Proxies Bastion RDS DB PrivateSubnet PrivateSubnet Virtual Private Cloud (VPC) Network sprawl AWS API Account us-east-1a us-east-1b PrivateSubnetPrivateSubnet Virtual Private Cloud (VPC)
  22. 22. Reasoning about networks Web service and CLI available in private beta Answers questions about your network No packets sent ?
  23. 23. Demo #3 – Network reasoning
  24. 24. Demo
  25. 25. Advanced options
  26. 26. Evolving the practice of security architecture Security architecture as a separate function can no longer exist Static position papers, architecture diagrams, and documents UI-dependent consoles and technologies Auditing, assurance, and compliance are decoupled, separate processes Current security architecture practice
  27. 27. Evolving the practice of security architecture Architecture artifacts (design choices, narrative, etc.) committed to common repositories Complete solutions account for automation Solution architectures are living audit/compliance artifacts and evidence in a closed loop Evolved security architecture practice AWS CodeCommit AWS CodePipeline Jenkins Security architecture can now be part of the “maker” team
  28. 28. Continuous monitoring and auto-remediation Self-managed  AWS CloudTrail -> Amazon CloudWatch Logs -> Amazon CloudWatch Alerts  AWS CloudTrail -> Amazon SNS -> AWS Lambda -> Network reasoning Compliance validation  AWS Config Rules Host-based compliance validation  Amazon Inspector Active change remediation  Amazon CloudWatch Events
  29. 29. More sophisticated machine learning models Train your model with your data  Real-world data specific to your application  Previous threats you have dealt with Considering modeling threats by clusters of logs  Identify threats more accurately than just a single log entry Build threat profiles that pattern typical attack stages  Reconnaissance, scanning, gaining access, maintaining access, and covering tracks
  30. 30. Tying it all together Amazon Machine Learning Trained model and endpoint for real- time predictions Log analytics stack AWS Lambda Trigger on each new log entry Amazon SNS notification Amazon S3 Stores machine learning dataset AWS Lambda Daily machine learning model training AWS Lambda Transform and store logs in S3 AWS Config Rules Network reasoning VPC, security groups, network ACLs
  31. 31. Next steps  Set up your log analytics stack: http://amzn.to/2dIZjIz  Blog post and AWS CloudFormation template  Build your first Amazon ML machine learning model: http://amzn.to/1K8HfRu  Stay tuned on the AWS Security Blog for more on this topic  We’re here all week! Come chat with us.
  32. 32. Thank you!
  33. 33. Remember to complete your evaluations!

×