Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ABD335_Real-Time Anomaly Detection Using Amazon Kinesis

4,206 views

Published on

Amazon Kinesis Analytics offers a built-in machine learning algorithm that you can use to easily detect anomalies in your VPC network traffic and improve security monitoring. Join us for an interactive discussion on how to stream your VPC flow Logs to Amazon Kinesis Streams and identify anomalies using Kinesis Analytics.

  • Be the first to comment

ABD335_Real-Time Anomaly Detection Using Amazon Kinesis

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Anomaly Detection Using Amazon Kinesis R y a n N i e n h u s , S r . P M , A m a z o n K i n e s i s A l l a n M a c I n n i s , K i n e s i s S o l u t i o n s A r c h i t e c t , A W S N o v e m b e r 2 0 1 7 AWS re:INVENT
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Most data is produced continuously Metering Records Mobile Apps Application LogsWeb Clickstream IoT Sensors Smart Buildings
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Diminishing value of data
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Durable • Continuous • Fast • Correct • Reactive • Reliable Processing real-time, streaming data What are the key requirements? Ingest Transform Analyze React Persist
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Amazon Kinesis Data Streams Amazon Kinesis Data Analytics Amazon Kinesis Data Firehose Build custom applications that process and analyze streaming data Easily process and analyze streaming data with standard SQL Easily load streaming data into AWS
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Data Streams • Easy administration and low cost • Build real-time applications with framework of choice • Secure, durable storage
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Data Firehose • Zero administration and seamless elasticity • Direct-to-data store integration • Serverless, continuous data transformations Amazon S3 Amazon Redshift
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Data Analytics • Powerful real-time applications • Easy to use, fully managed • Automatic elasticity
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon CloudWatch Monitor your AWS resources in near real time Monitor custom, application-specific metrics Monitor and store logs Set alarms View graphs and statistics Monitor and react to resource changes
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon CloudWatch Logs • Monitor logs from Amazon EC2 instances with CloudWatch Logs Agent • Archive logged data • Use other AWS services as data source: • Amazon Route 53 DNS queries • AWS CloudTrail logged events • VPC flow logs
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CloudWatch Logs Subscriptions Deliver near real-time feed of log events to Kinesis or AWS Lambda log data Application CloudWatch Logs Kinesis Lambda
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis benefits and CWL subscription • Use Kinesis Firehose to persist log data to another durable storage location: Amazon S3, Amazon Redshift, Amazon Elasticsearch Service • Use Kinesis Analytics to perform near real-time streaming analytics on your log data: • Anomaly detection • Aggregation • Use Kinesis Streams with a custom stream processing application to apply business logic to your log data: • Alternate data destinations • Data enrichment
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring application-specific metrics • Use CloudWatch Agent to send application logs to CloudWatch Logs • Analyze stream with Kinesis Analytics application • Persist raw log data to durable storage with Kinesis Firehose log data CloudWatch Logs Kinesis Streams Kinesis Analytics Kinesis Firehose DynamoDB • Active users over past 15 minutes? • Top 10 articles read in the past 30 minutes? • Filter unwanted log entries S3 log data Application
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring application-specific metrics • Don’t plan to use CloudWatch Events or Alarms? • Consider logging directly to Kinesis with the Kinesis Agent or Kinesis APIs event data event data CloudWatch Logs Kinesis Streams CloudTrail Kinesis Analytics Kinesis Firehose DynamoDB SNSTop 20 API calls over 1 min window? What service is getting called the most? What IAM user is making the most calls?
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring network activity • Use VPC Flow Logs to get visibility into application communication • VPC Flow Log records contain network data that can be analyzed 2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK Source IP Address Destination IP Address Action ACCEPT | REJECT
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring network activity • Enrich source and destination data in near real time • Aggregate data by specific dimensions and persist aggregated values network logs network logs CloudWatch Logs Kinesis FirehoseVPC Flow Logs DynamoDB Kinesis Analytics S3 map IP addresses to application names DynamoDB
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a Bad deployment?
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a Network problems?
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the application dependencies? Expected outbound dependencies for Service A MySQL Service A Redis
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the application dependencies? Identified outbound dependencies for Service A using traffic logs MySQL Service A Redis S3 Service B DynamoDB
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional Resources Learn more about Amazon Kinesis in our documentation. To easily send data to Amazon Kinesis, use the Kinesis Data Generator. For additional information, see Test Your Streaming Data Solution with the New Amazon Kinesis Data Generator. Learn more about Amazon CloudWatch in our documentation. For more ideas about log monitoring, see Implement Serverless Log Analytics Using Amazon Kinesis Analytics and Real-Time Clickstream Anomaly Detection with Amazon Kinesis Analytics.
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×