This document discusses using Amazon Kinesis to perform real-time anomaly detection on streaming data. It describes how Kinesis can ingest, transform, analyze, and react to streaming data. Specifically, it shows how to monitor application metrics and logs using CloudWatch Logs and Kinesis, and analyze network traffic using VPC flow logs and Kinesis. The goal is to detect issues and anomalies in real-time to maintain high service performance and reliability.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Anomaly Detection Using
Amazon Kinesis
R y a n N i e n h u s , S r . P M , A m a z o n K i n e s i s
A l l a n M a c I n n i s , K i n e s i s S o l u t i o n s A r c h i t e c t , A W S
N o v e m b e r 2 0 1 7
AWS re:INVENT

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Most data is produced continuously
Metering Records
Mobile Apps Application LogsWeb Clickstream
IoT Sensors Smart Buildings

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Diminishing value of data

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
Processing real-time, streaming data
What are the key requirements?
Ingest Transform Analyze React Persist

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Build custom
applications that process
and analyze streaming
data
Easily process and
analyze streaming data
with standard SQL
Easily load streaming
data into AWS

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Streams
• Easy administration and low cost
• Build real-time applications with framework of choice
• Secure, durable storage

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data transformations
Amazon S3
Amazon Redshift

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Analytics
• Powerful real-time applications
• Easy to use, fully managed
• Automatic elasticity

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch
Monitor your AWS resources in near real time
Monitor custom, application-specific metrics
Monitor and store logs
Set alarms
View graphs and statistics
Monitor and react to resource changes

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch Logs
• Monitor logs from Amazon EC2 instances with CloudWatch Logs
Agent
• Archive logged data
• Use other AWS services as data source:
• Amazon Route 53 DNS queries
• AWS CloudTrail logged events
• VPC flow logs

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs Subscriptions
Deliver near real-time feed of log events to Kinesis or AWS Lambda
log data
Application
CloudWatch
Logs
Kinesis
Lambda

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis benefits and CWL subscription
• Use Kinesis Firehose to persist log data to another durable
storage location: Amazon S3, Amazon Redshift, Amazon
Elasticsearch Service
• Use Kinesis Analytics to perform near real-time streaming
analytics on your log data:
• Anomaly detection
• Aggregation
• Use Kinesis Streams with a custom stream processing application
to apply business logic to your log data:
• Alternate data destinations
• Data enrichment

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Use CloudWatch Agent to send application logs to CloudWatch Logs
• Analyze stream with Kinesis Analytics application
• Persist raw log data to durable storage with Kinesis Firehose
log data
CloudWatch
Logs
Kinesis
Streams
Kinesis
Analytics
Kinesis
Firehose
DynamoDB
• Active users over past 15 minutes?
• Top 10 articles read in the past 30 minutes?
• Filter unwanted log entries
S3
log data
Application

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Don’t plan to use CloudWatch Events or Alarms?
• Consider logging directly to Kinesis with the Kinesis Agent or Kinesis
APIs
event data event data
CloudWatch
Logs
Kinesis
Streams
CloudTrail Kinesis
Analytics
Kinesis
Firehose
DynamoDB
SNSTop 20 API calls over 1 min window?
What service is getting called the most?
What IAM user is making the most calls?

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Use VPC Flow Logs to get visibility into application
communication
• VPC Flow Log records contain network data that can be analyzed
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
Source IP Address Destination IP Address Action
ACCEPT | REJECT

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Enrich source and destination data in near real time
• Aggregate data by specific dimensions and persist aggregated values
network logs network logs
CloudWatch
Logs
Kinesis
FirehoseVPC Flow Logs
DynamoDB
Kinesis
Analytics
S3
map IP addresses to
application names
DynamoDB

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Is something wrong with the network?
Service A Service A Service B Service B Service C
Account 1234567890, Zone us-east-1e
Service D Service D Service D Service E Service F
Account 0987654321, Zone eu-west-1a

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Is something wrong with the network?
Service A Service A Service B Service B Service C
Account 1234567890, Zone us-east-1e
Service D Service D Service D Service E Service F
Account 0987654321, Zone eu-west-1a
Bad
deployment?

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Is something wrong with the network?
Service A Service A Service B Service B Service C
Account 1234567890, Zone us-east-1e
Service D Service D Service D Service E Service F
Account 0987654321, Zone eu-west-1a
Network
problems?

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are the application dependencies?
Expected outbound dependencies for Service A
MySQL Service A Redis

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are the application dependencies?
Identified outbound dependencies for Service A using traffic logs
MySQL Service A Redis
S3 Service B DynamoDB

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional Resources
Learn more about Amazon Kinesis in our documentation.
To easily send data to Amazon Kinesis, use the Kinesis Data Generator. For
additional information, see Test Your Streaming Data Solution with the
New Amazon Kinesis Data Generator.
Learn more about Amazon CloudWatch in our documentation.
For more ideas about log monitoring, see Implement Serverless Log
Analytics Using Amazon Kinesis Analytics and Real-Time Clickstream
Anomaly Detection with Amazon Kinesis Analytics.

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!