Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Allan MacInnis, Solutions Architect
11/7/2017
Re...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Streaming data overview
• Services over...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Most data is produced continuously
Mobile Apps W...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The diminishing value of data
Recent data is hig...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Processing real-time, streaming data
• Durable
•...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis makes it easy to work with real-
...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Streams
• Reliably ingest and dur...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Firehose
• Reliably ingest and de...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Analytics
• Interact with streami...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch
Monitor your AWS resources in ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs
Monitor and store logs
• Monitor...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs Subscriptions
Deliver near real-...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Kinesis - CloudWatch Logs
subscripti...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Use Cl...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring application-specific metrics
• Don’t ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring access to AWS resources
• Deliver AWS...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Use VPC Flow Logs ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring network activity
• Enrich source and ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
Is something wrong with...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
What are all the depend...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor network activity
What are all the depend...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional Resources
Learn more about Amazon Kin...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Upcoming SlideShare
Loading in …5
×

Real-time Application Monitoring with Amazon Kinesis and Amazon CloudWatch - AWS Online Tech Talks

2,314 views

Published on

Learning Objectives:
- Understand how to build a real-time application monitoring system using network traffic logs
- Learn how to enrich and aggregate network flow logs data using Amazon Kinesis
- Learn how to visualize and analyze your network data to gain actionable insights

  • Be the first to comment

Real-time Application Monitoring with Amazon Kinesis and Amazon CloudWatch - AWS Online Tech Talks

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Allan MacInnis, Solutions Architect 11/7/2017 Real-time Application Monitoring with Amazon Kinesis and Amazon CloudWatch
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Streaming data overview • Services overview • Application monitoring scenarios • Application monitoring example
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming data overview
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Most data is produced continuously Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The diminishing value of data Recent data is highly valuable • If you act on it in time • Perishable Insights (M. Gualtieri, Forrester) Old + Recent data is more valuable • If you have the means to combine them
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Processing real-time, streaming data • Durable • Continuous • Fast • Correct • Reactive • Reliable What are the key requirements? Ingest Transform Analyze React Persist
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services overview
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis makes it easy to work with real- time streaming data Amazon Kinesis Streams • For Technical Developers • Collect and stream data for ordered, replayable, real-time processing Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into Amazon S3, Redshift, ElasticSearch Amazon Kinesis Analytics • For all developers, data scientists • Easily analyze data streams using standard SQL queries
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Streams • Reliably ingest and durably store streaming data at low cost • Build custom real-time applications to process streaming data
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Firehose • Reliably ingest and deliver batched, compressed, and encrypted data to S3, Redshift, and Elasticsearch • Point and click setup with zero administration and seamless elasticity
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Analytics • Interact with streaming data in real-time using SQL • Build fully managed and elastic stream processing applications that process data for real-time visualizations and alarms
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon CloudWatch Monitor your AWS resources in near real-time Monitor custom, application-specific metrics Monitor and store logs Set alarms View graphs and statistics Monitor and react to resource changes
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CloudWatch Logs Monitor and store logs • Monitor logs from Amazon EC2 instances with CloudWatch Logs Agent • Archive logged data • Use other AWS services as data source: • Amazon Route 53 DNS queries • AWS CloudTrail logged events • VPC flow logs
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CloudWatch Logs Subscriptions Deliver near real-time feed of log events to Kinesis or AWS Lambda log data Application CloudWatch Logs Kinesis Lambda
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of Kinesis - CloudWatch Logs subscription• Use Kinesis Firehose to persist log data to another durable storage location: S3, Redshift, Elasticsearch Service • Use Kinesis Analytics to perform near real-time streaming analytics on your log data: • Anomaly detection • Aggregation • Use Kinesis Streams with a custom stream processing application to apply business logic to your log data: • Alternate data destinations • Data enrichment
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Application monitoring scenarios
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring application-specific metrics • Use CloudWatch Agent to send application logs to CloudWatch Logs • Analyze stream with Kinesis Analytics application • Persist raw log data to durable storage with Kinesis Firehose log data CloudWatch Logs Kinesis Streams Kinesis Analytics Kinesis Firehose DynamoDB • Active users over past 15 minutes? • Top 10 articles read in the past 30 minutes? • Filter unwanted log entries S3 log data Application
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring application-specific metrics • Don’t plan to use CloudWatch Events or Alarms? • Consider logging directly to Kinesis with the Kinesis Agent or Kinesis APIs log data Kinesis Streams Kinesis Analytics Kinesis Firehose DynamoDB • Active users over past 15 minutes? • Top 10 articles read in the past 30 minutes? • Filter unwanted log entries S3 Application
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring access to AWS resources • Deliver AWS CloudTrail events to CloudWatch logs • Create Kinesis Streams subscription • Analyze Stream with Kinesis Analytics application event data event data CloudWatch Logs Kinesis Streams CloudTrail Kinesis Analytics Kinesis Firehose DynamoDB SNS Top 20 API calls over 1 min window? What service is getting called the most? What IAM user is making the most calls? S3
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring network activity • Use VPC Flow Logs to get visibility into application communication • VPC Flow Log records contain network data that can be analyzed 2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK Source IP Address Destination IP Address Action ACCEPT | REJECT
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring network activity • Enrich source and destination data in near real-time • Aggregate data by specific dimensions and persist aggregated values network logs network logs CloudWatch Logs Kinesis FirehoseVPC Flow Logs DynamoDB Kinesis Analytics S3 map IP addresses to application names DynamoDB
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor network activity Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor network activity Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a Bad deployment?
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor network activity Is something wrong with the network? Service A Service A Service B Service B Service C Account 1234567890, Zone us-east-1e Service D Service D Service D Service E Service F Account 0987654321, Zone eu-west-1a Network problems?
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor network activity What are all the dependencies of an application? MySQL Service A Redis Expected outbound dependencies for Service A
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor network activity What are all the dependencies of an application? MySQL Service A Redis Identified outbound dependencies for Service A using traffic logs S3 Service B DynamoD B
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional Resources Learn more about Amazon Kinesis in our documentation. To easily send data to Amazon Kinesis, use the Kinesis Data Generator. For additional information, see Test Your Streaming Data Solution with the New Amazon Kinesis Data Generator. Learn more about Amazon CloudWatch in our documentation. For more ideas about log monitoring, see Implement Serverless Log Analytics Using Amazon Kinesis Analytics and Real-time Clickstream Anomaly Detection with Amazon Kinesis Analytics.
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×