Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ABD203_Real-Time Streaming Applications on AWS

1,366 views

Published on

To win in the marketplace and provide differentiated customer experiences, businesses need to be able to use live data in real time to facilitate fast decision making. In this session, you learn common streaming data processing use cases and architectures. First, we give an overview of streaming data and AWS streaming data capabilities. Next, we look at a few customer examples and their real-time streaming applications. Finally, we walk through common architectures and design patterns of top streaming data use cases.

  • Be the first to comment

ABD203_Real-Time Streaming Applications on AWS

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Streaming Applications on AWS Use Cases and Patterns R a y Z h u , S r . P r o d u c t M a n a g e r , A W S AWS re:Invent N o v e m b e r 2 7 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Why real-time analytics? • What is real-time data? • What real-time services does AWS offer? • Who uses them? • How do I use them? • Q & A
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHY REAL-TIME ANALYTICS?
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Real-Time Analytics?
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A Day in the Life Error 500 The server encountered an internal error and was unable to complete your request.
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A Day in the Life 219.134.32.117 - - [16/Feb/1998:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR 3.8.23015.5)“ 95.169.41.62 - - [16/Feb/1998:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200 3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko“ 221.147.191.247 - - [16/Feb/1998:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/38.0.895.0 Safari/538.0.1“ 179.96.123.130 - - [16/Feb/1998:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6“ 16/Feb/1998 16/Feb/1998 16/Feb/1998 16/Feb/1998
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Diminishing Value of Data
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHAT IS REAL-TIME DATA?
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Real-Time Data? Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htdo cs/test
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Analytics Requirements • Durable • Continuous • Fast • Reactive • Available • Reliable Ingest Transform Analyze React Persist
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHAT REAL-TIME SERVICES DOES AWS OFFER?
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis: AWS Real-Time Services Kinesis Firehose Easily load streaming data into AWS Kinesis Analytics Easily process and analyze streaming data with standard SQL Kinesis Streams Build custom applications that process and analyze streaming data
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Streams? Kinesis Streams AWS SDK Mobile SDK Kinesis Agent CloudWatch AWS IoT AWS Lambda Kinesis Client Library Kinesis Firehose Kinesis Analytics Amazon EMR
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Firehose? Kinesis Firehose Amazon Redshift Amazon S3 Amazon Elasticsearch Service
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Analytics? Kinesis Analytics • Continuous anomaly detection • Continuous time series analysis • Continuous filtering • Continuous aggregation • Continuous enrichment • Continuous anomaly detection • Continuous time series analysis • Continuous filtering • Continuous aggregation • Continuous enrichment
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Third-Party Connectors
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHO USES THEM?
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Customers
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sushiro
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sushiro
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Game Studios
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Game Studios
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Netflix
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Netflix
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. HOW DO I USE THEM?
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time application • Continuous data processing by KCL application • Continuous data archival to Amazon S3 Amazon S3 Kinesis Client Library Kinesis Streams Kinesis Firehose
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time aggregation • Continuous metric aggregation over time periods • Continuous delivery to storage and analytics services SUM AVG MIN MAX Amazon Redshift Amazon S3 Amazon Elasticsearch Service Kinesis FirehoseKinesis Streams Kinesis Analytics
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time alarms and notifications • Continuous anomaly detection over a sequence of events • Real-time alarm and notification triggering Anomaly Detection Amazon CloudWatch Amazon SNS Kinesis Streams Kinesis Analytics AWS Lambda
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GETTING STARTED
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started • Kinesis Home Page: https://aws.amazon.com/kinesis/ • Kinesis Blog Post: https://aws.amazon.com/kinesis/blog-posts/ • Getting Started Page: https://aws.amazon.com/kinesis/getting- started/
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Q & A
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×