SlideShare a Scribd company logo
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Streaming Applications
on AWS
Use Cases and Patterns
R a y Z h u , S r . P r o d u c t M a n a g e r , A W S
AWS re:Invent
N o v e m b e r 2 7 , 2 0 1 7
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Why real-time analytics?
• What is real-time data?
• What real-time services does AWS offer?
• Who uses them?
• How do I use them?
• Q & A
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHY REAL-TIME
ANALYTICS?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Real-Time Analytics?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A Day in the Life
Error 500
The server encountered an internal error and
was unable to complete your request.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A Day in the Life
219.134.32.117 - - [16/Feb/1998:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; MSIE
10.0; Windows NT 6.1; Trident/5.1; .NET CLR 3.8.23015.5)“
95.169.41.62 - - [16/Feb/1998:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200 3883 "-" "Mozilla/5.0 (Windows NT 6.2;
Trident/7.0; rv:11.0) like Gecko“
221.147.191.247 - - [16/Feb/1998:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-" "Mozilla/5.0 (Windows; U; Windows
NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/38.0.895.0 Safari/538.0.1“
179.96.123.130 - - [16/Feb/1998:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 6.3; Win64;
x64; rv:5.4) Gecko/20100101 Firefox/5.4.6“
16/Feb/1998
16/Feb/1998
16/Feb/1998
16/Feb/1998
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Diminishing Value of Data
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT IS REAL-TIME
DATA?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Is Real-Time Data?
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client denied
by server configuration:
/export/home/live/ap/htdo
cs/test
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Analytics Requirements
• Durable
• Continuous
• Fast
• Reactive
• Available
• Reliable
Ingest Transform Analyze React Persist
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT REAL-TIME
SERVICES DOES AWS
OFFER?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis: AWS Real-Time Services
Kinesis Firehose
Easily load streaming
data into AWS
Kinesis Analytics
Easily process and
analyze streaming data
with standard SQL
Kinesis Streams
Build custom
applications that
process and analyze
streaming data
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Is Kinesis Streams?
Kinesis Streams
AWS SDK
Mobile SDK
Kinesis Agent
CloudWatch
AWS IoT
AWS Lambda
Kinesis Client Library
Kinesis Firehose
Kinesis Analytics
Amazon EMR
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Is Kinesis Firehose?
Kinesis Firehose
Amazon Redshift
Amazon S3
Amazon Elasticsearch Service
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Is Kinesis Analytics?
Kinesis Analytics
• Continuous anomaly detection
• Continuous time series analysis
• Continuous filtering
• Continuous aggregation
• Continuous enrichment
• Continuous anomaly detection
• Continuous time series analysis
• Continuous filtering
• Continuous aggregation
• Continuous enrichment
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Third-Party Connectors
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHO USES THEM?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Customers
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sushiro
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sushiro
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Game Studios
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Game Studios
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Netflix
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Netflix
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HOW DO I USE THEM?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reference Architecture
Real-time application
• Continuous data processing by KCL application
• Continuous data archival to Amazon S3
Amazon S3
Kinesis Client Library
Kinesis Streams
Kinesis Firehose
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reference Architecture
Real-time aggregation
• Continuous metric aggregation over time periods
• Continuous delivery to storage and analytics services
SUM
AVG
MIN
MAX Amazon Redshift
Amazon S3
Amazon Elasticsearch
Service
Kinesis FirehoseKinesis Streams Kinesis Analytics
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reference Architecture
Real-time alarms and notifications
• Continuous anomaly detection over a sequence of events
• Real-time alarm and notification triggering
Anomaly
Detection
Amazon CloudWatch
Amazon SNS
Kinesis Streams Kinesis Analytics AWS Lambda
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GETTING STARTED
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Started
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Started
• Kinesis Home Page: https://aws.amazon.com/kinesis/
• Kinesis Blog Post: https://aws.amazon.com/kinesis/blog-posts/
• Getting Started Page: https://aws.amazon.com/kinesis/getting-
started/
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Q & A
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!

More Related Content

What's hot

Getting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and ServerlessGetting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and Serverless
Amazon Web Services
 
Amazon Kinesis
Amazon KinesisAmazon Kinesis
Amazon Kinesis
Amazon Web Services
 
Real-Time Streaming: Intro to Amazon Kinesis
Real-Time Streaming: Intro to Amazon KinesisReal-Time Streaming: Intro to Amazon Kinesis
Real-Time Streaming: Intro to Amazon Kinesis
Amazon Web Services
 
Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)
Amazon Web Services
 
ElastiCache & Redis
ElastiCache & RedisElastiCache & Redis
ElastiCache & Redis
Amazon Web Services
 
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech TalksDeep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
Amazon Web Services
 
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
Amazon Web Services
 
실시간 스트리밍 분석 Kinesis Data Analytics Deep Dive
실시간 스트리밍 분석  Kinesis Data Analytics Deep Dive실시간 스트리밍 분석  Kinesis Data Analytics Deep Dive
실시간 스트리밍 분석 Kinesis Data Analytics Deep Dive
Amazon Web Services Korea
 
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
Amazon Web Services Korea
 
Visualizing Big Data Insights with Amazon QuickSight
Visualizing Big Data Insights with Amazon QuickSightVisualizing Big Data Insights with Amazon QuickSight
Visualizing Big Data Insights with Amazon QuickSight
Amazon Web Services
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
Amazon Web Services
 
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
Amazon Web Services
 
Amazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
Amazon RDS: Deep Dive - SRV310 - Chicago AWS SummitAmazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
Amazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
Amazon Web Services
 
Introduction to Amazon Relational Database Service
Introduction to Amazon Relational Database ServiceIntroduction to Amazon Relational Database Service
Introduction to Amazon Relational Database Service
Amazon Web Services
 
Amazon QuickSight
Amazon QuickSightAmazon QuickSight
Amazon QuickSight
Amazon Web Services
 
Architecting a Serverless Data Lake on AWS
Architecting a Serverless Data Lake on AWSArchitecting a Serverless Data Lake on AWS
Architecting a Serverless Data Lake on AWS
Amazon Web Services
 
AWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon KinesisAWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon Kinesis
Amazon Web Services
 
Real Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS LambdaReal Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS Lambda
Amazon Web Services
 
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
Amazon Web Services
 
Introduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-ITIntroduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-IT
Chitpong Wuttanan
 

What's hot (20)

Getting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and ServerlessGetting Started with AWS Lambda and Serverless
Getting Started with AWS Lambda and Serverless
 
Amazon Kinesis
Amazon KinesisAmazon Kinesis
Amazon Kinesis
 
Real-Time Streaming: Intro to Amazon Kinesis
Real-Time Streaming: Intro to Amazon KinesisReal-Time Streaming: Intro to Amazon Kinesis
Real-Time Streaming: Intro to Amazon Kinesis
 
Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)
 
ElastiCache & Redis
ElastiCache & RedisElastiCache & Redis
ElastiCache & Redis
 
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech TalksDeep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
Deep Dive on Amazon QuickSight - January 2017 AWS Online Tech Talks
 
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
Next-Generation e-Commerce Architectures (RET207) - AWS re:Invent 2018
 
실시간 스트리밍 분석 Kinesis Data Analytics Deep Dive
실시간 스트리밍 분석  Kinesis Data Analytics Deep Dive실시간 스트리밍 분석  Kinesis Data Analytics Deep Dive
실시간 스트리밍 분석 Kinesis Data Analytics Deep Dive
 
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
AWS Backup을 이용한 데이터베이스의 백업 자동화와 편리한 복구방법
 
Visualizing Big Data Insights with Amazon QuickSight
Visualizing Big Data Insights with Amazon QuickSightVisualizing Big Data Insights with Amazon QuickSight
Visualizing Big Data Insights with Amazon QuickSight
 
Amazon Aurora
Amazon AuroraAmazon Aurora
Amazon Aurora
 
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...
 
Amazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
Amazon RDS: Deep Dive - SRV310 - Chicago AWS SummitAmazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
Amazon RDS: Deep Dive - SRV310 - Chicago AWS Summit
 
Introduction to Amazon Relational Database Service
Introduction to Amazon Relational Database ServiceIntroduction to Amazon Relational Database Service
Introduction to Amazon Relational Database Service
 
Amazon QuickSight
Amazon QuickSightAmazon QuickSight
Amazon QuickSight
 
Architecting a Serverless Data Lake on AWS
Architecting a Serverless Data Lake on AWSArchitecting a Serverless Data Lake on AWS
Architecting a Serverless Data Lake on AWS
 
AWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon KinesisAWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon Kinesis
 
Real Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS LambdaReal Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS Lambda
 
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
Building Your Own ML Application with AWS Lambda and Amazon SageMaker (SRV404...
 
Introduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-ITIntroduce AWS Lambda for newbie and Non-IT
Introduce AWS Lambda for newbie and Non-IT
 

Similar to ABD203_Real-Time Streaming Applications on AWS

From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time AnalyticsFrom Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
Amazon Web Services
 
ABD335_Real-Time Anomaly Detection Using Amazon Kinesis
ABD335_Real-Time Anomaly Detection Using Amazon KinesisABD335_Real-Time Anomaly Detection Using Amazon Kinesis
ABD335_Real-Time Anomaly Detection Using Amazon Kinesis
Amazon Web Services
 
Building a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWSBuilding a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWS
Injae Kwak
 
Analyzing Streaming Data in Real-time with Amazon Kinesis
Analyzing Streaming Data in Real-time with Amazon KinesisAnalyzing Streaming Data in Real-time with Amazon Kinesis
Analyzing Streaming Data in Real-time with Amazon Kinesis
Amazon Web Services
 
MCL306_Making IoT Smarter with AWS Rekognition
MCL306_Making IoT Smarter with AWS RekognitionMCL306_Making IoT Smarter with AWS Rekognition
MCL306_Making IoT Smarter with AWS Rekognition
Amazon Web Services
 
MCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdfMCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdf
Amazon Web Services
 
Introduction to AWS for Mobile Developers
Introduction to AWS for Mobile DevelopersIntroduction to AWS for Mobile Developers
Introduction to AWS for Mobile Developers
Amazon Web Services
 
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game BalancingGAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
Amazon Web Services
 
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
Amazon Web Services
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
Amazon Web Services
 
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
Amazon Web Services
 
Intro To AWS for Mobile Developers: Collision 2018
Intro To AWS for Mobile Developers: Collision 2018Intro To AWS for Mobile Developers: Collision 2018
Intro To AWS for Mobile Developers: Collision 2018
Amazon Web Services
 
Reactive Architectures with Microservices
Reactive Architectures with MicroservicesReactive Architectures with Microservices
Reactive Architectures with Microservices
AWS Germany
 
ARC201_Scaling Up to Your First 10 Million Users
ARC201_Scaling Up to Your First 10 Million UsersARC201_Scaling Up to Your First 10 Million Users
ARC201_Scaling Up to Your First 10 Million Users
Amazon Web Services
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
Amazon Web Services
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
Amazon Web Services
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
Amazon Web Services
 
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
New Relic
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
Amazon Web Services
 
Journey Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million UsersJourney Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million Users
Adrian Hornsby
 

Similar to ABD203_Real-Time Streaming Applications on AWS (20)

From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time AnalyticsFrom Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
 
ABD335_Real-Time Anomaly Detection Using Amazon Kinesis
ABD335_Real-Time Anomaly Detection Using Amazon KinesisABD335_Real-Time Anomaly Detection Using Amazon Kinesis
ABD335_Real-Time Anomaly Detection Using Amazon Kinesis
 
Building a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWSBuilding a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWS
 
Analyzing Streaming Data in Real-time with Amazon Kinesis
Analyzing Streaming Data in Real-time with Amazon KinesisAnalyzing Streaming Data in Real-time with Amazon Kinesis
Analyzing Streaming Data in Real-time with Amazon Kinesis
 
MCL306_Making IoT Smarter with AWS Rekognition
MCL306_Making IoT Smarter with AWS RekognitionMCL306_Making IoT Smarter with AWS Rekognition
MCL306_Making IoT Smarter with AWS Rekognition
 
MCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdfMCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdf
 
Introduction to AWS for Mobile Developers
Introduction to AWS for Mobile DevelopersIntroduction to AWS for Mobile Developers
Introduction to AWS for Mobile Developers
 
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game BalancingGAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
 
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
Deliver Voice Automated Serverless BI Solutions in Under 3 Hours - ABD325 - r...
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
 
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
NEW LAUNCH! AWS IoT Analytics from Consumer IoT to Industrial IoT - IOT211 - ...
 
Intro To AWS for Mobile Developers: Collision 2018
Intro To AWS for Mobile Developers: Collision 2018Intro To AWS for Mobile Developers: Collision 2018
Intro To AWS for Mobile Developers: Collision 2018
 
Reactive Architectures with Microservices
Reactive Architectures with MicroservicesReactive Architectures with Microservices
Reactive Architectures with Microservices
 
ARC201_Scaling Up to Your First 10 Million Users
ARC201_Scaling Up to Your First 10 Million UsersARC201_Scaling Up to Your First 10 Million Users
ARC201_Scaling Up to Your First 10 Million Users
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
 
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
ARC207 Monitoring Performance of Enterprise Applications on AWS: Understandin...
 
Use Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition SystemUse Amazon Rekognition to Build a Facial Recognition System
Use Amazon Rekognition to Build a Facial Recognition System
 
Journey Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million UsersJourney Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million Users
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
Amazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
Amazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
Amazon Web Services
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Amazon Web Services
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
Amazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
Amazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Amazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
Amazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Amazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
Amazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Amazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
Amazon Web Services
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
Amazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
Amazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

ABD203_Real-Time Streaming Applications on AWS

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Streaming Applications on AWS Use Cases and Patterns R a y Z h u , S r . P r o d u c t M a n a g e r , A W S AWS re:Invent N o v e m b e r 2 7 , 2 0 1 7
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Why real-time analytics? • What is real-time data? • What real-time services does AWS offer? • Who uses them? • How do I use them? • Q & A
  • 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHY REAL-TIME ANALYTICS?
  • 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Real-Time Analytics?
  • 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A Day in the Life Error 500 The server encountered an internal error and was unable to complete your request.
  • 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A Day in the Life 219.134.32.117 - - [16/Feb/1998:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR 3.8.23015.5)“ 95.169.41.62 - - [16/Feb/1998:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200 3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko“ 221.147.191.247 - - [16/Feb/1998:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/38.0.895.0 Safari/538.0.1“ 179.96.123.130 - - [16/Feb/1998:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6“ 16/Feb/1998 16/Feb/1998 16/Feb/1998 16/Feb/1998
  • 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Diminishing Value of Data
  • 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHAT IS REAL-TIME DATA?
  • 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Real-Time Data? Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htdo cs/test
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Analytics Requirements • Durable • Continuous • Fast • Reactive • Available • Reliable Ingest Transform Analyze React Persist
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHAT REAL-TIME SERVICES DOES AWS OFFER?
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis: AWS Real-Time Services Kinesis Firehose Easily load streaming data into AWS Kinesis Analytics Easily process and analyze streaming data with standard SQL Kinesis Streams Build custom applications that process and analyze streaming data
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Streams? Kinesis Streams AWS SDK Mobile SDK Kinesis Agent CloudWatch AWS IoT AWS Lambda Kinesis Client Library Kinesis Firehose Kinesis Analytics Amazon EMR
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Firehose? Kinesis Firehose Amazon Redshift Amazon S3 Amazon Elasticsearch Service
  • 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Is Kinesis Analytics? Kinesis Analytics • Continuous anomaly detection • Continuous time series analysis • Continuous filtering • Continuous aggregation • Continuous enrichment • Continuous anomaly detection • Continuous time series analysis • Continuous filtering • Continuous aggregation • Continuous enrichment
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Third-Party Connectors
  • 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHO USES THEM?
  • 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Customers
  • 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sushiro
  • 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sushiro
  • 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Game Studios
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Game Studios
  • 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Netflix
  • 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Netflix
  • 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. HOW DO I USE THEM?
  • 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time application • Continuous data processing by KCL application • Continuous data archival to Amazon S3 Amazon S3 Kinesis Client Library Kinesis Streams Kinesis Firehose
  • 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time aggregation • Continuous metric aggregation over time periods • Continuous delivery to storage and analytics services SUM AVG MIN MAX Amazon Redshift Amazon S3 Amazon Elasticsearch Service Kinesis FirehoseKinesis Streams Kinesis Analytics
  • 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference Architecture Real-time alarms and notifications • Continuous anomaly detection over a sequence of events • Real-time alarm and notification triggering Anomaly Detection Amazon CloudWatch Amazon SNS Kinesis Streams Kinesis Analytics AWS Lambda
  • 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GETTING STARTED
  • 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started
  • 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started • Kinesis Home Page: https://aws.amazon.com/kinesis/ • Kinesis Blog Post: https://aws.amazon.com/kinesis/blog-posts/ • Getting Started Page: https://aws.amazon.com/kinesis/getting- started/
  • 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Q & A
  • 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!