SlideShare a Scribd company logo
1 of 22
Download to read offline
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Don’t Wait Until Tomorrow: How to
Use Streaming Data to Gain Real-
time Insights into Your Business
A B D 3 2 1
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A g e n d a 3 : 1 5 P M - 5 : 4 5 P M
I n t r o t o s t r e a m i n g d a t a o n A W S
S e c t i o n 1 : C a p t u r e a n d V i s u a l i z e r e a l - t i m e s e n s o r d a t a ( 1 h o u r )
S e c t i o n 2 : R e a l - t i m e A n a l y s i s + A l e r t i n g w i t h K i n e s i s A n a l y t i c s
( 1 h o u r )
Data is produced continuously
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/h
tdocs/test
The diminishing value of data
Amazon Kinesis makes it easy to work with
real-time streaming data
Amazon Kinesis
Streams
• For technical developers
• Collect and stream data
for ordered, replayable,
real-time processing
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into Amazon S3, Amazon
Redshift, Amazon ES
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries
Amazon Kinesis Streams
• Reliably ingest and durably store streaming data at low
cost
• Build custom real-time applications to process streaming
data
Amazon Kinesis Firehose
• Reliably ingest and deliver batched, compressed, and
encrypted data to S3, Amazon Redshift, and Amazon ES
• Point and click setup with zero administration and seamless
elasticity
Amazon Kinesis Analytics
• Interact with streaming data in real-time using SQL
• Build fully managed and elastic stream processing
applications that process data for real-time
visualizations and alarms
Amazon Kinesis Data Producers
SDKs
• Publish directly from application code via PutRecord and PutRecords APIs
Kinesis Agent
• Tail log files and forward lines as messages to Kinesis Streams
Kinesis Producer Library (KPL)
• Background process aggregates and batches messages
• Producer application calls PutUserRecord method
Third-party and open source
• Log4j appender
• Flume, fluentd source libraries
Amazon Kinesis Data Consumers
Direct API access
• Custom application, using GetShardIterator and GetRecords APIs
• Application responsible for shard processing, check-points, reshard operations
Kinesis Client Library (KCL)
• Open source library, available in several languages
• Manages stream checkpointing
• Manages shard-worker relationships on reshard, or consumer instance scaling
AWS Lambda
• Serverless stream processing
• Lambda function is invoked only when messages exist on stream
• One Lambda function instance per shard
Third-party and open source
• Spark Streaming
• Storm Spout
Tools:
http://bit.ly/2okWPnH
Workshop Questions
1. Utilization: What is the busiest toll station?
2. Promotions: Who are the most active users?
3. Support: Detect failing Toll sensors.
Resources
Workshop Quick start http://amzn.to/2zO4J2I
Recap
Data Visualization example
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Analytics & Alerting
Section 2:
Connect to streaming source
• Streaming data sources include Kinesis
Firehose or Kinesis Streams
• Input formats include JSON, .csv, variable
column, unstructured text
• Each input has a schema; schema is inferred,
but you can edit
• Reference data sources (S3) for data
enrichment
Amazon Kinesis Analytics Core Concepts
Write SQL code
• Build streaming applications with one-to-
many SQL statements
• Robust SQL support and advanced analytic
functions
• Extensions to the SQL standard to work
seamlessly with streaming data
• Support for at-least-once processing
semantics
Amazon Kinesis Analytics Core Concepts
Continuously deliver SQL results
• Send processed data to multiple destinations
• Amazon S3, Amazon Redshift, Amazon ES
(through Firehose)
• Streams (with AWS Lambda integration for
custom destinations)
• End-to-end processing speed as low as sub-
second
• Separation of processing and data delivery
Amazon Kinesis Analytics Core Concepts
Kinesis
Firehose
Ingestion
Stream
Kinesis
Analytics
Kinesis
Anomaly
Stream
Lambda
Function
S3 Bucket
Sample Architecture
Toll Data
(From KDG)
SMS Alerts on
Anomaly
Anomaly Detection
Streaming Analytics Demo
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
A d d i t i o n a l R e s o u r c e s :
h t t p : / / a m z n . t o / 2 A U a P v i W h i t e p a p e r : “ S t r e a m i n g D a t a S o l u t i o n s o n
A W S ”

More Related Content

More from Amazon Web Services

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 
Come costruire un'architettura Serverless nel Cloud AWS
Come costruire un'architettura Serverless nel Cloud AWSCome costruire un'architettura Serverless nel Cloud AWS
Come costruire un'architettura Serverless nel Cloud AWSAmazon Web Services
 
AWS Serverless per startup: come innovare senza preoccuparsi dei server
AWS Serverless per startup: come innovare senza preoccuparsi dei serverAWS Serverless per startup: come innovare senza preoccuparsi dei server
AWS Serverless per startup: come innovare senza preoccuparsi dei serverAmazon Web Services
 
Crea dashboard interattive con Amazon QuickSight
Crea dashboard interattive con Amazon QuickSightCrea dashboard interattive con Amazon QuickSight
Crea dashboard interattive con Amazon QuickSightAmazon Web Services
 
Costruisci modelli di Machine Learning con Amazon SageMaker Autopilot
Costruisci modelli di Machine Learning con Amazon SageMaker AutopilotCostruisci modelli di Machine Learning con Amazon SageMaker Autopilot
Costruisci modelli di Machine Learning con Amazon SageMaker AutopilotAmazon Web Services
 
Migra le tue file shares in cloud con FSx for Windows
Migra le tue file shares in cloud con FSx for Windows Migra le tue file shares in cloud con FSx for Windows
Migra le tue file shares in cloud con FSx for Windows Amazon Web Services
 
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?La tua organizzazione è pronta per adottare una strategia di cloud ibrido?
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?Amazon Web Services
 
Protect your applications from DDoS/BOT & Advanced Attacks
Protect your applications from DDoS/BOT & Advanced AttacksProtect your applications from DDoS/BOT & Advanced Attacks
Protect your applications from DDoS/BOT & Advanced AttacksAmazon Web Services
 
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用Amazon Web Services
 

More from Amazon Web Services (20)

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 
Come costruire un'architettura Serverless nel Cloud AWS
Come costruire un'architettura Serverless nel Cloud AWSCome costruire un'architettura Serverless nel Cloud AWS
Come costruire un'architettura Serverless nel Cloud AWS
 
AWS Serverless per startup: come innovare senza preoccuparsi dei server
AWS Serverless per startup: come innovare senza preoccuparsi dei serverAWS Serverless per startup: come innovare senza preoccuparsi dei server
AWS Serverless per startup: come innovare senza preoccuparsi dei server
 
Crea dashboard interattive con Amazon QuickSight
Crea dashboard interattive con Amazon QuickSightCrea dashboard interattive con Amazon QuickSight
Crea dashboard interattive con Amazon QuickSight
 
Costruisci modelli di Machine Learning con Amazon SageMaker Autopilot
Costruisci modelli di Machine Learning con Amazon SageMaker AutopilotCostruisci modelli di Machine Learning con Amazon SageMaker Autopilot
Costruisci modelli di Machine Learning con Amazon SageMaker Autopilot
 
Migra le tue file shares in cloud con FSx for Windows
Migra le tue file shares in cloud con FSx for Windows Migra le tue file shares in cloud con FSx for Windows
Migra le tue file shares in cloud con FSx for Windows
 
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?La tua organizzazione è pronta per adottare una strategia di cloud ibrido?
La tua organizzazione è pronta per adottare una strategia di cloud ibrido?
 
Protect your applications from DDoS/BOT & Advanced Attacks
Protect your applications from DDoS/BOT & Advanced AttacksProtect your applications from DDoS/BOT & Advanced Attacks
Protect your applications from DDoS/BOT & Advanced Attacks
 
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用
Track 6 Session 6_ 透過 AWS AI 服務模擬、部署機器人於產業之應用
 

ABD321_Don’t Wait Until Tomorrow How to Use Streaming Data to Gain Real-time Insights into Your Business

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Don’t Wait Until Tomorrow: How to Use Streaming Data to Gain Real- time Insights into Your Business A B D 3 2 1
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A g e n d a 3 : 1 5 P M - 5 : 4 5 P M I n t r o t o s t r e a m i n g d a t a o n A W S S e c t i o n 1 : C a p t u r e a n d V i s u a l i z e r e a l - t i m e s e n s o r d a t a ( 1 h o u r ) S e c t i o n 2 : R e a l - t i m e A n a l y s i s + A l e r t i n g w i t h K i n e s i s A n a l y t i c s ( 1 h o u r )
  • 3. Data is produced continuously Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/h tdocs/test
  • 5. Amazon Kinesis makes it easy to work with real-time streaming data Amazon Kinesis Streams • For technical developers • Collect and stream data for ordered, replayable, real-time processing Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into Amazon S3, Amazon Redshift, Amazon ES Amazon Kinesis Analytics • For all developers, data scientists • Easily analyze data streams using standard SQL queries
  • 6. Amazon Kinesis Streams • Reliably ingest and durably store streaming data at low cost • Build custom real-time applications to process streaming data
  • 7. Amazon Kinesis Firehose • Reliably ingest and deliver batched, compressed, and encrypted data to S3, Amazon Redshift, and Amazon ES • Point and click setup with zero administration and seamless elasticity
  • 8. Amazon Kinesis Analytics • Interact with streaming data in real-time using SQL • Build fully managed and elastic stream processing applications that process data for real-time visualizations and alarms
  • 9. Amazon Kinesis Data Producers SDKs • Publish directly from application code via PutRecord and PutRecords APIs Kinesis Agent • Tail log files and forward lines as messages to Kinesis Streams Kinesis Producer Library (KPL) • Background process aggregates and batches messages • Producer application calls PutUserRecord method Third-party and open source • Log4j appender • Flume, fluentd source libraries
  • 10. Amazon Kinesis Data Consumers Direct API access • Custom application, using GetShardIterator and GetRecords APIs • Application responsible for shard processing, check-points, reshard operations Kinesis Client Library (KCL) • Open source library, available in several languages • Manages stream checkpointing • Manages shard-worker relationships on reshard, or consumer instance scaling AWS Lambda • Serverless stream processing • Lambda function is invoked only when messages exist on stream • One Lambda function instance per shard Third-party and open source • Spark Streaming • Storm Spout
  • 12.
  • 13. Workshop Questions 1. Utilization: What is the busiest toll station? 2. Promotions: Who are the most active users? 3. Support: Detect failing Toll sensors.
  • 14. Resources Workshop Quick start http://amzn.to/2zO4J2I
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Analytics & Alerting Section 2:
  • 17. Connect to streaming source • Streaming data sources include Kinesis Firehose or Kinesis Streams • Input formats include JSON, .csv, variable column, unstructured text • Each input has a schema; schema is inferred, but you can edit • Reference data sources (S3) for data enrichment Amazon Kinesis Analytics Core Concepts
  • 18. Write SQL code • Build streaming applications with one-to- many SQL statements • Robust SQL support and advanced analytic functions • Extensions to the SQL standard to work seamlessly with streaming data • Support for at-least-once processing semantics Amazon Kinesis Analytics Core Concepts
  • 19. Continuously deliver SQL results • Send processed data to multiple destinations • Amazon S3, Amazon Redshift, Amazon ES (through Firehose) • Streams (with AWS Lambda integration for custom destinations) • End-to-end processing speed as low as sub- second • Separation of processing and data delivery Amazon Kinesis Analytics Core Concepts
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. THANK YOU! A d d i t i o n a l R e s o u r c e s : h t t p : / / a m z n . t o / 2 A U a P v i W h i t e p a p e r : “ S t r e a m i n g D a t a S o l u t i o n s o n A W S ”