© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Don’t Wait Until Tomorrow: How to
Use Streaming Data to Gain Real-
time Insights into Your Business
A B D 3 2 1

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A g e n d a 3 : 1 5 P M - 5 : 4 5 P M
I n t r o t o s t r e a m i n g d a t a o n A W S
S e c t i o n 1 : C a p t u r e a n d V i s u a l i z e r e a l - t i m e s e n s o r d a t a ( 1 h o u r )
S e c t i o n 2 : R e a l - t i m e A n a l y s i s + A l e r t i n g w i t h K i n e s i s A n a l y t i c s
( 1 h o u r )

Data is produced continuously
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/h
tdocs/test

The diminishing value of data

Amazon Kinesis makes it easy to work with
real-time streaming data
Amazon Kinesis
Streams
• For technical developers
• Collect and stream data
for ordered, replayable,
real-time processing
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into Amazon S3, Amazon
Redshift, Amazon ES
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries

Amazon Kinesis Streams
• Reliably ingest and durably store streaming data at low
cost
• Build custom real-time applications to process streaming
data

Amazon Kinesis Firehose
• Reliably ingest and deliver batched, compressed, and
encrypted data to S3, Amazon Redshift, and Amazon ES
• Point and click setup with zero administration and seamless
elasticity

Amazon Kinesis Analytics
• Interact with streaming data in real-time using SQL
• Build fully managed and elastic stream processing
applications that process data for real-time
visualizations and alarms

Amazon Kinesis Data Producers
SDKs
• Publish directly from application code via PutRecord and PutRecords APIs
Kinesis Agent
• Tail log files and forward lines as messages to Kinesis Streams
Kinesis Producer Library (KPL)
• Background process aggregates and batches messages
• Producer application calls PutUserRecord method
Third-party and open source
• Log4j appender
• Flume, fluentd source libraries

Amazon Kinesis Data Consumers
Direct API access
• Custom application, using GetShardIterator and GetRecords APIs
• Application responsible for shard processing, check-points, reshard operations
Kinesis Client Library (KCL)
• Open source library, available in several languages
• Manages stream checkpointing
• Manages shard-worker relationships on reshard, or consumer instance scaling
AWS Lambda
• Serverless stream processing
• Lambda function is invoked only when messages exist on stream
• One Lambda function instance per shard
Third-party and open source
• Spark Streaming
• Storm Spout

Tools:
http://bit.ly/2okWPnH

Workshop Questions
1. Utilization: What is the busiest toll station?
2. Promotions: Who are the most active users?
3. Support: Detect failing Toll sensors.

Resources
Workshop Quick start http://amzn.to/2zO4J2I

Recap
Data Visualization example

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Analytics & Alerting
Section 2:

Connect to streaming source
• Streaming data sources include Kinesis
Firehose or Kinesis Streams
• Input formats include JSON, .csv, variable
column, unstructured text
• Each input has a schema; schema is inferred,
but you can edit
• Reference data sources (S3) for data
enrichment
Amazon Kinesis Analytics Core Concepts

Write SQL code
• Build streaming applications with one-to-
many SQL statements
• Robust SQL support and advanced analytic
functions
• Extensions to the SQL standard to work
seamlessly with streaming data
• Support for at-least-once processing
semantics
Amazon Kinesis Analytics Core Concepts

Continuously deliver SQL results
• Send processed data to multiple destinations
• Amazon S3, Amazon Redshift, Amazon ES
(through Firehose)
• Streams (with AWS Lambda integration for
custom destinations)
• End-to-end processing speed as low as sub-
second
• Separation of processing and data delivery
Amazon Kinesis Analytics Core Concepts

Kinesis
Firehose
Ingestion
Stream
Kinesis
Analytics
Kinesis
Anomaly
Stream
Lambda
Function
S3 Bucket
Sample Architecture
Toll Data
(From KDG)
SMS Alerts on
Anomaly
Anomaly Detection

Streaming Analytics Demo

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
A d d i t i o n a l R e s o u r c e s :
h t t p : / / a m z n . t o / 2 A U a P v i W h i t e p a p e r : “ S t r e a m i n g D a t a S o l u t i o n s o n
A W S ”