Processing streaming data is becoming increasingly important to many organisations who need to analyse incoming data both in near real-time and in batch. In this session we will look at the best practices and patterns for analysing streaming data with AWS Kinesis Streams, Kinesis Firehose and Kinesis Analytics. Speaker: Johnathon Meichtry, Principal Solutions Architect, Amazon Web Services

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Johnathon Meichtry
Principal Solutions Architect, Amazon Web Services
Level 300
Analysing All Your Streaming Data

2. Most Data is Produced Continuously
Mobile apps Web clickstream Application logs
Metering records IoT sensors Smart buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/h
tdocs/test

3. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Modern Data Architecture

4. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Real-time Pipeline
Amazon
Kinesis
Machines
Devices
Mobile
Clickstream

5. Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon Kinesis
Analytics
Kinesis Family

6. Amazon Kinesis Streams
• Ingest & store streaming data at low cost, build custom real-time
applications to process streaming data

7. Amazon Kinesis Firehose
• Reliably ingest and deliver batched, compressed, and encrypted data to
S3, Amazon Redshift, and Amazon Elasticsearch Service (Amazon ES)
• Point-and-click setup with zero administration and seamless elasticity

8. Amazon Kinesis Analytics
• Interact with streaming data in real time using SQL
• Build fully managed and elastic stream processing applications that
process data for real-time visualisations and alarms

9. Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon Kinesis
Analytics
Kinesis Family

10. Availability
Zone
Availability
Zone
Availability
Zone
Amazon Kinesis
Stream
AWS Lambda
KCL App
Amazon EMR
Streaming
Logs
Alerts
Analysis
Dashboards
Predictions

11. SHARD1000 TPS or 1MB 5 TPS or 2MB
SHARD
2000 TPS or 2MB 10 TPS or 4MB
SHARD
3000 TPS or 3MB 15 TPS or 6MB
Retention: 24 hours to 7 Days
Amazon Kinesis Stream

12. Creating a Kinesis Stream

13. SHARD
SHARD
SHARD
EVENT
PRODUCERS
Kinesis
Endpoint
Specify Partition Key
Amazon Kinesis Stream

14. Kinesis Producers use a PUT call to store data in
a Stream. Each record <= 1 MB
PutRecord {Data,StreamName,PartitionKey}
PutRecords {Records{Data,PartitionKey}, StreamName}
A Partition Key is supplied by producer and used
to distribute the PUTs across Shards
A unique Sequence # is returned to the Producer
upon a successful PUT call
Aggregates
Kinesis Producer Library

15. Kinesis Agent
• Monitors files and sends new data records to your delivery stream
• Monitor multiple directories
• Write to multiple streams
• Handles file rotation, checkpointing, and retry upon failures
• Delivers all data in a reliable, timely, and simple manner
• Pre-process data
• Emits AWS CloudWatch metrics

16. Availability
Zone
Availability
Zone
Availability
Zone
Amazon Kinesis
Stream
AWS Lambda
KCL App
Amazon EMR
Streaming
Logs
Alerts
Analysis
Dashboards
Predictions

17. Kinesis Data Out – Kinesis Client Library
SHARD 1
SHARD 2
SHARD 3
SHARD N
EC2 Instance
Worker 1
Worker 2
EC2 Instance
Worker 3
Worker N
KCL: Java, Node.js, Python, .NET, Ruby

18. twitter-trends.com
twitter-trends.com Website

19. Too Big to Handle on One Box

20. twitter-trends.com
The Solution: Local Top 10
My top-10
My top-10
My top-10
Global top-10

21. K
I
N
E
S
I
S
twitter-trends.com
Challenges Using the Kinesis API Directly
Kinesis
application
Manual creation of workers and
assignment to shards
How many workers
per EC2 instance?
How many EC2 instances?

22. K
I
N
E
S
I
S
twitter-trends.com
Using the Kinesis Client Library
Kinesis
application
Shard mgmt
table

23. K
I
N
E
S
I
S
twitter-trends.com
Elasticity and Load Balancing
Shard mgmt
table
Auto scaling
Group

24. K
I
N
E
S
I
S
twitter-trends.com
Fault Tolerance Support in KCL
Shard mgmt
table
X
Availability Zone
1
Availability Zone
3

25. Checkpoint, Replay Design Pattern
KINESIS
1417182123
Shard-i
235810
Shard
ID
Lock Seq
num
Shard-i
Host A
Host B
Shard
ID
Local top-10
Shard-i
0
10
18
X2
3
5
8
10
14
17
18
21
23
0
310
Host AHost B
{#Movies: 10235, #Weather: 9835, …}{#Movies: 10235, #Weather: 9910, …}
1023
14
17
18
21
23

26. Availability
Zone
Availability
Zone
Availability
Zone
Amazon Kinesis
Stream
AWS Lambda
KCL App
Amazon EMR
Streaming
Logs
Alerts
Analysis
Dashboards
Predictions

27. Kinesis & Lambda
SHARD 1
SHARD 2
SHARD 3
SHARD N
AWS Lambda: Node.js, Java, Python, C#
AWS Lambda

28. Lambda
Blueprints

29. Lambda
Configure
Trigger

30. Lambda
Add Code

31. Lambda
Test

32. Lambda
Monitor

33. Availability
Zone
Availability
Zone
Availability
Zone
Amazon Kinesis
Stream
AWS Lambda
KCL App
Amazon EMR
Streaming
Logs
Alerts
Analysis
Dashboards
Predictions

34. Spark Core
Spark
SQL
Spark
Streaming
Spark
R
Spark
ML
Graph X

35. Spark Core
Spark
SQL
Spark
Streaming
Spark
R
Spark
ML
Graph X

36. Stream
Micro
Batches
Results
Amazon
Kinesis
Apache
Kafka

37. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Amazon
Kinesis
AWS Lambda
Application
Amazon EMR
Streaming
S3
(Logs)
Amazon
ElasticSearch
(Dashboards)
Amazon EMR
(Predictions)
ML
Amazon SNS
(Alerts)
Real-time Pipeline
Amazon
Redshift
(Analytics)

38. Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon Kinesis
Analytics
Kinesis Family

39. S3
Redshift
Elasticsearch
Auto provisioning
Auto partition keys
End to End Elastic
Batch
Compress
Encrypt
Amazon Kinesis Firehose

40. Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon Kinesis
Analytics
Kinesis Family

41. Kinesis Analytics
Stream or Firehose
Kinesis
Analytics
Data OutData In
SQL
Stream or
Firehose

43. • Digital Media
• Publication brands
• Mobile Apps
• Re-targeting platforms
Hearst’s World of Streaming Data

44. Answers
& Insights
Data
Ingest/Collect Consume/VisualizeStore Process/Analyze
BR TCO INV
Amazon
EMR
Amazon S3
Amazon
Kinesis
Amazon
Elastic
Beanstalk
Amazon
Elasticsearch
Buzzing
@Hearst
Amazon
Redshift

45. Thank you!