Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. In this chalk talk, we dive deep into best practices for Kinesis Data Streams and how to optimize for low-latency, multi-consumer solutions. Please join us for a speaker meet-and-greet following this session at the Speaker Lounge (ARIA East, Level 1, Willow Lounge). The meet-and-greet starts 15 minutes after the session and runs for half an hour.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Amazon Kinesis Data
Streams as a Low-latency Message
Bus
Allan MacInnis
Principal Solutions Architect
AWS
A N T 3 6 1

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stream New Data in Seconds
Get actionable insights quickly
Streaming
Ingest data as
it’s generated
Real-time
analytics/ML,
alerts, actions
Process data
on the fly

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Most Common Uses of Streaming
Industrial
Automation
Smart Home
Smart City
Data
Lakes
IoT
Analytics
Log
Analytics

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process, and store
video streams
Kinesis Video
Streams
Load data streams into
data stores
Kinesis Data
Firehose
SQL
Analyze data streams with
SQL
Kinesis Data
Analytics
Capture, process, and store
data streams
Kinesis Data
Streams

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
amzn.to/bigdata

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Architecture

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams Producers and Consumers
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer Library
Get* APIs
Kinesis Client Library
+ Connector Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Standard consumers
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
Application A
GetRecords()
Data
GetRecords():
5 transactions per second, per shard
Data:
2MB per second, per shard
Data
Producer
up to 1 MB or
1000 records
per second,
per shard
With only one
consumer
application,
records can be
retrieved every
200 ms.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Standard consumers
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
Application A
Data
Producer
Consumer
Application B
Consumer
Application C
Consumer
Application D
Consumer
Application E
With more
consumer
applications,
propagation delay
increases.
For example, with
5 consumer
applications, each
can only retrieve
records once per
second, and less
than 400 KBps.
<= 400 KBps

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Enhanced fan-out consumers
Consumers do not poll. Messages are pushed to the consumer as they arrive.
Shard 1
Kinesis Data Stream
Data
Producer
Consumer
Application A
SubscribeToShard()
Uses HTTP/2
• Up to 5 mins connection
• Data pushed to consumer
persist
One or more records pushed in
an array records

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Enhanced fan-out consumers
Each consumer application gets dedicated 2MB per second egress, per shard
Shard 1
Kinesis Data Stream
Data
Producer
Consumer
Application B
Consumer
Application A
RegisterStreamConsumer()
EFO Pipe
RegisterStreamConsumer()
EFO Pipe

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda supports Kinesis Data Streams
Enhanced Fan-Out and HTTP/2 for faster streaming
Enhanced fan-out allows customers to scale the
number of functions reading from a stream in
parallel while maintaining performance.
HTTP/2 data retrieval API improves data delivery
speed between data producers and Lambda
functions by more than 65%
Amazon Kinesis
Data Streams

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Enhanced fan-out Lambda
consumers
Shard 1
Kinesis Data Stream
Data
Producer
EFO Pipe
EFO Pipe
LambdaserviceLambdaservice
Invoke function
Records: <= configured
batch size

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Message bus example: New account creation
Account
managemen
t service PutRecord()
Email

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams: Enhanced fan-out
When to use standard consumers:
• Total number of consuming applications is low (< 3)
• Consumers are not latency-sensitive
• Minimize cost
When to use enhanced fan-out consumers:
• Multiple consumer applications for the same Kinesis Data Stream
• Default limit of 5 registered consuming applications. More can be
supported with a service limit increase request.
• Low-latency requirements for data processing
• Messages are typically delivered to a consumer in less than 70 ms

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
More questions?
Speaker meet & greet
When: Immediately following this session (1:30 p.m. – 2:00
pm)
Where: Aria East, Level 1, Willow Lounge

20. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. Time: 15 minutes after this session
Location: Speaker Lounge (ARIA East, Level 1, Willow Lounge)
Duration: 30 min.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.