The document discusses best practices for high performance data streaming using Amazon Kinesis. It covers introducing Amazon Kinesis and its capabilities, different consumer types (standard vs enhanced fan-out), considerations for scaling a Kinesis data stream, and Comcast's streaming data platform called Headwaters. The key points are how to address producer limits, consumer limits, shard management, consumption speed, and maximum acceptable latency.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High Performance Data Streaming
with Amazon Kinesis: Best Practices
Allan MacInnis
Principal Solutions Architect
AWS
Gabriel Commeau
Data Platforms Architect
Comcast
A N T 3 2 2 - R 1

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Streaming data overview
• Interactive demo
• Introduction to Amazon Kinesis
• Standard consumers vs. enhanced fan-out consumers
• Headwaters: Comcast streaming data platform
• Five considerations to scale an Amazon Kinesis Data Streams
with standard consumers
• Impacts of enhanced fan-out

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Timely decisions require new data in minutes
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
Real time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stream new data in seconds
Get actionable insights quickly
Streaming
Ingest data as
it’s generated
Real-time
analytics/ML,
alerts, actions
Process data
on the fly

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Most common uses of streaming
Industrial
Automation
Smart Home
Smart City
Data
Lakes
IoT
Analytics
Log
Analytics

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real-time
Capture, process, and store
video streams
Amazon Kinesis
Video Streams
Load data streams into
data stores
Amazon Kinesis
Data Firehose
SQL
Analyze data streams with
SQL
Amazon Kinesis
Data Analytics
Capture, process, and store
data streams
Amazon Kinesis
Data Streams

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
amzn.to/bigdata

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo architecture

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams producers and
consumers
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK for
iOS
Amazon Kinesis
Producer Library
Get* APIs
Amazon Kinesis Client
Library + Connector
Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Standard
consumers
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
application A
GetRecords()
Data
GetRecords():
Five transactions per second, per shard
Data:
2MB per second, per shard
Data
producer
up to 1 MB or
1000 records
per second,
per shard
With only one
consumer
application,
records can be
retrieved every
200 ms

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Standard consumers
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
application
A
Data
producer
Consumer
application
B
Consumer
application
C
Consumer
application
D
Consumer
application
E
With more
consumer
applications,
propagation delay
increases
For example, with
five consumer
applications, each
can only retrieve
records once per
second, and less
than 400 KBps
<= 400 KBps

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Enhanced fan-out
consumers
Consumers do not poll. Messages are pushed to the consumer as they arrive
Shard 1
Kinesis Data Stream
Data
producer
Consumer
application A
SubscribeToShard()
Uses HTTP/2
• Up to five mins connection
• Data pushed to consumer
persist

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Enhanced fan-out
consumers
Each consumer application gets dedicated 2MB per second egress, per shard
Shard 1
Kinesis Data Stream
Data
producer
Consumer
application B
Consumer
application A
RegisterStreamConsumer()
EFO Pipe
RegisterStreamConsumer()
EFO Pipe

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Enhanced fan-out
When to use standard consumers:
• Total number of consuming applications is low (< 3)
• Consumers are not latency-sensitive
• Minimize cost
When to use enhanced fan-out consumers:
• Multiple consumer applications for the same Kinesis Data Stream
• Default limit of five registered consuming applications. More can be
supported with a service limit increase request
• Low-latency requirements for data processing
• Messages are typically delivered to a consumer in less than 70 ms

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. Over time we came to call […] the
concept of managing this [stream
data] centrally a “streaming platform.”
Jay Kreps

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Headwaters: Comcast streaming data platform
• Decouple data producer from its consumers
• Scale data systems independently
• Act as buffer
• Formalize data exchanges
• Assist with data stream management
• Scale data stream
• Data retention period management
• Consumer on-boarding governance
• Foster real-time data exchange
• Stream metadata
• Data schema management
• Cost model

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Headwaters architecture
Manages
Creates
Headwaters control plane
Headwaters Data StreamHeadwaters Data StreamHeadwaters data stream
CI/CD pipeline

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Headwaters producer on-boarding
Headwaters
control plane
Headwaters data stream
(4) Welcome Email
(1) Registers
(2) Creates
Writes
Monitors
Communicates using
(3) Grants
Alerts
Data
Producer

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#1: Amazon Kinesis Data Stream producer limits
Kinesis
Data Stream
1 MB/sec
1000 PUT/sec
Kinesis Producer Limits:
Shards
per shard

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#1: Amazon Kinesis Data Streams producer limits
IP video analytics use case:
HTTP
Gateway
Headwaters
Streaming
Data
Platform

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#1: Amazon Kinesis Data Streams producer limits
Bandwidth surge Average bandwidth
Maximum bandwidth

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#1: Amazon Kinesis Data Streams producer limits
Bandwidth limitation:
Average bandwidth
Maximum bandwidth
Cost
Data latency
Back-pressure scale

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#1: Amazon Kinesis Data Streams producer limits
n: Number of shards in Kinesis Data Streams
n ∈ ℕ*
• 1000 PUT/sec/shard
n ≥
MaxBandwidth
1 MB/sec
n ≥
MaxThroughput
1000 messages/sec

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#2: Amazon Kinesis Data Streams consumer limits
Kinesis Limitations:
n ≥
MaxBandwidth × NbConsumers
2 MB/sec
2 MB / sec / shard
5 GET / sec / shard
Consumers
Copy
Slave
Consumers
Master

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#3: Split/merge shards
0
1
2
3
4
5
6
1 2 3 4 5
Number of Shards
Data Stream (1 day retention)

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#3: Update shard count
0
0 1
0
4
1
5
0
4
1
5 6
0
24
1
5 6
0
24
1
5 6
9 10

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#4: Consumption speed
Kinesis stream
(1000 msg/sec)
Consumer 1
(10000 msg/sec)
Consumer 2
(500 msg/sec)
Producer
(1500 msg/sec)
3000 msg/sec
3000 msg/sec
1500 msg/sec

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#4: Consumption speed
n : Number of shards in Kinesis Data Streams
N : Number of consumers
AverageThroughput : Average throughput of the producer
Ci(MaxThroughput) : Maximum throughput of consumer Ci
n ≥ maxi=1
N AverageThroughput
Ci(MaxThroughput)

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Producer:
• Constant 5MB/s
• Surge @ 20MB/s
 20-shard data stream
Four Consumers
Each consumer has:
20 × 2MB/s
4
= 10MB/s
Time
Bandwidth (MB/s)
#5: Max acceptable latency

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On the producer side:
SurgeDur × SurgeBw + CiMaxLat
× MaxInBw
On the consumer side:
CiMaxBw × (SurgeDur + CiMaxLat)
Also:
CiMaxBw =
n × 2MB/sec
NbConsumers
Time
Bandwidth (MB/s)
#5: Max acceptable latency

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#5: Max acceptable latency
Therefore:
Yields:
Example: ൜
Surge: 10s
CiMaxLat: 10s
 n ≥
4× 10×20+10×5
2× 10+10
= 25 shards
n × 2MB/sec
NbConsumers
× SurgeDur + CiMaxLat ≥
SurgeDur × SurgeBw
+ CiMaxLat × MaxInBw
n ≥
NbConsumers × (SurgeDur × SurgeBw + CiMaxLat × MaxInBw)
2MB/sec × (SurgeDur + CiMaxLat)

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introducing enhanced fan-out consumers
Fan-out consumers:
- Dedicated 2MB/sec/shard
Isolate a consumer from the other consumers
Unchanged considerations:
#1: Kinesis Data Streams producer limits
#3: Split/merge operations
#4: Consumption speed

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
#2: Amazon Kinesis Data Streams consumer limits
Kinesis limitations:
n ≥
MaxInputBandwidth × NbConsumers
2 MB/sec
n ≥
MaxInputBandwidth × NbRegularConsumers
2 MB/sec

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fan-out consumers: No lag
For regular consumers:
n ≥
NbConsumers × (SurgeDur × SurgeBw + CiMaxLat × MaxBw)
2MB/sec × (SurgeDur + CiMaxLat)
CiMaxLat = Maximum latency for regular consumers
n ≥
NbRegularConsumers × (SurgeDur × SurgeBw + CiMaxLat × MaxBw)
2MB/sec × (SurgeDur + CiMaxLat)
#5: Max acceptable latency

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams Shard Calculator
https://t2m.io/vcAFLW8U

42. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Allan MacInnis
ajmac@amazon.com
Gabriel Commeau
Gabriel_Commeau@comcast.com

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.