Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Summit Tel Aviv 2019

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Need for Speed – Intro To Real-Time Data
Streaming Analytics on AWS
Roy Ben-Alta
Head of worldwide practice Data Analytics
Amazon Web Services
Benny Tsui
Senior Software Engineer
Singular
D A T 2 0 2

Agenda
Real-Time Analytics Overview
Data Streaming Solutions on AWS
Choosing Solution for Streaming Data Storage
Choosing Solution for Streaming Data Processing
Singular Case Study

#Real-time

All data originates in real-time!

{
"payerId": "Joe",
"productCode": "AmazonS3",
"clientProductCode": "AmazonS3",
"usageType": "Bandwidth",
"operation": "PUT",
"value": "22490",
"timestamp": "1216674828"
}
Metering Record
127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET
/apache_pb.gif HTTP/1.0" 200 2326
Common Log Entry
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com
evntslog - ID47 [exampleSDID@32473 iut="3"
eventSource="Application" eventID="1011"][examplePriority@32473
class="high"]
Syslog Entry
“SeattlePublicWater/Kinesis/123/Realti
me” – 412309129140
MQTT Record
<R,AMZN,T,G,R1>
NASDAQ OMX Record
Smart Buildings
Beacons
Smart Textiles
Health Monitors

But, analytics to gain insights is usually done
much, much later.

Batch analytics operations take too long
BusinessValue
Time To Action
Data
originated
Analytics
performed
Insights
gleaned
Action
taken
Outdated
insights
Can be harmful
actions
PositiveNegative
Decision
made
Poor decision

#Streaming

Most Common Uses of Streaming
Security
Monitoring
Industrial
Automation
Data
Lakes
IoT Device
Monitoring
Log
Analytics

Data Streaming Use Cases
• Streaming ETL into Data Lakes for
near real-time analytical insights
• Messaging for decoupled micro-
services
• Log ingestion and monitoring
• Continuous metric generation
• Responsive analytics
• Change Data Capture (CDC)
• Anomaly Detection
Real-time (milliseconds) Near Real-time (seconds-minutes)

An Overview of Data Streaming Technology
• Stream Ingestion
• Deliver large volumes of high velocity data from a variety of sources into a stream
• Stream Storage
• Store large volumes of high velocity data and make it highly available for processing
• Stream Processing
• Real-time streaming analytics and machine learning
• Analyze data streams in real-time, use ML (e.g. Anomaly detection use cases)
• Streaming ETL
• Transform and deliver data into data lakes and warehouses for near real-time analysis (or durable
storage)

Source Stream Ingestion Stream Storage Stream Processing Destination
Devices and or
applications that produce
real-time data at high
velocity.
The process by which data
is ingested into the
stream.
The way in which the data
is stored within the
stream.
The way in which the
stream is processed to
provide analytical insight
Databases where data is
stored for near real-time
or longer term analysis.
An Overview of Data Streaming Technology
Destination
Real-Time Applications
(seconds)
Analyze streaming data
to generate real-time
insights and
notifications
Streaming ETL
(minutes)
Compress, encrypt and
transform data in near
real-time before it is
delivered to its
destination
Stream Storage
Stream Ingestion
[Wed Oct 11 14:32:52 2018]
[error] [client 127.0.0.1]
client denied by server
configuration:
/export/home/live/ap/htdocs
/test
Mobile device
Metering
Click streams
IoT sensors
Logs

Challenges of Data Streaming
Difficult to setup
Hard to achieve high availability
Tricky to scale
Integration requires development
Error prone and complex to manage Expensive to maintain

Overview
Amazon Kinesis

Amazon Kinesis is a Foundational Service Used
Across Amazon
AWS
metering

Amazon Kinesis is a Foundational Service Used
Across Amazon
Amazon Go
video analytics
Amazon.com
online catalog
Amazon
CloudWatch
logs
Amazon
S3 events
AWS
metering

Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams
Load data streams
into AWS data stores
Analyze data
streams in real-time
Capture, and store
data streams

Source Stream Ingestion Stream Storage Stream Processing Destination
Devices and or
applications that produce
real-time data at high
velocity.
The process by which data
is ingested into the
stream.
The way in which the data
is stored within the
stream.
The way in which the
stream is processed to
provide analytical insight
Databases where data is
stored for near real-time
or longer term analysis.
• Reliable
• Low latency
• Flexible
• Durable
• Elastic
• Secure
• Real-time
• Fully managed
• Scalable
An Overview of Data Streaming with Amazon Kinesis
Amazon S3
Amazon Redshift
Amazon Elasticsearch
Splunk
Real-Time Applications (seconds)
Streaming ETL (minutes)
Stream Ingestion
[Wed Oct 11 14:32:52 2018]
[error] [client 127.0.0.1]
client denied by server
configuration:
/export/home/live/ap/htdocs
/test
Mobile device
Metering
Click streams
IoT sensors
Logs
AWS SDKsAmazon
KinesisAgent
AmazonKinesis
ProducerLibrary
AmazonKinesis
ConsumerLibrary

Amazon Kinesis Data Streams
• Easy administration and low cost
• Real-time, elastic performance
• Secure, durable storage
• Available to multiple real-time analytics applications

Amazon Kinesis Data Analytics
• Interact with streaming data in real-time using SQL or integrated Java applications
• Build fully managed and elastic stream processing applications

Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless continuous data transformations
• Near real-time

Amazon Managed Streaming for Kafka (Preview)
• At re:Invent 2018 the public preview of Amazon MSK was announced.
• Apache Kafka is a popular open source streaming solution.
Many AWS customers deploy Kafka clusters on EC2, but this
comes with several challenges.

Challenges operating Apache Kafka
Difficult to setup
Hard to achieve high availability
Tricky to scale
AWS integrations = development
No console, no visible metrics ! "#!"#$%#&' = )
*+,
-
./0

Amazon Managed Streaming for Kafka (Preview)
• Fully compatible with Apache Kafka v1.1.1 and v2.1.0
• AWS Management Console and AWS API for
provisioning
• Clusters are setup automatically
• Provision Apache Kafka brokers and storage
• Create and tear down clusters on-demand

Comparing Amazon Kinesis Data Streams to MSK
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Shard 2
Shard 1
Shard 3
Writes
from
Producers
Stream with 3 shards
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Partition 2
Partition 1
Partition 3
Writes
from
Producers
Topic with 3 partitions

Comparing Amazon Kinesis Data Streams to MSK
• AWS API experience
• Throughput provisioning
model
• Seamless scaling
• Typically lower costs
• Deep AWS integrations
• Open-source compatibility
• Strong third-party tooling
• Cluster provisioning model
• Apache Kafka scaling isn’t
seamless to clients
• Raw performance

Comparing Kinesis Data Streams to Amazon MSK
Attribute Apache Kafka Kinesis Data Streams Managed Streaming for Kafka
(MSK)
Cost $$$ $ (pay for what you use) $$ (pay for infrastructure)
Ease of use Advanced setup required Get started in minutes Get started in minutes
Management
Overhead
High Low Low
Scalability Difficult to scale Scale in seconds with one click Scale in minutes with one click
Throughput Infinite Scales with shards, supports up to
1mb payloads
Infinite
Durability Configurable 3x by default Configurable
Infrastructure You manage AWS manages AWS manages
Write-to-Read
Latency
<100 ms is achievable <100 ms (with HTTP/2) <100 ms is achievable
Open Sourced? Yes No Yes

Data Streaming Processing
• AWS Lambda
• Apache Spark
• Apache Flink
• Others (Native KCL App, Kafka Streaming, Samza, Storm
and more)

Processing a data stream with AWS Lambda
data
producer
Kinesis Data
Streams
Amazon
SNS
Continuously stream data
Lambda
service
Lambda
functionA
Lambda
function B
Continuously polls for new data,
1 poll per second
Automatically invokes your
function(s) when data found
• Stateless
• Lambda polls each shard once per second
• Scales with your data

Filter, enrich and convert data while it is streaming
data
producer Kinesis Data
Firehose
Elasticsearch
Service
[Wed Oct 11 14:32:52 2017] [error] [client 127.0.0.1]
[Wed Oct 11 14:32:53 2017] [info] [client 127.0.0.1]
geo-IP
service
{
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
{
"recordId": "1",
"result": "Ok",
"data": {
"date": "2017/10/11 14:32:52",
"status": "error",
"source": "127.0.0.1",
"city": "Boston",
"state": "MA"
}
},
{
"recordId": "2",
"result": "Dropped"
}

Processing a data stream with Apache Spark

Processing a data stream with Apache Spark
https://spark.apache.org/docs/2.3.1/streaming-kinesis-integration.htm
l

Processing a data stream with Apache Flink
Simple
programming
model
High
performance
Stateful
Processing
Strong data
integrity
Easy to use and
flexible APIs make
building apps fast
In-memory
computing provides
low latency, high
throughput
Efficient, incremental,
durable and
consistent application
state saves
Exactly-once
processing and
consistent state

Apache Flink supports over 25 operators
Example Operators Typically usage
Map, FlatMap, Filter, Iterative Basic transformations
Key By, Split, Shuffle, Custom
Partition
Change logical or physical
structure of the stream
Window, Reduce, Fold, Sum,
Min, Max
Analytics and aggregations
Join, Union, coGroup,
Combine multiple data streams
into
… and much, much more.

Processing Data With Kinesis Data Analytics Java Applications
Streaming operators are applied to data streams in a pipeline
Source
Sink
DataStream
KeyedDataStream
DataStream
Sink
keyBy,
window
filter
apply

Summary
Attribute AWS Lambda Kinesis Data Analytics Spark Streaming with EMR
Stateless Yes Yes Yes
Stateful No Yes Partial
Windowing
Function
No Yes Yes
ML capabilities No Yes Yes
Streaming ETL Yes Yes Yes
Serverless Yes Yes No
Exactly Once
Processing
No Yes No
Native integration
with AWS services
Yes Yes Yes
Programing
Language
Node, Python, Java and
more
Java/SQL Scala/Java/Python

Benny Tsui, Sr. Software Engineer
Amazon Kinesis at Singular
Using Golang with Enhanced fan-out & KCL 2.0

Marketing Intelligence Platform
Singular

Unified Marketing Data
Automatically connect and combine campaign data from thousands of sources
with attribution data for a single source of truth.
What does Singular do?
Automation
Streamline workflows and focus on strategy by automating tedious and
manual tasks.
Audiences
Tap into attribution data to create high value audience segments and
automatically distribute across ad networks.

Mobile App Events
Purchases, installs, sessions, custom events etc.
Audiences at a Glance
Segmentation Criteria
For example, “Users who spend more than $10 a month”
Audience Distribution
Resulting devices used for retargeting, etc.

Handle “Lots of Data”
Currently at 13TB daily and around 30,000 events per second. And growing...
Requirements
Near Real-time
Eg. “Users who have not signed up yet” is useless if it’s not an up-to-date list.
Not Just Volume...
How to get the data there? How do we deal with failures?
Easy to Scale and Add Capacity
At mercy of customers’ data...

Option: Build our own
NSQ was existing stack but required maintenance. Durability concerns.
Decisions...
Option: Apache Kafka
Solid product but also requires additional maintenance. Initial setup cost.
Option: Kinesis
Already in AWS ecosystem. Easily scalable out of the box.

Throughput
Predictable and tunable knobs for throughput.
Why Amazon Kinesis?
Real-time Streaming
With added benefit of 7 days of lookback -- just in case.
Easy to Scale
Tune capacity through shards. Monitoring built-in.

No Official KPL Package
Used API and followed KPL guidelines, especially regarding batch
Learnings: Golang
No Official KCL Package
Forked Python KCL package to run Go binary.
MultiLangDaemon
Keep in mind: one golang process per Kinesis shard.

Multiple Applications
Without EFO and KCL 2.0, applications share throughput of shards.
Learnings: Enhanced fan-out & KCL 2.0
Enhanced fan-out & KCL 2.0
Applications get their own dedicated shards. EFO automatically enabled when
using a KCL 2.0 client.
No Golang
Official Java and Python support. Required our forked MLD.

Thank you!
Reach out to Benny Tsui at benny@singular.net for more information

Learn more about Amazon Kinesis:
aws.amazon.com/kinesis
Get started with Amazon Kinesis:
aws.amazon.com/kinesis/getting-started
Learn more about Amazon MSK:
aws.amazon.com/msk
Get started with Amazon MSK:
aws.amazon.com/msk/getting-started
Reach out to your account team to learn more
Learn More

Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Roy Ben-Alta
Email: benaltar@amazon.com
Twitter: @benalt
Benny Tsui
Email: benny@singular.net
http://bit.ly/2SEC4S4

Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Summit Tel Aviv 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Summit Tel Aviv 2019

Similar to Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Summit Tel Aviv 2019 (20)

More from AWS Summits

More from AWS Summits (20)

Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Summit Tel Aviv 2019