Stream Processing – Concepts and Frameworks

gschmutz
Stream Processing –
Concepts and Frameworks
JEE Conf 2019
Guido Schmutz (guido.schmutz@trivadis.com)
gschmutz http://guidoschmutz.wordpress.com

gschmutz
Agenda
Stream Processing – Concepts and Frameworks
1. Motivation for Stream Processing?
2. Capabilities for Stream Processing
3. Implementing Stream Processing Solutions
4. Demo
5. Summary

gschmutz
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer, Software Architect for Java, AWS, Azure,
Oracle Cloud, SOA and Big Data / Fast Data
Platform Architect & Head of Trivadis Architecture Board
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
155th edition

gschmutzStream Processing – Concepts and Frameworks
Motivation for Stream Processing?

gschmutz
Bulk Source
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
BI Tools
Enterprise Data
Warehouse
SQL
Search / Explore
Search
SQL
Export
Service
Parallel
Processing
Storage
Storage
RawRefined
Results
high latency
Enterprise Apps
Logic
{ }
API
File Import / SQL Import
DB
Extract
File
DB
Big Data solves Volume and Variety – not Velocity

gschmutz
Bulk Source
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
BI Tools
Enterprise Data
Warehouse
SQL
Search / Explore
Search
SQL
Export
Service
Parallel
Processing
Storage
Storage
RawRefined
Results
high latency
Enterprise Apps
Logic
{ }
API
DB
Extract
File
DB
Event Source
Location
Telemetry
IoT
Data
Mobile
Apps
Social
Event Stream

gschmutz
Bulk Source
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
BI Tools
Enterprise Data
Warehouse
SQL
Search / Explore
Search
SQL
Export
Service
• Machine Learning
• Graph Algorithms
• Natural Language Processing
Parallel
Processing
Storage
Storage
RawRefined
Results
high latency
Enterprise Apps
Logic
{ }
API
DB
Extract
File
DB
Event Stream
Event Source
Location
IoT
Data
Mobile
Apps
Social
Event
Hub
Event
Hub
Event
Hub
Telemetry

gschmutz
"Data at Rest" vs. "Data in Motion"
Data at Rest Data in Motion
Store
(Re)Act
Visualize/
Analyze
StoreAct
Analyze
11101
01010
10110
11101
01010
10110
vs.
Visualize

gschmutz
Event
Hub
Event
Hub
Hadoop Clusterd
Hadoop Cluster
Stream Analytics
Platform
Stream Processing Architecture solves Velocity
BI Tools
Enterprise Data
Warehouse
Event
Hub
SQ
L
Search / Explore
Enterprise Apps
Search
ServiceResults
Stream Analytics
Reference /
Models
Dashboard
Logic
{ }
API
Event
Stream
Event
Stream
Event
Stream
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Low(est) latency, no history
Telemetry

gschmutz
Hadoop Clusterd
Hadoop Cluster
Stream Analytics
Platform
Big Data for all historical data analysis
BI Tools
Enterprise Data
Warehouse
SQ
L
Search / Explore
Enterprise Apps
Search
ServiceResults
Stream Analytics
Reference /
Models
Dashboard
Logic
{ }
API
Event
Stream
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
Parallel
Processing
Storage
Storage
RawRefined
Results
Data FlowEvent
Hub
Event
Stream
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Telemetry

gschmutz
Hadoop Clusterd
Hadoop Cluster
Stream Analytics
Platform
Integrate existing systems with lower latency through CDC
BI Tools
Enterprise Data
Warehouse
SQ
L
Search / Explore
Enterprise Apps
Search
ServiceResults
Stream Analytics
Reference /
Models
Dashboard
Logic
{ }
API
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
Parallel
Processing
Storage
Storage
RawRefined
Results
Event
Stream
Event
Stream
Data FlowEvent
Hub
Event
Stream
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Change Data
Capture
Telemetry

gschmutz
New systems participate in event-oriented fashion
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Platform
Microservice State
{ }
API
Stream Analytics Platform
Stream
Processor
State
{ }
API
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
SQL
Export
Search
Service
Enterprise Apps
Logic
{ }
API
Event
Stream
Data FlowEvent
Hub
Event
Stream
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Change Data
Capture
Event
Stream
Event
Stream
Telemetry

gschmutz
Edge computing allows processing close to data sources
Hadoop Clusterd
Hadoop Cluster
Big Data Platform
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Platform
Microservice State
{ }
API
Stream Analytics Platform
Stream
Processor
State
{ }
API
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
SQL
Export
Search
Service
Enterprise Apps
Logic
{ }
API
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Edge Node
Change DataCapture D
ata
Flow
Event
Hub
Data Flow
Event
Stream
Event
Stream
Event Stream
Telemetry
Rules
Event Hub
Storage

gschmutz
Hadoop Clusterd
Hadoop Cluster
Big Data
Unified Architecture for Modern Data Analytics Solutions
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Event
Hub
D
ata
Flow
D
ata
Flow
Change DataCapture Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Stream Analytics
Microservices
Enterprise Apps
Logic
{ }
API
Edge Node
Rules
Event Hub
Storage
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Event Stream
Telemetry

gschmutz
Two Types of Stream Processing
(by Gartner)
Stream Data Integration
• focuses on the ingestion and processing of
data sources targeting real-time extract-
transform-load (ETL) and data integration
use cases
• filter and enrich the data
Stream Analytics
• targets analytics use cases
• calculating aggregates and detecting
patterns to generate higher-level, more
relevant summary information (complex
events)
• Complex events may signify threats or
opportunities that require a response from
the business
Gartner: Market Guide for Event Stream Processing, Nick Heudecker, W. Roy Schulte

gschmutz
Stream Processing & Analytics Ecosystem
Stream Analytics
Event Hub
Open Source Closed Source
Stream Data Integration
Source: adapted from Tibco
Edge

Important Capabilities for Stream
Processing

gschmutz
Capabilities: Stream Data Integration vs. Stream Analytics
Stream Data Integration Stream Analytics
Support for Various Data Sources yes -
Streaming ETL (Transformation/Format Translation, Routing, Validation) yes partial
Execution Mode: Native Streaming yes yes
Execution Mode: Non-Native Streaming - Micro-Batching yes partial
Delivery Guarantees yes yes
API : GUI-Based API / Declarative API / Programmatic yes yes
API: Streaming SQL - yes
Event Time vs. Ingestion / Processing Time - yes
Windowing - yes
Stream-to-Static Joins (Lookup/Enrichment) partial yes
Stream-to-Stream Joins - yes
State Management - yes
Queryable State (aka Interactive Queries) - yes
Event Pattern Detection - Yes

gschmutz
Integrating Data Sources
Sensor Stream
SQL Polling
Change Data Capture
(CDC)
File Polling
File Stream (File Tailing)
File Stream (Appender)

gschmutz
Streaming ETL
• Streaming Extract – Transform – Load
• Flow-based ”programming”
• High-Throughput, straight-through
data flows
• Visual coding with flow editor
• Stream Data Integration but not
Stream Analytics

gschmutz
Execution Mode: Native Streaming
Ingestion
Event
Source
Event
Source
Individual Event
PPPPPPPPPPPP
• Events processed as they arrive
• low(est)-latency
• fault tolerance expensive

gschmutz
Execution Mode: Non-Native Streaming - Micro-Batching
Ingestion
Event
Source
Event
Source
PPPPPP
• Splits incoming stream in small batches
• Fault tolerance easier to achieve
• Higher latency

gschmutz
Delivery Guarantees
At most once (fire-and-forget)
§ message is sent, but the sender doesn’t care if it’s received or lost.
At least once
§ Retransmission of messages can cause messages to be sent one or
more times
Exactly once
§ ensures that a message is received once and only once (never lost
and never repeated)
[ 0 | 1 ]
[ 1+ ]
[ 1 ]

gschmutz
API
GUI-based / Drag-and-Drop
• A graphical way of designing a
pipeline
• Often web-based
Declarative
• An streaming engine configured
declaratively
• JSON, YML
"config": {
"connector.class": "..MqttSourceConnector",
"tasks.max": "1",
"mqtt.server.uri": "tcp://mosquitto-1:1883",
"mqtt.topics": "truck/+/position",
"kafka.topic":"truck_position",
...

gschmutz
Programmatic
• Low-level (class) or high-level fluent
API
• Higher order function as operators
(filter, mapWithState …)
API (II)
Streaming SQL
• use stream in FROM clause
• Extensions support windowing, pattern
matching, spatial, ….
val filteredDf = truckPosDf.
where("eventType !='Normal'")
SELECT *
FROM truck_position_s
WHERE eventType != 'Normal'

gschmutz
Event Time vs. Ingestion / Processing Time
Event time
• time at which events actually occurred
Ingestion time / Processing Time
• time at which events are ingested into /
processed by the system
Not all use cases care about event times, but lot’s do!

gschmutz
Windowing
Computations over events done using windows of data
not feasible to keep entire stream of data in memory
window represents a certain amount of data to perform computations on
Time
Stream of Data Window of Data

gschmutz
Sliding / Hopping Window
eviction & trigger based on
window length and sliding interval
length
Fixed / Tumbling Window
eviction based on window being full
and trigger based on either count of
items or time
Session Window
sequences of temporarily related
events terminated by gap of
inactivity > than some timeout
Windowing
Time TimeTime

gschmutz
Joining – Stream-to-Static
Challenges of joining streams
• Data streams need to be aligned
because of their different
timestamps
• joins must be limited; otherwise
they will never end
• join needs to produce results
continuously
• there is no end to the data
Stream-to-Static (Table) Join
Stream-to-
Static Join
Time

gschmutz
Joining –Stream-to-Stream
Stream-to-Stream Join (one window
join)
Stream-to-Stream Join (two window
join)
Stream-to-
Stream
Join
Stream-to-
Stream
Join
Time
Time

gschmutz
State Management
Needed if use case is dependent
on previously seen data
Windowing, Joining and Pattern
Detection use State Management
behind the scenes
State needs to be as close to the
stream processor as possible
How does it handle failures?
Options for State Management
In-Memory
Replicated,
Distributed
Store
Local,
Embedded
Store
Operational Complexity and Features
Low high

gschmutz
Queryable State (aka. Interactive Queries)
Exposes state managed by
Stream Analytics solution
Allows application to query
managed state, i.e. to
visualize it
can eliminate need for an
external database to keep
results
Stream Processing Infrastructure
Reference
Data
Stream Analytics
{ }
Query API
State
Stream
Processor
Search /
Explore
Online &
Mobile Apps
Model
Dashboard

gschmutz
Event Pattern Detection
• Streaming Data often
contain interesting
patterns
• Special operators allow
finding complex
relationships between
events
• Absence Pattern - event A not followed by
event B within time window
• Sequence Pattern - event A followed by event
B followed by event C
• Increasing Pattern - up trend of a value of a
certain attribute
• Decreasing Pattern - down trend of a value of
a certain attribute
• …

Implementing Stream Processing
Solutions

gschmutz
Event Hub: Apache Kafka
Kafka Cluster
Consumer Consumer Consumer
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer Producer Producer
kafkacat
Data Retention
• Never
• Time (TTL) or Size-based
• Log-Compacted based

gschmutz
Stream Data Integration: Kafka Connect
curl -X "POST" "http://192.168.69.138:8083/connectors"
-H "Content-Type: application/json" -d $'{
"name": "mqtt-source",
"config": {
"connector.class": "io.confluent.connect.mqtt.MqttSourceConnector",
"tasks.max": "1",
"mqtt.server.uri": "tcp://mosquitto:1883",
"mqtt.topics": "truck/+/position",
"kafka.topic":"truck_position" }
}'
• declarative style data flows
• framework is part of Kafka
• Many connectors available
• Single Message Transforms
(SMT)

gschmutz
Stream Data Integration: StreamSets
• GUI-based, drag-and drop Data
Flow Pipelines
• Both stream and batch processing
• special option for Edge computing
• custom sources, sinks, processors
• Monitoring and Error Detection

gschmutz
Stream Analytics: Kafka Streams
• Programmatic API, “just” a Java library
• Native streaming
• fault-tolerant local state
• Fixed, Sliding and Session Windowing
• Stream-Stream / Stream-Table Joins
• At-least-once and exactly-once
KTable<Integer, Customer> customers = builder.stream(”customer");
KStream<Integer, Order> orders = builder.stream(”order");
KStream<Integer, String> joined = orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams

gschmutz
Stream Analytics: KSQL
• Stream Processing with zero coding using
SQL-like language
• part of Confluent Platform (community
edition)
• built on top of Kafka Streams
• interactive (CLI) and headless (command file)
CREATE STREAM customer_s WITH (kafka_topic='customer', value_format='AVRO');
SELECT * FROM customer_s WHERE address->country = 'Switzerland';
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands

gschmutz
Stream Analytics: Spark Structured Streaming
• 2nd gen Spark Streaming, using
DataFrame instead of RDD
• Programmatic API
• Code reuse between batch and
streaming
• Supports Java, Scala, Python, R
and SQL
val oderDf = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "broker-1:9092")
.option("subscribe", ”order")
.load()
val orderFilteredDf = orderDf.where(”address.county = ‘Switzerland'")

Demo

gschmutz
Sample Use Case
detect_dangero
us_driving
truck/nn/
position
mqtt-to-
kafka
truck_
position
Stream
Stream
dangerous_
driving
count_by_
eventType
Table
dangergous_
driving_coun
t
{"timestamp":1537343400827,"truckId":87,
"driverId":13,"routeId":987179512,"eventType":"Normal",
,"latitude":38.65,"longitude":-90.21, "correlationId":"-
3208700263746910537"}
Position &
Driving Info
Source: https://github.com/gschmutz/iot-truck-demo

Summary

gschmutz
Summary
• Stream Processing is the solution for low-latency
• Event Hub, Stream Data Integration and Stream Analytics are the main
building blocks in your architecture
• Kafka is currently the de-facto standard for Event Hub
• Various options exists for Stream Data Integration and Stream Analytics
• SQL becomes a valid option for implementing Stream Analytics

Technology on its own won't help you.
You need to know how to use it properly.

Stream Processing – Concepts and Frameworks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Stream Processing – Concepts and Frameworks

Similar to Stream Processing – Concepts and Frameworks (20)

More from Guido Schmutz

More from Guido Schmutz (20)

Recently uploaded

Recently uploaded (20)

Stream Processing – Concepts and Frameworks