There is something very interesting about stream processing. It builds upon messaging, rather than using a file system, as a more typical database does. But stream processing engines themselves are really a type of database. A database designed specifically to blend streams and tables so you can query continuous results. As such they span an architectural no-mans-land that sits between Database and Distributed Systems fields.
This talk will look at Stateful Stream Processing. Can a streaming engine provide the guarantees of a database? When is a streaming engine best? How do they work, under the covers?
4. Today we have 2 goals
1. Understand Stateful Stream
Processing (now & near future)
2. Case for SSP as a general framework
for building data-centric systems.
5. Data systems come in
different forms
• Database (OLTP)
• Analytics Database (OLAP/Hadoop)
• Messaging
• Distributed log
• Stream Processing
• Stateful Stream Processing
6. Database (OLTP)
Focuses on providing a consistent view that
supports updates and queries on individual tuples.
14. Windows bound a computation
How many items will we bring into the machine at
one time?
15. Buffering allows us to handle
late events
How many items will we bring into the machine at
one time?
16. Some query
Over some time window
Emitting at some frequency
Continually executing query
Stream(s)
Stream Processing Engine
Derived Stream
17. Avg(p.time – o.time)
From orders, payment
Group by payment.region
over 1 day window
emitting every second
Stream Processing
orders!
payments!
Completion time,
by region!
18. Avg(o.time – p.time)
From orders, payment
Group by payment.region
over 1 day window
emitting every second
Materialised View (DB )
Query
orders!
payments!
Completion time,
by region!
19. Avg(o.time – p.time)
From orders, payment, user
Group by user.region
over 1 day window
emitting every second
Stateful Stream Processing
Streams
Stream Processing Engine
Derived Stream
Query
Derived “Table”
Table
“View” is output as
table or stream
20. Table == Stream + Window0
n
== 0 N
Table is a stream with an infinite window (i.e. buffer from 0 -> now)
window !
21. SSP is about creating
materialised views.
Materialised as a table, or
materialised as a stream
22. Features: similar to database query
engine
Join Filter
Aggr-
egate
View
Windowed
Streams
23. Can distribute over many machines
in two dimensions
Join Filter
Aggr-
egate
View
Join Filter
Aggr-
egate
View
Join Filter
Aggr-
egate
View
Scale Out Scale Forward
24. Stateful Stream Processing engines typically
use Kafka (a distributed commit log)
Join Filter
Aggr-
egate
View
Kafka (a distributed log)
25. A log is very simple idea
Messages are added at the end of the log
Just think of the log as a file
Old New
26. Readers have a position & scan
Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan
28. Compacted Log
(Tabular View)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1
Version 2
Version 3
Version 5
STEAM
(All versions)
COMPACTED STREAM
(Latest Key only)
29. The log is a
Distributed System
For scalability and fault tolerance
33. The Log reassigns data on failure
Producers
Kafka
Many consumers
share partitions in
one topic
34. Kafka supplies two levels of
leader election
Replicas in Kafka have
an elected leader
Consumers in Kafka
have an elected leader
35. The log is important for SSP
Maintains History: Acts like a “push based” distributed file system
36. The log is important: Two Primitives
Stream
Compacted Stream (‘table’)
37. The Log is, to a streaming
engine, what HDFS is to Hadoop
38. But it’s a bit more than a HDFS
replacement: Processors inherit the
idea of “membership” from the log
39. So stateful Stream Processors use
the Log
Join Filter
Aggr-
egate
View
Kafka (Distributed Log)
40. They also use local storage
Join Filter
Aggr-
egate
View
(1) a Kafka
(2) Local KV Store
41. Local KV store has a few uses
(1) It caches streams on disk
(2) It caches “tables” on disk
Join Filter
Aggr-
egate
View
This makes join operations fast as they’re entirely local
Streams just cache recent
messages to help with joins
Tables are fully
“realised” locally
43. e.g. Useful for Enrichment
stream
Compacted
stream
Join
Orders
Customers
KafkaKafka Streams
Local DB
44. Aggregates need intermediary state
stream
Compacted
stream
Join
Orders
Customers
KafkaSum(orders)
group by region
Persist current value,
in case we fail
45. State store inherits durability from
the log
State store flushes
back to the log
Join Filter
Aggr-
egate
View
46. Separate Data, Processing & View
View
OrdersPayments View
View
Storage Layer
(a Kafka)
Processing & View
Query
47. You can query the views from
anywhere
View
OrdersPayments View
View
Storage Layer
(a Kafka)
Processing & View
Query
48. So what happens on failure?
View
OrdersPayments View
View
Storage Layer
(a Kafka)
Processing & View
49. Clustering Reroutes Data to
surviving node
View
OrdersPayments View
View
Storage Layer
(Kafka)
Ownership of partitions is re-routed from dead node
Processing & View
50. But what about state?
View
OrdersPayments View
View
Storage Layer
(Kafka)
“Cold” replica of state
takes over
Processing & View
51. Primitives for sharding &
replication
Stock
OrdersPayments Stock
Stock
Redundant copies are
cached on other nodes
Sharding spread data
over processors
52. So processors inherit much
from the log
Clustering comes
from the log
You just write the
functional bit
53. General framework for distributed, realtime data
computation
Protection from
broker failure
Protection from
engine failure
Join tables & streams
(in process)
Event Driven
Create views which
can be queried
Query
79. General framework for distributed, event-
driven data computation
Protection from
broker failure
Protection from
engine failure
Join tables & streams
(in process)
Event Driven
Create views which
can be queried
Query