Learn how the combination of Apache Kafka and Apache Flink is making stateful stream processing even more expressive and flexible to support applications in streaming that were previously not considered streamable.
The new world of applications and fast data architectures has broken up the database: Raw data persistence comes in the form of event logs, and the state of the world is computed by a stream processor. Apache Kafka provides a strong solution for the event log, while Apache Flink forms a powerful foundation for the computation over the event streams.
In this talk we discuss how Flink’s abstraction and management of application state have evolved over time and how Flink’s snapshot persistence model and Kafka’s log work together to form a base to build ‘versioned applications’. We will also show how end-to-end exactly-once processing works through a smart integration of Kafka’s transactions and Flink’s checkpointing mechanism.
3. What is Apache Flink?
3
Batch Processing
process static and
historic data
Data Stream
Processing
realtime results
from data streams
Event-driven
Applications
data-driven actions
and services
Stateful Computations Over Data Streams
4. Apache Flink in a Nutshell
4
Queries
Applications
Devices
etc.
Database
Stream
File / Object
Storage
Stateful computations over streams
real-time and historic
fast, scalable, fault tolerant, in-memory,
event time, large state, exactly-once
Historic
Data
Streams
Application
5. Everything Streams
5
Apache Flink handles everything as streams internally.
Continuous streaming and applications use "unbounded streams".
Batch processing and finite applications use "bounded streams".
6. Layered abstractions
6
Process Function (events, state, time)
DataStream API (streams, windows)
Stream SQL / Tables (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Navigate simple to complex use cases
9. High Level: SQL (ANSI)
9
SELECT
campaign,
TUMBLE_START(clickTime, INTERVAL ’1’ HOUR),
COUNT(ip) AS clickCnt
FROM adClicks
WHERE clickTime > ‘2017-01-01’
GROUP BY campaign, TUMBLE(clickTime, INTERVAL ‘1’ HOUR)
Query
past futurenowstart of
the stream
19. Why Checkpoints?
No barriers / boundaries low latency
No intermediate stream/state replication necessary
• High throughput
• Shuffles are very cheap! No load on brokers.
Handles very large state well (TBs)
Supports fast batch processing
Supports flexibly types of states and timers
19
21. Localized State Recovery (Flink 1.5)
21
Piggybags on internal multi-version
data structures:
• LSM Tree (RocksDB)
• MV Hashtable (Fs / Mem State Backend)
Setup:
• 500 MB state per node
• Checkpoints to S3
• Soft failure (Flink fails, machine survives)
22. Checkpoints for Program Evolution
22
Restore to different
programs
Bugfixes, Upgrades, A/B testing, etc
24. Replay from Savepoints to Drill Down
24
time
Incident of Interest
"Debug Job"
(modified version of original Job)
Filter
(events of interest only)
Extra sink for
trace output
25. Pause / Resume style execution
25
time
Bursty Event Stream (events only at only end-of-day )
26. Pause / Resume style execution
26
time
Bursty Event Stream (events only at only end-of-day )
Checkpoint / Savepoint
Store
28. Flink Kafka Reader
Supports version 0.8 – 0.11/1.0
Exactly-once semantics
• Flink checkpoints manage offsets
• Can optionally participate in reader groups offset committing
Topic and partition discovery
Multiple topics at the same time
Per-partition watermarking
28
29. Flink Kafka Writer
Supports version 0.8 – 0.11/1.0
Exactly-once via Kafka Transactions (0.11+)
• Details in a later sections
Supports partitioners, timestamps, and the usual
29
30. Transaction Coordination
Similar to a distributed 2-phase/3-phase commit
Coordinated by asynchronous checkpoints
==> non-blocking, no voting delays
Basic algorithm:
• Between checkpoints: Produce into transaction or Write Ahead Log
• On operator snapshot: Flush local transaction (vote-to-commit)
• On checkpoint complete: Commit transactions (commit)
• On recovery: check and commit any pending transactions
30
43. Sophisticated Time Semantics
43
Erik de Nooij / ING
"StreamING models, how ING adds models at runtime to catch fraudsters"
Low-latency event-time joins/aggregations
47. Flink 1.5
Big change to process model
• Better support for Framework and Library modes
• Dynamic resource acquisition
• All communication with clients is REST
Special network protocol to speed up checkpoint alignments
Lower shuffle latency with same throughput
Faster recovery of large state
Managed broadcast state
Interactive SQL Client (beta)
… much more …
47
Apache Flink is the only system that handles the full breadth of stream processing: from exploration of bounded data over streaming analytics to streaming data applications