The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Co...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

The Flux Capacitor of
Kafka Streams and ksqlDB
Matthias J. Sax | Software Engineer
@MatthiasJSax

Back to the Time Topic
2@MatthiasJSax
https://www.confluent.io/kafka-summit-san-francisco-2019/whats-the-time-and-why/

Stream Processing is our Density.

Recap: Time 101
4@MatthiasJSax
Event Time
• When an event happened (embedded in the message/record)
• Ensures deterministi...

First, you turn the time circuits on.

Tracking Time
Stream-time: the maximum observed input event timestamp (aka ROWTIME)
• Monotonically increasing
• Allows to...

Yeah, well, history is gonna change
Input records with descending event timestamp are considered out-of-order
• Out-of-ord...

You are not thinking fourth-dimensionally
8@MatthiasJSax
14:11…14:05…
14:03…14:04…
14:01…
14:02… 14:08…
Topic-A, Partition...

You are not thinking fourth-dimensionally
9@MatthiasJSax
14:11…Topic-A, Partition 0
Topic-B, Partition 0 empty
Pause proce...

When the hell are they?

Tumbling Windows
• fixed size / non-overlapping / grouped (i.e, GROUP BY)
Time Windows
11@MatthiasJSax
14:00 14:05 14:1514...

Time Windows
12@MatthiasJSax
Hopping Windows
• fixed size / overlapping / grouped (i.e., GROUP BY)
• Different to a slidin...

Different use-case: aggregate the data of the last (e.g.) 10 minutes
• Window boundaries are data dependent and unknown up...

When we are processing, we don’t need watermarks
Grace period: defines a cut-off for out-of-order records that are (too) l...

Retention Time
How long to store data in a (windowed) table.
TimeWindows.of(Duration.ofMinutes(5L)).grace(Duration.ofMinut...

If my calculations are correct…
16@MatthiasJSax
Table is continuously updated, but when to emit data to the result stream?...

Crossing Timelines (aka Joins)

Stream-Stream Join
18@MatthiasJSax
Streams are conceptually unbounded
• Limited join scope via a sliding time window
leftS...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Chaining stream-stream joins is not associative!
• Order matters: ⨝(s1,s2,s3) != (s1 ⨝ s2) ⨝ s3 != (s1 ⨝ s3) ⨝ s2 != (s2 ⨝...

Stream-Table Join
27@MatthiasJSax
Stream-Table join is a temporal join
14:01a 14:03b 14:05c 14:08b 14:11a
14:02… 14:04… 14...

Time Traveling is just too Dangerous
28@MatthiasJSax
Is it? Well, mind compaction!
14:05c 14:08b 14:11a
14:02… 14:04… 14:0...

You Need to Know your History
29@MatthiasJSax
Table Changelog
Stream
append
new data
(tail)
truncation
retention time
comp...

You Need to Know your History
30@MatthiasJSax
Table Changelog
Stream
truncation
retention time
Lost History
fully compacte...

You are the doc, Doc
31@MatthiasJSax
Wrapping up
• Event time vs processing time
• Stream-time, grace period, and retentio...

Hope, it was educational.

Thanks! We are hiring!
@MatthiasJSax
matthias@confluent.io | mjsax@apache.org

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 33 Ad

View on Slideshare

1 of 33 Ad