"Writing a Kafka Streams application is only the first step for your event driven applications. Eventually, you will deploy and tune your applications with different goals in mind (throughput, latency, robustness, high availability, you name it). To run and configure your applications efficiently, it is paramount to understand the internal architecture, and the dependencies and interactions of the different internal components. Otherwise, you end up with a trial-and-error approach that is cumbersome, potentially frustrating, and for sure not fun. In this talk, we will explore the internal architecture of Kafka Streams to set you up for successfully running and tuning your applications. -- What does the internal threading model look like? How are partitions assigned and mapped to tasks? Why are there multiple internal consumers? What are the most important/interesting configurations and how do they interact with each other? -- Those are just a few questions we will answer in this session, enabling you to run and tune your applications in record time!"

1. The Nuts and Bolts of Kafka Streams:
An Architectural Deep Dive
Matthias J. Sax
Software Engineer | Apache Kafka Committer and PMC member
Twitter/𝕏 @MatthiasJSax

2. 2
10,000ft View
The DSL Program
(let’s go down the rabbit hole)

3. Program
3
StreamsBuilder builder = …
KStream words = builder.stream(…);
KTable result = words.groupBy(…)
.count();
result.toStream().to(…);
Topology t = builder.build();

4. Program
4
StreamsBuilder builder = …
KStream words = builder.stream(…);
KTable result = words.groupBy(…)
.count();
result.toStream().to(…);
Topology t = builder.build();
(Sub-)Topology
src
grpBy
snk
src
cnt
snk
shuffle
input
result
log
1
2
3
4
1
2 3
4

5. Program
5
StreamsBuilder builder = …
KStream words = builder.stream(…);
KTable result = words.groupBy(…)
.count();
result.toStream().to(…);
Topology t = builder.build();
1
2
3
4
1
2 3
4
(Sub-)Topology
src
grpBy
snk
src
cnt
snk
shuffle
input
result
log

6. (Sub-)Topology
6
src
grpBy
snk
src
cnt
snk
shuffle
input
result
log

7. (Sub-)Topology Tasks
src
grpBy
snk
src
cnt
snk
shuffle
input
result
log
7
shuffle-0
input-0
result-1
shuffle-1
0_0 0_1
1_0 1_1
input-1
result-0
log-1
log-0

8. (Sub-)Topology Tasks
src
grpBy
snk
src
cnt
snk
shuffle
input
result
log
8
shuffle-0
input-0
result-1
shuffle-1
0_0 0_1
1_0 1_1
input-1
result-0
log-1
log-0

9. Instance B
Instance A
9
Kafka Streams Instances and StreamThreads
KafkaStreams streamsClient = new KafkaStreams(…); // num.stream.threads
streamsClient.start();
streamsClient.addStreamThread();
streamsClient.removeStreamThread();
// error handling: uncaught exception handler

10. 10
StreamThreads and Tasks
KafkaConsumer KafkaProducer
Kafka Cluster
1_0 1_1
0_0
0_1

11. 11
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords

12. 12
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms

13. 13
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms

14. 14
Task Execution
1_1
src
cnt
snk
Record
Buffer
Record
Buffer
Record
Buffer
“head” records
(deserialized)
- exception handling
- ts-extraction
max.task.idle.ms
src.process(record)
ctx.forward(r) {
cntProcessor.process(r)
}
store.put(…) {
changelogger.put(…)
}
sink.process(record) {
recordCollector.send(…)
}

15. 15
Task Execution
1_1
src
cnt
snk
Record
Buffer
Record
Buffer
Record
Buffer
“head” records
(deserialized)
- exception handling
- ts-extraction
max.task.idle.ms
src.process(record)
ctx.forward(r) {
cntProcessor.process(r)
}
store.put(…) {
changelogger.put(…)
}
sink.process(record) {
recordCollector.send(…)
}
depth-first
execution

16. 16
Task Execution
1_1
src
cnt
snk
Record
Buffer
Record
Buffer
Record
Buffer
“head” records
(deserialized)
- exception handling
- ts-extraction
max.task.idle.ms
src.process(record)
ctx.forward(r) {
cntProcessor.process(r)
}
store.put(…) {
changelogger.put(…)
}
sink.process(record) {
recordCollector.send(…)
}
depth-first
execution
RecordCollector
• Serialization
• Error handling
• Partitioning
KafkaProducer

17. 17
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms

18. 18
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms

19. Committing
19
At-least-once:
• flush state stores
• flush producer
• avoid data
• get offset / consumer position
• record buffers
• commitSync()
Exactly-once:

20. Committing
20
At-least-once:
• flush state stores
• flush producer
• avoid data
• get offset / consumer position
• record buffers
• commitSync()
Exactly-once:
• flush state stores
• flush producer
• avoid data
• get offsets / consumer position
• record buffers
• addOffsetsToTransaction()
• commitTransaction()
• configs:
• transaction.timeout.ms

21. 21
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms

22. 22
Main Processing Loop
while (running)
records = mainConsumer.poll() // config: poll.ms
// max.poll.records
// fetch configs
addRecordsToTask(records); // no deserialization
// config: buffered.records.per.partition
while (keepProcessing)
for each task:
if (task.isReady) // data available for all inputs?
task.process(nRecords) // -> ts-extraction
// -> (de)serialization + exception-handling
maybePunctuate()
maybeCommit() // config: commit.interval.ms [ctx.commit()]
adjust nRecords max.poll.interval.ms
transaction.timeout.ms

23. Fault-Tolerance, High-Availability, and Scaling
23

24. Instance A
24
State Stores and Changelog Topics
Task 1_1
cnt
log-1

25. Instance A
25
State Stores and Changelog Topics
Task 1_1
cnt
log-1

26. 26
State Stores and Changelog Topics
log-1
Instance B
Task 1_1
restore

27. 27
State Stores and Changelog Topics
log-1
Instance B
Task 1_1
cnt

28. Restore Consumer
28
KafkaConsumer (restore)
no consumer group
read changelog topics
used to restore state
no offset commits
while (running) {
mainConsumer.poll()
if (restoring)
restoreConsumer.poll()
restoreState()
continue;
processActiveTasks()
}

29. Instance A
29
Standby Tasks and High-Availability
Task 1_1
cnt
log-1

30. Instance A
30
Standby Tasks and High-Availability
Task 1_1
cnt
log-1
Instance B
Standby 1_1
restore

31. Instance A
31
Standby Tasks and High-Availability
Task 1_1
cnt
log-1
Instance B
Standby 1_1
restore
config: num.standby.tasks

32. Restore Consumer
32
KafkaConsumer (restore)
no consumer group
read changelog topics
used to restore state
no offset commits
while (running) {
mainConsumer.poll()
if (restoring)
restoreConsumer.poll()
restoreState()
continue;
else
restoreConsumer.poll()
updateStandbys()
processActiveTasks()
}

33. 33
Smooth Scaling with ”Warmup” Tasks
Instance A
Task 1_0
cnt
log-0
Task 1_1
cnt
log-1

34. 34
Smooth Scaling with ”Warmup” Tasks
Instance A
Task 1_0
cnt
log-0
Instance B
Standby 1_1
(warmup)
restore
Task 1_1
cnt
log-1

35. 35
Smooth Scaling with ”Warmup” Tasks
Instance A
Task 1_0
cnt
log-0
Instance B
Task 1_1
cnt
log-1

36. 36
Smooth Scaling with ”Warmup” Tasks
Instance A
Task 1_0
cnt
log-0
Instance B
Task 1_1
cnt
log-1
config: max.warmup.replicas

37. GlobalKTables/Stores and Threading
37

38. Instance B
Instance A
38
Global State Stores and GlobalStreamThread
builder.globalTable(…); // or .addGlobalStore(…)
KafkaStreams streamsClient = new KafkaStreams(…);
streamsClient.start()
GlobalStreamThread

39. GlobalStreamThread
39
KafkaConsumer (global)
no consumer group
reads all global topic partitions
no offset commits
Global StateStore updating

40. GlobalStreamThread
40
KafkaConsumer (global)
no consumer group
reads all global topic partitions
no offset commits
Global StateStore updating
// bootstrap global state
start() {
while(!endOffsetsReached)
globalConsumer.poll()
updateGlobalState()
}
// start StreamThreads
// regular processing
while(running) {
globalConsumer.poll()
updateGlobalState()
}

41. Thread Refactoring Project
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429334)
41

42. Instance B
Instance A
42
Thread Refactoring – StateUpdaterThread (AK 3.7 ?)
GlobalStreamThread
StateUpdaterThread

43. Instance B
Instance A
43
Thread Refactoring – Phase 2
Decouple consumer and processing threads (WIP)
GlobalStreamThread ProcessingThread
StateUpdaterThread

44. The Nuts and Bolts of Kafka Streams:
An Architectural Deep Dive
Matthias J. Sax
Software Engineer | Apache Kafka Committer and PMC member
Twitter/𝕏 @MatthiasJSax