Exactly-once semantics is the holy grail in data stream processing, and Apache Kafka (including its stream processing library Kafka Streams) supports it. However, there is a lot of misunderstanding what exactly-once really is, what Kafka technically offers, where the limitations are, and how to use it correctly. In this talk, we will dive into technical details to shed some light on the above questions. We approach the topic from a conceptual point of view, explain the challenges Kafka Connect faces when it comes to exactly-once, discuss how external source and sink systems can be integrated, and provide practical guidelines for implementing end-to-end exactly-once data pipelines correctly.

1. Exactly-once Stream Processing
Matthias J. Sax, Software Engineer
Apache Kafka committer and PMC member
matthias@confluent.io | @MatthiasJSax

2. @MatthiasJSax
Exactly-once: Delivery vs Semantics
Exactly-once Delivery
• Academic distributed system problem:
• Can we send a message an ensure it’s delivered to the receiver exactly once?
• Two Generals’ Problem (https://en.wikipedia.org/wiki/Byzantine_fault)
• Provable not possible!
Deliver != Semantics
2

3. @MatthiasJSax
Take input record, process it, update result, and record progress.
No Error. No Problem.
What is Exactly-once Semantics About?
3

4. @MatthiasJSax
What happens if something goes wrong?
Error during read, processing, write, or record progress.
We retry!
But is it safe?
What is Exactly-once Semantics About?
4

5. @MatthiasJSax
5
Are retries safe? With exactly-once, yes!
Exactly-once is about masking errors via safe retries.
The result of an exactly-once retry,
is semantically the same as if no error had occurred.
What is Exactly-once Semantics About?

6. @MatthiasJSax
Common Misconceptions
Kafka as an intermediate
• Pattern: Produce -> Kafka -> Consume
• No exactly-once semantics:
• Upstream write-only producer!
6

7. @MatthiasJSax
There is no* Write-only Exactly-once!
(*) Write-only exactly-once is possible for idempotent updates (but Kafka is append-only…)

8. @MatthiasJSax
Common Misconceptions
Kafka as an intermediate
• Pattern: Produce -> Kafka -> Consume
• No exactly-once semantics:
• Upstream write-only producer!
• Downstream read-only consumer!
8

9. @MatthiasJSax
There is NO Read-only Exactly-once!

10. @MatthiasJSax
Common Misconceptions
Kafka as an intermediate
• Pattern: Produce -> Kafka -> Consume
• No exactly-once semantics.
Kafka for processing
• Pattern: Consume -> Process -> Produce
• Built-in exactly-once via Kafka Streams (or DIY).
• Also possible with external source/target system!
10

11. @MatthiasJSax
Let’s Break it Down
Steps in a Processing Pipeline
• Read input:
• Does not modify state; re-reading is always safe.
• Process data:
• Stateless re-processing (filter, map etc) is always safe.
• Stateful re-processing: need to roll-back state before we can retry.
• Update result:
• Need to “retract” (partial) results.
• Or: rely on idempotent updates. (There are dragons!)
• Record progress:
• Modifies state in the source system (or does it?)
11

12. @MatthiasJSax
Exactly-once
==
At-least-once + Idempotency
It depends…

13. @MatthiasJSax
Idempotent Updates (Internal State)?
Stateful processing
Stateful processing is usually a “read and modify” pattern, e.g., increase a counter.
• It’s context sensitive!
13
Cnt: 73 Cnt: 74
73+1
Cnt: 74 Cnt: 75
74+1
Retry: L

14. @MatthiasJSax
Idempotent Updates? Maybe…
Stateful processing
Stateful processing is usually a “read and modify” pattern, e.g., increase a counter.
• It’s context sensitive!
• Idempotency requires context agnostic state modifications, e.g., set a new address.
14
City: LA City: NY
Set “NY”
City: NY City: NY
Set “NY”
Retry: J

15. @MatthiasJSax
Idempotent Updates (External State)
The issue of time travel…
15
City: LA City: NY
Set “NY”
City: BO
Set “BO”
Read: NY Read: BO
Read: LA

16. @MatthiasJSax
Idempotent Updates (External State)
Retrying a sequence of updates:
16
City: BO City: NY
Set “NY”
City: BO
Set “BO”
Read: NY L
Read: BO J Read: BO J

17. @MatthiasJSax
Idempotency is not enough.
All State Changes must be Atomic!

18. @MatthiasJSax
All State Changes must be Atomic
What is ”state”?
• Internal processing state.
• External state, i.e., result state.
• External state, i.e., source progress.
Transactions for the rescue!
Do we want to (can we) do a cross-system distributed transaction?
Good news: we don’t have to…
18

19. @MatthiasJSax
Exactly-Once with Kafka and External Systems
19
Example: Downstream target RDBMS
(Async) offset update
(not part of the transaction)
Atomic write via
ACID transaction
State
Result
Offsets

20. @MatthiasJSax
Exactly-Once with Kafka and External Systems
20
Example: Downstream target RDBMS
State
Result
Offsets
Reset offsets
and retry

21. @MatthiasJSax
Kafka Connect (Part 1)
Exactly-once Sink
• Has “nothing” to do with Kafka:
• Kafka provides source system progress tracking via offsets.
• Connect provide API to fetch start offsets from target system.
• Depends on targe system properties / features.
• Each individual connector must implement it.
21

22. @MatthiasJSax
How does Kafka Tackle Exactly-once?
22
Kafka Transactions
Multi-partition/multi-topic atomic write:
0 0
0 0 0
1 1 1 1
2
2
2
3
4
3
1
2
t
1
-
p
0
t
1
-
p
1
t
2
-
p
0
t
2
-
p
1
t
2
-
p
2
2
3

23. @MatthiasJSax
How does Kafka Tackle Exactly-once?
23
Kafka Transactions
Multi-partition/multi-topic atomic write:
producer.beginTransaction();
// state updates (changelogs + result)
producer.send(…);
producer.send(…);
…
producer.commitTransaction(); // or .abortTransaction()

24. @MatthiasJSax
Exactly-Once with Kafka
24
Kafka as Sink
Requirement: ability to track source system progress.
result
state (via changelogs)
source progress (via custom metadata topic)

25. @MatthiasJSax
Kafka Connect (Part 2)
•
•
•
•
•
Exactly-once Source
• “Exactly-once, Again: Adding EOS Support for Kafka Connect Source Connectors”
• Tomorrow: 2pm
• Chris Egerton, Aiven
• KIP-618 (Apache Kafka 3.3):
• https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
25

26. @MatthiasJSax
Kafka Streams
26
Kafka Transactions
Atomic read-process-write pattern:

27. @MatthiasJSax
Kafka Streams
27
__consumer_offsets
changelogs
result
Kafka Transactions
Multi-partition/multi-topic atomic write:

28. @MatthiasJSax
Kafka Streams
28
Kafka Transactions
Multi-partition/multi-topic atomic write:
producer.beginTransaction();
// state updates (changelogs + result)
producer.send(…);
producer.send(…);
…
producer.addOffsetsToTransaction(…);
producer.commitTransaction(); // or .abortTransaction()

29. @MatthiasJSax
Kafka Streams
Single vs Multi-cluster
Kafka Streams (current) only works against a single broker cluster:
• Does not really matter. We still rely on the brokers as target system.
• Need source offsets but commit them via the producer.
• Single broker cluster only avoids “dual” commit of source offsets.
Supporting cross-cluster EOS with Kafka Streams is possible:
• Add custom metadata topic to targe cluster.
• Replace addOffsetsToTransaction() with send().
• Fetch consumer offset manually from metadata topic.
• Issues:
• EOS v2 implementation (producer per thread) not possible.
• Limited to single target cluster.
29

30. @MatthiasJSax
The Big Challenge
Error Handling in a (Distributed) Application
Kafka transaction allow to fence “zombie” producers.
Any EOS target system needs to support something similar (or rely on idempotency if possible).
Kafka Connect Sink Connectors:
• Idempotency or sink system fencing required—Connect framework cannot help at all.
Kafka Connect Source Connectors:
• Relies on producer fencing.
• Does use a producer per task (similarly to Kafka Streams’ EOS v1 implementation).
Kafka Streams:
• Relies on producer fencing (EOS v1) or consumer fencing (EOS v2).
• EOS v2 implementation (producer per thread) relies on consumer/producer integration inside the same broker cluster.
30

31. @MatthiasJSax
What to do in Practice?
Publishing with producer-only app?
The important thing is to figure out where to resume on restart:
• Is there any “source progress” information you can store?
• You need to add a consumer to your app!
• On app restart:
• Initialize producer to fence potential zombie and to force any pending TX to complete.
• Use consumer (in read-committed mode) to inspect the target cluster’s data.
Reading with consumer-only app?
• If there is no target data system, only idempotency can help.
• With no target data system, everything is basically a side-effect.
31

32. @MatthiasJSax
Exactly-once Key Takeaways
(A) no producer-only EOS
(B) no consumer-only EOS
(C) read-process-write pattern
(1) need ability to track source system read progress
(2) require target system atomic write (plus fencing)
(3) source system progress is recorded in target system
Kafka built-in support via transactions + Zero coding with Kafka Streams
✅

33. @MatthiasJSax