2. ● Delivery guarantees provided by Kafka:
○ At most once
○ At least once
○ Exactly once
● Kafka features:
○ Kafka Idempotent Producer
○ Kafka Transactional Consumer and Producer
Outline
23. Transactional Producer
m1 m5 C
m3 m4 m6 C
m2 C
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
atomic commit
24. m1 m5 A
m3 m4 m6 A
m2 A
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
atomic abort
Transactional Producer
25. m1 m5 C
m3 m4 m6 C
m2 C
Consumer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
read committed
Transactional Consumer
26. m11 m12 C
m3 A m8 C
A m2 m9 C
Consumer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
read committed
Transactional Consumer
27. Why are the Transactions Important?
Read
Write
Process
Input
Output
28. Write all results
Mark all messages as consumed
Write to the output topic
Write offsets to the commit topic
Why are the Transactions Important?
All or nothing!
31. ● Delivery guarantees provided by Kafka:
○ At most once
○ At least once
○ Exactly once
● Depending on your needs, choose the right delivery guarantee
Putting it all Together