Message reliability in Kafka

1. Message Reliability in Kafka
Mihai Ghita
Data Engineer

2. ● Delivery guarantees provided by Kafka:
○ At most once
○ At least once
○ Exactly once
● Kafka features:
○ Kafka Idempotent Producer
○ Kafka Transactional Consumer and Producer
Outline

3. Kafka Producer - Example

4. Kafka Producer - Example

5. Kafka Consumer - Example

6. Kafka Consumer - Example

7. Delivery
Guarantees
At most once At least once Exactly once
Message Delivery Guarantees

8. “At Most Once” Guarantee

9. Kafka Consumer - “At Most Once” Guarantee
Consumer
1 2 3 4 5 6 7 8 9
Batch Size
1.Read records
2.Commit unprocessed
records
Kafka
3. Data
Processing

10. Kafka Consumer - “At Most Once” Guarantee
Consumer
1 2 3 4 5 6 7 8 9
Batch Size
1.Read records
2.Commit unprocessed
records
Kafka
Consumer
4.Read next
records
5.Commit records
Batch Size
3. Data
Processing
6. Data
Processing

11. Kafka Producer - “At Most Once” Guarantee
(x,y)
Producer Broker
(x,y)
Producer Broker
1. Send(x,y)
2. Append(x,y)
1. Send(a,b)
2. Append(a,b)

12. “At Least Once” Guarantee

13. Kafka Consumer - “At Least Once” Guarantee
Consumer
1 2 3 4 5 6 7 8 9
Batch Size
1.Read records
3.Commit records
Kafka
2. Data
Processing

14. Kafka Consumer - “At Least Once” Guarantee
Consumer
1 2 3 4 5 6 7 8 9
Batch Size
1.Read records
3.Commit records
Kafka
Consumer
4.Read next records
6.Commit records
Batch Size
2. Data
Processing
5. Data
Processing

15. Kafka Producer - “At Least Once” Guarantee
(x,y)
Producer Broker
(x,y) (a,b)
Producer (a,b)
Broker
1. Send(x,y) 2. Append(x,y)
3. Ack
1. Send(a,b)
2. Append(a,b)
3. Ack
5. Append(a,b)
4. Resend(a,b)
6. Ack

16. “Exactly Once” Guarantee

17. Idempotent Producer

18. Acknowledgement failure leads to message duplication
(x,y)
Producer Broker
(x,y) (a,b)
Producer (a,b)
Broker
1. Send(x,y) 2. Append(x,y)
3. Ack
1. Send(a,b)
2. Append(a,b)
3. Ack
5. Append(a,b)
4. Resend(a,b)
6. Ack

19. The idempotent producer will eliminate message duplication from producer side
(x,y)
Seq = 0
PID = 100
Producer Broker
1. Send(x,y)
Seq = 0
PID = 100
2. Append(x,y)
Seq = 0
PID = 100
3. Ack
(x,y)
Seq = 0
PID = 100
Producer Broker
4. Send(a,b)
Seq = 1
PID = 100
2. Append(a,b)
Seq = 1
PID = 100
3. Ack
(a,b)
Seq = 1
PID = 100
5. Ack-Duplicate
1. Send(a,b)
Seq = 1
PID = 100

20. Kafka Transactions

21. Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
Transactional Producer

22. m1 m5
m3 m4 m6
m2
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
Transactional Producer

23. Transactional Producer
m1 m5 C
m3 m4 m6 C
m2 C
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
atomic commit

24. m1 m5 A
m3 m4 m6 A
m2 A
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
atomic abort
Transactional Producer

25. m1 m5 C
m3 m4 m6 C
m2 C
Consumer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
read committed
Transactional Consumer

26. m11 m12 C
m3 A m8 C
A m2 m9 C
Consumer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1
read committed
Transactional Consumer

27. Why are the Transactions Important?
Read
Write
Process
Input
Output

28. Write all results
Mark all messages as consumed
Write to the output topic
Write offsets to the commit topic
Why are the Transactions Important?
All or nothing!

29. End-to-End Exactly-Once

30. Kafka Transactions - Demo

31. ● Delivery guarantees provided by Kafka:
○ At most once
○ At least once
○ Exactly once
● Depending on your needs, choose the right delivery guarantee
Putting it all Together

33. Kafka Transactional - Internal Implementation

34. Kafka Consumer - Internal Implementation