In any sufficiently complex IT system, at some point, we realize that we have to answer the question: what if something goes wrong. We can do nothing and swallow the erroneous events. We can also stop processing and wait for the fix. But what if we want some sensible retry policy, and in the absence of success - postponing events in a Dead Letter Queue (DLQ). It turns out that Kafka has no support for such a scenario. In this short presentation, I will talk about a pattern based on three topics: operational, retry, and DLQ, and how it can be handled programmatically. Based on a real-world example of a messaging system created for one of the biggest European banks.

1. Reliable Event Delivery
in Apache Kafka
Based on Retry Policy
and Dead Letter Topics

2. Jacek Grobelny
Tech Lead @ Consdata
Reliable Event Delivery in Apache Kafka
Based on Retry Policy and Dead Letter Topics
jgrobelny@consdata.com
ynleborg.github.io
jacek-grobelny

3. Mailbox and Rearchitecture
Error Handling Strategies
Summary
Additional Considerations
PRESENTATION
AGENDA

4. Reliable Event Delivery 4
Santander Mailbox 2.0

5. Reliable Event Delivery 5
Santander Mailbox 2.0
Mass message distribution

6. Reliable Event Delivery 6
Event Sourcing to the Rescue!

7. Reliable Event Delivery 7
Santander Mailbox 2.0
Mass message distribution - asynchronous version

8. Why is there no
DLQ in Kafka?

9. Reliable Event Delivery 9
Topics

10. Reliable Event Delivery 10
Error handling
Strategy #1: Lack of error handling

11. Reliable Event Delivery 11
Error handling
Strategy #2: Infinite retry

12. Reliable Event Delivery 12
Error handling
Strategy #3: Finite retry at the point of an error

13. Reliable Event Delivery 13
Error handling
Strategy #4: Finite retry on a separate topic

14. Reliable Event Delivery 14
Additional Considerations
Retry Service
Skippable Offset Service

15. Reliable Event Delivery 15
Working example
https://github.com/ynleborg/kafka-reliable-consumers

16. Reliable Event Delivery 16
Summary
● Don't be afraid to re-architecture.
● Have good monitoring in place.
● Pick the right tool for the job.

17. THANK YOU
jgrobelny@consdata.com
ynleborg.github.io
jacek-grobelny