In distributed systems, it is common to encounter the double write problem, where the same data is written to multiple locations, leading to data inconsistencies. Apache Kafka is no exception to this, and these inconsistencies can cause significant issues in applications that rely on Kafka for data processing and storage. One effective solution to this problem is the outbox pattern, which involves storing all writes in a separate outbox table before they are replicated to the main data store. This allows the system to ensure that each write is only performed once, eliminating the risk of double writes. In this talk, we will discuss the double write problem in Apache Kafka and how the outbox pattern can be implemented to solve it. We will also demonstrate the use of the outbox pattern in a sample Kafka application and show how it can be used to ensure data consistency and integrity in distributed systems. Attendees will come away with a thorough understanding of the double write problem and how to use the outbox pattern to solve it in Apache Kafka and other distributed systems.

2. Rafael Roman
Principal Engineer

3. Double write
problem?

5. Everything will ultimately
fail.
Michael Nygard

7. This message is forever lost

8. Wrap in transaction

9. Wrap in transaction

10. Wrap in transaction

13. Thank you
Rafael Roman - Principal Engineer